How a 100 Nanosecond Timestamp Glitch Almost Wrecked a Data Center Migration
Cloud migrations are supposed to be straightforward—at least in theory. You move your data, validate its integrity, and switch over. But what happens when an oversight at the nanosecond level nearly derails your entire operation? This is the story of one such close call and the lessons learned along the way.
The Setup
Our migration was standard fare: a bulk data transfer using AWS Snowball, followed by AWS DataSync to synchronize any changes. Snowball handled the lion’s share, and DataSync was set up to catch incremental updates. This approach is well-documented and generally reliable. However, an essential piece of advice came up early: Snowballs perform significantly better with large, single files rather than repositories containing hundreds of thousands of small files.
The solution was tar. By combining many small files into a single archive, tar allowed us to streamline the transfer process. Tar preserves metadata, including modification times (m-time), ensures data consistency, and simplifies handling.
It looked like everything was falling into place.
The Issue in Disguise: Nanosecond-Level Precision
Here’s where the seemingly minor details unraveled. Windows file systems, historically designed with business use in mind, store timestamps with a precision of 100 nanoseconds-a level introduced with NTFS to support high-resolution timekeeping for detailed logs.
A file’s time is a 64-bit value that represents the number of 100-nanosecond intervals that have elapsed since 12:00 A.M. on January 1, 1601, Coordinated Universal Time (UTC). The system records file times when applications create, access, and write to files.
On Windows, the built in tar binary sourced from libarchive had a bug that slipped between the cracks (which has now been resolved)
There was some bad math that truncated the last digit of the time value for the file system metadata. For outdated libarchive libraries on Windows, the file's modified time could become truncated at the 100-nanosecond level:
• 0.1234567: Correct m-time
• 0.1234560: Tar-extracted m-time
Furthermore, this is very tricky to identify with scripting because native software time modules often don’t check time to the 100-nanosecond level, opting instead for millisecond or microsecond time comparisons. In our case, this resulted in the metadata validation pre and post-extraction being overlooked when comparing down to the millisecond using default Python libraries.
Why Nanoseconds Matter
AWS DataSync uses m-time to determine whether a file has changed. If even a single nanosecond differs, DataSync assumes the file has been modified and queues it for transfer. Only after the transfer does DataSync validate file integrity (as a hash). This makes sense in several ways for efficiency. If you know a file has been modified from its file metadata, do you really need to spend the extra compute time calculating the hash? No: you would just transfer it, and check the integrity of the file after.
On migration day, we noticed an alarming trend: files that had already been transferred via Snowball were being re-transferred by DataSync. Our old data center’s bandwidth was limited, so these unnecessary transfers risked blowing our migration window.
The culprit? The extracted tar files. Despite identical content, the files’ m-time had been altered ever so slightly during extraction. To DataSync, this correctly made them appear as new files.
The Workaround
We had to act fast as the migration window was closing. The first step was configuring DataSync to transfer only files marked as "new" so we did not have to stop the transfer of files entirely.
Then, using DataSync’s logs from the tasks that were transferring only new files, we could view the metadata of the “skipped” files and then identify files with m-times differing only at the 7th decimal place. From there, we generated a manifest of true list of files for DataSync to transfer, excluding the ones that had truncated m-times.
Lessons Learned
This experience highlighted the importance of dry runs and meticulous validation:
- Be cautious of your filesystem’s time precision for data validation. If your scripts rely on timestamp comparisons, ensure they’re using precision-compatible libraries.
- Always dry-run data transfers. Analyze the number of files flagged for transfer and those skipped. Compare this with your estimates to catch anomalies early.
- Replicate production conditions in testing. Synthetic or test data may not surface the same issues as production data, especially if generated using problematic libraries.
- Understand your tools. Knowing how AWS DataSync validates files and handles metadata saved us from unnecessary transfers.
Closing Thoughts
What seemed like an innocuous detail—nanosecond-level precision—nearly brought our migration to a halt. By staying flexible, using DataSync’s built-in options, and leveraging precise validation scripts, we overcame the challenge.
Next time you’re planning a migration, remember: the devil’s in the details, and sometimes, in the nanoseconds. Stay vigilant, and your cloud journey will be smoother for it.
This post is written by Mason Shuler, Solutions Architect and Lead, Cloud Center of Competency.