Two Nights That Show the Method

AI Projects › The two nights

Both happened the same week, and both were found by checking results against the source rather than trusting a job’s exit status.

Night one: the truncated copies

What happened

A read-only join of every compressed file to its original’s hash flagged two copy mismatches. A wider sweep found four truncated files. One camera original should have been 15.21 GB and was 12.73 GB. Three of the four carried modification times from the same seven-minute window, which pointed to one interrupted copy event. A name-and-size check had passed all of them.

How I handled it

1. Located full copies on three other disks.
2. Wrote the fixes to temporary names, checked sha256 and modification time, and only then swapped the files, stopping at the first failure.
3. Had a second machine re-hash all four over the network, bypassing the server’s cache.
4. Result: 4 of 4 matched.

The takeaway: the join between a child file and its parent hash is a cheap, powerful audit. It paid for itself in one night.

Night two: the flat 360 videos

What happened

A 360 render played flat. Within minutes, a probe showed the pixels intact at 5760×2880 and the spherical tag gone. The encoder’s missing flag was found and reproduced on a five-second test. A header-only sweep of all 3,639 compressed files bounded the damage to exactly 10.

How I handled it

1. Repaired all 10 by re-adding the tag only, not re-encoding.
2. Checked that the video stream checksum matched before and after. Each file grew by 487 bytes of metadata, and every date was kept.
3. Added a guard that copies any 360 source as-is instead of encoding it, in all eight compressor scripts.
4. Sequenced the repair ahead of a 2.9 TB copy reading the same share (copy at row 3, first affected file at row 45).

The takeaway: the quality gate measured pixels and missed metadata. 360 videos passed VMAF at 96 and still played flat. The next build is a metadata parity gate.

What both nights have in common

• The result was checked against the source, not against the tool’s own report.
• The damage was bounded before anything was fixed.
• The fix was proven by a second, independent check.
• A rule or a guard came out of it, so it cannot happen the same way twice.

Sources: decisions log, history log, encode and lineage reports, and the Nov 9 oddity sweep.

← Field lessons  |  Six AI skill areas →