AI Projects › Field lessons
Twelve lessons, grouped by what they change. Each one changed how the next job was built, and each would change how I run a client engagement from day one.
Identity and proof
1. Name and size never prove a copy
A 6.6 TB folder matched another 100% by name and size, and the match was circular. Only a sha256 proves a copy. On Nov 9 the hash caught truncated files that name and size would have passed.
2. “Done” migrations were 6% to 24% done
The five-drive rebuild found 12 to 420 times more data on the source drives than their “verified” migrations showed. The verification had only covered what was copied.
3. A success check must prove the result
An exit code of 0 is not proof. A drive export is green only when the pool is really gone. A copy is verified only when hash, size and date all match.
Process and time
4. Cheapest proof first
Metadata before bytes. A listing of names, sizes and dates showed 1.8 TB changed out of about 64 TB, which avoided about 97% of a rescan. Every database query is priced before it runs.
5. The slowest box sets the finish line
A single ARM core, not the 28-thread server, decided when the job ended. ETAs from a 30-second burst were wrong by 2 to 4 times, so use the average rate and re-check against fixed windows.
6. Timestamps are part of the record
Every copy keeps the original modification time and is checked afterward. Missing dates are inferred into the database and never written back over the file.
Agents and structure
7. Durable messages beat relaying
Prompts went unsubmitted and agents kept stale briefs. Cards, check-backs and nudges to re-read the current brief fixed a month of stalls.
8. One home per fact
When the board and the workers fall out of sync, things unravel quickly. Layered context, numbered stage folders and short context files keep a multi-week, multi-agent effort coherent.
9. Diagnose before fixing
Three hard resets looked like a dying server. Console photos showed a corrupt pool panicking at boot. The reformat was cancelled and the unique data salvaged.
Measurement and evals
10. Gates measure what they measure
VMAF proved the picture and missed the 360 flag, flight telemetry tracks and spatial audio. A compressed child must pass a metadata parity check before it can stand in for its original.
11. Report both denominators
Once triage removes the worst candidates, “% saved on passes” rises mechanically. Always report whole-pool savings beside it.
12. Recency is not reliability
The newest batch was the corrupted one. Score health and completeness first, and use the date only as a tiebreaker.
Sources: decisions log, history log, encode and lineage reports.
← Guardrails | The two nights →