AI Projects › The journey
Thirteen phases in three stages. Each phase was a full project with its own risk, tooling and proof, and each had to run safely beside the others on the same few machines.
Stage A: recovery and inventory (Sep 21 to Oct 24)
Goal: know what exists before touching anything.
| Date | Phase | What happened |
|---|---|---|
| Sep 21 | Brief and first agent | The archive was framed as a recovery problem, not a storage problem. |
| Oct 1 | A drive fails its health test | A 24 TB pool disk failed a SMART extended test, which started the clock. Replica copying began in batches: batch 1 was 50,000 files and 1,341.6 GB. |
| Oct 13 | Stakes become rules | “I have waited 29 years to do this” became a shared operating protocol every agent reads. |
| Oct 16 | Identify before acting | Nothing destructive ran until the identify-and-learn phase finished. The loop: read deep, ask questions, take one safe step. A “40 of 48 matched” result was questioned, not accepted. |
| Oct 17 | The rebuild that changed the plan | The old five-drive NAS was rebuilt by hand and every share checked against its migrated copy. “Done” migrations could no longer be trusted without per-share proof. |
| Oct 18 to 19 | GPU lane and relocations | A compression lane was set up for HEVC with VMAF gates. At matched quality it beat the laptop GPU on size. |
| Oct 24 | DB2, the inventory engine | A Postgres design: full sha256, md5, quick hash and media probe per file, one pass per disk with all disks running at once. Three gates per volume (G1 counts, G2 every file hashed, G3 a 50-file independent re-hash). The same day an agent started five scanners without approval, and “never start a long job without approval” became a standing rule. |
Stage B: fingerprint, protect, compress (Oct 25 to Nov 9)
Goal: prove every copy, then reclaim only what is proven.
| Date | Phase | What happened |
|---|---|---|
| Oct 25 to Nov 7 | Fingerprinting the fleet | Seventeen volumes across the server, the five-drive NAS, a mini-PC and laptops were hashed: 14.7 million files. On Nov 7 the report produced the copy map. That evening, three hard resets were traced from console photos to a corrupt single-disk pool, and a reformat plan was cancelled. |
| Nov 1 | Compression at scale | Two machines encoded in parallel. 3,501 files passed the quality gate. 5.69 TB became 0.83 TB, about 85% smaller, with VMAF logged for every file. |
| Nov 3 to 4 | Observability and the front door | A live console showed GPU load, jobs and disks on one page. Public access was designed as one forward into a hostname-routed reverse proxy, after logged intrusion attempts on an old forward. |
Stage C: structure and walk-away operations (Nov 7 to Nov 10)
Goal: move the proven library into its final home.
| Date | Phase | What happened |
|---|---|---|
| Nov 7 to 8 | Structure and the walk-away morning | A documentation method was adopted after work kept falling apart without one. AI workers moved into containers with a 24/7 card dispatcher. Then a 4.5-hour walk-away morning with 30 minutes of my decisions: a failing 1.52 TB drive emptied and verified (118,599 files), 8.43 TB of single-copy exposure mapped, and about 97% of a rescan avoided by checking metadata first. That afternoon, lineage traced 98.9% of the compressed library to source runs. |
| Nov 9 | Parent and child proof pays off | Joining each compressed file’s hash to its original’s proved 7,429 copies bit-perfect and linked 3,501 encodes. It also caught 4 truncated camera originals. That night, flat-playing 360 videos were found and fixed. See the two nights. A 2.9 TB copy into the final structure ran overnight from a 495-row sheet. |
| Nov 10 | Landing home | As of Nov 10 the structured library is about 89% landed and the ReadyNAS hash is at 39%. Next: hash the new structure as its own database volume, so every file in its final home is proven against its source. |
How the lanes stayed safe
The hard part was never one project. It was five of them on shared disks, a shared network and shared RAM, on different clocks. Every lane had an owner, a console, a stop rule and interlocks with its neighbors.
| Lane | What it did |
|---|---|
| Inventory | Hashing the fleet, 17 volumes, 14.7 M files, writing to the database in real time. |
| Protection | Replica batches and evacuation of the failing drive, each batch verified before the next. |
| Compression | GPU encodes with quality gates and triage rules, in parallel streams. |
| Structure | Copying the proven library into its final layout, one sheet at a time. |
| Operations | Consoles, the run launcher, decision logs, AI workers and check-ins. |
The interlocks
- No encode without hashes. Compression waits for a proven fingerprint.
- No delete without two proven copies. Removal needs hash-matched copies on two other disks and written approval.
- Repair before row 45. The 360 repair rewrote files on the same share a 2.9 TB copy was reading. The copy was on row 3 and the first affected file was at row 45, so the repair ran ahead of it.
- Burn-in waits on evacuation. A drive is not tested or reused until its data is verified elsewhere.
The run launcher also caught a human error. On Nov 9 I pasted a copy’s launch command a second time. The launcher refused, because a job with that ID was already running.
Source: project history log, decisions log, fingerprint report (Nov 7), lineage and encode reports (Nov 8 to 9). Items still moving on Nov 10 are marked.