AI Projects › The copy map
Once every file had a sha256, the archive could answer a basic question: how many copies of this do I really have? The answer was lopsided.
The map
Each bar pair shows distinct content (colored) against the total bytes stored for it (grey), in TiB. Source: fingerprint report, Nov 7.
What the map means
- The 777,579 contents stored six or more times took 36.5 TiB for 4.4 TiB of unique material. That is the long tail of twenty drive generations copying forward.
- Meanwhile 771,861 contents existed exactly once, each one a single disk failure from gone.
- Theoretical reclaim across the fleet was 82.6 TiB, before keeping any safety copy.
- Of 126 TiB stored, about 43.5 TiB was distinct.
That one chart turns “my storage is a mess” into a ranked work order with a dollar figure attached: protect the red bar first, reclaim from the tall grey bars last, and only with proof.
How the work order falls out
- Protect: copy the single-copy content to a second home, verified by hash.
- Choose golden records: by health and completeness first. Recency is only a tiebreaker, because the newest copy was once the corrupted one.
- Reclaim: only where a content hash survives on at least two other disks, with written approval for each batch.
- Compress: only after hashes exist, with quality gates and parent-to-child links.
Try it on your own data
You do not need 126 TiB for this to pay off. A marketing library, an engineering share or twenty years of project folders all have the same shape. The first step is a read-only walk of names, sizes and dates. In my measurements, 12,871 files and 2.88 TiB were walked in two seconds.
Figures from the Nov 7 fingerprint report. Percentages in the chart are to scale against a 40 TiB full width.
← Hardware limits | Guardrails →