Andrew Fribush

fig. 2.20.2: the first fleet’s 32 finished files were banked, not redone: 37,458,383 records, a quarter of the job

fig. 2.20.2 · the first fleet’s 32 finished files were banked, not redone: 37,458,383 records, a quarter of the job.
32 FILES BANKED, NOT REDONE: 37,458,383 RECORDS, 25% OF THE JOBONE SQUARE IS ONE PART-FILE, A COLUMN A BOX; ACCENT IS A BANKED FILE.BOX0123456789101112131415FIRST FLEET,BANKEDRELAUNCHEDFLEETEACH BOX’S TWO BANKED FILES HOLD 2,338,359 TO 2,343,786 RECORDS, A 0.23%SPREAD; 37,458,383 OF 150,000,000 RECORDS, 25% OF THE JOB, THE OTHER 96 FILESON THE RELAUNCHED FLEET.

128 part-files of about 1,170,000 records each; box k took every part-file whose index mod 16 is k. The salvage script uploaded nothing unless the vectors file was exactly count × 3,076 bytes, the ID file had exactly count lines and the dimension header read 768. First to last vector upload: 15:42:00 to 21:40:38 UTC, 5.98 hours; 150,000,000 vectors counted exact from object sizes.

source: credit_150m_embedding_per_shard.csv lines 2-18; credit_150m_embedding_metrics.csv lines 4, 15, 18-23 · bankedParts.data.ts

View in context: Embedding 150 million records for about $230 → · All figures

Andrew Fribush, software and computer engineer in Austin. I build consequential systems where software meets institutions: finance, healthcare, commerce, government data, and the mechanisms that govern AI agents. Taking new engagements: hello@andrewfribush.com.