Embedding 150 million records for about $230

I embedded 150 million identity records on 16 rented GPUs for about $230, and the last vector landed 5.98 hours after the first. An embedding model reads a piece of text and turns it into a list of numbers, so two records that describe the same person come out as lists that point the same way. This one was a 600-million-parameter Qwen3 model fine-tuned for record matching, and each record came out as 768 numbers. (A GPU is the graphics chip, built to do thousands of small sums at once. Each of the 16 rented boxes had one NVIDIA L40S.)

The run was part of an identity-resolution system, and the vectors went into a head-to-head between two cloud platforms with the same model on each. The plan gave embedding a window of 6 to 10 hours, and the handoff was exactly 150,000,000 vectors, with none missing and none extra. That is what came back, counted from the byte sizes of the output files in cloud storage without downloading a single one.

The size of the fleet comes from one number, the records per second one box can embed, so before launching anything I measured it three times. The development box said 44. Synthetic test strings said 858. Real records said 383. Only the last came from the records the fleet would embed, on the GPU it would rent.

Records Divided by Rate Times Hours

Sizing a fleet is one line of arithmetic. Divide the records by the per-box rate and you get box-hours, the hours one box would need to do the whole job alone. Divide box-hours by the window and round up for the number of boxes. Multiply box-hours by the hourly price for the bill. At 383 records a second, 150,000,000 records is 150,000,000 / (383 x 3,600) = 108.8 box-hours, and 16 boxes split that into about 6.8 hours each, inside the window.

The box count barely moves the bill. Five boxes for eight hours cost about what eight boxes for five hours do, because you pay for GPU-hours and the rate fixes the GPU-hours.

The runbook priced four GPU options per hour on demand: an A10G at $1.006, an L4 at $0.805, an L40S at $1.861, and eight A100s in one box at $32.77. I picked the L40S. The runbook puts it at two to three times an A10G for a model this size, so it does more work per dollar even at nearly twice the hourly price. Sixteen of them on demand cost $29.78 an hour, and the 5.98 hours from first vector to last come to $178. The run log marks that as an estimate that still needs the first fleet's hours added. Those are the 32 files encoded before the span began, about 27.2 box-hours, which brings the job to about 122.9 box-hours and about $229 at $1.861. The runbook's formula on the whole job gives $202 (108.8 box-hours at $1.861). None of the three is an invoice figure. Each comes from the logged rate and the hourly price.

On spot instances, the spare capacity a cloud rents out cheap and can take back on short notice, the runbook's L40S price is about $0.65 an hour. That puts the same 108.8 box-hours near $71, against the run log's own estimate of about $40. The run itself was on demand, so neither spot figure was billed. Embedding is safe to interrupt anyway, since a box that loses its spot instance just redoes the file it was on.

Synthetic Strings Overstated Throughput by 2.2x

The development box was a new GPU the software had no ready-made code for, so it translated generic GPU code on the fly the whole time and managed 44 records a second. At that rate one box would need 39 days for the job. The fleet was always going to run on L40S boxes, so that number never sized anything.

The second number came from the L40S itself, on synthetic strings: 858 records a second. The strings were short and all identical, and the run used batches of 64. Real records run longer than those strings, and no two of them are the same. A model doesn't read letters, it reads tokens, each one a word or a piece of a word, and the median real record is 143 tokens. One in twenty runs past 171, and the longest is 195. Records also go through the GPU in batches, and every record in a batch gets padded with filler tokens out to the length of the longest one so the batch makes a neat rectangle. Identical strings never need padding, since each one is already as long as the longest. A batch of real records nearly always needs some. Put a 195-token record in a batch with a 143-token one and the shorter gets 52 filler tokens, and the GPU runs the model over those 52 the same as over the text.

On a sample of real records the same L40S did 383 a second, and 858 / 383 = 2.24. The run log blames the short identical strings for the gap and gives no measurement that splits it between length and padding, so I'll say only that both are in play, along with the batch size, which was 64 for the synthetic run and 128 for the real one. Sized on 858 for the same 6.8-hour window, the formula says 48.6 box-hours, which is 8 boxes. Eight boxes at the real 383 take 13.6 hours, twice the plan.

A bigger batch is the obvious way to get more out of a GPU, so I swept it on the real sample. Batches of 128 beat 512 and 1,024, and the run log records only "optimal / slower / slower" with no rates for the other two. The reason is the padding again: a bigger batch is more likely to hold one long record, and that record sets the padding for every other record in it. The fleet ran at 128.

At 383 records a second the L40S was doing about 36% of the arithmetic it's rated for. More than that would take a different serving stack, Hugging Face's Text Embeddings Inference with flash attention (a faster way to run the step where the model weighs each token against the others), and no batch setting gets there.

Decide What Gets Cut Before the Model Does

A model like this reads up to a fixed number of tokens and drops the rest. The cap here was 160. With a median of 143 and a 95th percentile of 171, 14% of records run past it. The embedding text puts each record's own fields first and the derived matching tokens, the lowest-priority layer, last, so a record over the cap loses tokens off its end, which is where the lowest-priority ones sit.

The same thinking set the vector size. The model natively produces 1,024 numbers per record, and it was trained Matryoshka-style, meaning the first few hundred numbers already make a usable shorter embedding on their own, like nesting dolls. So the question was how many to keep. I measured recall at 1, the share of test records whose closest match came back as the right one, on 10,000 test records with planted variations. It was 0.9994 at 384 numbers, 0.9998 at 768 and 0.9999 at 1,024. The run kept 768, which makes the output 461.4 GB instead of about 615 GB. It needed no retraining, only keeping the first 768 and rescaling each vector to length one.

Bank Every File the Moment It's Done

The records came as 128 part-files of about 1.17 million records each. Box k took every file whose position, divided by 16, left remainder k, so each box got 8 files at about 51 minutes of embedding apiece.

The first fleet ran a worker that kept every vector for its shard (its share of the files) in one file on its own disk and uploaded at the end. A box that died in hour six would take six hours of work with it, and the fleet only finishes when its slowest box does. I replaced that worker mid-run with one that uploads each part-file's vectors as soon as they're done and, on start-up, checks cloud storage and skips every part already there. Here it is, reduced, from embed_shard.py:

done = set(s3names(SHARD_PREFIX))                    # parts already uploaded -> resumetodo = [f for f in mine        if not ({os.path.basename(f).split(".")[0] + e for e in (".fvecs", ".ids")} <= done)]# ... embed one part-file ...# upload BOTH before deleting local -> the part is durable the instant this returnssubprocess.check_call(["aws", "s3", "cp", fp, f"{SHARD_PREFIX}{part}.fvecs"])subprocess.check_call(["aws", "s3", "cp", ip, f"{SHARD_PREFIX}{part}.ids"])

A part counts as done only when both its vectors file and its ID file are in storage. The ID file says which record each vector belongs to, and without it a vector is 3 KB of anonymous numbers.

When I made the swap, every box in the first fleet had already finished two files, and those vectors were sitting inside the old worker's single local file. Throwing them away would have meant redoing a quarter of the job. So a salvage script ran on each box. It read the box's own log for the list of finished files and their record counts, cut those files out of the big one by offset, and uploaded each under the new layout, where the new worker would find it and skip it. It refused to upload anything unless three checks held. This is the reduced version from salvage_done_parts.sh:

[ "$BYTES" -eq "$((SUM*V))" ] || { echo "REFUSE shard $KP: fvecs bytes != Σcount*$V"; exit 3; }[ "$LINES" -eq "$SUM" ]       || { echo "REFUSE shard $KP: ids lines != Σcount";       exit 4; }[ "$HDR"  -eq "$DIM" ]        || { echo "REFUSE shard $KP: dim header $HDR != $DIM";    exit 5; }

V is 3,076, the bytes one vector takes on disk: a 4-byte header saying "768" and then 768 numbers at 4 bytes each. If a file's size isn't exactly the record count times 3,076, something is torn or missing, and that box gets re-embedded. Thirty-two part-files came across, two per box, 37,458,383 records, which is 25% of the job. The relaunched fleet did the other 96, and its last vector landed at 21:40:38 UTC, 5.98 hours after the first one landed at 15:42:00.

Those numbers don't multiply out to 150 million on their face. 383 x 16 x 5.98 hours x 3,600 is 131,923,584, because the span starts at the first object in storage and the 32 salvaged files were encoded before that. The other 96 files hold 112,541,617 records, which at 6,128 a second (16 boxes at 383) need 5.10 hours. The remaining 0.88 hours of the 5.98 went to booting, syncing the model and relaunching, which the run log says the span includes.

Count the Output Without Downloading It

The output was 461.4 GB of vectors plus an ID map. Every vector takes exactly 3,076 bytes, so the total count is the sum of the file sizes divided by 3,076, and cloud storage reports file sizes without sending the files. That sum came to 150,000,000 exact. The sharding was even too, with the salvaged files per box ranging from 2,338,359 to 2,343,786 records, a spread of 0.23%. I enjoyed that number more than the speed.

The count can't see inside a vector, so a QA script read 250,000 vectors from five shards by byte range, pulling slices of files rather than whole ones. All 250,000 had 768 numbers. Their median length was 1.0000, with the 1st and 99th percentiles at 0.9996 and 1.0004, which is the rounding you expect from half-precision arithmetic (fp16, the model computing with 16-bit numbers instead of 32-bit ones). None held a NaN or an infinity, the values a computer produces when a sum blows up, and none was all zeros.

The step before embedding, turning each raw record into its embedding text, ran on an existing box in about an hour of CPU time and added nothing to the bill.

Calibrate on the Records You'll Pay For

If I ran it again I'd do these in order. First, calibrate on a sample of real records on the GPU you'll rent, never on synthetic strings and never on the dev box. Second, sweep the batch size on that same sample, and expect the winner to be small when record lengths vary. Third, size the fleet from the measured rate and let the window set the box count, since the rate already set the bill.

Fourth, make the worker bank each file the moment it's done, and do it before the first launch. I built mine during the run, after the first fleet was already going. Fifth, write the measured span, the truncation rate and the output checks into the run log as they happen. Mine holds all three in a 47-line metrics file.