Andrew Fribush

fig. 2.20.3: field order puts the 160-token cap on the lowest-priority tail; the 14% of records past it lose only that

fig. 2.20.3 · field order puts the 160-token cap on the lowest-priority tail; the 14% of records past it lose only that.
FIELD ORDER PUTS THE 160-TOKEN CAP ON THE TAIL: 14% LOSE ONLY THATTOKENS PER RECORD; TICKS: MEDIAN, P95, LONGEST; ACCENT: THE CAP.0 TOKENS50100150200MEDIAN 14395TH PERCENTILE 171LONGEST 195CAP 160 TOKENS14% OF RECORDS RUN PAST THE CAP AND LOSE ONLYTHEIR TAILWRITTEN FIRST TO LAST: NAME · ADDRESS · DOB · GEO · DERIVED SEMANTIC TOKENS,THE TAIL THE CAP CUTS

Token lengths of the embedding text under the model’s own tokenizer; max_seq_length 160. The derived semantic tokens are the enrichment’s L2 layer, written after the record’s own fields.

source: credit_150m_embedding_metrics.csv lines 7, 31-34; the vector handoff note in docs/, line 39 · tokenCap.data.ts

View in context: Embedding 150 million records for about $230 → · All figures

Andrew Fribush, software and computer engineer in Austin. I build consequential systems where software meets institutions: finance, healthcare, commerce, government data, and the mechanisms that govern AI agents. Taking new engagements: hello@andrewfribush.com.