Darkset

Identity resolution at scale across consumer, credit-bureau, and device-graph data: a fine-tuned embedding model, a GPU pipeline that embedded 150M records for about $230, graph clustering into stable IDs, and an FCRA gate on every credit-bureau read.

Identity resolutionEmbeddingsGPUGraph clusteringFCRA

0.9387F1 on 8.1M records

150Mrecords embedded

6.8hon 16 GPUs, about $230

Focus

Consumer, credit-bureau, and device-graph data

Darkset resolves identities across consumer, credit-bureau, and device-graph data. A Qwen3 embedding model fine-tuned for record matching, Matryoshka-truncated from 1024 to 384 dimensions, scores 0.9387 F1 on an 8.1M-record evaluation. The labeled pairs come from a deterministic 107-rule corruption engine that replaced an LLM-based generator, so the right answer for every pair is known from the rule that made it, with no hand labeling.

Highlights

▹150M records embedded on 16 L40S GPUs in 6.8 hours: 6,128 records per second, about $230, recall@1 0.9998, run head to head across cloud platforms with the same model on each
▹Search on a Milvus 2.6 distributed HNSW index behind a Text Embeddings Inference backend: 3.2x faster in wall-clock time at under 0.001 F1 drift
▹Deterministic attribute blocking and graph clustering (stateless Union-Find, Spark GraphFrames, Neptune weakly-connected components) resolve the match graph into stable IDs
▹An FCRA permissible-purpose gate on every credit-bureau read, covering the ten Section 604 purpose categories, with short-lived per-request claims and a scope registry that tiers credit as the only critical data class
▹An immutable audit trail with retention classes on every state-changing call, plus hashed email and device-ID suppression lists

Built with

Qwen3 embeddingsMilvusText Embeddings InferenceSpark GraphFramesNeptuneL40S GPUs

Source in a private repo