How to choose a matching algorithm for your data
A six-step procedure for picking a record matching method: tag a labelled sample by difference type, map types to methods, block, and tune on a hold-out.
A leaderboard built on someone else’s data may not transfer to yours. When we benchmarked seven methods on 1,150,000 synthetic company-name pairs, the top four finished within 0.015 F1 of each other, yet they differed in which error types they missed: Jaro-Winkler handled typos but not word-order swaps, token overlap the reverse, and trigram TF-IDF was the only one of them to recover dropped words. The Fellegi-Sunter style model overlapped heavily with Jaro-Winkler (typo 0.67 vs 0.73, word order 0.42 vs 0.49, dropped word 0.05 vs 0.07). The full results are in deterministic vs probabilistic matching, benchmarked. So the useful question is not “which algorithm is best” but “which kinds of difference dominate my data, and which methods recover those”. This post is a procedure for answering it.
Step 1: Profile a labelled sample by difference type
Pull a sample of record pairs you can label by hand: something like 100-200 pairs, including obvious matches, obvious non-matches, and the awkward ones in between. Mark each pair as match or non-match. Then, for every true match, write down how the two records differ.
Use a fixed list of tags so the counts are comparable. This is the taxonomy the benchmark used:
| Tag | Example |
|---|---|
| case | Acme Inc / ACME Incorporated |
| suffix | Atlas Global Trading / Atlas Global Trading LLC |
| punctuation | Coastal Shipping Co. / Coastal Shipping Corporation |
| abbreviation | Evergreen Paper Products / Evergreen Paper Prods Corp |
| typo | a single-character error in one word |
| word_order | two neighbouring words swapped |
| dropped_word | Lakeshore Media / Lakeshore Media Group |
| acronym | Johnson & Johnson Medical / J&J Medical Devices LLC |
The named examples are real pairs from the sample_data/companies files in the Match Data Studio repo. The typo and word-order rows have no example there.
A pair can carry several tags. Add tags when the list does not fit; an other bucket that keeps growing means your data has a pattern worth naming.
Here is the tally for the 15 true pairs in the repository’s sample company files, using our hand tags, with the benchmark’s best per-type recall attached. The tags are judgement calls: we tagged Meridian Financial Group / Meridian Financial Holdings as other (a swapped generic word rather than a dropped one), Healthcare vs Health as other rather than abbreviation, and J&J Medical Devices as acronym plus dropped word.
The two files are also public downloads: sample_companies_crm.csv and sample_companies_registry.csv (names, cities and states only). If you download them, change the file paths in the Step 4 snippet to match.
from collections import Counter
# The 15 true pairs from sample_data/companies with our hand tags (one pair can carry several tags).
labelled = [
("Acme Inc", "ACME Incorporated", ["case", "suffix"]),
("Johnson & Johnson Medical", "J&J Medical Devices LLC", ["acronym", "dropped_word", "suffix"]),
("Pacific Northwest Tech", "Pacific NW Technology Inc", ["abbreviation", "suffix"]),
("Green Valley Farms LLC", "Green Valley Farms", ["suffix"]),
("Meridian Financial Group", "Meridian Financial Holdings", ["other"]), # generic word swapped
("Bright Horizons Education", "Bright Horizons Education Inc.", ["suffix", "punctuation"]),
("Summit Healthcare Partners", "Summit Health Partners LLC", ["other", "suffix"]), # Healthcare -> Health
("Red Rock Construction", "Red Rock Construction Co", ["suffix"]),
("Coastal Shipping Co.", "Coastal Shipping Corporation", ["suffix", "punctuation"]),
("BlueSky Analytics", "BlueSky Analytics Inc", ["suffix"]),
("Evergreen Paper Products", "Evergreen Paper Prods Corp", ["abbreviation", "suffix"]),
("Atlas Global Trading", "Atlas Global Trading LLC", ["suffix"]),
("Pinnacle Insurance Corp", "Pinnacle Insurance Corporation", ["suffix"]),
("Metro Dining Group", "Metro Dining Group Inc", ["suffix"]),
("Lakeshore Media", "Lakeshore Media Group", ["dropped_word"]),
]
# Best recall per difference type in the synthetic benchmark (Table 2).
BEST = {"case": "Jaro-Winkler 0.92", "punctuation": "trigram TF-IDF 0.88", "suffix": "trigram TF-IDF 0.91",
"abbreviation": "trigram TF-IDF 0.95", "typo": "Jaro-Winkler 0.73", "word_order": "token overlap 0.81",
"dropped_word": "trigram TF-IDF 0.85", "acronym": "none (0.00 for all)"}
tally = Counter(tag for _, _, tags in labelled for tag in tags)
for tag, n in tally.most_common():
print(f"{tag:13} {n:2}/{len(labelled)} pairs benchmark best: {BEST.get(tag, 'not in taxonomy')}")
Output:
suffix 13/15 pairs benchmark best: trigram TF-IDF 0.91
dropped_word 2/15 pairs benchmark best: trigram TF-IDF 0.85
abbreviation 2/15 pairs benchmark best: trigram TF-IDF 0.95
other 2/15 pairs benchmark best: not in taxonomy
punctuation 2/15 pairs benchmark best: trigram TF-IDF 0.88
case 1/15 pairs benchmark best: Jaro-Winkler 0.92
acronym 1/15 pairs benchmark best: none (0.00 for all)
Fifteen pairs is far too few to trust the shares (one pair is 6.7 percentage points), but the shape is already informative: almost every pair differs by a legal suffix, and a handful differ in actual words. Replace labelled with your own sample and the same tally tells you where to look.
While you label, also note the non-matches that look like matches: two different companies with near-identical names. You will need them in Step 5.
Step 2: Map difference types to methods
The benchmark’s per-type recall (Table 2 in the benchmark post linked above) gives a map from tag to method. It is synthetic and from a single run, so treat it as a map of failure modes rather than a forecast:
| If your tally is dominated by... | Try first | Benchmark recall on that type | Known blind spot in the benchmark |
|---|---|---|---|
| case, punctuation, suffix | Normalized exact match | 0.79 to 0.84; precision 1.000 only because entities were unique after normalization | typos 0.01, dropped words 0.00 |
| typo | Jaro-Winkler on normalized names | 0.73 | word order 0.49, dropped words 0.07 |
| word_order | Token overlap (Jaccard) | 0.81 | typos 0.14 |
| dropped_word or abbreviation | Character 3-gram TF-IDF cosine | 0.85 and 0.95 | typos 0.33; precision 0.902 overall |
| acronym | Alias table, other fields, or a semantic method you test | 0.00 for every method | string similarity has nothing to compare |
Benchmark: 1,000 vs 1,150 synthetic company names, thresholds tuned on a separate seed, one run.
Whatever the tally says, normalize first. In the benchmark, lowercasing, stripping punctuation and legal suffixes, and expanding a short list of abbreviations raised exact-match recall from 0.414 to 0.717 at precision 1.000. That precision is a property of the benchmark’s data design: every synthetic entity is unique after normalization, so an equality link could not be wrong. Real data with duplicates or same-name entities can produce wrong equality links. In the sample files, the same idea (without the abbreviation list) took exact matching from 0 to 9 of 15. Normalization is a deterministic rule, it is free, and it makes every later method’s job easier. Data cleaning before matching lists the usual transformations.
Read the sample tally through this table: suffix dominates, and normalization handles it. What is left is two abbreviations and two dropped words (trigram territory), two word changes that fit no tag (Group vs Holdings, Healthcare vs Health), and one acronym that no string method recovered in the benchmark.
Step 3: Decide how candidates are generated
Scoring every pair is the simplest option and the most expensive. In the benchmark, side A had 1,000 records and side B 1,150, so all pairs means 1,150,000 comparisons:
| Method | How candidates were generated | Pairs scored | Share of all pairs | Time (pure Python) |
|---|---|---|---|---|
| Jaro-Winkler | every pair | 1,150,000 | 100% | about 15 s |
| Character 3-gram TF-IDF | pairs sharing a trigram | 554,988 | 48.3% | 0.2 s |
| Fellegi-Sunter style | trigram stage scored 554,988 pairs; rescored the top 30 per record | 554,988 (trigram stage) | 48.3% | 0.6 s |
| Token overlap | inverted index: pairs sharing a normalized word | 154,435 | 13.4% | 0.1 s |
Timings are relative only: pure-Python implementations on one machine.
Two consequences matter more than the timings. First, candidate generation sets a ceiling on recall: a pair that is never generated can never be linked. A word index can never propose an acronym match, because initials share no word with the full name. Second, cost scales with the number of pairs that reach the expensive step, which matters most once AI is involved (Step 6). Blocking on a field such as state or ZIP code is the other common lever; it was not tested in this benchmark, and setting up blocking keys covers how to choose one.
Step 4: Pick thresholds on a hold-out, not on the data you report
Every scored method needs a cut-off, and the tuned values in the benchmark show why you cannot carry one over from another method: Jaro-Winkler 0.943, token Jaccard 0.667, trigram cosine 0.828, and 10.810 for the Fellegi-Sunter log-likelihood sum. A “0.8” means something different on each scale. The benchmark tuned each threshold on one random seed and reported results on another; do the same with two separate labelled samples.
Then check whether any single threshold can work at all. This script scores all 300 CRM-registry pairs in the sample files (15 × 20), not just each record’s best candidate:
import csv, re
from difflib import SequenceMatcher
LEGAL = {"inc", "incorporated", "llc", "corp", "corporation", "co", "company", "ltd"}
def normalize(name):
s = re.sub(r"[^a-z0-9& ]", " ", name.lower())
return " ".join(t for t in s.split() if t not in LEGAL)
crm = list(csv.DictReader(open("sample_data/companies/companies_crm.csv")))
reg = list(csv.DictReader(open("sample_data/companies/companies_registry.csv")))
truth = {f"CRM{i:03d}": f"REG{i:03d}" for i in range(1, 15)} | {"CRM015": "REG020"}
# Score every CRM x registry pair (15 x 20 = 300), not just each record's best candidate.
pairs = [(SequenceMatcher(None, normalize(c["company_name"]), normalize(r["registered_name"])).ratio(),
truth[c["crm_id"]] == r["reg_id"], c["company_name"], r["registered_name"])
for c in crm for r in reg]
for t in (0.5, 0.7, 0.9):
kept_true = sum(s >= t and ok for s, ok, *_ in pairs)
kept_wrong = sum(s >= t and not ok for s, ok, *_ in pairs)
print(f"threshold {t}: true pairs kept {kept_true}/15, wrong pairs kept {kept_wrong}/285")
s, _, a, b = max(p for p in pairs if not p[1])
print(f"highest-scoring wrong pair: {a} | {b} ({s:.2f})")
Output:
threshold 0.5: true pairs kept 15/15, wrong pairs kept 6/285
threshold 0.7: true pairs kept 13/15, wrong pairs kept 0/285
threshold 0.9: true pairs kept 11/15, wrong pairs kept 0/285
highest-scoring wrong pair: Meridian Financial Group | Metro Dining Group Inc (0.67)
The true pair Johnson & Johnson Medical / J&J Medical Devices LLC scores 0.50, while the unrelated Meridian Financial Group / Metro Dining Group Inc scores 0.67. No cut-off keeps the first and rejects the second. (At 0.7 the second missed true pair is Pacific NW Technology Inc, which scores 0.698 and prints as 0.70 at two decimals.) That is the signal to stop tuning and change something else: a method suited to the failing tag, another field, or human review for a middle band. Understanding similarity thresholds goes deeper on the trade-off.
Step 5: Evaluate links, and stress-test with hard negatives
Measure on the hold-out sample, counting links:
- Precision = correct links / links made.
- Recall = correct links / true matches.
- F1 = the harmonic mean of the two.
The guide to precision, recall and F1 covers the details. Three habits make the numbers trustworthy:
- Match the evaluation to how links are made. The sample files show the gap. Picking each CRM record’s best candidate found all 15 pairs, but only because every CRM record has a match. Scored pair by pair, a threshold low enough for all 15 also admitted 6 wrong pairs.
- Include hard negatives. The benchmark’s side B included 150 “sibling” records, each an A name with one business word changed, precisely so methods had to reject near-identical names. Your labelled sample needs the same kind of near-miss: different branches, parent and subsidiary, same name in a different city.
- Report recall by tag, not just overall. Overall F1 hid the most useful result in the benchmark: Jaro-Winkler and token overlap scored 0.890 and 0.879 overall, yet differed by 0.32 on word-order swaps (0.49 vs 0.81).
Step 6: Decide where deterministic rules stop and AI starts
After normalization and a string method suited to your tally, look at the true matches that are still missed. In the benchmark, two types stand out in Table 2: every string method found 0 of 30 acronyms, and every string method except trigram TF-IDF stayed at 0.07 or below on dropped words. In the sample files, the hard cases are an acronym and those two word changes.
That residue is where embeddings or an LLM confirmation step might help, because those cases might need meaning, not characters. We ran both on this benchmark in a separate post: embeddings also found 0 of the 30 acronyms and 0.36 of the dropped-word matches (trigram TF-IDF found 0.85), and adding an LLM yes/no step lowered F1 from 0.892 to 0.866 with the prompt we used. That is one dataset and one prompt, so test them on exactly those labelled residual pairs before adopting them.
Cost depends on how many pairs reach the model. As arithmetic on assumptions, not a measurement: assume about 150 input tokens and 10 output tokens per pair, at gemini-2.5-flash-lite standard prices as of October 2026 ($0.10 per million input tokens, $0.40 per million output tokens, per Google’s pricing page):
| Pairs sent to the model | Input tokens | Output tokens | Cost |
|---|---|---|---|
| 10,000 (after candidate filtering) | 1.5M → $0.15 | 0.1M → $0.04 | $0.19 |
| 1,150,000 (every benchmark pair) | 172.5M → $17.25 | 11.5M → $4.60 | $21.85 |
Assumes 150 input and 10 output tokens per pair. Output prices include thinking tokens, so a model that reasons at length uses more output tokens than assumed. The Batch API is priced 50% lower. Prices change.
The ratio is what matters: cost tracks the pair count, so Step 3 decides most of it.
Where Match Data Studio fits
Match Data Studio follows the same order. Stage 1 normalizes columns and applies string pre-filters (exact, prefix, contains) and numeric pre-filters (percentage difference) without AI. AI extraction and gemini-embedding-001 embeddings run only on rows that survived. Pairs are scored by cosine similarity against thresholds, and optional LLM confirmation uses gemini-2.5-flash-lite, capped at 50,000 candidate pairs per full run. A sample run processes 50 rows from one side against 1,000 from the other first, which is a convenient place to check your labelled pairs before a full run. The product pipeline itself has not been benchmarked. Its embedding and LLM steps were measured only in a separate offline benchmark run, described in AI embeddings vs rule-based matching.
The benchmark behind the tables in this post is our own and its code is not published. The sample-file steps above are runnable, and the procedure applies to any labelled sample of your own data.
Keep reading
- Deterministic vs probabilistic matching, benchmarked
- Fuzzy matching algorithms explained: Levenshtein, Jaro-Winkler, and beyond
- Five matching mistakes that silently ruin your results
- Matching at scale: strategies for millions of records
- Best data matching tools in 2026: a feature-by-feature comparison
Ready to match your data?
Create a free account, upload your CSV files, and configure matching in plain English. 100 free credits — no credit card required.
Free account No credit card Match your files in minutes