Deterministic vs probabilistic matching, benchmarked
Deterministic vs probabilistic record matching, defined and tested on 1.15M synthetic company-name pairs: what each method finds, misses, and when to use it.
Take a CRM export with 15 companies and a business registry with 20. The CRM says Acme Inc, the registry says ACME Incorporated. The CRM says Johnson & Johnson Medical, the registry says J&J Medical Devices LLC. Every CRM company is in the registry, but not one name is spelled the same way on both sides.
We ran three approaches on exactly these two files (they ship as sample_data/companies/ in the Match Data Studio repo, and we labelled the true pairs by hand):
| Approach | Rule | Correct pairs found |
|---|---|---|
| Exact match | names identical | 0 / 15 |
| Normalized exact match | identical after lowercasing, stripping punctuation and legal suffixes | 9 / 15 |
| Fuzzy (difflib ratio) | best-scoring registry name for each CRM name | 15 / 15 |
Real run on the repository's sample files. A tiny dataset: it shows failure modes, it does not measure accuracy.
That table is the deterministic vs probabilistic question in miniature. Rules that demand equality are predictable but blind to anything they were not written for. Scoring approaches find more, but they hand you a number, and the number has to be turned into a decision. The rest of this post defines the terms precisely, then puts them through a larger benchmark that we measured and published.
The sample-company experiment, in full
The two files are also public downloads: sample_companies_crm.csv and sample_companies_registry.csv (names, cities and states only). If you download them, change the file paths in the snippet to match.
The script uses only the Python standard library. Run it from the repository root:
import csv, re
from difflib import SequenceMatcher
LEGAL = {"inc", "incorporated", "llc", "corp", "corporation", "co", "company", "ltd"}
def normalize(name):
s = re.sub(r"[^a-z0-9& ]", " ", name.lower())
return " ".join(t for t in s.split() if t not in LEGAL)
def sim(a, b):
return SequenceMatcher(None, normalize(a), normalize(b)).ratio()
crm = list(csv.DictReader(open("sample_data/companies/companies_crm.csv")))
reg = list(csv.DictReader(open("sample_data/companies/companies_registry.csv")))
# Hand-labelled truth: CRM001-014 -> REG001-014, CRM015 -> REG020
truth = {f"CRM{i:03d}": f"REG{i:03d}" for i in range(1, 15)} | {"CRM015": "REG020"}
by_exact = {r["registered_name"]: r["reg_id"] for r in reg}
by_norm = {normalize(r["registered_name"]): r["reg_id"] for r in reg}
hits = {"exact": 0, "normalized": 0, "fuzzy": 0}
for c in crm:
name, t = c["company_name"], truth[c["crm_id"]]
best = max(reg, key=lambda r: sim(name, r["registered_name"]))
hits["exact"] += by_exact.get(name) == t
hits["normalized"] += by_norm.get(normalize(name)) == t
hits["fuzzy"] += best["reg_id"] == t
if by_norm.get(normalize(name)) != t:
print(f"{name:27} -> {best['registered_name']} ({sim(name, best['registered_name']):.2f})")
print(hits)
Output (it prints only the six pairs normalization missed):
Johnson & Johnson Medical -> J&J Medical Devices LLC (0.50)
Pacific Northwest Tech -> Pacific NW Technology Inc (0.70)
Meridian Financial Group -> Meridian Financial Holdings (0.78)
Summit Healthcare Partners -> Summit Health Partners LLC (0.92)
Evergreen Paper Products -> Evergreen Paper Prods Corp (0.93)
Lakeshore Media -> Lakeshore Media Group (0.83)
{'exact': 0, 'normalized': 9, 'fuzzy': 15}
Three things stand out. Normalization alone took exact matching from 0 to 9 correct pairs, and it is still a pure rule. The six it missed all differ in words, not formatting: an acronym (J&J), abbreviations (NW, Prods), a swapped generic word (Group vs Holdings), a shortened word (Health), a dropped word (Group). And the fuzzy scores for correct pairs run from 0.50 to 1.00, so a fixed cut-off of 0.9 would have discarded four of these six correct pairs, and only a cut-off of 0.50 or lower keeps J&J Medical Devices LLC.
Two caveats keep this honest. Every CRM record has a true match and the registry has only 20 rows, so “pick the best candidate” flatters the fuzzy approach: it never has to say “no match”, and false positives are not tested. And 15 pairs illustrate failure modes; they do not measure accuracy. That is what the benchmark below is for.
Definitions, precisely
Deterministic matching links two records when explicit rules are satisfied: fields are equal, possibly after transformations such as lowercasing or stripping Inc. The output is yes or no, and you can always say which rule fired. A shared reliable identifier (a tax ID, a SKU, an email address you trust) is the cleanest case: an exact join on that key is a deterministic match.
Similarity-threshold matching (fuzzy matching) computes a continuous score with a string metric such as Jaro-Winkler, token overlap or character n-gram cosine, then applies a cut-off. The computation is deterministic: the same inputs always give the same score. But the score is not a probability, and the cut-off has to be chosen.
Probabilistic matching, in the technical sense, means the framework that goes back to Fellegi and Sunter (1969, “A Theory for Record Linkage”, Journal of the American Statistical Association). It asks, for each field comparison, how likely that outcome is among true matches versus among non-matches, and combines the evidence.
How Fellegi-Sunter weights work
The explanation of m and u below follows the Splink documentation. For each comparison (city agrees, state agrees, name similarity in a given bucket) the model uses two probabilities:
- m-probability: the probability of that outcome when the pair is a true match.
- u-probability: the probability of that outcome when the pair is not a match.
The weight of the outcome is the log of the ratio, log(m / u). Agreement on a field that is rarely shared by chance gets a large positive weight. Disagreement gets log((1 - m) / (1 - u)), which is negative. Assuming the comparisons are independent given match status, the weights add up to a total log-likelihood ratio for the pair, and that total is compared to thresholds. Fellegi and Sunter’s 1969 decision rule uses two: above the upper one the pair is a link, below the lower one a non-link, and the band in between is a possible link for clerical review.
An illustrative example, with invented probabilities rather than measured ones: if city agrees in 90% of true matches (m = 0.9) and in 2% of non-matching pairs (u = 0.02), city agreement is worth ln(0.9 / 0.02) = 3.81 in natural-log units and city disagreement is worth ln(0.1 / 0.98) = -2.28. A field that agrees by chance more often gets a smaller agreement weight, which is the whole point: the model learns how informative each field is instead of treating every agreement as equal.
The benchmark below estimates m and u from labelled pairs.
We measured it
We built an offline benchmark and ran it on our own machine. The benchmark code is not published, so these figures are our measurements and cannot be rerun from this page; the sample-file snippets elsewhere in this post can.
The data is synthetic. Side A has 1,000 company records; side B has 1,150, of which 700 are true matches for A records, 150 are “siblings” (an A name with one business word changed, a deliberate hard non-match) and 300 are unrelated. That is 1,000 × 1,150 = 1,150,000 possible pairs. Names are generated from word lists (Cedar Foods Group Inc style). The generator gives 35% of true matches an identical name on both sides, which is an assumption, not a fact about real data (in the test seed 264 of the 700 ended up uncorrupted: in the generator some corruptions are skipped when they cannot apply to a name, for example an acronym needs at least three words). The other 65% get one or two corruptions from: case change, legal-suffix change, punctuation, abbreviation (Industries to Inds), a one-character typo, word-order swap, a dropped generic word (Group, Holdings), or an acronym (initials only). Every entity is unique after normalization, so the ground truth is unambiguous.
Protocol. Thresholds are tuned on one random seed and evaluated on a different seed. Each A record is linked to its single best-scoring B record if the score clears the threshold. Precision is correct links divided by links made; recall is correct links divided by the 700 true matches; F1 is their harmonic mean. If those terms are new, this guide to precision, recall and F1 walks through them.
| Method | Family | Precision | Recall | F1 | Threshold | Pairs scored | Time |
|---|---|---|---|---|---|---|---|
| Exact match (raw name) | Deterministic | 1.000 | 0.414 | 0.586 | equality | 1,150,000 | 0.0 s |
| Normalized exact match | Deterministic | 1.000 | 0.717 | 0.835 | equality | 1,150,000 | 0.0 s |
| Normalized name + same state | Deterministic | 1.000 | 0.717 | 0.835 | equality | 1,150,000 | 0.0 s |
| Jaro-Winkler on normalized names | Similarity threshold | 0.971 | 0.821 | 0.890 | 0.943 | 1,150,000 | about 15 s |
| Token overlap (Jaccard) | Similarity threshold | 1.000 | 0.784 | 0.879 | 0.667 | 154,435 | 0.1 s |
| Character 3-gram TF-IDF cosine | Similarity threshold | 0.902 | 0.853 | 0.877 | 0.828 | 554,988 | 0.2 s |
| Fellegi-Sunter style weights | Probabilistic | 1.000 | 0.806 | 0.892 | 10.810 | 554,988 | 0.6 s |
Synthetic data, one run, no confidence intervals. Times are pure-Python implementations on one arm64 Mac and only show relative cost.
The Fellegi-Sunter style method here works in two steps: trigram TF-IDF proposes the top 30 candidates for each A record, then each candidate is scored by summing log-likelihood weights for four comparisons (Jaro-Winkler name similarity in four buckets; state agrees; city agrees; revenue within 25%). The weights were estimated from labelled tuning data.
Overall scores hide the interesting part. Table 2 splits recall by the kind of corruption each true match carries.
| Method | None (264) | Case (90) | Punct. (77) | Suffix (75) | Abbrev. (80) | Typo (85) | Word order (77) | Dropped word (55) | Acronym (30) |
|---|---|---|---|---|---|---|---|---|---|
| Exact (raw) | 1.00 | 0.17 | 0.14 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| Normalized exact | 1.00 | 0.81 | 0.79 | 0.84 | 0.80 | 0.01 | 0.31 | 0.00 | 0.00 |
| Normalized + same state | 1.00 | 0.81 | 0.79 | 0.84 | 0.80 | 0.01 | 0.31 | 0.00 | 0.00 |
| Jaro-Winkler | 1.00 | 0.92 | 0.86 | 0.89 | 0.85 | 0.73 | 0.49 | 0.07 | 0.00 |
| Token overlap | 1.00 | 0.87 | 0.86 | 0.88 | 0.85 | 0.14 | 0.81 | 0.00 | 0.00 |
| 3-gram TF-IDF | 1.00 | 0.91 | 0.88 | 0.91 | 0.95 | 0.33 | 0.66 | 0.85 | 0.00 |
| Fellegi-Sunter style | 1.00 | 0.90 | 0.84 | 0.89 | 0.84 | 0.67 | 0.42 | 0.05 | 0.00 |
Numbers in parentheses are how many true matches carry that corruption. A match can carry two corruptions, so the counts add up to more than 700, and a method can miss a 'case' match because of its second corruption.
What the numbers say
Raw exact matching found 41% of true matches and made no wrong links. It only finds names that are already identical. The zero wrong links is a design artifact of the synthetic data: every entity is unique after normalization, so an equality link cannot be wrong here. Real data with duplicates or different entities sharing a name can produce wrong equality links; that was not tested.
Normalization did most of the work. Lowercasing, stripping punctuation and legal suffixes, and expanding a short list of abbreviations raised recall from 0.414 to 0.717 with precision still at 1.000. It is still a deterministic rule and it costs nothing; in the benchmark, each similarity method normalizes names first too. If you skip one step, do not let it be this one; data cleaning before matching goes through the usual transformations.
The best four methods are effectively tied. Fellegi-Sunter style 0.892, Jaro-Winkler 0.890, token overlap 0.879, trigram TF-IDF 0.877. On one synthetic run, a 1.5-point F1 gap is not a reason to choose one over another.
They tie for different reasons. Each method fails on different corruption types:
- Typos are almost invisible to equality rules (recall 0.01). Jaro-Winkler finds 73% of them; token overlap finds 14%, because a misspelled word is a different token.
- Word-order swaps hurt Jaro-Winkler (0.49) but barely bother token overlap (0.81), which treats a name as a set of words.
- Dropped words are found almost only by trigram TF-IDF (0.85). Every other string method is at 0.07 or below.
- The Fellegi-Sunter style model’s profile tracks Jaro-Winkler’s (typo 0.67 vs 0.73, word order 0.42 vs 0.49). In the benchmark, bucketed Jaro-Winkler name similarity is one of its four features.
Nothing found an acronym. Every method scored 0.00 on the 30 acronym matches. Initials share no words and almost no characters with the full name, so string similarity has nothing to work with. Recovering them needs knowledge of the alias, a lookup table, or other fields.
Precision differs more than F1 suggests. Trigram TF-IDF had the highest recall (0.853) but precision 0.902, so roughly 1 in 10 of its links were wrong. Jaro-Winkler’s 0.971 means about 3 in 100. Token overlap and the Fellegi-Sunter style model made no wrong links on this run. The Fellegi-Sunter model also looks at state, city and revenue, but the run does not isolate whether those fields or the name bands made the difference.
The “same state” rule added nothing here, because names are unique after normalization, so there was nothing for a second field to disambiguate. In real data, where two different entities can share a name, a second field is how you separate them, but this benchmark did not test that case.
Run time differs, but for two reasons at once. Jaro-Winkler compared all 1,150,000 pairs and took about 15 seconds on our machine (12 to 17 seconds across runs). Token overlap used an inverted index, scored only the 154,435 pairs that share at least one word (13.4% of all pairs), and took 0.1 seconds. Trigram TF-IDF scored 554,988 pairs (48.3%). The runs differ in both the similarity metric and the number of pairs scored, and this benchmark does not separate the two effects. Reducing the pairs scored is what blocking keys are for.
Limits of this benchmark
Read the numbers with these in mind:
- Synthetic data. Names come from word lists and corruptions come from a fixed menu. Real names have distributions this does not model.
- One run, no confidence intervals. Differences of a few points are not meaningful.
- The 35% identical-name share is an assumption. The overall recall figures depend on it; whether the per-type recalls do was not tested.
- Thresholds were tuned on a separate seed of the same generator, which is better than tuning on the test data but much easier than transferring a threshold to a new real dataset.
- Pure-Python timings on one machine. They show relative cost, not what an optimized library would do.
- Embeddings and LLM confirmation are not in the tables above. They were run separately, and the results and their caveats are in AI embeddings vs rule-based matching.
- No same-name different-entity cases, so equality rules could not make wrong links, and the value of a second field for disambiguation is untested.
Where embeddings and LLMs fit
Neither is in the tables above, so this section is conceptual. A separate run on the same data is reported in AI embeddings vs rule-based matching: embeddings reached an F1 of 0.892, in the range of the best string methods, and found 0 of the 30 acronyms; adding an LLM yes/no step lowered F1 to 0.866 with the prompt we used.
An embedding model turns a piece of text into a vector of numbers, and similar texts should land near each other, which you then compare with cosine similarity. Google’s gemini-embedding-001, for example, is text-only, accepts up to 2,048 input tokens and returns 3,072 dimensions by default (configurable from 128 to 3,072). Structurally it is another similarity score with a threshold to choose. The hope is that it captures meaning the string metrics cannot, such as the acronym and swapped-word cases above. Whether it does on your data is something to measure, not assume; AI embeddings vs rule-based matching discusses the trade-offs.
LLM confirmation asks a language model whether two records describe the same entity. A model may know that J&J stands for Johnson & Johnson, which no string metric can know. It also costs money per pair and is not guaranteed to be consistent, so it belongs on a small, pre-filtered set of candidates, with its decisions checked against labelled pairs.
What to use when
Derived from the measured failure patterns, not from a universal rule:
| If your true matches mostly differ by... | Start with | Benchmark evidence |
|---|---|---|
| Formatting only: case, punctuation, legal suffixes | Normalized exact match | Recall 0.79 to 0.84 on these types; precision 1.000, but only because entities were unique after normalization |
| Typos | Jaro-Winkler (or another character-level metric) | Typo recall 0.73 vs 0.01 for normalized exact |
| Word order | Token overlap | 0.81 vs 0.49 for Jaro-Winkler |
| Dropped generic words | Character 3-gram TF-IDF | 0.85; every other string method 0.07 or below |
| Acronyms and aliases | Not string similarity: lookup tables, other fields, or a semantic method you test | 0.00 for every method |
| Several of the above | Normalization first, then a scored method, with a held-out labelled sample to compare | Top four methods within 0.015 F1 |
Synthetic benchmark, one run. Treat it as a map of failure modes, not a leaderboard.
Two more rules follow from the results. If wrong links are expensive, the methods that made none here (normalized exact, token overlap, the Fellegi-Sunter style model) are a starting point, but check them on labelled near-duplicates from your own data, since the benchmark had no same-name entities; send the rest to review rather than lowering a threshold. And since the methods fail on different error types, combining them is worth testing, but this benchmark did not measure a combination, and a union of methods also collects all of their false positives. The step-by-step version of this decision, starting from a labelled sample of your own data, is in how to choose a matching algorithm.
Where Match Data Studio fits
Match Data Studio’s pipeline puts the deterministic work first. It normalizes columns and applies string pre-filters (exact, prefix, contains) and numeric pre-filters (percentage difference) without any AI. It then runs AI extraction and gemini-embedding-001 embeddings only on rows that survived, scores candidate pairs by cosine similarity against thresholds, and can optionally ask gemini-2.5-flash-lite to confirm pairs. Those AI steps are not in the tables above. The embedding and LLM steps were measured separately in AI embeddings vs rule-based matching, and this post makes no accuracy claim for the product pipeline itself.
Keep reading
Ready to match your data?
Create a free account, upload your CSV files, and configure matching in plain English. 100 free credits — no credit card required.
Free account No credit card Match your files in minutes