On the sample company files in our repository, Johnson & Johnson Medical in a CRM export has to be matched to J&J Medical Devices LLC in a business registry. Python’s difflib still ranks the right registry record first, but scores the pair at 0.50, so any cutoff of 0.7 or higher would throw the correct match away. A string metric sees characters. It cannot know that J&J stands for Johnson & Johnson.

Each method in a modern matching pipeline addressed a specific problem, and each left something unsolved. Those leftovers are still in your data, which makes the history a practical guide to choosing methods, not trivia. Below is a timeline built from primary sources, with what each step addressed and what it did not.

The timeline at a glance

Milestones in record matching (primary sources)
YearMilestoneSourceWhat it addressed
1918, 1922SoundexR. C. Russell, US Patents 1,261,167 and 1,435,663Filing names by sound, so spelling variants share a code
1965 (Russian), 1966 (English)Levenshtein edit distanceDoklady Akademii Nauk SSSR; Soviet Physics DokladyCodes correcting deletions, insertions and reversals; the edit distance named after him
1969A Theory for Record LinkageFellegi and Sunter, JASALink / possible link / non-link decisions with controlled error levels
1989Advances in Record-Linkage Methodology as Applied to Matching the 1985 Census of Tampa, FloridaJaro, JASARecord-linkage methodology applied to a census
1990String Comparator Metrics and Enhanced Decision Rules in the Fellegi-Sunter Model of Record LinkageWinkler, Proceedings of the Section on Survey Research Methods, American Statistical AssociationExtends the Jaro string comparator; enhanced decision rules
2013word2vecMikolov et al., arXivEstimating vector representations of words
2016MagellanKonda et al., PVLDBBuilding entity matching management systems
2018Deep Learning for Entity Matching: A Design Space ExplorationMudgal et al., SIGMODWhere deep learning helps: textual and dirty data, not structured
2019Sentence-BERTReimers and Gurevych, EMNLPSentence embeddings from Siamese BERT networks
2021 (arXiv 2020)DittoLi et al., PVLDBEntity matching with pre-trained language models
2022Can Foundation Models Wrangle Your Data?Narayan et al., PVLDBFoundation models on five data cleaning and integration tasks, entity matching among them

Full citations and links are in each section below.

1918 and 1922: Soundex, matching by sound

Robert C. Russell was granted US Patent 1,261,167, titled “Index”, on 2 April 1918, and a second patent with the same title, US 1,435,663, on 14 November 1922 (1918 patent, 1922 patent). The 1918 patent describes mapping letters to eight phonetic groups represented by numerals, so names that sound alike are filed under the same code. The US National Archives still documents the Soundex coding rules on its census research pages.

What it solved: finding a name when you do not know how it was spelled. What it left open: Soundex produces a code, not a degree of similarity, so it groups records rather than ranking them. Anything that is not a sound-alike spelling (abbreviations, reordered words, initials) is outside what it was designed for.

1965–1966: Levenshtein, counting edits

V.I. Levenshtein’s paper on codes capable of correcting deletions, insertions and reversals appeared in Russian in Doklady Akademii Nauk SSSR 163(4), 845–848, in 1965 (original), and in English as “Binary codes capable of correcting deletions, insertions, and reversals”, Soviet Physics Doklady 10(8), 707–710, in 1966. The edit distance that carries his name is the minimum number of single-character insertions, deletions and substitutions needed to turn one string into another.

What it solved: a graded measure of how far apart two strings are, which is exactly what typos need. What it left open: a distance between characters says nothing about meaning. J&J and Johnson & Johnson are far apart in edits and identical in fact. For how edit-based and other string metrics compare in practice, see fuzzy matching algorithms explained.

1969: Fellegi and Sunter, from comparisons to decisions

I.P. Fellegi and A.B. Sunter (1969), “A Theory for Record Linkage”, Journal of the American Statistical Association 64(328), 1183–1210 (doi:10.1080/01621459.1969.10501049). The paper frames matching as a decision for each pair with three outcomes, link, possible link or non-link, chosen so that error levels are controlled. The modern explanation of the m- and u-probabilities behind the weights is well documented in the Splink docs.

What it solved: combining many field comparisons into one decision, with an explicit middle band for pairs that need review instead of a forced yes or no. What it left open: the model needs good comparison functions for each field and good parameter estimates.

That approach still holds up. In our synthetic benchmark (1,000 × 1,150 company records, thresholds tuned on one seed and tested on another), a Fellegi-Sunter style scorer, with weights for name similarity, state, city and revenue estimated from labelled data, made no wrong links (precision 1.000) and found 80.6% of true matches. That was within about 1.5 F1 points of the best string-only methods, too close to rank on one run. The deterministic vs probabilistic matching post covers the full results.

1989–1990: Jaro and Winkler, comparators built for names

M.A. Jaro (1989), “Advances in Record-Linkage Methodology as Applied to Matching the 1985 Census of Tampa, Florida”, JASA 84(406), 414–420 (doi:10.1080/01621459.1989.10478785). W.E. Winkler (1990), “String Comparator Metrics and Enhanced Decision Rules in the Fellegi-Sunter Model of Record Linkage”, Proceedings of the Section on Survey Research Methods, American Statistical Association (PDF). Its abstract describes extending the Jaro string comparator.

The Jaro-Winkler implementation in our benchmark adds a bonus for up to four matching leading characters, with a scaling factor of 0.1, to Jaro scores above 0.7.

What it solved: string comparison inside a Fellegi-Sunter linkage. What it left open: anything that is not a character-level variation. The benchmark makes this concrete:

Share of true matches found, by corruption type (synthetic benchmark, test seed)
MethodTypo (n=85)Word order (n=77)Dropped word (n=55)Acronym (n=30)
Normalized exact match0.010.310.000.00
Jaro-Winkler on normalized names0.730.490.070.00
Token overlap (Jaccard)0.140.810.000.00
Character 3-gram TF-IDF cosine0.330.660.850.00
Fellegi-Sunter style weights0.670.420.050.00

Source: our offline benchmark, one run on synthetic data. Acronyms are initials only, such as CFG for Cedar Foods Group. Differences of a few points are not meaningful.

In this benchmark, Jaro-Winkler found most typos while equality rules found almost none. Word order and dropped words hurt different methods in different ways. And no string method found a single initials-only acronym. That last column is the gap that later, learned approaches were meant to address.

2013: word2vec, words become vectors

T. Mikolov, K. Chen, G. Corrado and J. Dean (2013), “Efficient Estimation of Word Representations in Vector Space”, arXiv preprint arXiv:1301.3781 (arXiv). As the title says, the work is about estimating vector representations of words.

What it solved: a way to represent words as vectors, so similarity can be computed between words that share no characters. What it left open: it represents words, not whole records, which mix names, addresses and codes.

2016: Magellan, matching as a system

P. Konda et al. (2016), “Magellan: Toward Building Entity Matching Management Systems”, PVLDB 9(12), 1197–1208 (doi:10.14778/2994509.2994535). As the title says, the paper is about building systems to manage entity matching as a whole, rather than about one more similarity function.

The DeepMatcher/Magellan dataset index lists public entity-matching benchmark datasets across products (Abt-Buy, Amazon-Google, Walmart-Amazon), citations (DBLP-ACM, DBLP-Scholar), restaurants (Fodors-Zagats), music (iTunes-Amazon), beer and companies. They are a useful second check alongside your own data.

2018: Mudgal et al., where deep learning helps

S. Mudgal, H. Li, T. Rekatsinas, A. Doan, Y. Park, G. Krishnan, R. Deep, E. Arcaute and V. Raghavendra (2018), “Deep Learning for Entity Matching: A Design Space Exploration”, Proc. SIGMOD 2018, pp. 19–34 (doi:10.1145/3183713.3196926).

Their finding is directly useful to practitioners: deep learning did not outperform existing solutions on structured data, but did on textual and dirty data. What it left open: whether a particular learned matcher beats a simpler baseline on your data is still something to measure.

2019: Sentence-BERT, comparable sentence embeddings

N. Reimers and I. Gurevych (2019), “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks”, EMNLP 2019 (arXiv:1908.10084). In a Siamese setup, the same network encodes each input separately, so every record gets its own fixed-length vector and pairs are compared afterwards with a vector similarity measure such as cosine.

What it solved: embeddings for whole sentences, computed once per record rather than once per pair, which makes semantic comparison workable for large lists. What it left open: a similarity score is not a decision. Look-alike non-matches, model numbers and quantities can sit close together in vector space, and the threshold still has to be tuned on labelled pairs. Those mechanics, with a runnable example, are in AI embeddings vs rule-based matching.

2020–2021: Ditto, pre-trained language models for matching

Y. Li, J. Li, Y. Suhara, A. Doan and W.-C. Tan, “Deep Entity Matching with Pre-Trained Language Models”, PVLDB 14(1), 50–60, 2021, first posted to arXiv in April 2020 (arXiv:2004.00584, doi:10.14778/3421424.3421431).

What it addressed: as its title says, bringing pre-trained language models to entity matching. What it left open: as with every learned matcher, how well it does on your records is an empirical question.

2022: foundation models enter data wrangling

Narayan, A., Chami, I., Orr, L., Arora, S., Re, C., “Can Foundation Models Wrangle Your Data?”, PVLDB 16(4), 738–746, 2022 (arXiv:2205.09911, doi:10.14778/3574245.3574258). The paper studies five data cleaning and integration tasks, entity matching among them.

What it opened: the question in its title, whether general-purpose foundation models can do data-wrangling work, entity matching included. LLM confirmation steps in today’s pipelines rest on the same idea: asking a general-purpose model whether two records describe the same thing. What it leaves open for you: every pair you send adds its own prompt and response tokens, so cost grows with the number of pairs, and accuracy on your data is something to measure, not assume. We ran exactly that test on our benchmark: on synthetic company names, embeddings scored about as well as the best string methods (F1 0.892), and a gemini-2.5-flash-lite yes/no check on top of them scored lower (0.866) with the prompt we used. That is one dataset and one prompt, so read the details and caveats rather than the headline.

What the history implies for choosing a method today

In this benchmark, each method missed a different kind of error. Typos, word-order swaps, dropped words and acronyms each defeat a different subset of methods. Start by finding out which errors your data contains, then pick methods that cover them. How to choose a matching algorithm walks through that decision.

The decision framework is still 1969’s. Fellegi and Sunter’s three outcomes map directly onto modern pipelines: confident links, confident non-links, and a middle band. Today that middle band can go to a human reviewer or to an LLM confirmation step; either way, you decide its width, and therefore its cost.

Learned models have the clearest case on textual and dirty data. That is where Mudgal et al. found deep learning ahead. On name data, the deterministic vs probabilistic matching benchmark shows how far rules and Fellegi-Sunter style scoring get without any learned model.

Labelled pairs never went away. Fellegi-Sunter weights need estimates; the benchmark’s weights came from labelled tuning data. Splink, the UK Ministry of Justice’s open-source Python package for probabilistic record linkage based on the Fellegi-Sunter model, needs no training data and runs on DuckDB locally and on Spark and other backends (docs, repository, MIT license). Even so, you need labelled pairs to know how well any method, including embeddings and LLMs, works on your files.

Candidate generation is its own problem. Every method above compares pairs, and pairs grow as the product of the two list sizes: 1,000 × 1,150 records is 1,150,000 pairs. Indexing and blocking decide how many of them you can afford to score.

Further reading

  • P. Christen (2012), Data Matching: Concepts and Techniques for Record Linkage, Entity Resolution, and Duplicate Detection, Springer (doi:10.1007/978-3-642-31164-2, ISBN 978-3-642-31163-5).
  • G. Papadakis, D. Skoutas, E. Thanos and T. Palpanas, “Blocking and Filtering Techniques for Entity Resolution: A Survey”, ACM Computing Surveys 53(2), 2020 (doi:10.1145/3377455).
  • Splink documentation, for a working Fellegi-Sunter implementation.

In Match Data Studio: a run applies string and numeric pre-filters with no AI first, then AI extraction and gemini-embedding-001 embeddings only for surviving rows, then cosine similarity with thresholds and optional LLM confirmation using gemini-2.5-flash-lite. The product’s matching accuracy has not been benchmarked.

Try it free →

Keep reading