ON THIS PAGE
By Efe Berke Colaker, Founder at GetleadReviewed by the Getlead editorial team for accuracy. Last updated August 2026.
Duplicates arrive quietly. Two exports, one CRM sync, a conference list and a form fill later, the same person exists three times with different capitalisation and one stale title.
The cost shows up somewhere else: two reps working the same account, a prospect receiving two sequences in a week, and a report that counts the same company twice.
Where duplicates actually come from
A duplicate is two or more records that refer to the same person or the same account, even when no field matches exactly.
- Multiple sources. A database export, a scraped list and a form submission describe the same person differently.
- Formatting drift. Trailing whitespace, capitalisation and punctuation in company names.
- Alias domains. A company using both example.com and example.co.uk, or a rebrand that kept the old domain alive.
- Personal variants. jane@ and jane.smith@ at the same company, both valid, one person.
- Re-imports. The same file loaded twice because nobody was sure the first run finished.
For example, a list assembled from three sources typically contains a few percent of exact duplicates and a larger share of near duplicates that no exact match query will ever find.
Choosing the matching key
Everything depends on this choice, and most bad dedupes come from matching on the wrong field.
A canonical domain is the registrable root of a company's web address, stripped of subdomains and normalised to lowercase, and it is the most stable identifier a B2B record carries.
For example, mail.example.com, www.example.com and example.com are one account. Matching on the raw value in the field creates three, and every downstream count inherits the error.
Deciding which record survives
Merging is not choosing a row, it is choosing a field at a time. The best record is usually assembled from several.
- Email: keep the address with the most recent successful verification, not the newest record.
- Name and title: keep the most recently sourced values, since people are promoted.
- Company fields: keep the canonical domain and the most recent headcount band.
- Provenance: keep the earliest acquisition date and every source, not just the last one.
- Suppression state: if any duplicate opted out, the merged record is opted out. This one is not negotiable.
That last rule prevents the worst failure mode in this whole exercise. A dedupe that keeps the newer row and discards an older opt-out quietly reinstates someone who asked to be left alone.
Verification state is worth keeping per address rather than per person. When two addresses exist for one contact, the one that verified most recently is the one to send to, and the other is worth retaining as a fallback.
Safeguards that stop a dedupe from deleting pipeline
A deduplication run is a bulk delete wearing a friendly name. Four safeguards make it reversible.
- Never hard delete. Archive the discarded record for at least 90 days with a pointer to the survivor.
- Dry run first. Produce the merge report and read a sample of 50 before executing anything.
- Exclude open opportunities. Records attached to a live deal or an ongoing conversation stay untouched.
- Log every merge. Which records, which key matched, which fields won, and when.
For example, a fuzzy match on first name plus domain will occasionally merge two genuine people, such as two colleagues both called Daniel. The archive is what turns that from a lost contact into a five minute fix.
When to run it
Deduplication belongs at the boundary where records enter, not as a quarterly cleanup project.
Run exact matching at import, before anything is written. Run fuzzy matching monthly across the active database, when there is time to read the report. Run an account level pass before any campaign that targets companies rather than individuals.
The import time check is the one that pays for itself, because a duplicate that never enters the database never has to be found later, and never sends anyone a second sequence.
Track the duplicate rate as a standing metric. Under 5% on canonical domain is healthy, and a rising number usually means a new import path is bypassing the check rather than that the data got worse.
Sources and method
First-party data (Getlead, 2026): the verification split of 43.4% confirmed valid, 23.9% invalid, 16.7% catch-all and 16.0% unknown comes from 383,368 addresses analyzed through live SMTP verification, and the 0.51% bounce rate comes from 34,973 tracked sends, aggregated and anonymized at campaign level. Full method in our cold email benchmark study.
External sources: the requirement to honor an opt-out and the accompanying record keeping obligations come from the FTC CAN-SPAM compliance guide; where records originate with a third party, Article 14 of the GDPR sets out what you must be able to tell the person.
Figures were checked in August 2026.
Frequently asked questions
What should I match on when deduplicating leads?
Normalised email address for exact person level matching, first name plus canonical company domain for fuzzy person matching, and canonical domain for account level matching. Never match on the company name string, since punctuation and capitalisation defeat it.
What is a canonical domain?
The registrable root of a company's web address, stripped of subdomains and lowercased, so mail.example.com, www.example.com and example.com all resolve to one account key. It is the most stable identifier a B2B record carries.
Which record should survive a merge?
None of them entirely. Merge field by field: the address with the most recent successful verification, the most recently sourced title, the canonical domain, the earliest acquisition date with all sources retained, and the suppression state of any duplicate that opted out.
Can deduplication break compliance?
Yes, and this is the most serious risk. If a merge keeps a newer record and discards an older opt-out, you have reinstated someone who asked to be left alone. Opt-out state must always propagate to the surviving record.
How often should I deduplicate?
Exact matching at import before anything is written, fuzzy matching monthly across the active database, and an account level pass before any campaign targeting companies. The import time check is the one that prevents rather than repairs.
What duplicate rate is acceptable?
Under 5% on canonical domain is healthy for an active database. A rising rate usually indicates a new import path bypassing the deduplication step rather than a decline in the underlying data quality.