Dedupe & Merge

Find duplicate rows, decide which one survives, and get a file listing every row that went.

Excel's Remove Duplicates does three things this tile deliberately does differently:

What it accepts. CSV, TSV and other delimited text, Excel workbooks, Parquet, JSON and NDJSON. What it writes. CSV, Excel, Parquet or JSON.

How to use it

1. Open Dedupe & Merge and add a file.
2. Choose the key columns — the columns that decide whether two rows are the same thing. Leave it on all columns to remove whole-row duplicates.
3. Choose a matching mode.
4. Choose which row survives.
5. Preview — this reads the file and shows you the duplicate groups without writing anything.
6. Run.

The three matching modes

• Exact — the values must be identical.
• Normalized — case and punctuation are ignored and runs of whitespace collapse to one, so Acme Corp. and ACME Corp are the same company. This is the default, and it is the mode that catches the pair everybody has in their data.
• Fuzzy — near-matches, scored by similarity (Jaro-Winkler or Levenshtein) against a threshold you set. To keep it tractable on large files, rows are first grouped by a blocking strategy — the first few normalized characters, a sound-alike code, or the value's length — and only rows in the same group are ever compared. The threshold and blocking strategy used are recorded in the report.

Fuzzy matching takes the full transitive closure: if A matches B and B matches C, all three end up in one group. That is what "groups, not pairs" means, and at a low threshold it is how a group gets larger than you expected — raise the threshold if a group has swept up rows that do not belong together.

Which row survives

• First seen / Last seen — by position in the file.
• Most complete — the row with the fewest empty values.
• Highest / Lowest in a column you name — the newest date, the largest amount.

Merge mode

Instead of dropping the losers, merge fills the survivor's blanks from the rest of the group: take the first non-empty value, the last non-empty value, or the longest value, per column. This is how three partial records of one customer become one complete record.

The removals file and the report

Every run writes … - removals.csv beside the output unless you turn it off — every removed row, with the key it matched on and a short reason. Ask for a report and you also get an HTML summary: "the total rows, the duplicate groups found, the rows removed, the mode and settings used," and a sample of the groups.

One note on the preview in fuzzy mode: exact and normalized previews mark the surviving row exactly as the run will. Fuzzy groups are assembled differently, so the preview marks the likely survivor advisorily and the final choice is made when the run happens. The panel says so rather than letting the preview and the run disagree quietly.

Output

Saved to the Deduped subfolder of your output folder.

If something goes wrong here

• "Deduplication needs at least one key column." Pick the columns that define sameness, or select all of them for whole-row duplicates.
• Fuzzy matching grouped rows that are not the same thing. Raise the threshold, or pick a tighter blocking strategy. Check the removals file — it names what was matched.
• The output has the same number of rows as the input. There were no duplicates under the mode and keys you chose. Try normalized if you were on exact.
• Merge mode produced the same file as a plain dedupe. Merge only fills blanks. If the surviving row was already complete, there is nothing to fill.