Remove Duplicates
Remove duplicate lines from text instantly.
100% client-side processing
Input lines: 6
Output lines: 0 (0 removed)
How Deduplication Works
The tool normalizes line endings first, converting both Windows \r\n and classic Mac \r to a single \n, so a file from any operating system behaves the same. It then splits the text on newlines and processes each line according to your options: trimming surrounding whitespace and/or dropping blanks if those are enabled. After that it walks the remaining lines in order and keeps a line only if its comparison key has not been seen. The key is the line itself in case-sensitive mode, or the lowercased line in case-insensitive mode. Because the first occurrence wins, the relative order of unique lines is preserved and no line is moved until you ask for sorting.
What Each Option Changes
| Option | Default | What it does |
|---|---|---|
| Case sensitive | On | Apple and apple are different lines; when off, only the first casing is kept |
| Trim whitespace | On | Leading and trailing spaces are ignored for comparison and removed from output |
| Remove empty lines | On | Blank lines are deleted before deduplication |
| Sort output | Off | Deduplicates first, then sorts the survivors alphabetically |
The interaction between the first two matters. With trimming on, the line " apple " and "apple" are treated as duplicates, and the retained output is the trimmed apple regardless of which came first. With trimming off, they are different lines and both survive.
Worked Example: Cleaning a Pasted List
Suppose you paste six lines, where the last one separates two groups:
apple
Banana
apple
cherry
banana
With the defaults (case sensitive, trim, remove empty lines, no sort), the blank line is dropped, apple appears twice and keeps only its first occurrence, and the output is apple, Banana, cherry, banana — four lines. The header reports 6 input lines, 4 output lines, and 2 removed: one true duplicate plus one blank line.
Now turn case sensitivity off and keep everything else. banana now duplicates Banana, so the output shrinks to apple, Banana, cherry — three lines. Notice the second word retained is still spelled Banana, because that was the first occurrence; the lowercase banana was discarded.
Case Sensitivity, Blank Lines, and Line Endings
These three details cause most of the confusion, so it is worth being explicit:
- Case-insensitive deduplication preserves the first spelling. The lowercasing is only a comparison key. If the first occurrence is
USAand a later line isusa, the output keepsUSA. - Blank lines are not duplicates. They are removed by their own checkbox. Even with empty-line removal off, a run of blank lines collapses to a single blank line because the first is unique and the rest are duplicates of it.
- A whitespace-only line becomes blank under trimming. Indentation or trailing tabs do not protect a line from empty-line removal, because trimming happens before the blank check.
- Line endings are unified. A CRLF file and an LF file with identical content produce identical output, so you can compare exports from different tools without pre-converting them.
Common Mistakes
- Expecting word-level deduplication. The unit is the line.
red, blue, redstays as-is because it is one line. - Forgetting that trimming is on by default. Reformatting that relies on a leading space, such as indented code, will lose that indentation in the output.
- Turning off case sensitivity and then wanting the lowercase version. The tool keeps the first occurrence, not the lowercase one.
- Deduping before extracting a key column. As noted above, whole-line comparison will not collapse rows that share only one field.
- Assuming sort removes more lines. Sorting only reorders the already-unique lines and never changes the count.
Real-World Scenarios
Email list cleanup. Export subscribers and paste one address per line with case sensitivity off and trimming on. This collapses Jane@Example.com and jane@example.com, as well as entries that differ only by a stray trailing space. It will not merge aliases such as jane+news@example.com with jane@example.com, because those are genuinely different strings.
Server log triage. Paste a block of repeated error lines and keep case sensitivity on so distinct messages stay separate, then enable sorting to group identical errors together before you count them. Duplicate suppression here is exact, so log lines that differ by a timestamp or request ID are preserved.
Keyword lists. Scraped keywords often repeat with different capitalization. Case-insensitive mode collapses SEO Tools and seo tools into the first spelling. It does not merge plurals, stems, or near-matches, so tool and tools both survive.
Indented code or configuration. Here it pays to turn trimming off. If you are deduplicating an indented YAML or Python block, trimming on the default setting will strip the indentation and change the meaning of the output.
It is not a word-level or token-level deduplicator, it cannot detect fuzzy or near-duplicates such as typos and singular/plural pairs, and it cannot compare by a key column the way a spreadsheet’s Remove Duplicates command can. It also does not preserve information about how many blank lines were present. When you need keyed, row-based comparison, a spreadsheet or database is the better instrument; when you have raw, line-oriented text and no schema, this tool is faster and format-agnostic.
Related Tools
Frequently Asked Questions
Does case-insensitive mode change the capitalization of the line it keeps?
No. The tool lowercases each line only to build the comparison key; the text stored in the output is the first occurrence exactly as it appeared. So with case-insensitive mode on, a list containing APPLE, then Apple, then apple collapses to whichever came first in the original order, preserving that line's original casing.
Are blank lines treated as duplicates of one another?
No. Blank lines are handled by the separate Remove empty lines option and are never counted as duplicates. When Remove empty lines is on (the default), every blank line is dropped before deduplication begins. When it is off, blank lines survive as their own entries, but a single blank line is not repeated in the output because the first one is kept as unique.
Will it remove duplicate words that appear inside a single line?
No. Comparison is whole-line, so the tool cannot see two identical words within one line. A line reading buy now buy now is one unique line and passes through unchanged. To dedupe words you would need a word-splitting tool or a script, not this line-level utility.
Why did lines disappear when I had no exact duplicates?
Two defaults are usually responsible. Trim whitespace is on, so a line indented with spaces is compared and emitted without that indentation, and Remove empty lines is on, so any line that becomes empty after trimming is deleted. Whitespace-only lines vanish entirely. Turn both options off to keep the input byte-for-byte except for true duplicates.
Can I remove duplicates using only one column of a CSV or spreadsheet export?
Not directly. The tool compares entire lines, so two rows that differ in any other field are kept even if their email column matches. To dedupe by a single column, first extract that column into its own list, dedupe it here, then rejoin it to the other data with a spreadsheet lookup.
Does the sort option change which duplicate survives?
No. Deduplication runs first and keeps each unique line in its first-seen position; sorting runs afterward and merely reorders the surviving lines. If you want a specific copy to survive, order the input that way before deduplicating, because sorting will not swap which occurrence was retained.
Reviewed by the HandyTools Hub team · Last updated