How to remove duplicate lines
- Paste your list into the top box, one item per line.
- The Unique lines box fills in immediately. Each line appears once, in the order it first appeared.
- Adjust the options if needed, then click Copy result or Download .txt (saved as
unique-lines.txt).
The line above the result summarises what happened, for example 7 lines → 5 unique lines · 1 duplicate removed · 1 empty line removed. The first number counts every line in your input, blank ones included.
What the options do
- Ignore case (off by default) – “Apple”, “apple” and “APPLE” count as the same line. The spelling of the first occurrence is the one kept.
- Ignore leading and trailing spaces (on) – spaces and tabs at the start and end of a line are ignored when comparing, and they are also removed from the result. Spaces inside a line still matter: “New York” and “New York” with two spaces are different lines.
- Remove empty lines (on) – blank lines, including lines that contain only spaces, are left out. When this is off, a blank line is treated like any other line, so only the first blank line survives.
- Sort A→Z (off) – sorts the unique lines alphabetically, ignoring case and accents, with numbers in natural order (“item 2” before “item 10”). For more sort orders, use Sort Lines.
Worked example
Take this seven-line input, where the third line has a space after apple and the fourth line is empty:
apple · Banana · apple␣ · (empty) · cherry · banana · Apple
- Default settings give
apple,Banana,cherry,banana,Apple. The trailing space is ignored, so the third line counts as a repeat of the first, and the empty line is dropped. - With Ignore case the result shrinks to
apple,Banana,cherry, and the summary reports 3 duplicates removed. - With leading and trailing spaces no longer ignored,
apple␣is kept as a separate line. A stray space is the most common reason two lines that look identical both survive.
A second example: a mailing list containing [email protected], [email protected], [email protected], [email protected] and [email protected] becomes three addresses with Ignore case on. The part after the @ is case-insensitive by definition, and in practice almost every mail provider ignores case before the @ as well, so merging them is usually safe.
How it works
The tool reads your lines from top to bottom and keeps a record of every line it has already seen (after applying the case and space options). A line that is already on record is skipped; anything new is added to the result. Because the first occurrence always wins, the original order is preserved, unlike sorting-based approaches. If you work on the command line, the equivalent is awk '!seen[$0]++' list.txt, while sort -u removes duplicates but also reorders the lines. In spreadsheets the same job is done by the UNIQUE function or Excel's Remove Duplicates command.
Practical uses
- Merging two mailing lists or customer lists without sending anyone the same message twice: paste one list below the other and copy the result.
- Cleaning keyword lists for SEO or advertising, hashtags, tags and product codes.
- Reducing a log file or error report to the distinct messages it contains.
- Building a vocabulary or name list from several sources.
Tips and common mistakes
- One value per line is required. If your values are separated by commas, put each on its own line first. To go the other way afterwards and turn the result into a single comma-separated line, use Remove Line Breaks with “a comma”.
- Invisible differences keep lines apart: a tab instead of a space, a non-breaking space, curly versus straight quotes, or full-width versus half-width characters all make two lines that look the same technically different.
- Accents are not ignored, even with Ignore case: “café” and “cafe” stay separate lines.
FAQ
Which copy of a duplicate is kept?
The first one. Later copies are removed and the remaining lines keep their original order, unless you turn on Sort A→Z.
Can I see which lines were removed?
The summary tells you how many were removed. To see exactly which ones, paste the original list and the result into the Diff Checker: every removed line is highlighted.
Does it work with Korean, Japanese, Chinese or other scripts?
Yes. Lines are compared character by character in any language. Ignore case simply has no effect on scripts that don't have capital letters.
Is there a size limit?
There is no fixed limit. Lists with hundreds of thousands of lines are processed in a moment on a typical computer, because the whole job runs locally in your browser; only very large files are limited by your device's memory.