Remove duplicate lines
Clean up a list: remove repeats, sort it properly, strip blank lines. It tells you which lines were duplicated and how often, rather than silently shrinking your list.
- Never stored
- No queue, no waiting
- No signup, no watermark
Result appears here.
How it works
Paste your list
Emails, URLs, product codes, anything one per line. Or drop a text file in.
Choose what "the same" means
Ignore trailing spaces, ignore capitalisation, keep the first or the last copy — or remove every line that had a duplicate at all.
Sort and tidy
Sort naturally, reverse, shuffle, number the lines or strip blanks — then copy the result.
What happens to the lines you keep
Two different kinds of whitespace handling sit on this page and it is worth not confusing them. Trimming for comparison builds the key and never edits your lines — the copy that survives keeps its original spacing exactly. The tidy options do the opposite: removing indentation, collapsing runs of spaces and tabs, and stripping the ends of lines are real edits, and they happen before anything is deduplicated, so switching one on quietly changes what counts as a duplicate as well as what the lines look like.
A blank line is compared like any other line, which means every blank after the first disappears along with the other duplicates — a list with paragraph spacing in it comes back with one blank line in the position the first one held. And the report of what repeated shows the key rather than your line: trimmed, and lowercased if you asked for capitalisation to be ignored. An entry appearing in your list as both Smith and SMITH is therefore reported once, in lower case, with a count of two. That list is capped at the fifty commonest.
Whatever goes in, what comes out is joined with line feeds only, so a list pasted from a Windows file loses its carriage returns on the way through — usually what you wanted, and worth knowing if you are pasting the result back into something that cares. One further side effect: switching on "ignore capitalisation" changes the sorting as well as the comparison, and the sort then stops distinguishing accented letters from plain ones too, so resume and résumé come out together rather than apart.
When you want something else
When the list is really a table, line comparison is the wrong shape of tool. A CSV where "duplicate" means two rows agreeing on one column is not a question about strings, and treating it as one either misses the duplicates or removes rows that only looked alike. A spreadsheet's own Remove Duplicates does this properly for a file you already have open, and mlr uniq -g does it on the command line by naming the field that decides, at any size.
For a list too large to sit in a tab, or one you will clean again next week, the command line is a single expression that streams: awk '!seen[$0]++' keeps the first copy and the original order, which is exactly what this page does by default, on a file bigger than your memory. It has no trimming, no case folding and no report of what repeated — that is the trade, and for a file of ten million lines it is the right one.
Frequently asked questions
Is my list stored anywhere?
No. Nothing is stored and no request is made. You can disconnect from the internet after the page loads and it keeps working.
Why are there still duplicates after removing them?
Almost always trailing whitespace. Two lines that end with a different number of spaces look identical on screen and are not, so an exact comparison keeps both — and you conclude the tool is broken. Trimming before comparing is switched ON here by default for that reason. Importantly it only affects the COMPARISON: the line that survives keeps its original text, because quietly editing your lines is not what "remove duplicates" means.
Which copy is kept?
The first, by default, and the order of the rest is untouched — that matters when a list is already in a meaningful order. You can keep the last instead, or remove every line that had a duplicate, leaving only the lines that appeared exactly once. That last one is how you find the entries unique to a list.
Why does sorting put item10 before item2?
That is ordinary alphabetical sorting, where "1" comes before "2" and the rest of the number is never reached. Switch on natural sorting and digit runs are compared as numbers, so item2 comes before item10 — the order a person expects. Accented text is handled properly too: a plain comparison sorts "Zürich" after "Zyzzyva", because it compares the underlying numbers rather than the letters.
Does it tell me what was removed?
Yes — the lines that appeared more than once are listed with their counts, commonest first. Knowing WHICH entry appeared four times is usually more useful than knowing that three lines went away.
Is the shuffle actually random?
Yes. It uses a Fisher-Yates shuffle with the browser's cryptographic random source. The common shortcut — sorting with a random comparison — is not a shuffle at all: sorting algorithms assume a consistent comparison, and with a random one the result is measurably biased towards the original order. If you are drawing a winner from a list, that difference matters.
Is there a size limit?
Available memory, rather than a fixed size cap. Lists of hundreds of thousands of lines are fine.
Good to know: Sorting uses your browser's language rules, so accented letters and different alphabets order the way a reader expects rather than by byte value. That also means the order can differ slightly between browsers. Very large lists are held in memory whole.
Put this tool on your website
Free for any blog, class page or help article. Paste one snippet and your visitors can use it right on your page.
More Text Utilities tools
Word Counter
Words, sentences and reading time
Case Converter
UPPER, lower, Title Case, camelCase and the rest
Text Compare
See exactly what changed between two versions
Character Counter
Characters, bytes and what your limit really counts
HTML Minifier
Shrink HTML without breaking the page
CSS Minifier
Shrink a stylesheet, or be told why not