How to use the Remove Duplicate Lines
Remove duplicate lines in three steps:
1
Paste your list
Paste any multi-line text with duplicate lines into the input area, or upload a .txt or .csv file.
2
Set matching options
Toggle case-sensitive matching (on = 'Apple' and 'apple' are different; off = they are duplicates). Toggle 'Trim whitespace' to ignore leading/trailing spaces before comparing. Toggle 'Keep last occurrence' to keep the last copy instead of the first.
3
Copy or download
Click 'Remove Duplicates' and copy or download the deduplicated output. The stats pills show exactly how many lines were removed.
When to use this tool
Use to clean up lists and data exports with repeated entries:
- →Removing duplicate email addresses or user IDs from an exported mailing list before sending
- →Deduplicating log lines to identify unique events and filter out repeated noise entries
- →Cleaning up a scraped list of URLs, keywords, or product names that contains duplicates
- →Removing repeated entries from merged or concatenated text files
- →Deduplicating a list of CSS class names, import statements, or configuration lines
- →Cleaning survey or form response exports where the same answer appears on multiple rows
Frequently asked questions
Q:Is the original order of lines preserved?
Yes — by default, the first occurrence of each unique line is kept and all subsequent duplicates are discarded. The remaining lines appear in the same order they appeared in the original input. Enable 'Keep last occurrence' to instead keep the last copy of each duplicate, which also preserves the relative order of the retained lines.
Q:What does case-insensitive matching do exactly?
With case-insensitive matching (the default), lines are compared after converting both to lowercase. This means 'Apple', 'apple', and 'APPLE' are all treated as the same line and only the first occurrence is kept (in its original casing). With case-sensitive matching enabled, 'Apple' and 'apple' are treated as distinct lines and both are kept.
Q:What does 'Trim whitespace before compare' do?
When enabled (the default), leading and trailing whitespace is stripped from each line before comparison, so ' apple ' and 'apple' are treated as duplicates. The actual line content in the output is preserved in its original form — only the comparison key is trimmed, not the output itself.
Q:Does it handle very large lists efficiently?
Yes — the deduplication uses a JavaScript Set for O(n) time complexity, meaning it processes one million lines nearly as fast as one thousand. There is no size limit enforced. All processing happens in your browser with no server round-trip.
Q:What is the difference between 'keep first' and 'keep last' occurrence?
Keep first (the default) retains the earliest appearance of each unique line and discards all later duplicates. Keep last retains the most recent appearance and discards all earlier duplicates. In most list-cleaning scenarios, keep first is appropriate. Keep last is useful when your list is ordered by date or recency and you want the most recent value for each key.
Q:Can I use this to deduplicate CSV rows?
Yes — if each CSV row is on its own line, the tool deduplicates entire rows. Note that it does not understand CSV structure, so it compares the entire raw line including all columns and commas. If you need to deduplicate by a specific column only, a spreadsheet tool or the dedicated CSV tools on this site would be more appropriate.