Remove duplicate lines

How matching works

Each line is compared as a complete string. Differences in capitalization, leading or trailing spaces, punctuation, or Unicode representation mean two lines are not exact matches. When duplicate lines occur, the earliest one is retained and later copies are omitted. The sequence of retained lines follows their first appearance in the input.

This behavior is appropriate for exact repeated entries but is not fuzzy deduplication. “Example” and “example” remain distinct; a row with an extra space also remains distinct. If you want case-insensitive or whitespace-normalized matching, apply an explicit normalization step first and review what distinctions that would erase.

When removing duplicates can be wrong

Repeated lines may represent repeated events, transactions, observations, or legitimate quantities. Removing them changes record count and can change meaning. Before deduplicating, establish whether repetition is a data error or part of the real dataset. Preserve the original and compare counts before and after.

Safe checklist

1. Confirm each line is an independent record.
2. Decide whether case or surrounding whitespace should matter.
3. Check that repeated occurrences are not meaningful events.
4. Verify the resulting number of lines and inspect representative values.

Trim accidental edge spaces with Trim Each Line, or reorder a list with Sort Lines after deduplication if that is appropriate.