Here’s a little awk trick to remove duplicate lines from files, while preserving the original order of the file.

awk '!visited[$0]++' your_file > deduplicated_file

Credits to Lazarus Lazaridis