1) sort -u original > de-duplicated
2) sort original | uniq > de-duplicated
3) awk '!x[$0]++' original > de-duplicated
2) sort original | uniq > de-duplicated
3) awk '!x[$0]++' original > de-duplicated
Published June 28th, 2011
3 different ways to remove duplicate lines from a file at the command line.
Similar Articles
Command line openssl
Linux disk space usage 1-liner
Conditionally sum up response sizes in Apache logs
Backup lvs snapshots and scp to remote server
Xen hypervisor reset guest root password from host
Claw back netbook performance after upgrading to Debian Jessie
Recover lost form posts in Firefox
Linux find and replace in files
Copy Debian installed packages
wget show only response headers
Clear Linux buffers, cache when benchmarking filesystem
Linux increase console resolution
Using vi as a hex editor
Debian upgrade to Linux kernel 3.2
Linux high load when writing to slow block devices (SD cards, USB hard drives)
Linux reset root password using GRUB
Linux wipe SD card partition table
Concatenate 2 files in Linux with cat
Debian Linux netbook / laptop / Raspberry Pi power saving with acpitool
java.io.IOException: Too many open files
ls sort by date and time
init.d script to create essential dirs (apache, postgresql, debconf) when mounting /var on tmpfs
qmail alias for email address with dot
Debian dist-upgrade lenny to squeeze
Find open ssh servers on a network using nmap on Linux
Using Linux tmpfs to reduce disk writes for power saving and longer CF lifespan
Linux find and move old files
bash rename files using a for loop
Linux show routing table without being root
Debian bash can auto-complete program switches
tar: time stamp is in the future
Change the hostname on a Linux system
tar list contents of file
svnserve start on boot using update-rc.d on Debian Squeeze
Binary file (standard input) matches when grepping tail
Java return code in Linux shell script
Debian Linux 2-NIC router
Debian N220 netbook power saving script
Linux find grep and rm with a massive number of files
Sort files by size in Debian
Linux redirect standard out and standard err to dev null
/etc/apt/sources.list for dist-upgrade lenny to squeeze
Samsung SyncMaster 913N 1280x1024 resolution in Debian and Ubuntu
Disable exim4 MTA startup on Debian squeeze
Remove .svn folders on Linux
Which process is using port with lsof or netstat (solve bind errors)
[SOLVED] Linux route command takes a long time to complete
Debian software RAID 1 on non boot disk
Find and Replace with nano
Create ISO from CD or DVD in Linux
Linux disk space usage 1-liner
Conditionally sum up response sizes in Apache logs
Backup lvs snapshots and scp to remote server
Xen hypervisor reset guest root password from host
Claw back netbook performance after upgrading to Debian Jessie
Recover lost form posts in Firefox
Linux find and replace in files
Copy Debian installed packages
wget show only response headers
Clear Linux buffers, cache when benchmarking filesystem
Linux increase console resolution
Using vi as a hex editor
Debian upgrade to Linux kernel 3.2
Linux high load when writing to slow block devices (SD cards, USB hard drives)
Linux reset root password using GRUB
Linux wipe SD card partition table
Concatenate 2 files in Linux with cat
Debian Linux netbook / laptop / Raspberry Pi power saving with acpitool
java.io.IOException: Too many open files
ls sort by date and time
init.d script to create essential dirs (apache, postgresql, debconf) when mounting /var on tmpfs
qmail alias for email address with dot
Debian dist-upgrade lenny to squeeze
Find open ssh servers on a network using nmap on Linux
Using Linux tmpfs to reduce disk writes for power saving and longer CF lifespan
Linux find and move old files
bash rename files using a for loop
Linux show routing table without being root
Debian bash can auto-complete program switches
tar: time stamp is in the future
Change the hostname on a Linux system
tar list contents of file
svnserve start on boot using update-rc.d on Debian Squeeze
Binary file (standard input) matches when grepping tail
Java return code in Linux shell script
Debian Linux 2-NIC router
Debian N220 netbook power saving script
Linux find grep and rm with a massive number of files
Sort files by size in Debian
Linux redirect standard out and standard err to dev null
/etc/apt/sources.list for dist-upgrade lenny to squeeze
Samsung SyncMaster 913N 1280x1024 resolution in Debian and Ubuntu
Disable exim4 MTA startup on Debian squeeze
Remove .svn folders on Linux
Which process is using port with lsof or netstat (solve bind errors)
[SOLVED] Linux route command takes a long time to complete
Debian software RAID 1 on non boot disk
Find and Replace with nano
Create ISO from CD or DVD in Linux