Uniq command in Linux – What is it and How to Use it?

Linux offers really good text processing and editing tools. One of these tools is the uniq command. The uniq command helps you detect and delete adjacent occurrences of the same line. That means it deals with repetitions of sentences in a piece of text.

Using the uniq command in Linux

In this tutorial we will see how the uniq command works. Let’s get started.

1. Create a sample text file

We will create a sample text file with a few repeated lines.

The text for the file is given below :

The weather is good today.
The weather is good today.
The Weather Is Good Today.
Today is a sunny day.
Today is a sunny day.
Today is a sunny day.

Here the first two lines are the same. The third line is different and the remaining lines are all alike.

To create a file with this text use the cat command.

cat > file.txt

Output :

Text File
Text File

2. Using uniq to delete repeated lines from the text

To delete repeated lines from the text, use :

uniq file.txt

Output :

The weather is good today.
The Weather Is Good Today.
Today is a sunny day.

There are no repetitions in the text anymore. As you can see, the output displays line 1 and line 2 as unique lines even though the content is the same. That’s because Linux is case sensitive.

3. Get a count of the number of repetitions

To count the number of repetitions use the following line of code :

uniq -c file.txt

Output :

 2 The weather is good today.
 1 The Weather Is Good Today.
 3 Today is a sunny day.

The output contains lines from the text with the count at the beginning.

4. Only print the repeated lines

The uniq command gives you an option to only print the lines that occur more than once. To print only the repeated lines use :

uniq -D file.txt 

Output :

The weather is good today.
The weather is good today.
Today is a sunny day.
Today is a sunny day.
Today is a sunny day.

5. Only print the non-repeated lines

This is the opposite of the example above. When you use the -u flag along with the uniq command then only the lines that occur once are printed. To print only the non-repeated lines use :

uniq -u file.txt 

Output :

The Weather Is Good Today.

6. How to delete repeated lines that don’t occur together?

If you want to delete multiple occurrences of a line that don’t occur together then you can sort the text first.

For example, consider the text given below :

The weather is good today
Today is a sunny day
The weather is good today
The Weather Is Good Today
Today is a sunny day
Today is a sunny day

Let’s see what happens when we run the uniq command on this text.

uniq file.txt

Output :

The weather is good today
Today is a sunny day
The weather is good today
The Weather Is Good Today
Today is a sunny day

There is no change in the text. Let’s use sorting so that same lines occur together.

We can sort the file and store the output in another file with the use of the sort command:

sort file.txt > file1.txt

After sorting the text looks like this :

The Weather Is Good Today
The weather is good today
The weather is good today
Today is a sunny day
Today is a sunny day
Today is a sunny day

Now we can use the uniq command to delete the repeated lines.

uniq file1.txt

Output :

The Weather Is Good Today
The weather is good today
Today is a sunny day

We can also count the number of occurrences for each line.

uniq -c file1.txt

Output :

  1 The Weather Is Good Today
  2 The weather is good today
  3 Today is a sunny day

7. How to store the output to a file?

When you run the uniq command on a file, the contents of the original file are not modified. To save the output of the uniq command you can redirect it to a file. You can do that using :

uniq file.txt > [filename]

Conclusion

This tutorial was about uniq command in Linux. We learned how to use this command for deleting repeated occurrences of a line. Hope you had fun learning with us! You can learn more about the uniq command using the man command.