# uniq

## PROLOG



           This manual page is part of the POSIX Programmer's Manual.  The
           Linux implementation of this interface may differ (consult the
           corresponding Linux manual page for details of Linux behavior), or
           the interface may not be implemented on Linux.
    


## NAME



           uniq — report or filter out repeated lines in a file
    


## SYNOPSIS



           uniq **[** -c|-d|-u**] [** -f _fields_**] [** -s _char_**] [**_input_file_ **[**_output_file_**]]**
    


## DESCRIPTION



           The _uniq_ utility shall read an input file comparing adjacent
           lines, and write one copy of each input line on the output. The
           second and succeeding copies of repeated adjacent input lines
           shall not be written.  The trailing <newline> of each line in the
           input shall be ignored when doing comparisons.
    
           Repeated lines in the input shall not be detected if they are not
           adjacent.
    


## OPTIONS



           The _uniq_ utility shall conform to the Base Definitions volume of
           POSIX.1‐2017, _Section 12.2_ , _Utility Syntax Guidelines_ , except that
           **'+'** may be recognized as an option delimiter as well as **'-'**.
    
           The following options shall be supported:
    
           **-c** Precede each output line with a count of the number of
                     times the line occurred in the input.
    
           **-d** Suppress the writing of lines that are not repeated in
                     the input.
    
           **-f** _fields_ Ignore the first _fields_ fields on each input line when
                     doing comparisons, where _fields_ is a positive decimal
                     integer. A field is the maximal string matched by the
                     basic regular expression:
    
                         [[:blank:]]*[^[:blank:]]*
    
                     If the _fields_ option-argument specifies more fields than
                     appear on an input line, a null string shall be used for
                     comparison.
    
           **-s** _chars_  Ignore the first _chars_ characters when doing
                     comparisons, where _chars_ shall be a positive decimal
                     integer. If specified in conjunction with the **-f** option,
                     the first _chars_ characters after the first _fields_ fields
                     shall be ignored. If the _chars_ option-argument specifies
                     more characters than remain on an input line, a null
                     string shall be used for comparison.
    
           **-u** Suppress the writing of lines that are repeated in the
                     input.
    


## OPERANDS



           The following operands shall be supported:
    
           _input_file_
                     A pathname of the input file. If the _input_file_ operand
                     is not specified, or if the _input_file_ is **'-'** , the
                     standard input shall be used.
    
           _output_file_
                     A pathname of the output file. If the _output_file_
                     operand is not specified, the standard output shall be
                     used. The results are unspecified if the file named by
                     _output_file_ is the file named by _input_file_.
    


## STDIN



           The standard input shall be used only if no _input_file_ operand is
           specified or if _input_file_ is **'-'**.  See the INPUT FILES section.
    


## INPUT FILES



           The input file shall be a text file.
    


## ENVIRONMENT VARIABLES



           The following environment variables shall affect the execution of
           _uniq_ :
    
           _LANG_      Provide a default value for the internationalization
                     variables that are unset or null. (See the Base
                     Definitions volume of POSIX.1‐2017, _Section 8.2_ ,
                     _Internationalization Variables_ for the precedence of
                     internationalization variables used to determine the
                     values of locale categories.)
    
           _LC_ALL_    If set to a non-empty string value, override the values
                     of all the other internationalization variables.
    
           _LC_CTYPE_  Determine the locale for the interpretation of sequences
                     of bytes of text data as characters (for example,
                     single-byte as opposed to multi-byte characters in
                     arguments and input files) and which characters
                     constitute a <blank> in the current locale.
    
           _LC_MESSAGES_
                     Determine the locale that should be used to affect the
                     format and contents of diagnostic messages written to
                     standard error.
    
           _NLSPATH_   Determine the location of message catalogs for the
                     processing of _LC_MESSAGES_.
    


## ASYNCHRONOUS EVENTS



           Default.
    


## STDOUT



           The standard output shall be used if no _output_file_ operand is
           specified, and shall be used if the _output_file_ operand is **'-'** and
           the implementation treats the **'-'** as meaning standard output.
           Otherwise, the standard output shall not be used.  See the OUTPUT
           FILES section.
    


## STDERR



           The standard error shall be used only for diagnostic messages.
    


## OUTPUT FILES



           If the **-c** option is specified, the output file shall be empty or
           each line shall be of the form:
    
               "%d %s", <_number of duplicates_ >, <_line_ >
    
           otherwise, the output file shall be empty or each line shall be of
           the form:
    
               "%s", <_line_ >
    


## EXTENDED DESCRIPTION



           None.
    


## EXIT STATUS



           The following exit values shall be returned:
    
            0    The utility executed successfully.
    
           >0    An error occurred.
    


## CONSEQUENCES OF ERRORS



           Default.
    
           _The following sections are informative._
    


## APPLICATION USAGE



           If the collating sequence of the current locale has a total
           ordering of all characters, the _sort_ utility can be used to cause
           repeated lines to be adjacent in the input file. If the collating
           sequence does not have a total ordering of all characters, the
           _sort_ utility should still do this but it might not. To ensure that
           all duplicate lines are eliminated, and have the output sorted
           according the collating sequence of the current locale,
           applications should use:
    
               LC_ALL=C sort -u | sort
    
           instead of:
    
               sort | uniq
    
           To remove duplicate lines based on whether they collate equally
           instead of whether they are identical, applications should use:
    
               sort -u
    
           instead of:
    
               sort | uniq
    
           When using _uniq_ to process pathnames, it is recommended that
           LC_ALL, or at least LC_CTYPE and LC_COLLATE, are set to POSIX or C
           in the environment, since pathnames can contain byte sequences
           that do not form valid characters in some locales, in which case
           the utility's behavior would be undefined. In the POSIX locale
           each byte is a valid single-byte character, and therefore this
           problem is avoided.
    


## EXAMPLES



           The following input file data (but flushed left) was used for a
           test series on _uniq_ :
    
               #01 foo0 bar0 foo1 bar1
               #02 bar0 foo1 bar1 foo1
               #03 foo0 bar0 foo1 bar1
               #04
               #05 foo0 bar0 foo1 bar1
               #06 foo0 bar0 foo1 bar1
               #07 bar0 foo1 bar1 foo0
    
           What follows is a series of test invocations of the _uniq_ utility
           that use a mixture of _uniq_ options against the input file data.
           These tests verify the meaning of _adjacent_.  The _uniq_ utility
           views the input data as a sequence of strings delimited by **'\n'**.
           Accordingly, for the _fields_ th member of the sequence, _uniq_
           interprets unique or repeated adjacent lines strictly relative to
           the _fields_ +1th member.
    
            1. This first example tests the line counting option, comparing
               each line of the input file data starting from the second
               field:
    
                   uniq -c -f 1 uniq_0I.t
                       1 #01 foo0 bar0 foo1 bar1
                       1 #02 bar0 foo1 bar1 foo1
                       1 #03 foo0 bar0 foo1 bar1
                       1 #04
                       2 #05 foo0 bar0 foo1 bar1
                       1 #07 bar0 foo1 bar1 foo0
    
               The number **'2'** , prefixing the fifth line of output, signifies
               that the _uniq_ utility detected a pair of repeated lines. Given
               the input data, this can only be true when _uniq_ is run using
               the **-f 1** option (which shall cause _uniq_ to ignore the first
               field on each input line).
    
            2. The second example tests the option to suppress unique lines,
               comparing each line of the input file data starting from the
               second field:
    
                   uniq -d -f 1 uniq_0I.t
                   #05 foo0 bar0 foo1 bar1
    
            3. This test suppresses repeated lines, comparing each line of
               the input file data starting from the second field:
    
                   uniq -u -f 1 uniq_0I.t
                   #01 foo0 bar0 foo1 bar1
                   #02 bar0 foo1 bar1 foo1
                   #03 foo0 bar0 foo1 bar1
                   #04
                   #07 bar0 foo1 bar1 foo0
    
            4. This suppresses unique lines, comparing each line of the input
               file data starting from the third character:
    
                   uniq -d -s 2 uniq_0I.t
    
               In the last example, the _uniq_ utility found no input matching
               the above criteria.
    


## RATIONALE



           Some historical implementations have limited lines to be 1080
           bytes in length, which does not meet the implied {LINE_MAX} limit.
    
           Earlier versions of this standard allowed the **-**_number_ and **+**_number_
           options. These options are no longer specified by POSIX.1‐2008 but
           may be present in some implementations.
    


## FUTURE DIRECTIONS



           None.
    


* * *