Section 8. Article Field Access Commands

This section lists commands that may be used to access specific
   article fields; that is, headers of articles and metadata about
   articles.  These commands typically fetch data from an "overview
   database", which is a database of headers extracted from incoming
   articles plus metadata determined as the article arrives.  Only
   certain fields are included in the database.
   This section is based on the Overview/NOV database [ROBE1995]
   developed by Geoff Collyer.
8.1.  Article Metadata
   Article "metadata" is data about articles that does not occur within
   the article itself.  Each metadata item has a name that MUST begin
   with a colon (and that MUST NOT contain a colon elsewhere within it).
   As with header names, metadata item names are not case sensitive.
   When generating a metadata item, the server MUST compute it for
   itself and MUST NOT trust any related value provided in the article.
   (In particular, a Lines or Bytes header in the article MUST NOT be
   assumed to specify the correct number of lines or bytes in the
   article.)  If the server has access to several non-identical copies
   of an article, the value returned MUST be correct for any copy of
   that article retrieved during the same session.
   This specification defines two metadata items: ":bytes" and ":lines".
   Other metadata items may be defined by extensions.  The names of
   metadata items defined by registered extensions MUST NOT begin with
   ":x-".  To avoid the risk of a clash with a future registered
   extension, the names of metadata items defined by private extensions
   SHOULD begin with ":x-".
8.1.1.  The :bytes Metadata Item
   The :bytes metadata item for an article is a decimal integer.  It
   SHOULD equal the number of octets in the entire article: headers,
   body, and separating empty line (counting a CRLF pair as two octets,
   and excluding both the "." CRLF terminating the response and any "."
   added for "dot-stuffing" purposes).
   Note to client implementers: some existing servers return a value
   different from that above.  The commonest reasons for this are as
   follows:
   o  Counting a CRLF pair as one octet.
   o  Including the "." character used for dot-stuffing in the number.
   o  Including the terminating "." CRLF in the number.
   o  Using one copy of an article for counting the octets but then
      returning another one that differs in some (permitted) manner.
   Implementations should be prepared for such variation and MUST NOT
   rely on the value being accurate.
8.1.2.  The :lines Metadata Item
   The :lines metadata item for an article is a decimal integer.  It
   MUST equal the number of lines in the article body (excluding the
   empty line separating headers and body).  Equivalently, it is two
   less than the number of CRLF pairs that the BODY command would return
   for that article (the extra two are those following the response code
   and the termination octet).
8.2.  Database Consistency
   The information stored in the overview database may change over time.
   If the database records the content or absence of a given field (that
   is, a header or metadata item) for all articles, it is said to be
   "consistent" for that field.  If it records the content of a header
   for some articles but not for others that nevertheless included that
   header, or if it records a metadata item for some articles but not
   for others to which that item applies, it is said to be
   "inconsistent" for that field.
   The LIST OVERVIEW.FMT command SHOULD list all the fields for which
   the database is consistent at that moment.  It MAY omit such fields
   (for example, if it is not known whether the database is consistent
   or inconsistent).  It MUST NOT include fields for which the database
   is inconsistent or that are not stored in the database.  Therefore,
   if a header appears in the LIST OVERVIEW.FMT output but not in the
   OVER output for a given article, that header does not appear in the
   article (similarly for metadata items).
   These rules assume that the fields being stored in the database
   remain constant for long periods of time, and therefore the database
   will be consistent.  When the set of fields to be stored is changed,
   it will be inconsistent until either the database is rebuilt or the
   only articles remaining are those received since the change.
   Therefore, the output from LIST OVERVIEW.FMT needs to be altered
   twice.  Firstly, before any fields stop being stored they MUST be
   removed from the output; then, when the database is once more known
   to be consistent, the new fields SHOULD be added to the output.
   If the HDR command uses the overview database rather than taking
   information directly from the articles, the same issues of
   consistency and inconsistency apply, and the LIST HEADERS command
   SHOULD take the same approach as the LIST OVERVIEW.FMT command in
   resolving them.
8.3.  OVER
8.3.1.  Usage
   Indicating capability: OVER
   Syntax
     OVER message-id
     OVER range
     OVER
   Responses
   First form (message-id specified)
     224    Overview information follows (multi-line)
     430    No article with that message-id
   Second form (range specified)
     224    Overview information follows (multi-line)
     412    No newsgroup selected
     423    No articles in that range
   Third form (current article number used)
     224    Overview information follows (multi-line)
     412    No newsgroup selected
     420    Current article number is invalid
   Parameters
     range         Number(s) of articles
     message-id    Message-id of article
8.3.2.  Description
   The OVER command returns the contents of all the fields in the
   database for an article specified by message-id, or from a specified
   article or range of articles in the currently selected newsgroup.
   The message-id argument indicates a specific article.  The range
   argument may be any of the following:
   o  An article number.
   o  An article number followed by a dash to indicate all following.
   o  An article number followed by a dash followed by another article
      number.
   If neither is specified, the current article number is used.
   Support for the first (message-id) form is optional.  If it is
   supported, the OVER capability line MUST include the argument
   "MSGID".  Otherwise, the capability line MUST NOT include this
   argument, and the OVER command MUST return the generic response code
   503 when this form is used.
   If the information is available, it is returned as a multi-line data
   block following the 224 response code and contains one line per
   article, sorted in numerical order of article number.  (Note that
   unless the argument is a range including a dash, there will be
   exactly one line in the data block.)  Each line consists of a number
   of fields separated by a TAB.  A field may be empty (in which case
   there will be two adjacent TABs), and a sequence of trailing TABs may
   be omitted.
   The first 8 fields MUST be the following, in order:
      "0" or article number (see below)
      Subject header content
      From header content
      Date header content
      Message-ID header content
      References header content
      :bytes metadata item
      :lines metadata item
   If the article is specified by message-id (the first form of the
   command), the article number MUST be replaced with zero, except that
   if there is a currently selected newsgroup and the article is present
   in that group, the server MAY use the article's number in that group.
   (See the ARTICLE command (Section 6.2.1) and STAT examples
   (Section 6.2.4.3) for more details.)  In the other two forms of the
   command, the article number MUST be returned.
   Any subsequent fields are the contents of the other headers and
   metadata held in the database.
   For the five mandatory headers, the content of each field MUST be
   based on the content of the header (that is, with the header name and
   following colon and space removed).  If the article does not contain
   that header, or if the content is empty, the field MUST be empty.
   For the two mandatory metadata items, the content of the field MUST
   be just the value, with no other text.
   For all subsequent fields that contain headers, the content MUST be
   the entire header line other than the trailing CRLF.  For all
   subsequent fields that contain metadata, the field consists of the
   metadata name, a single space, and then the value.
   For all fields, the value is processed by first removing all CRLF
   pairs (that is, undoing any folding and removing the terminating
   CRLF) and then replacing each TAB with a single space.  If there is
   no such header in the article, no such metadata item, or no header or
   item stored in the database for that article, the corresponding field
   MUST be empty.
   Note that, after unfolding, the characters NUL, LF, and CR cannot
   occur in the header of an article offered by a conformant server.
   Nevertheless, servers SHOULD check for these characters and replace
   each one by a single space (so that, for example, CR LF LF TAB will
   become two spaces, since the CR and first LF will be removed by the
   unfolding process).  This will encourage robustness in the face of
   non-conforming data; it is also possible that future versions of this
   specification could permit these characters to appear in articles.
   The server SHOULD NOT produce output for articles that no longer
   exist.
   If the argument is a message-id and no such article exists, a 430
   response MUST be returned.  If the argument is a range or is omitted
   and the currently selected newsgroup is invalid, a 412 response MUST
   be returned.  If the argument is a range and no articles in that
   number range exist in the currently selected newsgroup, including the
   case where the second number is less than the first one, a 423
   response MUST be returned.  If the argument is omitted and the
   current article number is invalid, a 420 response MUST be returned.
8.3.3.  Examples
   In the first four examples, TAB has been replaced by vertical bar and
   some lines have been folded for readability.
   Example of a successful retrieval of overview information for an
   article (explicitly not using an article number):
      [C] GROUP misc.test
      [S] 211 1234 3000234 3002322 misc.test
      [C] OVER
      [S] 224 Overview information follows
      [S] 3000234|I am just a test article|"Demo User"
          <nobody@example.com>|6 Oct 1998 04:38:40 -0500|
          <45223423@example.com>|<45454@example.net>|1234|
          17|Xref: news.example.com misc.test:3000363
      [S] .
   Example of a successful retrieval of overview information for an
   article by message-id:
      [C] CAPABILITIES
      [S] 101 Capability list:
      [S] VERSION 2
      [S] READER
      [S] OVER MSGID
      [S] LIST ACTIVE NEWSGROUPS OVERVIEW.FMT
      [S] .
      [C] OVER <45223423@example.com>
      [S] 224 Overview information follows
      [S] 0|I am just a test article|"Demo User"
          <nobody@example.com>|6 Oct 1998 04:38:40 -0500|
          <45223423@example.com>|<45454@example.net>|1234|
          17|Xref: news.example.com misc.test:3000363
      [S] .
   Note that the article number has been replaced by "0".
   Example of the same commands on a system that does not implement
   retrieval by message-id:
      [C] CAPABILITIES
      [S] 101 Capability list:
      [S] VERSION 2
      [S] READER
      [S] OVER
      [S] LIST ACTIVE NEWSGROUPS OVERVIEW.FMT
      [S] .
      [C] OVER <45223423@example.com>
      [S] 503 Overview by message-id unsupported
   Example of a successful retrieval of overview information for a range
   of articles:
      [C] GROUP misc.test
      [S] 211 1234 3000234 3002322 misc.test
      [C] OVER 3000234-3000240
      [S] 224 Overview information follows
      [S] 3000234|I am just a test article|"Demo User"
          <nobody@example.com>|6 Oct 1998 04:38:40 -0500|
          <45223423@example.com>|<45454@example.net>|1234|
          17|Xref: news.example.com misc.test:3000363
      [S] 3000235|Another test article|nobody@nowhere.to
          (Demo User)|6 Oct 1998 04:38:45 -0500|<45223425@to.to>||
          4818|37||Distribution: fi
      [S] 3000238|Re: I am just a test article|somebody@elsewhere.to|
          7 Oct 1998 11:38:40 +1200|<kfwer3v@elsewhere.to>|
          <45223423@to.to>|9234|51
      [S] .
   Note the missing "References" and Xref headers in the second line,
   the missing trailing fields in the first and last lines, and that
   there are only results for those articles that still exist.
   Example of an unsuccessful retrieval of overview information on an
   article by number:
      [C] GROUP misc.test
      [S] 211 1234 3000234 3002322 misc.test
      [C] OVER 300256
      [S] 423 No such article in this group
   Example of an invalid range:
      [C] GROUP misc.test
      [S] 211 1234 3000234 3002322 misc.test
      [C] OVER 3000444-3000222
      [S] 423 Empty range
   Example of an unsuccessful retrieval of overview information by
   number because no newsgroup was selected first:
      [Assumes currently selected newsgroup is invalid.]
      [C] OVER
      [S] 412 No newsgroup selected
   Example of an attempt to retrieve information when the currently
   selected newsgroup is empty:
      [C] GROUP example.empty.newsgroup
      [S] 211 0 0 0 example.empty.newsgroup
      [C] OVER
      [S] 420 No current article selected
8.4.  LIST OVERVIEW.FMT
8.4.1.  Usage
   Indicating capability: OVER
   Syntax
     LIST OVERVIEW.FMT
   Responses
     215    Information follows (multi-line)
8.4.2.  Description
   See Section 7.6.1 for general requirements of the LIST command.
   The LIST OVERVIEW.FMT command returns a description of the fields in
   the database for which it is consistent (as described above).  The
   information is returned as a multi-line data block following the 215
   response code.  The information contains one line per field in the
   order in which they are returned by the OVER command; the first 7
   lines MUST (except for the case of letters) be exactly as follows:
       Subject:
       From:
       Date:
       Message-ID:
       References:
       :bytes
       :lines
   For compatibility with existing implementations, the last two lines
   MAY instead be:
       Bytes:
       Lines:
   even though they refer to metadata, not headers.
   All subsequent lines MUST consist of either a header name followed by
   ":full", or the name of a piece of metadata.
   There are no leading or trailing spaces in the output.
   Note that the 7 fixed lines describe the 2nd to 8th fields of the
   OVER output.  The "full" suffix (which may use either uppercase,
   lowercase, or a mix) is a reminder that the corresponding fields
   include the header name.
   This command MAY generate different results if it is used more than
   once in a session.
   If the OVER command is not implemented, the meaning of the output
   from this command is not specified, but it must still meet the above
   syntactic requirements.
8.4.3.  Examples
   Example of LIST OVERVIEW.FMT output corresponding to the example OVER
   output above, in the preferred format:
      [C] LIST OVERVIEW.FMT
      [S] 215 Order of fields in overview database.
      [S] Subject:
      [S] From:
      [S] Date:
      [S] Message-ID:
      [S] References:
      [S] :bytes
      [S] :lines
      [S] Xref:full
      [S] Distribution:full
      [S] .
   Example of LIST OVERVIEW.FMT output corresponding to the example OVER
   output above, in the alternative format:
      [C] LIST OVERVIEW.FMT
      [S] 215 Order of fields in overview database.
      [S] Subject:
      [S] From:
      [S] Date:
      [S] Message-ID:
      [S] References:
      [S] Bytes:
      [S] Lines:
      [S] Xref:FULL
      [S] Distribution:FULL
      [S] .
8.5.  HDR
8.5.1.  Usage
   Indicating capability: HDR
   Syntax
     HDR field message-id
     HDR field range
     HDR field
   Responses
   First form (message-id specified)
     225    Headers follow (multi-line)
     430    No article with that message-id
   Second form (range specified)
     225    Headers follow (multi-line)
     412    No newsgroup selected
     423    No articles in that range
   Third form (current article number used)
     225    Headers follow (multi-line)
     412    No newsgroup selected
     420    Current article number is invalid
   Parameters
     field         Name of field
     range         Number(s) of articles
     message-id    Message-id of article
8.5.2.  Description
   The HDR command provides access to specific fields from an article
   specified by message-id, or from a specified article or range of
   articles in the currently selected newsgroup.  It MAY take the
   information directly from the articles or from the overview database.
   In the case of headers, an implementation MAY restrict the use of
   this command to a specific list of headers or MAY allow it to be used
   with any header; it may behave differently when it is used with a
   message-id argument and when it is used with a range or no argument.
   The required field argument is the name of a header with the colon
   omitted (e.g., "subject") or the name of a metadata item including
   the leading colon (e.g., ":bytes"), and is case insensitive.
   The message-id argument indicates a specific article.  The range
   argument may be any of the following:
   o  An article number.
   o  An article number followed by a dash to indicate all following.
   o  An article number followed by a dash followed by another article
      number.
   If neither is specified, the current article number is used.
   If the information is available, it is returned as a multi-line data
   block following the 225 response code and contains one line for each
   article in the range that exists.  (Note that unless the argument is
   a range including a dash, there will be exactly one line in the data
   block.)  The line consists of the article number, a space, and then
   the contents of the field.  In the case of a header, the header name,
   the colon, and the first space after the colon are all omitted.
   If the article is specified by message-id (the first form of the
   command), the article number MUST be replaced with zero, except that
   if there is a currently selected newsgroup and the article is present
   in that group, the server MAY use the article's number in that group.
   (See the ARTICLE command (Section 6.2.1) and STAT examples
   (Section 6.2.4.3) for more details.)  In the other two forms of the
   command, the article number MUST be returned.
   Header contents are modified as follows: all CRLF pairs are removed,
   and then each TAB is replaced with a single space.  (Note that this
   is the same transformation as is performed by the OVER command
   (Section 8.3.2), and the same comment concerning NUL, CR, and LF
   applies.)
   Note the distinction between headers and metadata appearing to have
   the same meaning.  Headers are always taken unchanged from the
   article; metadata are always calculated.  For example, a request for
   "Lines" returns the contents of the "Lines" header of the specified
   articles, if any, no matter whether they accurately state the number
   of lines, while a request for ":lines" returns the line count
   metadata, which is always the actual number of lines irrespective of
   what any header may state.
   If the requested header is not present in the article, or if it is
   present but empty, a line for that article is included in the output,
   but the header content portion of the line is empty (the space after
   the article number MAY be retained or omitted).  If the header occurs
   in a given article more than once, only the content of the first
   occurrence is returned by HDR.  If any article number in the provided
   range does not exist in the group, no line for that article number is
   included in the output.
   If the second argument is a message-id and no such article exists, a
   430 response MUST be returned.  If the second argument is a range or
   is omitted and the currently selected newsgroup is invalid, a 412
   response MUST be returned.  If the second argument is a range and no
   articles in that number range exist in the currently selected
   newsgroup, including the case where the second number is less than
   the first one, a 423 response MUST be returned.  If the second
   argument is omitted and the current article number is invalid, a 420
   response MUST be returned.
   A server MAY only allow HDR commands for a limited set of fields; it
   may behave differently in this respect for the first (message-id)
   form from how it would for the other forms.  If so, it MUST respond
   with the generic 503 response to attempts to request other fields,
   rather than return erroneous results, such as a successful empty
   response.
   If HDR uses the overview database and it is inconsistent for the
   requested field, the server MAY return what results it can, or it MAY
   respond with the generic 503 response.  In the latter case, the field
   MUST NOT appear in the output from LIST HEADERS.
8.5.3.  Examples
   Example of a successful retrieval of subject lines from a range of
   articles (3000235 has no Subject header, and 3000236 is missing):
      [C] GROUP misc.test
      [S] 211 1234 3000234 3002322 misc.test
      [C] HDR Subject 3000234-3000238
      [S] 225 Headers follow
      [S] 3000234 I am just a test article
      [S] 3000235
      [S] 3000237 Re: I am just a test article
      [S] 3000238 Ditto
      [S] .
   Example of a successful retrieval of line counts from a range of
   articles:
      [C] GROUP misc.test
      [S] 211 1234 3000234 3002322 misc.test
      [C] HDR :lines 3000234-3000238
      [S] 225 Headers follow
      [S] 3000234 42
      [S] 3000235 5
      [S] 3000237 11
      [S] 3000238 2378
      [S] .
   Example of a successful retrieval of the subject line from an article
   by message-id:
      [C] GROUP misc.test
      [S] 211 1234 3000234 3002322 misc.test
      [C] HDR subject <i.am.a.test.article@example.com>
      [S] 225 Header information follows
      [S] 0 I am just a test article
      [S] .
   Example of a successful retrieval of the subject line from the
   current article:
      [C] GROUP misc.test
      [S] 211 1234 3000234 3002322 misc.test
      [C] HDR subject
      [S] 225 Header information follows
      [S] 3000234 I am just a test article
      [S] .
   Example of an unsuccessful retrieval of a header from an article by
   message-id:
      [C] HDR subject <i.am.not.there@example.com>
      [S] 430 No Such Article Found
   Example of an unsuccessful retrieval of headers from articles by
   number because no newsgroup was selected first:
      [Assumes currently selected newsgroup is invalid.]
      [C] HDR subject 300256-
      [S] 412 No newsgroup selected
   Example of an unsuccessful retrieval of headers because the currently
   selected newsgroup is empty:
      [C] GROUP example.empty.newsgroup
      [S] 211 0 0 0 example.empty.newsgroup
      [C] HDR subject 1-
      [S] 423 No articles in that range
   Example of an unsuccessful retrieval of headers because the server
   does not allow HDR commands for that header:
      [C] GROUP misc.test
      [S] 211 1234 3000234 3002322 misc.test
      [C] HDR Content-Type 3000234-3000238
      [S] 503 HDR not permitted on Content-Type
8.6.  LIST HEADERS
8.6.1.  Usage
   Indicating capability: HDR
   Syntax
     LIST HEADERS [MSGID|RANGE]
   Responses
     215    Field list follows (multi-line)
   Parameters
     MSGID    Requests list for access by message-id
     RANGE    Requests list for access by range
8.6.2.  Description
   See Section 7.6.1 for general requirements of the LIST command.
   The LIST HEADERS command returns a list of fields that may be
   retrieved using the HDR command.
   The information is returned as a multi-line data block following the
   215 response code and contains one line for each field name
   (excluding the trailing colon for headers and including the leading
   colon for metadata items).  If the implementation allows any header
   to be retrieved, it MUST NOT include any header names in the list but
   MUST include the special entry ":" (a single colon on its own).  It
   MUST still explicitly list any metadata items that are available.
   The order of items in the list is not significant; the server need
   not even consistently return the same order.  The list MAY be empty
   (though in this circumstance there is little point in providing the
   HDR command).
   An implementation that also supports the OVER command SHOULD at least
   permit all the headers and metadata items listed in the output from
   the LIST OVERVIEW.FMT command.
   If the server treats the first form of the HDR command (message-id
   specified) differently from the other two forms (range specified or
   current article number used) in respect of which headers or metadata
   items are available, then the following apply:
   o  If the MSGID argument is specified, the results MUST be those
      available for the first form of the HDR command.
   o  If the RANGE argument is specified, the results MUST be those
      available for the second and third forms of the HDR command.
   o  If no argument is specified, the results MUST be those available
      in all forms of the HDR command (that is, it MUST only list those
      items listed in both the previous cases).
   If the server does not treat the various forms differently, then it
   MUST ignore any argument and always produce the same results (though
   not necessarily always in the same order).
   If the HDR command is not implemented, the meaning of the output from
   this command is not specified, but it must still meet the above
   syntactic requirements.
8.6.3.  Examples
   Example of an implementation providing access to only a few headers:
      [C] LIST HEADERS
      [S] 215 headers supported:
      [S] Subject
      [S] Message-ID
      [S] Xref
      [S] .
   Example of an implementation providing access to the same fields as
   the first example in Section 8.4.3:
      [C] CAPABILITIES
      [S] 101 Capability list:
      [S] VERSION 2
      [S] READER
      [S] OVER
      [S] HDR
      [S] LIST ACTIVE NEWSGROUPS HEADERS OVERVIEW.FMT
      [S] .
      [C] LIST HEADERS
      [S] 215 headers and metadata items supported:
      [S] Date
      [S] Distribution
      [S] From
      [S] Message-ID
      [S] References
      [S] Subject
      [S] Xref
      [S] :bytes
      [S] :lines
      [S] .
   Example of an implementation providing access to all headers:
      [C] LIST HEADERS
      [S] 215 metadata items supported:
      [S] :
      [S] :lines
      [S] :bytes
      [S] :x-article-number
      [S] .
   Example of an implementation distinguishing the first form of the HDR
   command from the other two forms:
      [C] LIST HEADERS RANGE
      [S] 215 metadata items supported:
      [S] :
      [S] :lines
      [S] :bytes
      [S] .
      [C] LIST HEADERS MSGID
      [S] 215 headers and metadata items supported:
      [S] Date
      [S] Distribution
      [S] From
      [S] Message-ID
      [S] References
      [S] Subject
      [S] :lines
      [S] :bytes
      [S] :x-article-number
      [S] .
      [C] LIST HEADERS
      [S] 215 headers and metadata items supported:
      [S] Date
      [S] Distribution
      [S] From
      [S] Message-ID
      [S] References
      [S] Subject
      [S] :lines
      [S] :bytes
      [S] .
   Note that :x-article-number does not appear in the last set of
   output.