Section 9. Augmented BNF Syntax for NNTP

9.1.  Introduction
   Each of the following sections describes the syntax of a major
   element of NNTP.  This syntax extends and refines the descriptions
   elsewhere in this specification and should be given precedence when
   resolving apparent conflicts.  Note that ABNF [RFC4234] strings are
   case insensitive.  Non-terminals used in several places are defined
   in a separate section at the end.
   Between them, the non-terminals <command-line>, <command-datastream>,
   <command-continuation>, and <response> specify the text that flows
   between client and server.  A consistent naming scheme is used in
   this document for the non-terminals relating to each command, and
   SHOULD be used by the specification of registered extensions.
   For each command, the sequence is as follows:
   o  The client sends an instance of <command-line>; the syntax for the
      EXAMPLE command is <example-command>.
   o  If the client is one that immediately streams data, it sends an
      instance of <command-datastream>; the syntax for the EXAMPLE
      command is <example-datastream>.
   o  The server sends an instance of <response>.
      *  The initial response line is independent of the command that
         generated it; if the 000 response has arguments, the syntax of
         the initial line is <response-000-content>.
      *  If the response is multi-line, the initial line is followed by
         a <multi-line-data-block>.  The syntax for the contents of this
         block after "dot-stuffing" has been removed is (for the 000
         response to the EXAMPLE command) <example-000-ml-content> and
         is an instance of <multi-line-response-content>.
   o  While the latest response is one that indicates more data is
      required (in general, a 3xx response):
      *  the client sends an instance of <command-continuation>; the
         syntax for the EXAMPLE continuation following a 333 response is
         <example-333-continuation>;
      *  the server sends another instance of <response>, as above.
   (There are no commands in this specification that immediately stream
   data, but this non-terminal is defined for the convenience of
   extensions.)
9.2.  Commands
   This syntax defines the non-terminal <command-line>, which represents
   what is sent from the client to the server (see section 3.1 for
   limits on lengths).
     command-line = command EOL
     command = X-command
     X-command = keyword *(WS token)
     command =/ article-command /
           body-command /
           capabilities-command /
           date-command /
           group-command /
           hdr-command /
           head-command /
           help-command /
           ihave-command /
           last-command /
           list-command /
           listgroup-command /
           mode-reader-command /
           newgroups-command /
           newnews-command /
           next-command /
           over-command /
           post-command /
           quit-command /
           stat-command
     article-command = "ARTICLE" [WS article-ref]
     body-command = "BODY" [WS article-ref]
     capabilities-command = "CAPABILITIES" [WS keyword]
     date-command = "DATE"
     group-command = "GROUP" [WS newsgroup-name]
     hdr-command = "HDR" WS header-meta-name [WS range-ref]
     head-command = "HEAD" [WS article-ref]
     help-command = "HELP"
     ihave-command = "IHAVE" WS message-id
     last-command = "LAST"
     list-command = "LIST" [WS list-arguments]
     listgroup-command = "LISTGROUP" [WS newsgroup-name [WS range]]
     mode-reader-command = "MODE" WS "READER"
     newgroups-command = "NEWGROUPS" WS date-time
     newnews-command = "NEWNEWS" WS wildmat WS date-time
     next-command = "NEXT"
     over-command = "OVER" [WS range-ref]
     post-command = "POST"
     quit-command = "QUIT"
     stat-command = "STAT" [WS article-ref]
     article-ref = article-number / message-id
     date = date2y / date4y
     date4y = 4DIGIT 2DIGIT 2DIGIT
     date2y = 2DIGIT 2DIGIT 2DIGIT
     date-time = date WS time [WS "GMT"]
     header-meta-name = header-name / metadata-name
     list-arguments = keyword [WS token]
     metadata-name = ":" 1*A-NOTCOLON
     range = article-number ["-" [article-number]]
     range-ref = range / message-id
     time = 2DIGIT 2DIGIT 2DIGIT
9.3.  Command Continuation
   This syntax defines the further material sent by the client in the
   case of multi-stage commands and those that stream data.
     command-datastream = UNDEFINED
       ; not used, provided as a hook for extensions
     command-continuation = ihave-335-continuation /
           post-340-continuation
     ihave-335-continuation = encoded-article
     post-340-continuation = encoded-article
     encoded-article = multi-line-data-block
       ; after undoing the "dot-stuffing", this MUST match <article>
9.4.  Responses
9.4.1.  Generic Responses
   This syntax defines the non-terminal <response>, which represents the
   generic form of responses; that is, what is sent from the server to
   the client in response to a <command> or a <command-continuation>.
     response = simple-response / multi-line-response
     simple-response = initial-response-line
     multi-line-response = initial-response-line multi-line-data-block
     initial-response-line =
           initial-response-content [SP trailing-comment] CRLF
     initial-response-content = X-initial-response-content
     X-initial-response-content = 3DIGIT *(SP response-argument)
     response-argument = 1*A-CHAR
     trailing-comment = *U-CHAR
9.4.2.  Initial Response Line Contents
   This syntax defines the specific initial response lines for the
   various commands in this specification (see section 3.1 for limits on
   lengths).  Only those response codes with arguments are listed.
     initial-response-content =/ response-111-content /
           response-211-content /
           response-220-content /
           response-221-content /
           response-222-content /
           response-223-content /
           response-401-content
     response-111-content = "111" SP date4y time
     response-211-content = "211" 3(SP article-number) SP newsgroup-name
     response-220-content = "220" SP article-number SP message-id
     response-221-content = "221" SP article-number SP message-id
     response-222-content = "222" SP article-number SP message-id
     response-223-content = "223" SP article-number SP message-id
     response-401-content = "401" SP capability-label
9.4.3.  Multi-line Response Contents
   This syntax defines the content of the various multi-line responses;
   more precisely, it defines the part of the response in the multi-line
   data block after any "dot-stuffing" has been undone.  The numeric
   portion of each non-terminal name indicates the response code that is
   followed by this data.
     multi-line-response-content = article-220-ml-content /
           body-222-ml-content /
           capabilities-101-ml-content /
           hdr-225-ml-content /
           head-221-ml-content /
           help-100-ml-content /
           list-215-ml-content /
           listgroup-211-ml-content /
           newgroups-231-ml-content /
           newnews-230-ml-content /
           over-224-ml-content
     article-220-ml-content = article
     body-222-ml-content = body
     capabilities-101-ml-content = version-line CRLF
           *(capability-line CRLF)
     hdr-225-ml-content = *(article-number SP hdr-content CRLF)
     head-221-ml-content = 1*header
     help-100-ml-content = *(*U-CHAR CRLF)
     list-215-ml-content = list-content
     listgroup-211-ml-content = *(article-number CRLF)
     newgroups-231-ml-content = active-groups-list
     newnews-230-ml-content = *(message-id CRLF)
     over-224-ml-content = *(article-number over-content CRLF)
     active-groups-list = *(newsgroup-name SPA article-number
           SPA article-number SPA newsgroup-status CRLF)
     hdr-content = *S-NONTAB
     hdr-n-content = [(header-name ":" / metadata-name) SP hdr-content]
     list-content = body
     newsgroup-status = %x79 / %x6E / %x6D / private-status
     over-content = 1*6(TAB hdr-content) /
           7(TAB hdr-content) *(TAB hdr-n-content)
     private-status = token ; except the values in newsgroup-status
9.5.  Capability Lines
   This syntax defines the generic form of a capability line in the
   capabilities list (see Section 3.3.1).
     capability-line = capability-entry
     capability-entry = X-capability-entry
     X-capability-entry = capability-label *(WS capability-argument)
     capability-label = keyword
     capability-argument = token
   This syntax defines the specific capability entries for the
   capabilities in this specification.
     capability-entry =/
           hdr-capability /
           ihave-capability /
           implementation-capability /
           list-capability /
           mode-reader-capability /
           newnews-capability /
           over-capability /
           post-capability /
           reader-capability
     hdr-capability = "HDR"
     ihave-capability = "IHAVE"
     implementation-capability = "IMPLEMENTATION" *(WS token)
     list-capability = "LIST" 1*(WS keyword)
     mode-reader-capability = "MODE-READER"
     newnews-capability = "NEWNEWS"
     over-capability = "OVER" [WS "MSGID"]
     post-capability = "POST"
     reader-capability = "READER"
     version-line = "VERSION" 1*(WS version-number)
     version-number = nzDIGIT *5DIGIT
9.6.  LIST Variants
   This section defines more specifically the keywords for the LIST
   command and the syntax of the corresponding response contents.
     ; active
     list-arguments =/ "ACTIVE" [WS wildmat]
     list-content =/ list-active-content
     list-active-content = active-groups-list
     ; active.times
     list-arguments =/ "ACTIVE.TIMES" [WS wildmat]
     list-content =/ list-active-times-content
     list-active-times-content =
           *(newsgroup-name SPA 1*DIGIT SPA newsgroup-creator CRLF)
     newsgroup-creator = U-TEXT
     ; distrib.pats
     list-arguments =/ "DISTRIB.PATS"
     list-content =/ list-distrib-pats-content
     list-distrib-pats-content =
           *(1*DIGIT ":" wildmat ":" distribution CRLF)
     distribution = token
     ; headers
     list-arguments =/ "HEADERS" [WS ("MSGID" / "RANGE")]
     list-content =/ list-headers-content
     list-headers-content = *(header-meta-name CRLF) /
           *((metadata-name / ":") CRLF)
     ; newsgroups
     list-arguments =/ "NEWSGROUPS" [WS wildmat]
     list-content =/ list-newsgroups-content
     list-newsgroups-content =
           *(newsgroup-name WS newsgroup-description CRLF)
     newsgroup-description = S-TEXT
     ; overview.fmt
     list-arguments =/ "OVERVIEW.FMT"
     list-content =/ list-overview-fmt-content
     list-overview-fmt-content = "Subject:" CRLF
           "From:" CRLF
           "Date:" CRLF
           "Message-ID:" CRLF
           "References:" CRLF
           ( ":bytes" CRLF ":lines" / "Bytes:" CRLF "Lines:") CRLF
           *((header-name ":full" / metadata-name) CRLF)
9.7.  Articles
   This syntax defines the non-terminal <article>, which represents the
   format of an article as described in Section 3.6.
     article = 1*header CRLF body
     header = header-name ":" [CRLF] SP header-content CRLF
     header-content = *(S-CHAR / [CRLF] WS)
     body = *(*B-CHAR CRLF)
9.8.  General Non-terminals
   These non-terminals are used at various places in the syntax and are
   collected here for convenience.  A few of these non-terminals are not
   used in this specification but are provided for the consistency and
   convenience of extension authors.
     multi-line-data-block = content-lines termination
     content-lines = *([content-text] CRLF)
     content-text = (".." / B-NONDOT) *B-CHAR
     termination = "." CRLF
     article-number = 1*16DIGIT
     header-name = 1*A-NOTCOLON
     keyword = ALPHA 2*(ALPHA / DIGIT / "." / "-")
     message-id = "<" 1*248A-NOTGT ">"
     newsgroup-name = 1*wildmat-exact
     token = 1*P-CHAR
     wildmat = wildmat-pattern *("," ["!"] wildmat-pattern)
     wildmat-pattern = 1*wildmat-item
     wildmat-item = wildmat-exact / wildmat-wild
     wildmat-exact = %x22-29 / %x2B / %x2D-3E / %x40-5A / %x5E-7E /
          UTF8-non-ascii  ; exclude ! * , ? [ \ ]
     wildmat-wild = "*" / "?"
     base64 = *(4base64-char) [base64-terminal]
     base64-char = UPPER / LOWER / DIGIT / "+" / "/"
     base64-terminal = 2base64-char "==" / 3base64-char "="
     ; Assorted special character sets
     ;   A- means based on US-ASCII, excluding controls and SP
     ;   P- means based on UTF-8, excluding controls and SP
     ;   U- means based on UTF-8, excluding NUL CR and LF
     ;   B- means based on bytes, excluding NUL CR and LF
     A-CHAR     = %x21-7E
     A-NOTCOLON = %x21-39 / %x3B-7E  ; exclude ":"
     A-NOTGT    = %x21-3D / %x3F-7E  ; exclude ">"
     P-CHAR     = A-CHAR / UTF8-non-ascii
     U-CHAR     = CTRL / TAB / SP / A-CHAR / UTF8-non-ascii
     U-NONTAB   = CTRL /       SP / A-CHAR / UTF8-non-ascii
     U-TEXT     = P-CHAR *U-CHAR
     B-CHAR     = CTRL / TAB / SP / %x21-FF
     B-NONDOT   = CTRL / TAB / SP / %x21-2D / %x2F-FF  ; exclude "."
     ALPHA = UPPER / LOWER   ; use only when case-insensitive
     CR = %x0D
     CRLF = CR LF
     CTRL = %x01-08 / %x0B-0C / %x0E-1F
     DIGIT = %x30-39
     nzDIGIT = %x31-39
     EOL = *(SP / TAB) CRLF
     LF = %x0A
     LOWER = %x61-7A
     SP = %x20
     SPA = 1*SP
     TAB = %x09
     UPPER = %x41-5A
     UTF8-non-ascii = UTF8-2 / UTF8-3 / UTF8-4
     UTF8-2    = %xC2-DF UTF8-tail
     UTF8-3    = %xE0 %xA0-BF UTF8-tail / %xE1-EC 2UTF8-tail /
                 %xED %x80-9F UTF8-tail / %xEE-EF 2UTF8-tail
     UTF8-4    = %xF0 %x90-BF 2UTF8-tail / %xF1-F3 3UTF8-tail /
                 %xF4 %x80-8F 2UTF8-tail
     UTF8-tail = %x80-BF
     WS = 1*(SP / TAB)
   The following non-terminals require special consideration.  They
   represent situations where material SHOULD be restricted to UTF-8,
   but implementations MUST be able to cope with other character
   encodings.  Therefore, there are two sets of definitions for them.
   Implementations MUST accept any content that meets this syntax:
     S-CHAR   = %x21-FF
     S-NONTAB = CTRL / SP / S-CHAR
     S-TEXT   = (CTRL / S-CHAR) *B-CHAR
   and MAY pass such content on unaltered.
   When generating new content or re-encoding existing content,
   implementations SHOULD conform to this syntax:
     S-CHAR   = P-CHAR
     S-NONTAB = U-NONTAB
     S-TEXT   = U-TEXT
9.9.  Extensions and Validation
   The specification of a registered extension MUST include formal
   syntax that defines additional forms for the following non-terminals:
   command
      for each new command other than a variant of the LIST command -
      the syntax of each command MUST be compatible with the definition
      of <X-command>;
   command-datastream
      for each new command that immediately streams data;
   command-continuation
      for each new command that sends further material after the initial
      command line - the syntax of each continuation MUST be exactly
      what is sent to the server, including any escape mechanisms such
      as "dot-stuffing";
   initial-response-content
      for each new response code that has arguments - the syntax of each
      response MUST be compatible with the definition of <X-initial-
      response-content>;
   multi-line-response-content
      for each new response code that has a multi-line response - the
      syntax MUST show the response after the lines containing the
      response code and the terminating octet have been removed and any
      "dot-stuffing" undone;
   capability-entry
      for each new capability label - the syntax of each entry MUST be
      compatible with the definition of <X-capability-entry>;
   list-arguments
      for each new variant of the LIST command - the syntax of each
      entry MUST be compatible with the definition of <X-command>;
   list-content
      for each new variant of the LIST command - the syntax MUST show
      the response after the lines containing the 215 response code and
      the terminating octet have been removed and any "dot-stuffing"
      undone.
   The =/ notation of ABNF [RFC4234] and the naming conventions
   described in Section 9.1 SHOULD be used for this.
   When the syntax in this specification, or syntax based on it, is
   validated, it should be noted that:
   o  the non-terminals <command-line>, <command-datastream>,
      <command-continuation>, <response>, and
      <multi-line-response-content> describe basic concepts of the
      protocol and are not referred to by any other rule;
   o  the non-terminal <base64> is provided for the convenience of
      extension authors and is not referred to by any rule in this
      specification;
   o  for the reasons given above, the non-terminals <S-CHAR>,
      <S-NONTAB>, and <S-TEXT> each have two definitions; and
   o  the non-terminal <UNDEFINED> is deliberately not defined.