  Whether a file is accepted by a single parser is not a reliable indication of
whether a file complies with its stated format. Bugs within both the parser and
the format specification mean that a compliant file may fail to parse, or that
a non-compliant file might be read without any apparent trouble. The latter
situation presents a significant security risk, and should be avoided. This
article suggests that a better way to assess format specification compliance is
to examine the set of error messages produced by a set of parsers rather than a
single parser. If both a sample of compliant files and a sample of
non-compliant files are available, then we show how a statistical test based on
a pseudo-likelihood ratio can be very effective at determining a file's
compliance. Our method is format agnostic, and does not directly rely upon a
formal specification of the format. Although this article focuses upon the case
of the PDF format (ISO 32000-2), we make no attempt to use any specific details
of the format. Furthermore, we show how principal components analysis can be
useful for a format specification designer to assess the quality and structure
of these samples of files and parsers. While these tests are absolutely
rudimentary, it appears that their use to measure file format variability and
to identify non-compliant files is both novel and surprisingly effective.
