  When a software transformation or software security task needs to analyze a
given program binary, the first step is often disassembly. Since many modern
disassemblers have become highly accurate on many binaries, we believe reliable
disassembler benchmarking requires standardizing the set of binaries used and
the disassembly ground truth about these binaries. This paper presents (i) a
first version of our work-in-progress disassembly benchmark suite, which
comprises 879 binaries from diverse projects compiled with multiple compilers
and optimization settings, and (ii) a novel disassembly ground truth generator
leveraging the notion of "listing files", which has broad support by Clang,
GCC, ICC, and MSVC. In additional, it presents our evaluation of four prominent
open-source disassemblers using this benchmark suite and a custom evaluation
system. Our entire system and all generated data are maintained openly on
GitHub to encourage community adoption.
