Hey, finally I'm catching a report like this at a point in time where I'm not myself working as a software security assessment consultant, which frees me to get publicly back on my hobby horse about public assessment reports.

I think third-party assessment reports like these are a real problem in our industry. There are firms that are worse about them and firms that are somewhat better but it's a near-universal problem.

I don't at all object to technical vulnerability reports being shared. It's good to know when third-party auditors have spotted problems in products (and it's also good for prospective customers of consulting firms to know what kinds of vulnerabilities that firm is likely to spot, and how padded out some of these reports can be with low-quality findings). I also think it's worth remembering that the overwhelming majority of security assessment engagements are never reported out to the public; even when vendors do publish reports, more often than not it's after previous unpublished engagements have been run.

But the way these reports get written creates a huge conflict of interest. They're not simply reports of (1) what was tested and (2) what was found. Too often, they're also product marketing documents, beginning and concluding with "overall assessments" of the target software that is almost invariably positive, even on projects that reduced the target to a smoking crater (to say the least, that didn't happen here, but when it does, it's always framed as "[vendor] has made great strides in improving their security in the wake of this important engagement").

There are firms that specialize in writing public audit reports, and firms that specialize in finding excellent vulnerabilities, and the Venn of those firms is practically two disjoint circles† --- at least in the sense that there's a sort of insider chatter about who the quietly bad-ass firms are, and who the "most credible for shutting down sales objections" firms are.

Even when they're not intended to be tools of persuasion, public audit reports tend to function that way systemically. That's because vendors control the terms on which these audits are done, what's to be tested, when the testing will occur, how much time will be allotted and who's staffing. Vendors pay for public-facing reports, as an extra line item in the SOW, and in some circumstances get to review drafts.

Meanwhile, the public that reads these reports is in no way qualified to weigh or contextualize the report. If you're reading an audit report from a commercial vendor, it is invariably presented as a "clean bill of health". But even if you somehow get principals at Azimuth to assess your system, there is no such thing as a clean-bill-of-health audit. Different teams of auditors will find different bugs (even different teams from the same vendor!).

I think we need a new norm in the industry, and that it can only come from the auditing firms themselves. I think that, roughly, that norm should be that public-facing reports can be provided only in the same dry, technical form they're presented to development teams in ordinary, non-public projects: a methodology, a scope and rules of engagement, and a list of findings. No editorializing and no editorial review by vendors. Probably, though I'm less clear on the mechanics of how this would work, it should also stop being OK to charge different amounts for projects that do and don't have public reports.

I know there are consultants that disagree with me about this; I look forward to reading their takes.

† I'm not editing this out but on reflection this is pretty imprecise and it's probably easy to come up with a counterexample.