Ask HN: What is the cost of unit test coverage in complex C++ projects
This question have been intriguing me for the past few weeks. A few project I'm involved in are supposed to get more unit test coverage. The project include some fault-tolerant distributed servers, proxies and various API proxies/converters (between data query languages and protocols). Some of the components that needs to run really fast are written in C++, and are using threading and asynchronous requests between components (potentially in different data centers). Some of these components use asio/beast and stackless and stackfull coroutines. Of course, most of the code is still trivial, sequential methods doing something well defined with a predictable outcome.

Articles and discussions on the net about the right balance for code coverage in unit tests are plentiful. But I have not found much relevant material about the actual cost of targeting say 50%, 70%, 90% or 99% unit test coverage. And it looks to me that most of the numbers regards interactive web pages and mobile apps. Not the more complex technology beneath. I'm not opposing testing - I think testing is great. I just want a realistic idea about the actual cost of various code coverage percentages in projects of some complexity.

My experience from smaller open source projects in modern C++ that I've written from scratch is that "proper" unit testing adds about 60 - 100% coding overhead (designing and writing the tests, and also being mindful about testing while designing and writing the actual code). I usually only test the public methods on an interface, unless there are some special reasons to test or access private methods. I try to avoid mocking. The aim of the unit test is to validate that the algorithm in a public method works as documented. Since some of these methods are interpreters with potentially indefinite variations in input, I'm testing for valid and invalid input, but only with a few workloads to test specific paths and error conditions in the code. The tests will typically catch logical regressions, but not security vulnerabilities or performance regressions. I have not used code coverage tools, but my gut feeling is that I typically cover < 70% of my code in my small open source projects with unit tests.

All projects of this nature are unique, and increasingly asynchronous and parallel algorithms does, in my experience (may be I'm doing something wrong), increase unit testing complexity and the time needed to design the code and write the tests. In my opinion, it's not acceptable to introduce performance regressions in hot paths in the code to facilitate testing (for example by using virtual classes in stead of templates).

So, with this language; C++ 17 (no, re-writing millions of lines of C++ code in Rust or Ruby is not an option right now) and this complexity (parallel and asynchronous algorithms in a fault-tolerant distributed system with massive disk/network IO and capacity for billions of low latency external requests per day) - what could the cost-curve for unit test coverage / coding time look like, from 50% to 99% code coverage?

Some people involved in the project are talking about "100% unit test coverage" as a target.

Note that I only focus on unit testing here. That's because the other forms of testing in this project are done by different teams, and don't have a direct impact on the time it takes to get a feature from a specification document to code complete. Also, please excuse my grammar ;) English is not my native language.