Even the best tests only catch like 50% of the bugs though.

That's not really correct.

Properly input space partitioned tests have something like 90 to 95 accuracy if properly written.

The issues always come from people not wanting to add sufficient tests.

Does that mean that 1 in 20 deploys breaks, then?

1 in 20 deploys has a false negative (IE, something went out without working).

Continous Integration is mostly about regression, so the new tests have less value than running old tests.

We tend to fix issues in production (bugs) by mandating a unit test with the failing issue to be written as part of the fix.

> Properly, if your code has an ‘if’, you need two tests, for the two branches. Same with every place the outcome may diverge. With this approach, it's basically impossible to botch the code

    if (x % 3 == 0)
      println "Fizz"
    if (x % 5 == 0)
      println "Buzz"

There, solved it! And it works fine in all cases you told me to test (e.g. 3, 4, and 5), which means it's impossible I botched anything! Surely I nailed this interview?

Seriously though, the criterion you mentioned is only one of many of increasing strictness (see e.g. https://en.wikipedia.org/wiki/Code_coverage#Basic_coverage_c...), namely branch coverage. Having branch coverage still says very little - the interaction between different branches can be trivially wrong. And desiring full path coverage immediately leads to combinatorial explosion (and the halting problem, once loops are involved).

> unless something slips your mind while writing both the code and the tests.

That is true for any choice of coverage metric and target.

The point is that you claim to have "the" (best?) way to do proper unit tests, but it still fails in basic cases like some combination of paths through if statements being buggy (even if you properly covered each one in isolation).

You normally cannot prove general correctness with unit tests. You can try to probe interesting points in the input space, and different coverage metrics encourage different levels of rigour and effort with this, leading to different fractions of bugs caught, but you'll never have a guarantee to catch all bugs (short of formal proof or exhaustive input coverage).

This is good in theory, and often in practice, but not always. If this practice is followed blindly, unit tests can become too tightly bound to the code to the point where every code change is assumed to require corresponding 1:1 unit test changes and it becomes difficult to distinguish between a broken test that signals broken functionality versus one that simply hasn't been 'synced' yet.

It's worse. You need two tests for every time your code should have had an if - maybe the fact that there isn't one is the bug.

And if the same unit has multiple branches, you need double the tests to cover all paths.

And that doesn't guarantee correctness, and it's only unit tests.

But the 50% was a number I vaguely recall from Code Complete.

> every time your code should have had an if

Yeah, I should've probably clarified this, seeing as we're on the topic of correctness and pedantry.

Personally I'm doing this in TDD style, before the code is written. And then while it's written, too. And for code that didn't have tests—the approach really helps catch bugs previously unknown.

I meant pow as an example to point out that there are logical errors in code that you cannot catch with unit tests.

those kinds of errors should be caught a little higher in the hierarchy... some kind of feature or integration test.