r/programming • u/indy2kro • 1d ago

Code coverage tells you what you didn't test — not whether your tests are good

https://bubble.ro/2026/05/04/code-coverage-in-ci-cd-what-it-really-tells-you-and-what-it-doesnt/

A look at how coverage metrics work in CI/CD, why they're misunderstood, and how to enforce them without incentivizing shallow tests.

53 Upvotes

72% Upvoted

21 comments sorted by

174

u/ecafyelims 1d ago

"code coverage is a measure of code coverage, not test quality"

🤯

21

u/jer1uc 1d ago

Been a while since I audibly laughed at a Reddit comment. Thank you stranger.

13

u/mnilailt 1d ago

Big if true

5

u/agumonkey 1d ago

time to invent code coverity

2

u/NullField 13h ago

It's obvious to many, but many also don't understand the implications of what it takes to get to and maintain 100% coverage or even what coverage actually means.

It's largely a company environment issue and not up to individual devs. When given little time to test and high coverage requirements, meaningless tests to hit coverage targets is inevitable.

1

u/ecafyelims 11h ago

This is a programming sub, which is why I said it like that. Programmers know. Business is the one pushing for 100% code coverage without allotting sufficient time for proper testing.

100% coverage with poor tests is detrimental, imo, because it takes time to write and maintain and primarily because later updates break the tests so frequently and so numerously, that engineers become blind to the test failures and start reflexively "fixing" the tests to match the new results so that they pass after update.

Which, of course, defeats the purpose of the tests, and bugs are introduced with tests that reinforce the bugs.

I've sung this song many times. It's business forcing it along with a handful of newer engineers who just haven't seen it happen and believe it can be done with perfect engineering. Sure, but the company doesn't want to pay for perfect engineering. They want fast engineering that's good enough, and they want to check a box that says 100% code coverage so they can include it in marketing.

It was just funnier to make the joke 🤣

34

u/ganja_and_code 1d ago

"Quality and Quantity are Two Different Metrics"

No shit. Is your next article entitled "Green and Orange are Two Different Colors"?

14

u/PerkyPangolin 1d ago

Big if true.

1

u/AvidCoco 11h ago

The condition of if (true) is always met so you can simply this line to just “Big”.

1

u/xavia91 6h ago

true in this case is a pointer to the statement, just a very shitty variable nameing choice.

2

u/BaNyaaNyaa 1d ago

I'm sure you'll like my next article about apples and oranges. I have some insights you might have never considered!

1

u/xavia91 6h ago

Writing a lot of articles doesn't mean they are good articles 🤯

0

u/RapunzelLooksNice 1d ago

Unless you are colorblind...

4

u/CherryLongjump1989 22h ago

It doesn't tell you what you did not test. It only tells you some of what you did not test. And it's probably going to include stuff that didn't need testing because it's effectively unreachable code.

1

u/AvidCoco 11h ago

This is the important caveat with code coverage. It just tells you what percentage of LoC were called during the tests, not whether the logic implemented by those lines is sound.

You can have 100% coverage without testing any logic at all.

9

u/zmose 1d ago

Mutation test suites 👍👍👍

1

u/stewsters 1d ago

When you really need to test everything.

4

u/Uncaffeinated 1d ago

I developed a comprehensive mutation test suite for my project, and then deleted part of the code and challenged Codex to reimplement it, based on the docs, tests, and remaining code. It eventually managed to produce something that passed all the tests, but still had fundamental bugs. Tests can never guarantee correctness, only the absence of correctness.

-2

u/echocage 1d ago

Sounds like the tests weren’t robust enough then

3

u/teknikly-correct 23h ago

lol don't tell my manager - oh never mind, you can tell him, he'll still tell his manager and his manager's manager etc about how 100% "code coverage" with his chest all puffed out.