We All Suck at Something

I want to let you in on a journey I’ve been on lately, just in case it can be as profound for you as it has been for me.

Picture this: You’ve been in auto racing for decades. You were self-taught in the beginning, quickly won several big races, led a racing team and even mentored many up-and-coming racers. Always in the back of your mind, you knew that you could win more races if you could just figure out how to keep from running out of fuel by the end of the race. You made your fuel capacity as big as feasible, read through several car manuals over the years and even had some small talk about the problem with several other drivers throughout your career, but hadn’t yet come across a great solution to your problem.

Then one day, you happened to be listening to the victory speech by another driver and they thanked their pit crew for their part in the win. This was an unremarkable part of the speech, but one that deeply triggered something for you.

A pit stop! That’s what you needed. How did you not know about this!? How did it never come up after all these years? You were obviously focused during your races, but how did you have such a blind spot that you never saw the other drivers heading to the pits? How much easier would your wins have been if you had just stopped during the race to fuel up? It was something that race drivers were just expected to know, so no one thought to tell you. You missed the rest of the speech, secretly embarrassed, watching your career flash before your eyes, questioning your life choices, worried that you’d be found out as a fraud and pondering how you could have worded your problem better so that others would have known about the gap in your knowledge.

This is what I felt like recently. I’m a formally trained and seasoned developer. I’ve learned and researched various software development techniques over my career. I teach a college course on the subject. I likely even Googled this particular problem at some point without finding satisfaction. Somehow, I missed a known solution to a problem I was having. This solution was mentioned, essentially in passing, during a talk I attended, and I had the same reaction as the race car driver above. It was a struggle to keep my attention on the rest of the talk. Even now, I can’t help but be humbled.

I’m consoled by the realization that we all have blind spots. No one can be an expert at everything. We all suck at something, even if it’s adjacent to something we are good at. This is evidenced by the fact that software bugs exist.

Striving for Perfection

As with any software development team, testing becomes vital to mitigating bugs. We’ve implemented quite the variety in testing platforms and approaches to try to catch all of the edge cases we could possibly think to test for… Except, bugs still make it through. Obviously, we are still missing some tests. If only there were a way to determine which tests we’re missing.

The Nuclear Solution

This is where mutation testing comes in. It’s something I did not have the explicit intention to learn about at the time, but was nonetheless thrust upon me during Nuno Maduro’s Pest 3.0 talk at the Laracon ’24 convention. It allows you to determine code coverage of your tests.

Assuming your existing tests are passing, mutation testing will apply so-called “mutations” to your code and verify that your tests should now fail. If the tests don’t fail with the mutation in place, then the tests aren’t testing that area of your code properly. In other words, mutation testing isn’t really testing your code, it’s testing your tests. No potassium iodide pills necessary.

Examples of mutations that can be made are flipping comparison operators or removing a single line of code. These are easy changes or mistakes a programmer can make and are equally easy to overlook during a code review. These are also prime candidates if you’re looking for 100% code coverage in your testing suite. If a line of code can be removed without a test failing, then either that line of code isn’t important and should be removed, or your tests are incomplete.

Most of the time, the mutation tests run multiple times against your code, applying a single mutation at a time. There is also a variation that applies multiple mutations per run to check for higher-order situations and find more complex bugs.

While mutation testing can be resource intensive, it is a great approach to verifying absolute product quality. It’s one that I’m playing catch-up with and regretting that I didn’t know about it earlier.

I checked, and apparently the concept has been around since the 1970s, so I can’t let myself off the hook for some emerging technology. However, the good news for my ego and bad news for the world is that it’s not exactly ubiquitous in testing frameworks. With that said, there are many different testing frameworks across different stacks that support mutation testing. I highly recommend looking into, or even building, a mutation testing framework for your stack/product.

While there’s a lot more to it than I’ve touched on here, if it’s a new topic to you, at least now you’ve got a name for the concept you can use in your research.

My next problem to solve: how to know what I don’t know so I know what else I need to learn.