Bug Rate vs Mistake Rate
When we talk about software quality, we often focus on bugs. We count them, track them across releases, and use them to evaluate reliability. While this perspective is useful, it misses an important aspect of software development: the effort spent correcting mistakes before bugs ever exist.
In the previous post we introduced the concept of model plasticity. This post we complement model plasticity with mistake rate, a concept aimed at describing how efficiently a development process converges toward a correct solution.
A bug is a defect that survives long enough to be observed, reported, or tested. A mistake, in contrast, is an incorrect step made during development that is corrected before the software is considered finished or released. Most mistakes never become bugs. They are discovered and fixed through compilation errors, failing tests, local experimentation, or often reasoning before even writing the code. Yet these mistakes still consume time and cognitive effort. Ignoring them hides an important part of the development cost.
Mistake rate is not a measure of failure, it is a measure of process friction
Bug rate primarily reflects product reliability and validation effectiveness. Mistake rate reflects productivity and convergence speed. The diffference can be summarised as “How many defects escape?” vs “How much effort is required to get things right?”
Can we define mistake rate?
At a high level, mistake rate describes how many corrections are required to complete a change. A simple working definition is:
Mistake rate is the number of failed attempts or corrective iterations required to implement a change successfully.
Mistake rate is not a precise metric, but rather a conceptual tool to reason about development efficiency. Depending on the context, it can be measured in different ways. In a test-driven workflow, this can be interpreted as the number of times the implementation fails after the test is already written and before the test finally passes. In processes that follow up-front design, a mistake rate could be the number of revisions or changes needed to refine the design until it meets requirements.
Types of mistakes
Depending on the context, mistakes can take various forms. Without trying to create an exhaustive list, here are some common examples:
- Syntax mistakes: Errors that prevent code from compiling or running.
- Logical mistakes: Errors in reasoning that lead to incorrect implementations.
- Design mistakes: Flaws in design that require rework.
- Architectural mistakes: Very expensive decisions that often require rewriting of the system.
- Communication mistakes: Misunderstandings of what needs to be built.
- Product design mistakes: Errors in interpreting the market needs.
Each type of mistake is associated with different development activity, different detection mechanisms, and different costs to fix.
Mistakes Rate and Model Plasticity
Together with the model plasticity, mistake rate helps create a more complete picture of software development cost. While mistake rate tells us how frequently we need to correct our course, model plasticity tells us the cost of each correction. And just like plasticity, different models will exibit different mistake rates.
Mistake rate is affected by many of the same factors that influence model plasticity:
- clarity of requirements and intent
- quality of abstractions
- coupling and system complexity
- tooling and automation
Understanding mistake rate can help teams choose the right level of abstraction for their models. A higher level model may have a lower mistake rate due to lack of detail, therefore opporunities to make mistakes, or it could have a higher mistake rate due to lack of feedback and visibility of aspects that are more than just implementation details. This gives us a way to reason about trade-offs of higher level models.
Why mistake rate matters?
Mistake rate is not just for measuring human developers. As AI-powered coding agents become more prevalent, understanding and optimising mistake rate allows us to design more effective coding agents, choose better models and optimise our development workflows.
Designing better coding agents
By monitoring mistake rate, we can identify where coding agents struggle. High mistake rates may indicate that the agent needs better context, more focused prompts, or improved reasoning strategies. Iteratively refining these aspects can lead to more efficient code generation.
Choosing the right abstraction models
Working at diffferent levels of abstraction, from high-level designs to low-level code, affects mistake rate. Higher-level models may have higher mistake rates due to ambiguity, while lower-level models may have lower mistake rates but higher complexity. Balancing these trade-offs is key to effective development. A higher abstraction model may increase model plasticity, but will it also increase or decrease mistake rate? That depends on the specifics and these two concepts help us understand these dynamics.
Optimising development workflows
Mistake rate highlights inefficiencies in the development process. By analysing where mistakes occur, teams can identify bottlenecks, improve communication, and streamline workflows. This leads to faster convergence on correct solutions and ultimately higher productivity.
Next Steps
To complete the picture we’re missing one last concept: feedback loops. Feedback loops determine how quickly we discover the need for change, and they play a crucial role in both mistake rate and model plasticity. In future work, I will explore feedback loops in detail and how they interact with these concepts to shape the software development life cycle.