I’ve worked with some software organizations that always meet their delivery schedules. When it becomes clear that the project won’t finish on time, the team enters a “rapid descoping phase.” They defer some planned functionality, whiz through testing, and release on schedule a crippled system with quality problems that provides no value to anyone. Then they declare success because they delivered on time. Yes, they delivered something on time, but it bore scant resemblance to expectations.
Defining your product’s release criteria is an essential part of laying the foundation for a successful project. Release criteria must be realistic, objectively measurable, documented, and aligned with what quality and success mean to your customers. Decide early on how you’ll determine when you’re done and then track progress toward those goals.
How Do You Know When You’re Done?
No one builds perfect software products that contain every imaginable function and function flawlessly. Every project team must decide when its product is good enough to go out the door. “Good enough” means the product has a blend of functionality, quality, timeliness, customer value, competitive positioning, and supporting infrastructure in place to achieve business success.
There’s no perfect, simple measure of software quality. The customer view of quality depends on functionality as well as factors such as usability, reliability, and performance. You need customer input to the release criteria. How would your customers judge whether the product was ready for use, whether they could cut over to a new application that replaces an existing system, or whether they’d be willing to pay for the new product?
Internal quality also is important. Can the software be modified and maintained efficiently during its operational life? Is documentation in place for the support staff? The project might end when you deliver the system, but the product could live on for years to come.
You shouldn’t have to change your release criteria unless business objectives or other key project realities change. But as the scheduled delivery date for a release, an iteration, or the full product fast approaches, you might reluctantly conclude that you can’t satisfy all release criteria. Identify the risks of failing to satisfy each criterion so the stakeholders understand the implications of relaxing or bypassing them.
Some Sample Release Criteria
Table 1 lists actual release criteria for two products. One is a revised internetwork operating system that runs on a family of routers; the second is the next release of a compiler. These criteria reflect important ideas, but some, such as “extensive customer exposure” and “high level of customer satisfaction,” are vague. You can’t judge whether or not such fuzzy criteria are satisfied. Some ambiguous words to avoid are: acceptable, adequate, appropriate, comprehensive, enough, extensive, high, improved, low, reasonable, robust, seamless, sufficient, and supported.
Consider writing SMART product release criteria that are:
Specific (not vague or imprecise)
Measurable (not qualitative or subjective)
Attainable (not unrealistic stretch goals that you’ll never achieve)
Relevant (related to customer needs and your business objectives)
Trackable (something we can monitor throughout the project)
Try applying these five properties to the two examples shown in Table 1. The release criteria for the internetwork operating system don’t make the grade. I anticipate energetic debates over what constitutes a “serious” defect and whether the internal testing and deployment were sufficiently comprehensive. The criteria shown for the new compiler are better, provided the meaning of supported is made clear in every case.
The following sections suggest possible release criteria in several categories. You might have other criteria that are pertinent to your situation. Tailor these examples for your own projects and keep the SMART characteristics in mind.
A buggy product can lead to excessive support costs, user disappointment, poor product reviews, returns, and even lawsuits. As one quality indicator, monitor the number of defects discovered during development and testing. Consider using historical data about defect densities (defects per thousand lines of code, or KLOC) from previous projects to estimate the number of potential defects.
Trends in discovery rates can suggest how many defects might remain undetected. Various software reliability engineering models can estimate the number of remaining defects based on the rate at which new ones are being discovered through testing. Consider adapting defect-related release criteria from the following general statements:
- There are no open defects of severity 1 or 2 on a 4-level severity scale.
- The number of open defects has decreased for X weeks and the estimated number of residual defects is less than Y.
- The rate of arrival of new defects is fewer than X defects per 100 testing hours.
- The density of known remaining defects is fewer than X defects per KLOC.
- All errors reported by source-code analyzers have been corrected.
- Known problems from previous releases were corrected without introducing new defects.
Testers might not really know just what it means when they declare testing complete. They might stop testing simply because they’re out of time. Some better testing-related release criteria are:
- All code compiles, builds, and passes smoke tests on all target platforms.
- 100 percent of integration and system tests are passed.
- Specific functionality has passed all system and user acceptance tests (for example, the normal flows and associated exceptions for top-priority use cases).
- The mean time between failures is at least 100 hours.
- Predetermined testing coverage targets for code or requirements have been achieved.
You need multiple, complementary criteria. If no one does any testing in a certain period, obviously you won’t find any defects that way. Therefore, the rate of defect discovery alone isn’t an adequate quality indicator.
Another aspect of software quality concerns various attributes that characterize the product’s behavior. Quality attributes include reliability, security, integrity, usability, safety, maintainability, efficiency, robustness, and interoperability. You can assess the product’s readiness for release partly by whether it satisfies its critical quality attribute requirements.
For instance, the product should satisfy quantitative performance and reliability goals on all platforms. A reliability goal might state that the mean time between failures is at an acceptable level and is increasing with each build. Your product must satisfy specified security goals and requirements. It might also need to meet specified conditions so it passes a needed certification audit or evaluation. My article “Specifying Quality Requirements with Planguage” describes a structured way to write precise quality requirements.
Every product needs a minimal set of functionality that provides sufficient value to the customer before it’s ready to ship. Consider the following requirements-related release criteria:
- All high-priority requirements committed for this release are functioning correctly.
- Specified customer acceptance criteria are satisfied.
- All accessibility requirements are satisfied.
- All localization and globalization tests are passed.
- Specified legal, contractual, standards-compliance and regulatory goals are met.
- All functional requirements trace to test cases.
Configuration management involves uniquely identifying the components that are assembled into the product, managing and controlling versions of these components, and building them into the deliverable files and documents. Before you push your product out the door, ask some configuration questions:
- Can you build the product reproducibly on all target platforms?
- Has a physical configuration audit confirmed that the correct versions of all components are present?
- Does the product install correctly on all target platforms? Installation checks should include reinstallation, upgrades, uninstall, and recovery from aborted or failed install attempts.
- Is the release image free from malware?
Just because the software is done the world might not be ready for it. Make sure you’ve lined up the other elements necessary for a smooth rollout and implementation. Prepare release notes that identify defects corrected in this release, enhancements added, any capabilities removed, and known uncorrected defects. Log any known uncorrected defects in the project’s defect-tracking system.
The support function must be ready to respond to customer problem reports, so confirm that software release and support policies and procedures are published and understood by affected stakeholders. The operating environment must have all necessary infrastructure in place to execute the software. In addition, the software manufacturing and distribution channels should be ready to receive the product. Manufacturing is especially important when the product includes software embedded into hardware components.
Making the Call
Determine early on who will monitor progress toward satisfying the release criteria and who will make the final release decision. The project’s sponsor should agree that the release criteria are appropriate indicators of likely business success. You’ll need the sponsor’s commitment to rely on these criteria to make the decision. If management overrides the objective indicators, the team won’t take release criteria seriously on future projects.
Track your release criteria in a binary fashion: each one either is or is not completely satisfied. You might create a color-coded tracking chart. Green indicates that the criterion is completely fulfilled, red means that the criterion is not yet fulfilled, and yellow indicates a risk of possibly not being able to completely satisfy the criterion. Don’t use yellow as an indicator for partial satisfaction. If the board isn’t all green, you aren’t done.
Agreeing on these end-game issues early on will help you lay a solid foundation for achieving project success.
This article is adapted from Practical Project Initiation: A Handbook with Tools by Karl Wiegers. If you’re interested in software requirements, business analysis, project management, software quality, or consulting Process Impact provides numerous useful publications and other resources. Karl’s latest book is The Thoughtless Design of Everyday Things.