Predicting the Future of Testing

As the end of the year approaches, psychics and pundits alike will start making their predictions about what's in store for us in 2004 and beyond. In this week's column, industry veteran Harry Robinson gives us his forecast on the future of software testing

"It is tough to make predictions, especially about the future."—Yogi Berra

Every December, tabloid fortune-tellers reveal what will happen in the coming year: “Madonna will fly on the space shuttle,” “the U.S. capital will move to Wichita, Kansas” and so forth. I’d like to jump on that bandwagon and make a few predictions of my own about where software testing is going in the next few years. And I can only hope my prophecies fare better than those of my esteemed tabloid colleagues!

My main prediction is that software testing in the future will look very different than it does today. My reasoning is straightforward: Software testing largely stinks today. It comes into a project too late, contributes too little, and costs too much. If we care about the quality of our products and the health of our bottom lines, we need to re-think our approach to testing and quality.

I will even go out on a limb and say that better methods, better training and a better appreciation for testers will revolutionize the software industry. To be specific, technologies such as executable specifications, model-based test generation, bug prevention, and system simulation will play important roles in the unfolding drama.

Here are some scenes we will see in the industry over the next few years. In fact, some of these trends are starting to emerge already.

Testers, Spec Writers, and Developers See Themselves as Partners

Testers Help Spec Writers
Testers work with spec writers to review and understand the spec as it is being written. Early review and modeling exposes many consistency, completeness, and ambiguity bugs while they are still cheap to fix.

Spec Writers Help Testers
The test team creates models to generate tests of its applications' behaviors. Spec writers review the models to ensure that they adequately cover the feature set. The resulting test model becomes an "executable spec."

Testers Help Developers
Because the spec is clean and unambiguous, the developers understand better what their code should accomplish. Testers provide lightweight models that developers can run against their code before they officially hand it over for testing.

Developers Help Testers
Proceeding on a feature-by-feature basis (to avoid the late-in-the-cycle, product-pounding approach of the past), developers and testers ensure that the code is easy to test with automated methods. Developer's code is now full of testability hooks, making errors more detectable.

Testers Help Testers
Tests are now modeled in a high-level language, so teams working on other features (and even other products) can help review and improve the test models. This creates a community of test generation experts.

Methods and Metrics Get Better

Bug Prevention and Early Detection
Because the emphasis is now on the quality of the deliverable (and not on how many bugs you found along the way), prevention practices and detection tools, such as static analyzers, are mainstream.

Simulation in Testing
Simulation tools become common, making it easy to “fake” computer environments. Testing for exceptions and error paths now happens early in the development process. After the code has stabilized, real environments verify that the simulations were accurate.

Just-in-time Test Cases
Massive test case management systems are a thing of the past. Most tests are generated on the fly. Test cases are no longer stored away like rotting inventory, so it is easy to keep tests up to date.

Positive Metrics
Misleading metrics, such as bug counts and test case counts, are dead. Useful metrics, such as spec coverage, model coverage, and code coverage, drive the projects.

Fewer Testers, Better Testers
Machines now perform much of the mundane work that testers previously did creating tests. Teams require fewer testers, and the testers who remain are more highly trained. Their work is more interesting to them because they are focused on bigger issues in their tests rather than slogging through grunt work.

More Tests, Better Tests
Testers can now generate millions of tests on any day, so the challenge becomes how to run the most useful tests first. Combinatorial tools allow testers to prioritize their testing and aim their test runs at the areas most likely to have significant bugs.

Roles Will Change for Testers

Distinctions Blur in Testing
Work in the testing field blurs the line between people who only specialize in hands-on testing and those who only create test tools. A new specialty emerges that encompasses both "testers who like to break things via programs" and "programmers who like to create programs that break things" – people in newsgroups debate endlessly about what to call the new specialty.

Boundaries Blur Between Testing and Development
Testers and developers work in tandem to produce testable, high-quality code. Testers help iron out spec issues to make the developer's job easier, and developers create cleaner, more testable code to make the tester's job easier.

Feedback from Customers Becomes Integrated with Testing
The quality of deliverables becomes higher. Testers routinely conduct root cause analyses. We ask questions such as "How did we miss this bug?" and "How can we prevent this type of bug in the future?" We work to delight our customers.

New Challenges Emerge
The sophisticated and interconnected environment of the computing world guarantees that new problems such as security testing continue to keep testers running hard. This is OK—testers find these challenges invigorating.

Testers Get Respect
Testers are no longer called in at the last moment to "pound on the product." They provide a visible, vital, value-added service throughout the software development process. People realize that testing can be rewarding, interesting, and even fun.

Testing Gets Trendy
Software testers start to hold their heads high. And, because breaking stuff is at least as much fun as building it, people begin to rotate between positions in development and testing. Everyone learns more about what makes good code.

Adrenaline Junkies Move On
The new process works so well that spec writers, developers, and testers end up having lives. This is disconcerting to some who were raised in the adrenaline-charged world of late-night, last-minute, firefighting sessions. These people gravitate to companies that remain out of control.

Elvis Presley Is Discovered Working as a Software Tester
The giveaway is his conference paper titled “Software Quality: It’s Now or Never”.

Prepare for the future today

Whether or not my predictions come true, the future is on its way. Here are five ideas for how you can prepare to meet it:

1. Get Actively Dissatisfied
Don't accept the current state of testing. Look around and think, "What are we doing that makes no sense?"

2. Push the Envelope
Figure out how to test better and share that knowledge. Overall quality will improve only if everyone seeks to make the code they are working on the best it can be.

3. Learn More about Testing
At this moment, the industry is exploding with innovative software testing ideas. Go to conferences, join mailing lists, and scour the Web to see what is happening on the cutting edge of testing.

4. Learn More about Development
Take a programming class. Even if you don't plan to write significant amounts of code, view the class as a reconnaissance flight over bug territory.

5. Change the World!
As PC pioneer Alan Kay said: "The best way to predict the future is to invent it."

About the Author

Harry Robinson is Test Architect for Microsoft's Enterprise Management Division. In addition to his day job, he teaches classes on advanced software test automation and is a driving force behind Microsoft's model-based testing initiative. He has been at Microsoft for five years and has a Bachelors and Masters in Electrical Engineering. You can reach him at harryr@microsoft.com.