When business owners discuss the best way to develop software, testing is always a scary topic. How much time and money you should spend on software testing is not an easy question to answer. What type of human resource you should commit to testing is an even harder one.
Test automation obviously requires some degree of programming skill, but software developers are a precious resource. Why waste developer’s time on automating tests when you’ve got new products and features that you need to roll out the door? Bringing on a dedicated test automation developer is no small expense, either. You could probably hire two or even three manual testers for about the same amount as one software engineer. What makes a test automation engineer that much more valuable?
These questions and their answers mean everything when creating a software testing strategy.
We recognize and advocate the benefits of both manual and automated testing in their proper contexts. We understand that the business must weigh the benefits of each type of testing to create a plan that returns the best ROI.
The key to creating the right test plan is to listen to your team. When your developers or manual testers come to you asking for more help, ask them what they need more help on and listen carefully to the problems they are having. Knowing what they are having trouble accomplishing will help you discern what type of help they need.
Do you need a manual tester?
There is one simple rule in testing any software. You always need someone besides the developers that made the software testing your product before release.
Your developers know how they designed their creation to work, and they test it to make sure it works that way. But developers have skills and a viewpoint on how their software works that users do not have, and never will. Consequently, they often struggle identifying the types of troubles users will have while trying to use the product. Only an external set of eyes can give developers a critical early view of how end users will react to their creation.
Exactly what type of tester will do the testing depends on the kind of product being tested. If it is a customer facing application such as a web or native mobile app, you should have a manual tester, ideally one without development skills. On the other hand, if it is a programming interface that will only be used by other computer programs, you will need someone with development skills.
Once you get people besides the developers testing the software, you need to make sure they are pulling their weight. Make sure to ask these questions:
- How well do your developers say the testers are communicating bugs and other problems? What is the average time for developers to work on a specific tester’s bug reports before they are fixed? Writing a concise bug report that points the developer to the root of the problem is a skill to be developed. If they are not performing this task well, consider enrolling them in a class or pointing them to test training literature.
- Has there been a drop in bugs reported out in the field since this tester came on board? If your testers are not providing a quality view into how customers will react to the product, perhaps they should spend some time in the helpdesk area to get a better idea of how end users are having trouble.
Automation is only a question of how much
All software will have some types of automated testing or else it likely will be broken every single day. The real questions you need to ask is how much more testing should you automate. Here are some things to consider that can indicate how much automation is missing:
- Is the Alpha testing environment unreliable for basic testing? Users expect quality enhancements/fixes in every release, and testers will expect a certain level too. When developers are not delivering working code, this is a red flag that quality unit testing is not being done. Good developers always build test automation for unit tests!
- Are your manual testers complaining about the amount of time it takes to simply verify that already-tested functionality has not broken? Are they managing long manual test scripts to remember the critical functionality to retest in each release? Almost all manual testing scripts can be automated. The longer they are, the more likely it is you need to consider automation to speed up the process.
- Is your release process slow and error-prone? For new developers who have never released software before, they may have no idea just how much of the process could be automated. Continuous Integration (CI) is one science under the mantra of the DevOps methodology. It can not only drastically lower the amount of time it takes to release, but it can also give consistently increasing confidence in the ability to release software more often. “What do you mean ‘more often’?” you say? The proper release strategy can change your release expectations from once or twice a year down to weekly. Some companies release new software every day. But don’t you dare expect any team without significant automated processes in place to deliver that kind of turnaround and maintain consistent quality. It is simply not practical.
Who should own your test strategy?
So after considering your situation, you have decided you need a lot of automated testing and a lot more than you have now. OK. Who do you assign to the task, and what should their priorities be?
Well, for starters, it will need to be a skilled developer. Sorry. There’s no way around it. We’re talking about building an internal product line, not for your clients, but for your workers, and that includes developers. If you expect the best ROI possible, you need an architect. Manual testers simply will not have the skills to create something that will stand the test of time. After all, aren’t they already swamped trying to make sure the latest release is good?
So which developer? You’ll naturally want to see if you can use one of your own. Is there one that has really been trying to include automated tests in their development already? Have you ever scolded one for it? “I’m paying you to develop software, not tests!” Well, there’s a good candidate.
But that doesn’t necessarily qualify them to lead the effort. You should quiz them on just how much they know about test automation. Is it just an idea they’ve been toying around with, or have they done their research? What test frameworks have they used or do they currently use? Do they know terms like CI and DevOps?
It will probably be a good idea for you to at least temporarily bring in someone who has experience setting up good CI environments and coaching your entire team on how to keep it rolling.
Are you prepared to make this a priority?
So you’ve got someone who’s on fire and knowledgeable about setting up a good testing environment. Who should they work with first to get automated tests going?
Well, once again, it should be the developers. Yes, all of them. I know. You’ll have to put off that next big release for who knows how long to let them set up the infrastructure and get into the habit of using it. Here are some reasons why that investment is worth it:
- Developers, not testers, should be primarily responsible for the quality of their software. Insisting that they start consistently writing unit tests alongside their new and modified code is the best way to get them working true quality into the foundation of your product.
- With the right tools and testing structures, unit tests run much faster than integration tests. This means your developers are much more likely to be alerted of a problem before they shift their focus to other tasks, which is critical for productivity.
- Unit tests are also easier to write and maintain. It’s best to start out simple, testing small bits of the code at a time. This is where someone with experience in mocking technologies can be very helpful. Once they have a handle on that, they can add increasingly complex integration tests.
- Unit tests are developers’ best documentation for other developers. More than likely, your developers are eventually going to hand their code off to someone else. At the very least, their colleagues will get their hands into that code at some time or another. Automated unit tests force other developers making changes to first consider all of the different angles the original developer has had to consider when writing and enhancing the code.
- Automating the manual tester’s regression scripts first, especially without the aid of the developers, is a practice doomed to failure. Once your test automation expert moves on, the other developers are much less likely to maintain his work, and it is unlikely that the manual testers will have gained sufficient ability to do it themselves.
Now a word of caution, or perhaps from your perspective, relief. Your developers should not try to go about setting up enough unit tests to cover all, or even most, of their legacy code. After all, while it may not be unit tested, it has probably been manually tested pretty well. So until it gets changed in some way, there is less of a chance that it will break. It’s enough for the developers to just set up their code scanners to ensure that new and changed code is always unit tested.
Now once you’ve given your developers all the tools they need to automate unit and integration tests for their code, you can then come around to setting up the infrastructure to allow your manual testers to automate those scripts they’ve been manually exercising. When you do, you might be surprised to find that they actually have the time and peace of mind to learn how. That’s because when your developers started writing unit and integration tests in the process of working through all of those annoying bug tickets, the testers stopped having to write so many.
Oops! You’ve run out of budget to support that top notch consultant. No problem! Your developers now have enough test automation skills to take up the design, creation, and maintenance of the unfinished part of the process.
And one more surprise. The testers and developers have actually started liking each other again! The unit and integration tests have helped the testers trust the developers more. These new testing tools are also allowing them to automate the boring parts of their job. Now they can focus on the indispensable (and fun) part: Testing sleek new functionality and giving feedback (including new automated tests) to make it the best product your customers have ever seen. What developer doesn’t like that kind of feedback?
That’s the icing on the cake of a solid product that will last for a long time to come.