Business companies constantly struggle to reduce software release life cycle; this increases both budget pressure and emphasis on quality. Automation is among the popular ways to boost the QA efficiency. It portends growing test coverage, eliminating tedious routine and error-prone human hand as such, and cost saving. But the initial automating effort is a significant investment, not a short-time solution. There are many successful attempts, but disappointments are not so rare.
Before introducing test automation, we must ensure that our product (and requirements to it) is relatively stable and no serious redesign is expected on a short-term horizon. Every time the software is modified, test suite needs to be updated. Next, what tests should be robotized? The more repetitive scenario is, the better candidate it is for automation. For example, smoke testing, basic acceptance and regression tests, ran upon every build or release.
Performance testing, like simulated load from a thousand users, can be executed only through test machines. Automated tests are able to generate millions of events and combinations limited just by hardware power and time allotted.
It is smart idea to automate the similar workflows but using varied input, that is data-driven tests. Applying random values also allows to do new things every time. With completing long forms, good test automation solutions are capable to process over two thousands dialog boxes overnight; imagine how long would this take manually. While testers are able to focus precisely on new features, exploration and sophistication of test scenarios.
Automated tests typically require substantial maintenance, particularly cheap express techniques, such as capture-playback. Sometimes they have a predicted “life” of only one or two runs, yet making them is three to ten times as expensive as equivalent manual work. Another factor of growing automated test maintenance expenses is the complicated environment. And, surely, to receive the best value from the solution, test results satisfying pass and fail conditions must be predictable and reliable. Many companies tend to automate their business-specific critical paths, particularly those which envisage multiple user roles. For such cases it is useful to have tailored automation test framework, seamlessly integrated and aligned with different development phases. It has to be extendable, scalable and re-usable.
Automation goes hand in hand with mobile app testing, which is not surprising, since this field always involves multiple OS, browser and device combinations, where, for example, a multitude of web pages should look and behave the same. User interface tests are especially fragile in mobile world. They require testing tools, robust before minor changes in the interface layout, sensitive to the principles of responsive web-design. Modern test automation solutions are typically compatible with popular architectures: Java, Python, Ruby, .Net, PHP. Today, test automation has been taken to the Cloud in secure Global Data Centers. Cloud-based solutions like Selenium WebDriver or Appium are used to ensure application quality on real mobile environments in real time against live end-user experiences. Other factors to consider when choosing tool may be compatibility with the task and issue management system in place, ubiquity in the market (wide user base means good customer support and other networking advantages), and smart features simplifying script maintenance. If the solution is open-source, good software testing provider will overcome its limitations, enhance it to the enterprise grade for their customers. Such tools as Cucumber, MS Visual Test, SOAP UI, jUnit, testNG, Calabash pay back their cost pretty well.
When calculating the return of investment (ROI) made into test automation, we consider number of runs, compare estimated values of shortening test cycle, increased coverage, accomplishing new value-adding test types, labor saving, to expected efforts for: establishing infrastructure, coverage, updating between releases, execution and results analysis. Test results have to be measurable, and reports should be provided in flexible manner according to the customer’s need.
Intangible benefits of automating testing include improved competency, expansion into advanced issues, faster confirmation of builds, and, above all ‒ improving product quality. However, the machine doesn’t care if its results are awkward or useless. Selecting improper test automation partner will lead to false and duplicate reports, unreproducible bugs, error cascades, and, ultimately, investing in “shelfware”. Finally, remember that automation will never substitute the first round of manual testing for new functions. It is intended at reducing expenses and risks associated with regression testing.
