How do we decide whether to automate and what to automate?

In test projects, a typical conundrum that a Test manager/Lead faces is whether to automate and what to automate? A high level guideline one must use in this case is whether Return on Investment (ROI) exists in this exercise. I would like to illustrate this by describing the decision making process we went through on a recent project that was successfully completed in our company. The project plan indicated that the testing activities for the Cloud based multi-tier Web application would span 2 months. A basic set of functional tests existed for the current version of the application. These tests would constitute the regression test suite for the existing functionality, even as the development team built new features encompassing UI screens, and APIs. We, the test team were contemplating the design of a test automation framework to complement structured and ad-hoc manual testing. The test plan included two full regression test passes at the most over the duration of the project. The highly compressed time frame (2 months) of the overall project, added to the fact that the UI would be unstable for the first several weeks of the project, led us to conclude that there was clearly no ROI in UI test automation. Having chucked off the UI test automation option, we moved on to the next automation candidate, which was test automation of the APIs. We were looking at several existing Restful APIs, with a plan of adding a few new ones. The API layer was evaluated to be more stable than the UI layer, and the automation team was more confident about building a robust test automation framework for the APIs in a short time frame. Moreover, we saw the opportunity to reuse the API test automation framework for Scalability testing also. We could potentially simulate several scenarios involving concurrent users, creating load/stress conditions etc. by leveraging the API test automation framework. Continuing on the API test automation path was expected to give us the desired coverage and the ROI!

Using a combination of the aforementioned test automation strategy, and manual regression and new feature testing at the UI level, we were able to flush out functional bugs, uncover scalability issues, memory leaks, and many other deep rooted issues. We had designed the API test automation framework such that it was extensible in order to support configurability of automation for future use. At the end of the testing engagement, we were able to release a stable and a high quality product, within the stipulated timelines and budget. That said, our recommendation to the end customer was to take up UI test automation framework development to automate their UI regression tests for future test passes in anticipation of getting the required ROI over the next several releases.