Flexible and Disposable Test Environment on Cloud

By VamsiKrishna Koundinya & Kapil Sahu

Introduction

The Selenium test automation suite of a typical web application consists of hundreds of test cases. If these test cases are interdependent then the executions of the same must be done sequentially. It may take a considerable amount of time to validate the execution for the complete suite.

In a contradictory scenario, if the suite can be designed in such a manner that the execution of any test case within the suite is independent of another, all the test cases in the suite can be executed in parallel and time efficient manner. This way the longest time to execute the complete suite is almost equal to the longest test case execution time.

This also gives the developers who have contributed to a certain build an opportunity to do a quick sanity check and validate the changes on the build.

Challenges

Running the test executions parallelly for a large number bring in a big challenge of having a large dedicated compute intensive infrastructure to handle these executions.

Reserving such a mammoth dedicated capacity within your On-Premise infrastructure would

• Incur a massive initial set up charge.
• Need some set up time before getting the ground running.
• Require a timely maintenance cost along with other operational infrastructural costs.

Additionally, in case one decides to go with the On-Premise infrastructure model, it cannot be scaled up or down on demand i.e. when the number of test cases in the suite boosts up to a larger extent, infra team would need to procure additional capacity which again would be a time and cost-intensive process.

Cloud to the rescue

Running the infrastructure on the cloud becomes an obvious choice for the playground to execute the test suite. When moving to cloud the initial setup time and cost are negligible, the maintenance and operational Infrastructure cost is also negligible.

Along with the cost and time parameters cloud gives a highly flexible and scalable environment that can be extended on demand i.e. even if the load to the infrastructure explodes, we can allocate the required capacity and pay only for that particular duration.

Even the initial required capacity need not be reserved and can be allocated only when one needs to execute the test suite. Hence the only cost incurred is the execution time of the suite on the cloud.

AWS with extensive features and vast instance flavors for running compute-intensive workloads can become a strong candidate to run the test suite.

Proposed Architecture

Component Details

Jenkins

Jenkins is an open source automation server, that provides hundreds of plugins to support building, deploying and automation of any project. The AWS CloudFormation plugin for Jenkins allows for the creation of cloud formation stacks before running the build and the deletion of them after the build is completed.

This plugin gives Jenkins the ability to spawn Amazon Cloud Formation stacks before running job for the build and stopping it at the end of execution.

AWS Cloud Formation

AWS CloudFormation is a service that helps in modeling the complete AWS infrastructure as a code that is basically a JSON/YAML containing detailed configurations for the different AWS resources. This way the time required to manage the resources is reduced, that can then be focused on the applications hosted in the AWS environment.

There is no need to individually create and configure the AWS resources and figure out the relevant interdependencies, AWS CloudFormation handles all of that.

This template will launch an EC2 instance running an Amazon AMI in a Public Subnet within a VPC with Internet Gateway attached. Ports in the Instance Security Groups are opened to allow access to SSH, http/s and Zalenium grid UI.

The route tables associated with the public subnet can be configured to be accessed only from the organization’s public IP as a security precaution. The role required for the instance to upload data to S3 bucket can also be defined within the template resources.

The user data scripts for the instance can be designed to create the complete custom platform such as installing the required software to run the test suite and initiate the test execution. Once the execution is complete the reports can be uploaded to an S3 bucket put request via AWS CLI.

Zalenium

A flexible and scalable container-based Selenium Grid (basically a docker image) that can be used to get a grid up and running in a few seconds along with the advantages of

• A flexible and scalable container-based Selenium Grid (basically a docker image) that can be used to get a grid up and running in a few seconds along with the advantages of
• Live preview of the test case execution and detailed informative dashboard.
• It’s a dynamic selenium grid that can expand and contract based on load.
• While using a selenium grid it becomes complicated to maintain it over time (keep up with the new browser, Selenium and drivers’ version). In the case of Zalenium, the docker image maintains it for the end user.
• Simple to set up and use. The ramp-up time is very low to get the grid up and running.
• Logs and video recordings of execution are easily available over the dashboard.

GIT

Git is an open source distributed version control system. The latest code for the sanity test suite is maintained in git that can be pulled as a part of user data scripts in the ec2-instance and then initiate the suite execution on the latest code.

AWS S3

Amazon S3 is an object storage service for storing large unstructured data with virtually unlimited scalability. With low cost, ease of data access and security of the buckets, S3 stand a strong candidate to store the date-based reports for the executions.

Once the execution of the complete suite is complete the reports/ recordings can be maintained under a date-based folder structure into a S3 bucket for any future references. Lifecycle policies for the S3 bucket can be used to move objects from S3 buckets to cold/ archival object storage i.e. Amazon Glacier to further reduce the cost.

Cost Benefits

With the above architecture, the only cost incurred to run the complete sanity will be the cost for the time for which ec2-instance is running.

The runtime and hence the cost are directly proportional to the run time of the longest running test case in the suite as they are running independently of each other.

Example Scenario

Consider there is one single run requested for the sanity suite in a day and the longest test case in the suite takes roughly around 15 minutes, then for the whole month it will take an approximate of 8 hours only, for execution.

Considering that even if we use a very compute intensive ec2 instance such as c4.8xlarge with 36 cores to handle the large parallel executions. The cost for 8-10 execution hours would only be $12-$15/ month.

Additional Cost Optimization with Spot Instances

A Spot Instance is an unused EC2 instance that is available for less than the On-Demand price. Because Spot Instances enable you to request unused EC2 instances at steep discounts, you can lower your EC2 instance costs significantly.

With spot instance, the rates for this particular instance go down by almost 50-60% i.e. only around $6-$7/ month. But, going for spot instances would require the test automation suite to be designed to handle interruption.

Interruption rates for Spot instances

It is always possible that your Spot Instance will be interrupted. For this EC2 instance, the interruption rates are as low as 5% and that can even be reduced further with the introduction of a spot instance fleet. Please check the AWS Spot Bid Advisor and pick instances that are having low interruption rate.( < 5%)