Why Performance Testing In The Cloud Is Amazing

Cloud computing has its roots back in 1970s when telecommunication providers started to utilize common infrastructure to serve the customers. It has come long way and has become mature offering as per Gartner’s Hype Cycle theory. Offering wise cloud Services can be categorized in three areas Infrastructure as a Service – IaaS (e.g. Amazon’s AWS), Platform as a Serivce – PaaS (e.g. Microsoft’s Azure) and Software as Service – SaaS (e.g. many day to day services that we use like email, file servers, instant messaging, etc.). Deployment wise Cloud Services can be categorized as Private Cloud, Public Cloud or Hybrid Cloud. In Private Cloud model entire software solution is hosted in the dedicated infrastructure, which is hosted either internally within the organization data center or externally in the dedicated third party data center. In Public Cloud model, underlying Hardware and Network infrastructure is shared among multiple customers. In Hybrid deployment model, mix of Private and Public infrastructure takes place.

With the Cloud at the core of System Under Test (SUT), testing approach needs to change. While Cloud Services bring in good things like cost optimization, efficiency via global distribution of the applications, etc. It also brings in dependency on Cloud Service providers for scalability, availability and reliability aspects. One has to take proactive measures to guard these aspects ongoing basis.

Conventionally performance testing has been done in two phases. In the first phase, testing is done on modest hardware to identify software bottlenecks. Consequently performance testing is done on larger hardware to measure performance metrics. Performance metrics is then used to extrapolate information regarding a) number of users given infrastructure can support b) required infrastructure to support given number of users.

With Cloud in the picture, one can take advantages of its just-in-time scalability and global availability benefits. When one conducts performance tests of the applications hosted in the cloud environment one can go for actual user load with desired load pattern at various times of the day; instead of going for small representative load and then extrapolating the results. When it comes to Cloud deployment, there are many varying parameters (e.g. network, storage, distributed components, non isolated network in the cloud environment, quality of hardware, etc.) which may impact quality of the application (SUT) and thus making it really difficult to extrapolate results. Hence testing against the actual user load makes more sense. In order to simulate actual user load patterns, load generation can be made close to real usage pattern by distributing load generation across different load generating machines (called load generators) hosted in the data centers of the cloud provider in different parts of the world. Since these machines (load generators) are hosted in the cloud, in order to save the cost, one can use and pay for these machines only during certain times of the day. Efficient DevOps strategies and good use of right DevOps tools can play a very effective role in this. Need of actual user load generation makes User Profiling activity a key component in performance testing activities of a cloud app. In order to derive realistic user load patterns, existing usage data can be used along with analysis techniques.

Application hosted in the cloud may not have direct access to hooks (APIs, utilities) provided by underlying platforms to measure performance metrics. This is especially true when SUT is dependent on PaaS or SaaS. In such cases one needs appropriate APIs which can provide the performance metrics indicators (e.g. processor, memory, I/O, etc.)

By integrating right DevOps processes one can conduct performance testing on regular basis and thus doubling it up for the monitoring activities of the SUT. For monitoring key system metrics such as resource utilization and key user metrics such as response times, throughput, etc. need to be measured at the minimum.

Numerous tools and platforms are already out in the market which can assist development and configuration of performance test scripts of applications hosted in the cloud. These tools are also provided as cloud offerings themselves. To name a few there are tools like CloudTest by SOASTA, LoadStorm, BlazeMeter, Nessus, App Thwack, Jenkins Dev@Cloud, Xamarin’s Test Cloud, etc. which can be considered for non functional (performance, security) as well as functional testing of the cloud applications or On Premises applications.

As per Gartner Hype Cycle, the Cloud Services have passed the initial phases like inflated expectations, disillusionment, enlightenment and now have entered the mature Productivity Phase. Cloud Services are there to stay here for a very long period. Businesses are orienting their needs and services around Cloud Services. High Availability, Scalability and Reliability are among critical quality attributes which application providers have to guard constantly. Hence these need to be part of product engineering and quality monitoring activities from the very beginning of the product engineering cycle. There are many tools and platforms which can be evaluated while planning performance testing approach. If the Cloud Application depends on third party PaaS / SaaS components, it needs right hooks to fetch key performance metrics to monitor the performance. DevOps will play key role in continuous deployment and monitoring of functional as well as non functional quality attributes of the Cloud Application.