Performance Testing

Background of the Field 

Due to the wide reach of the internet, cloud infrastructure, and mobility; every person in the world can be your potential customer if you are ready to cater them with your software solutions.

Limit of your service solely depends on how many people you can cater to. A few examples: Amazon, Twitter, Facebook, WhatsApp and Netflix etc.

One has to think through and design properly for developing a highly scalable and highly available software solution. 

Similarly validation and verification of highly scalable and highly available software system is also an elaborate activity.

We typically break down software performance testing into two phases. 

In the first phase we test the system with very modest hardware resources, then we stress the system up to that point till it breaks down. This process is called as stress testing.   

In this process we try to find the software and design bottlenecks in the system.

In the second phase, which is also called as Performance Testing phase, we simulate the production testing environment and generate the seed data. 

It is either done from the production system by removing personally identifiable information (PII) from it, or by generating production equivalent data using custom built utilities. 

We then simulate load according to the peak load of production environment and by keeping some buffer load on the top of it for a considerable period of time to mimic the production environment. 

Application metrics and system metrics are recorded and monitored while the performance test is being conducted. Application metrics such as page response time, system throughput, transaction time, etc. are monitored. 

System parameters such as processor, memory, I/O network utilisation, are also measured. 

At the end, loss of any test data is verified while checking for any errors or exceptions in the logs. 

Based on all this data we prepare a report which consists of important statistics and insights. We also present our recommendations regarding architectural design so as to improve the system performance.

Problems Solved for our Customers 

For a General Insurance customer we designed performance test strategy and conducted performance testing of their application user interface. 

Customer’s system was developed using Microsoft technology stack and was deployed over Windows platform. 

We used open source tools to conduct performance testing.

For a Health Information Exchange customer we designed performance test strategies for different test components like web series services, BizTalk Adaptor interface, Enterprise Master Patient Index (EMPI) component. 

We performed component stress testing as well as an end to end testing for the entire system. We also conducted stress testing of the whole system.  

Post that we carried out performance testing of end to end system with 1 million patient demographic hl7 messages, and 30 million clinical data hl7 messages. 

We also tested EMPI component in cloud as well as local data centre environment  and provided capacity planning inputs

For a Banking software product, we designed and conducted stress as well as performance testing in on-prem environment as well as over the AWS cloud.

The product was developed using Microsoft technology stack and was hosted on Windows server with IIS as web and application server.

For a mobile data management (MDM) customer we planned and conducted stress as well as load testing for the key user scenarios such as: bulk device enrolment,  bulk user enrolment, and enterprise application distribution. The solution was a Multi-Tenant solution and was hosted over AWS cloud environment.

For a distributed peer to peer data backup product, we performed stress and performance testing of the agent node components as well as Central coordinator component.

For an enterprise scale Retail Task Management customer’s product, we designed and performed end to end performance testing for their solution. It was deployed in their own data centre which was located remotely. 

This involved generating and feeding the system with complex base data and also resetting of system state at the end of every performance test run.

Services we Provide

Services we provide include: 

  • Consulting for architecture design review of the system. 
  • Performance Test Strategy, Design and Execution. 
  • Capacity planning. 
  • Implementing design and code changes for performance enhancement of the applications. 

Team Composition  

We have performance test architects who have around 10+ years of experience in the industry, and who have done performance strategizing, performance test development and performance test execution for various software products.

We have performance test engineers who can develop performance tests using open source as well as commercial tools. Build seed data for setting up production equivalent environment, execute performance test and prepare performance test reports.

We have software architects who can analyze the system, recommend design and code changes. 

We have software engineers who can help in fixing performance issues.

Tools & Technologies

We use AWS, Azure, Openstack based cloud to simulate cloud based production environments. Our engineers can deploy complex test environments from scratch.

We use Virtual Machines or Containers to deploy performance test environment according to production deployment architecture

On linux systems we use tools such as Prometheus, Graphite, Nagios and utilities like Top, nmon, vmstat, Iotop, iostat, Iftop  for recording and monitoring systems metrics.

On windows systems we use tools such as PERFMON, SCOM, Nagios for recording  and monitoring system metrics

We use web server logs to record server response time and throughput. We use load generation tools to record client response time.

We use open source tools such as Nagios, ELK Stack, IIS Web Server Log Analyzer for analysing web server logs

Based on the requirement we build custom load generation utilities or use tools like Apache JMeter, Microsoft Visual Studio, IBM Rational Performance Tester, AppPerfect Load Test tool.