XMPP Server Performance Testing

Recently, a customer requested us to carry out performance tests on their mobile application. Specifically, the chat functionality for which they had switched to using XMPP (using OpenFire) rather than the traditional Web Service model.

Read more on XMPP: http://en.wikipedia.org/wiki/XMPP

Problems faced using Web Service model:

• During peak load, the delay in receiving a chat message was as high as 8 minutes. This was unacceptable
• Message loss was as high as 22% during peak load
• Since the web service calls are fired over HTTP, and HTTP being synchronous, the messages are queued up on the client (mobile device). This lead to issues where the chat message was sent but never received.

After we did an initial Reliability test which found the above issues, the customer took the decision to move their chat to XMPP.

Challenges:

Besides being the most common solution for chat protocols (Skype, Gtalk, etc.) it is also asynchronous and implemented over TCP. So obviously, performance testing of such a service would be difficult since we will have to send requests over TCP and also receive the responses asynchronously. We also had to productize this since the customer wanted to run these tests.

How we planned to tackle these challenges:

• First and foremost was to understand the XMPP protocol inside out.
• Get a hold of the customer’s source code (which was luckily available) and understand how it was implemented.
• Is there any tool available (open-source) which we can readily use?
• Has anyone successfully performance tested this before? If yes, how was it done?
• Since, we had never done XMPP performance testing before, we were not sure what benchmarks we should target. Hence, we needed to get benchmark requirements from customer.
• We also needed to find out what was the server architecture and server resources available
• Brainstorm ways of productizing the end result.

How we finally cracked it:

1. We started with understanding the protocol. Since it is an open-source protocol, getting information about it over the internet was not quite difficult. The official website www.xmpp.org has tonnes of information on XMPP. That was a good starting point which gave us our basic understanding of what XMPP is all about. During this research we also found that there have been many people who have already successfully carried out performance tests (More on this later…)
2. We went through the customer’s code to understand how they have implemented XMPP. Which features were being used and which of those features would require to be performance tested. We made a list of features and designed performance test cases around them. After jotting down the test cases we decided to follow a two-pronged approach to test this new protocol:
   • Performance Testing: Plain old traditional performance tests where we hit the server with multiple requests and try to find out the peak load benchmarks. This should give us an idea of what the server and software is capable of handling. Also carry out CPU, Memory, Disk I/O and Network Usage profiling to find out bottlenecks if any.
   • Reliability Testing: Performance tests are good to stress the server, but what about stress testing the actual mobile app. Even though the message from the sender makes it to the server, it still has to pass through the application for the receiver to view it. For this we needed to test the reliability of chat messages on the actual app.
3. We had extensive experience in using JMeter for Performance tests, so we decided to research the possibility of writing our own plugin for XMPP. A little bit of googling led to this project which uses the smack library for XMPP connection. Using this library we were able to setup a connection, login and send chat messages to the XMPP server. We then bundled this into a jar file, deployed it into JMeter and voila! JMeter was able to spawn threads and we were able to hit the server with chat messages over multiple threads.Next step was to be able to automate the designed test case into JMeter and achieve the required throughput. The customer wanted to achieve around 1,000,000 chat messages per 24 hours (~ 11.5 chat messages per second) over 450 one-to-one chat sessions (~ 2,250 chat messages between 2 users over 24 hours). So, we decided to code the test case for 1 chat session and spawn 450 threads.We also decided to make sure that chats sent are also successfully received by a different client. We also coded a test case to get the sent messages. This would not only give us a list of lost messages by comparing sent and received messages, but also let us know if the sent message was delivered correctly to the desired recipient. Finally, using log4j, we set up logging of the output. To be able to compare the sent and received messages, we decided to output the log in csv format so that we can easily compare and generate reports in Excel. We got hold of around 900 active users and divided them into 2 sets of 450 each. One being the sender set and another being receiver set.
4. These are some of the important methods:
   Connection:

   public XMPPConnection doConnect(String userName, String password) {                XMPPConnection connection = null;try{                                Date date = new  Date();                                ConnectionConfiguration config = newConnectionConfiguration(host,port,domain);                                                           connection = new XMPPConnection(config);                                        connection.connect();                                connection.login(userName, password);                                }catch(Exception e){                                                 Date date=new Date(); logger.info(sdf.format(date) + “,” + userName + “,” + password + “,” + e.getMessage());                                 } return connection; }

   The sendChat method:

   publicboolean sendChat(String senderId, String reciverId, String authToken, String chat,String chatMin, String chatMax) {boolean isSend = false;                XMPPConnection conn = null;long sleeptime = Long.parseLong(“1000″);                Date date = new  Date();int rndNum=0;int i=0;                String chatMsg=“”;try { //Connect to the server and login using the given credentials                                 conn = doConnect(senderId+“@”+host, authToken); //Construct the message                                 Message msg = new Message(reciverId+“@”+host, Message.Type.chat); //A random number of chats per session will be sent   rndNum=Integer.parseInt(chatMin)+(int)(Math.random()*((Integer.parseInt(chatMax)-Integer.parseInt(chatMin))+1)); for(i=1;i<=rndNum;i++){                                                 chatMsg=chat + “_” + i;                                                 msg.setBody(chatMsg); //Send the chat                                                 conn.sendPacket(msg);                                                 isSend = true;                                                 date = new  Date(); //Write to the log fine logger.info(sdf.format(date) + “,Send,” + senderId + “,” + reciverId + “,” + chatMsg + “,OK”);                                                 Thread.sleep(sleeptime);                                 }                                                                                                               }catch(Exception e){                                 date = new  Date(); //In case of an error in sending, raise an exception in the log logger.error(sdf.format(date) + “,Send,” + senderId + “,” + reciverId + “,” + chat + “,” + e.getMessage());                                 isSend = false;                 }finally{ //Most importantly, close the session                                 removeConnection(conn);                 } return isSend;                         }

   This is how the jar was implemented in JMeter:
   
   

5. After running the above tests over 450 threads (chat sessions) for a long period of time we were able to simulate the load that was desired. A similar method for receiving chat was written and implemented as the “Receiver Group” (disabled in the above screenshot). The same jmx file was started on a different machine with Sender Group disabled and Receiver Group enabled which logged all the received messages. Once the run was completed, we compared both the sender and receiver log files (csv format) for missed and incorrect messages.
6. Next was setting up Reliability tests. These tests would prove that messages are being successfully sent and received through the actual mobile app itself. Since we had already automated a lot of functional regression tests using calabash, we decided to extend the same framework to carry out reliability tests. We simply created two separate scripts to be executed on two separate devices, again, one being sender and the other being a receiver. The sender will send unique messages with time stamp, while the receiver will capture the chats sent with the receiver’s timestamp. We ran this test for around 60 mins. and collated the results. We were able to verify whether the following issues existed:
   1. Are there any messages lost during transit?
   2. What is the average delay in receiving messages?
   3. Are the messages being received in correct sequence?

   While the performance tests confirm the load and stress levels of the server at a macro level, the above reliability tests were able to verify the load testing metrics at a micro level.

7. We then thought about productizing this so that anyone else with a similar requirement will be able to directly plug and play this code without having to “reinvent the wheel”. We achieved this by parameterizing all the variable inputs and packaging it with JMeter with a short readme file. We sent this package to the customer who could easily implement the package and start testing almost immediately. The customer now did not have to worry about “how” to test this, but they could now concentrate on setting the necessary parameters as per their business requirements and see how the server performs under that load.It was a complete turn-key implementation and now the customer is using it for the staging server and also plan to run this on their production test runs all by themselves.