Using Mock objects in load testing

Introduction

One of the main problems with performance testing services is the overall scope and in particular applying scrutiny to an individual software component.

The first issue with scope is that if multiple components are being tested, as in a full systems performance test, it is difficult to isolate the system resources used by a given component (process). By way of example consider a Web services architecture that has Common Gateway Interfaces CGI (or asp and jsp etc) running on the same server.

When load is put on the front end (Web browser) the Web Services are called via the CGI and the resource utilization is an aggregate of both the CGI and the other Web services. This issue does go away if the components are separated out onto different servers but in most cases some shared server resource, by components, will exist.

The second issue, concerning scope, with performance testing a system, is that the complete system has to be built before a complete test can be executed through the front end (i.e. Web Service, CGI or Web Browser).

In the case of Web services both the client (usually Web browser, CGI, asp, jsp etc) and the backend data base need to be available to perform the end-to-end performance test.

Both of the issues of scope for performance testing can be addressed with mock services (stubs) and a test harness to drive the load from.

Mock Web Services.

Mock Web services are place holders for any SOA request that has to be developed. Mock services are similar to Mock objects for unit testing . There are many papers on Mock Objects, in the software testing services directory, that go into further detail.

The advantages of mock services for load testing are:-

The called service does not have to be fully complete.
The calling component (i.e. Web service) can be isolated in terms of measuring performance.
It is also easier to control the behavior of the returned values, using a Mock service, as these (canned) returned values can be defined for the given purpose.

It is important to note that this test is not a system test and other functional tests will need to be performed to validate the component but for the purposes of performance testing the mock object will be able to ‘stand in’ for the real thing.

An open source Mock service tool.

SoapUI has a web service mocking feature that allows for quick and easy building of mock services. This tool set also includes building the various responses that the anticipated calling program expects from the complete delivered web service.

The harness or driver.

Given we can isolate the server component from the Web Services; we now look to isolate that component from the client. This isolation is needed in any event in order to recreate large volumes of simulated service calls to the component under test.

If the component under test is itself a Web service then a SOAP load testing tool can be used as the driving harness.

If the component is a CGI (or other web server side component) a HTTP load driver can be used to put load on the component under test.

In either case a suitable open source load testing tool can be found.

Measuring the component under test.

When isolated there are a number of useful monitoring tools that can be used to measure the performance of the given component.

Although CPU is a useful measure to determine the overall performance of a middleware component it is essential to establish that a server side component is not leaking memory or not returning other unused resources (such as threads) back to the pool when appropriate. The first type of load test that should be performed on a middleware component, and this can be done early in the SDLC with Mock services, is stability testing.

With stability testing the memory, thread pool and CPU are monitored over an extended period, about six hours, of consistent load to determine if performance degrades over time.

Optimizing the process of load testing for AJAX based web applications

This blog post examines classical engineering Test and Measurement process improvement strategies that have been successfully applied to manufacturing, in order to define an appropriate model for the performance verification of today’s Rich Internet Applications (RIA).

Although software applications continue to evolve in terms of scale and complexity the basic models of verification, derived from their manufacturing counterparts, still hold today as they did over 70 years ago.

Critical to all software verification is the establishment of a Test and Measurement function that is active through all phases of the software production process (be that Agile or Water Fall).

Establishing the most effective Test and Measurement activities is only the initial step, these activities need to be subjected to continuous improvement as enabling technologies (as well as the System Under Test SUT) continue to evolve.

Applying a Test and Measurement continuous process improvement framework, derived from manufacturing, to software production has several challenges due to the intangible nature of software. To begin with it is not obvious ‘what’ is to be measured. In manufacturing the diameter, length, weight or some other physical characteristic of the component can be measured and verified for compliance during every stage of the production process.

In software there is no physical representation of the product so a scheme of software quality characteristics has been devised in order to Test and Measure the finished product (or components during production) so that adherence to specification can be verified.

Although there have been a number of software quality characteristic models one of the most popular is the FURPS model (HP). The FURPS model breaks out software characteristics into:-

Functionality:

Feature set, Capabilities, Generality, Security

Usability:

Human factors, Aesthetics, Consistency, Documentation

Reliability:

Frequency/severity of failure, Recoverability, Predictability, Accuracy, Mean time to failure

Performance:

Speed, Efficiency, Resource consumption, Throughput, Response time

Supportability:

Testability, Extensibility, Adaptability, Maintainability, Compatibility, Configurability, Serviceability, Install-ability, Localizability, Portability

Having a perspective (such as the FURPS model) of ‘what’ is desired in the software product or component is only the first step in establishing an effective Test and Measurement process that can be subjected to continuous improvement.

The next step is to define the measurements (and tolerances, or ranges) for each of these characteristics. By way of example Functionality can be measured by simply writing a series of functional tests that verify the software specification has been correctly implemented.

In fact functional testing services is by far the most common Test and Measurement activity in software production today. Having decided the desired measurements for each of the software characteristics the next step is selecting the most useful tools and techniques to perform the Test and Measurement of the desired characteristic (at various stages of production). Finally the Test and Measurement process itself is subjected to continuous improvement, as is done in the best manufacturing quality management systems.

A practical example of a process improvement initiative for Test and Measurement of software performance.

Having examined a basic manufacturing analogy of a Test and Measurement process improvement strategy that could be applied to software, what follows is a summary of an implementation of this strategy (and decision process) for the verification of performance of Ajax based web applications.

The example scenario represents a simplified version of the steps that would be required to analyze, design and implement a suitable improvement for performance verification Test and Measurement activities.

The steps followed, in the example, are for:-

• Define the current problem
• Analyze and Measure the current process
• Analyze the current causes of the issue
• Design a process improvement
• Implement and measure the costbenefit of the process improvement
• Subject the process to continuous improvement

The scenario presented, by way of example, is for CompanyCRM ( a fictitious maker of CRM products) that wishes to address performance issues with their Ajax based CRM product.

Define the current problem

CompanyCRM has been seeing numerous performance issues (response times of > 8 seconds) just after product launch and has decided to embark upon a process improvement initiative to address this issue.

Analyze and Measure the current process

CompanyCRM needs to establish a benchmark, in terms of their current Test and Measurement performance verification process. It is important, as with any process improvement initiative, that the current situation is objectively measured (benchmarked) in order to be able to measure the effectiveness of any counter measures (changes) to the process.

The failure of a given Test and Measurement performance verification point can be identified by the presence of subsequent performance issues (defects) that the given performance verification point should have detected. By way of example consider a Test and Measurement performance verification point after a given jsp (or other GCI, php, asp etc.) has been developed.

The value of such a verification point, early in the delivery cycle, would be to verify component stability, including the lack of memory leaks.

If an application delivered into production leaked memory then the memory leak can be traced to a given component and the Test and Measurement process of the given component could be ‘improved’ so as future memory leaks could be detected at that particular Test and Measurement point in the software delivery cycle.

Example – Analyze and Measure the effectiveness of the current process:

Following our example scenario CompanyCRM is finding performance issues (slow response times) when the product has been shipped to the production servers. The application is Ajax based and these performance issues were not detected in their current system load testing just prior to release.

CompanyCRM is using Jmeter as their load testing tool and after further investigation it has been determined that Jmeter is not generating the same workloads (HTTP traffic) that are being experienced in production. CompanyCRM’s performance engineers believe (hypothesis) that if they could generate more realistic web traffic during performance system testing then they would uncover the defects that the current process misses.

Analyze the current causes of the issue

Given the hypothesis, that it is the nature of the generated HTTP traffic that can be changed to improve the effectiveness of CompanyCRM performance Test and Measurement capability, CompanyCRM’s performance engineers begin to analyze the short comings of traffic that Jmeter (and other HTTP testing tools) generate.

CompanyCRM’s performance engineers examine HTTP requests made against the production server and compare these requests with what Jmeter has been reproducing and they see significant differences when generating traffic originating from an Ajax based Web Browser.

Ajax enabled applications, challenges for HTTP generation.

Ajax places more processing logic in the browser which in turn has moved the http traffic structure, and timing, away from simple Get resource (URL). By way of example consider the typical Ajax type ahead (auto complete) function that is popular in searches.

In the Ajax enabled CompanyCRM application the user might start to enter the customer contact’s surname and as the first letters of this surname are entered a list of qualifying customers are returned (auto complete) for the user to select. The http traffic for this Ajax search operation would look something like:-

http://mycrm.com/search?lang=en&search=j

http://mycrm.com/search?lang=en&search=jo

http://mycrm.com/search?lang=en&search=jon

http://mycrm.com/search?lang=en&search=jone

http://mycrm.com/search?lang=en&search=jones

The Jmeter script only had the final http request being called:-

http://mycrm.com/search?lang=en&search=jones

The Jmeter script was data driven so the the search term was given as a variable i.e &search=$lastname.

Having identified the issue the CompanyCRM performance engineers examined potential improvements for the performance verification process.

Design a process improvement

The CompanyCRM performance engineers knew that they could write scripts to read and parse out names one letter at a time (they are Jmeter experts) in order to recreate the desired requests. However, they decided to experiment with another approach to web traffic generation which they believe would be more cost effective (in terms of their labor) at generating the required traffic. The new approach they decided to research was utilizing the Headless browser strategy to performance testing.

The Headless browser approach to web traffic generation for load testing

One recent innovation that has enabled the load driver traffic to move closer to the ‘Real user’ experience is the Headless browser. The headless browser concept allows for a browser to be automatically executed without the user interface (GUI) portion.

The advantage of running a browser without the GUI component is that all of the underlying API (that communicates with the web server) is still available but the process executes with a much lower CPU and Memory resource requirement.

Utilizing the Headless browser approach enables multiple browsers to be executed simultaneously on a single computer (or VM). The execution of simultaneous (headless) browsers combined with the scaling capability of the public (or private) Cloud has proved useful for generating large amounts of web based traffic that is close to the real user experience.

An added bonus of using the headless browser approach, for load testing services, is that it allows for the re purposing of automated scripts that have been traditionally used for functional verification (i.e. Selenium Web driver scripts).

One of the reasons why the headless browser load driving approach has become so popular, in recent years, is the popularity of Ajax which brings with it a significant change in the format and timing of http requests as well as dynamically changing html content.

Following their own analysis and further research, which included a Proof of Concept testing experiment, CompanyCRM performance engineers implemented the change to the current performance verification process by utilizing a Headless Browser approach.

Implement and measure the costbenefit of the process improvement

Cost

The cost of Test and Measurement performance verification is basically time (labor) and materials, with materials being the verification architecture (load testing tools) and the required hardware to execute the tests.

Although this is a simple breakdown the labor costs will depend on the testing tool’s ease of use (usability). Usability, for all software, impacts the labor cost drivers of:- Training (time to become proficient with the testing technology), efficient tool utilization (how many hours does it take to produce the test cases).

Benefit

The Benefit of Test and Measurement performance verification process can be measured by the number of performance defects discovered after the product has been moved to production.

In this way the performance defects, found in production, should be documented and reviewed, in order to determine if this type of defect could have been discovered earlier in the SDLC and if needed the verification (load testing) process should be changed.

In the above example CompanyCRM performance engineers should be able to identify performance defects earlier in the SDLC given the improved software testing services process.

Subject the process to continuous improvement

That said, the verification process (both cost and benefit) should be continually monitored and scrutinized for process improvement as new tools and techniques come onto the market and the nature of the software being produced (and it’s environment) evolves.

Conclusion

Identifying the right test automation services framework to validate web based applications is an ongoing search for the approach that aligns the value (purpose) of the verification process with the lowest cost.

Within the SDLC no single load driver (load testing tool) will be appropriate for all the given performance verification points.

By identifying, measuring and documenting the appropriate load driver for the given performance verification point, an end to end Test and Measurement process can be established that is analogous to it’s manufacturing counterpart.

For driving load that is close to the Real User, for AJAX based application, there is a compelling framework that runs multiple ‘headless’ browsers.

The AJAX illustration is just one example of identifying the most appropriate framework for the given Test and Measurement performance verification task and in any event all verification processes should be subjected to continuous improvement as we all move toward the ultimate goal of zero defects (and satisfied customers).

What is Performance Testing?

What is Performance Testing?

Performance Testing services is a type of testing to ensure that the applications will behavior as expected with the workload on the customer application. Most important factor are

• Response time.
• CPU Utilization.
• Memory Utilization.
• DB Utilization.
• Capacity.
• Reliability

---

Why is Performance Testing is needed?

It is done to provide customer with the information about their application regarding response time, stability and capacity. More importantly, it is done to improve the application behavior before it goes into market.  It is important because, without it the application will suffer the issues such as:

• Running slow if multiple users access the application simultaneously.
• Inconsistencies across different operating systems and usability.
• Capacity of the application.

It will determine if the customer application is meeting the  response time, speed, capacity and stability. It will give the confidence to customer about the behavior in the market.
Customer application sent to market without performance testing will gain bad reputation and will not meet the market sales.

---

Different type of Performance Testing

Load Test

It will help to check the application's  behavior to perform under anticipated loads on application. The objective is to identify performance bottlenecks of the application under expected load.

Endurance Test

It is done to make sure the software can handle the expected load on the application over a long period of time. Because the memory leak will be observed when the application is running for longer duration. So the most important objective of the endurance test is to find the memory leak and behavior during the load for longer duration.

Stress Test

Stress testing is one on the application too find the breaking point of the application. It will determine how much load the application can handle. How will the response time and resource behavior when the extreme load is applied on the system. Important it will give the customer the need confidence how much load the system can handle with existing resources.

Spike Test

Spike testing is done to check if the application can handle, if there is any sudden spike of user on the application. It will let the customer know if the system will crash or it will handle the spike with some  degradation on the application behavior.

Volume Test

Under Volume Testing large number of data is populated in database and the overall software system's behavior is monitored. The objective is to check software application's performance under varying database volumes.

Scalability Test

Capacity/scalability testing is done to determine the software effectiveness in scaling pattern of the user load on the application. It will help the customer to plan the capacity of the application by adding the resources to your software system.

Performance Testing for Ecommerce – Is your Ecommerce Website ready this Holiday Season?

Getting ready this holiday season

The holiday season is just around the corner and it is time for retailers to get busy for the next couple of months.

One of the biggest sales events of the year Black Friday and Christmas is just days away now. As an online retailer, this is just about time to ensure your site is ready for this holiday season.

For a brick and mortar store, it is more about stocking up and ramping up staffs. As for online retailers, it is more about the site performance and making sure the site runs well throughout the season.

Why site performance is important?

Though people still like to visit the brick and mortar store, the number of transactions that happen online is on the rise. Last year, the Black Friday online sales hit a record USD 6.22 billion, which is 23.6% more than the previous year.

The amount of traffic you get will be doubled or even more in most situations, during the holiday season sale. It is important or at least in this case mandatory to test your website for performance before the holiday season begins.

Did you know? That 40% of people will abandon your site if your site takes more than 3 secs to load.

YES! webpage loading time plays a vital role in your business. Performance testing services from a leading software testing service provider will ensure your website will be ready to handle the traffic and execute multiple transactions simultaneously.

Test your website for performance

Irrespective of the exciting offers on your website, your customers can abandon you for web page loading time. Hence, performance testing is important. There are 2 important types of performance testing needs to be done.

Load testing – Increase in website traffic can be a good thing for your business as you will make a lot of money.

However, it may turn ugly, if your website crashes due to the sudden influx in traffic. This will affect your business very badly. Especially in the digital world where people take it up in social media. However, it may turn ugly, if your website crashes due to the sudden influx in traffic. This will affect your business very badly. Especially in the digital world where people take it up in social media.

A 2-second delay in loading time could cost you fortune. Hence, you have to make sure your eCommerce website has optimal loading time.

Load testing will help you optimize the performance of your website. Schedule a load test prior to the sale so that you will have ample time to fix the issues from load testing results.

Spike testing – Sudden spike in user load could crash your website.

Introducing Spike testing within your performance testing plan will ensure your eCommerce website is ready to handle the sudden rise and fall of user load. This is mandatory especially before the holiday season and during special discount days.

End-to-end Testing

Now that you have tested the performance of your website, it is time now to do complete end-to-end testing of your eCommerce website.

As a first step, prepare a checklist with the list of testing that has to be completed before the holiday season.

Here is a list of testing an eCommerce website must undergo,

Functional Testing – Testing all the functionalities just before the sale begins ensures the website is ready to make money.

A lot of new functionalities could have been added to your website just for the holiday season. For instance, integration with other new applications, features, etc. It is worthwhile to test all the functionalities of the website before the sale begins.

Security testing – Security testing is not an option anymore. It is mandatory to ensure your website is secure from security threats. While other types of testing can be skipped (I strongly recommend not to skip), serious security threats may impact your business heavily. Hence, Security testing is a must.

Performance testing – Similar to security testing, Performance testing services is mandatory for your eCommerce website, especially before the holiday sale.

Usability Testing – – Though there won’t be many changes to the features of your eCommerce website, it is advisable to perform usability testing, just to make sure everything is fine.

This will be particularly helpful if you have added new features to your website either to improve the shopping experience or for holiday sale purpose.

Mobile app testing – People are not interested to turn on their laptops or PCs anymore.

The smartphone revolution has given the power to people to access everything from their handheld device. For the same reason, many eCommerce businesses have their own android or IOS app in order to retain their loyal customers. In these situations, testing the mobile app thoroughly will be a good remuneration.

Mobile first (Bonus tip)

According to a recent survey by Boston Consulting Group, almost a third of Americans will give-up sex instead of their smartphones.

I don’t have to stress the importance of mobile-friendly websites here.

Google has made it clear on so many occasions that websites should be mobile-friendly, and it is one of the primary weightage for your website to rank in Google’s search results.

If you don’t care about Google, then you should at least be worried about your future customers. According to Statista, the number of smartphone users is expected to reach 2.87 billion by 2020

You would have probably designed your website for mobile devices during the launch of your website.

If not, it is best to make sure your website is responsive to different screens just before the sale begins.

The focus should be on giving the best experience to your shoppers this holiday season. Since webpages optimized for mobile screens boost conversions and sale.

Stock up

The digital revolution has opened the doors for a lot of small businesses. There are many external factors (which is not in your control) that directly affect your business, such as competition and economy.

You never know, this might be the best or worst holiday season yet for you.

The least you can do is to be prepared and deliver the best experience to your customers. This blog should help you prepare for this holiday season and keep your website crash-free.

View the original source: https://www.indiumsoftware.com/blog/performance-testing-for-ecommerce-websites/

So, go ahead, stock up and happy selling!

How to Do Performance Testing for Mobile Applications – The Ultimate Guide

Introduction

The usage of mobile phones has increased drastically in today’s digital world. The usage of the desktop has gone down since the rise of mobile phone usage. According to Statista, By 2020, the number of smartphone users is projected to reach 2.87 billion.

What Does this Mean to Businesses?

You have to ensure your customers are getting the best possible experience while using your application. Be it the usability or performance. Though businesses are carful while developing the application, it is best to do a complete mobile application testing before it gets into your customer’s hands.

Whether it is on a browser or an application, these days mobile users expectations are very high when it comes to user experience. Especially for mobile apps, it has become very competitive. There are plenty of apps in the apps store, you just have to be the best to earn a place in your customer’s phone.

Types of Mobile Applications

Not all applications are the same. Each one is created keeping in mind the devices and technologies in which they will be used. There are totally 3 types of applications.

Native mobile applications

Web-based mobile applications

Hybrid applications

Native Mobile Applications

The applications that run on your smartphone or tablets are called native applications. These applications run on the OS of the device. These are generally reliable and provide high performance. Native applications can also access other applications such as address book, photos etc from your smartphone.

Web-based Mobile Applications

These applications are run directly from the browser and are not installed on the device.

Hybrid Applications

Hybrid applications are a mixture of both native and web-based applications. The application runs inside a native container as native applications. It uses the mobile device browser to render applications and not the actual browser.

The Mobile App Performance Testing Approach

The mobile app performance testing approach in smartphone different from web application performance testing. The desktops and laptops used to access web apps usually don’t face problems with poor network conditions such as latency or packet loss.

However, in mobile applications all factors such as latency, packet loss, network conditions, device type (smart device or smartphone) and bandwidth matters. We should take all the above factors into consideration while performing mobile application performance testing.

The first step of mobile app performance testing services should be to understand the type of application you are running on the device. The three types of mobile applications are

Browser-based applications

Native Applications

Hybrid applications

Browser-Based Applications

Browser-based applications can be accessed directly via a mobile browser. These types of applications don’t need additional software to be installed or downloaded. These applications are built lightweight and are developed on an identical stack in all device types. Moreover the responsive design makes it easier to create a multi browser-based application in order to make the application render on different screen sizes.

The major drawback for browser-based application is it cannot be accessed without an internet connection. Also, the application’s speed is based on the internet connection.

Hence, it is important to replicate the user load of a mobile browser during performance testing. It is equally important to test the web page rendering on target devices.

Native Applications

Native applications are built using the software on a particular platform that is installed directly on the device.

These applications are more responsive and faster than browser based applications as they are built locally on the device. During performance testing of native applications, one has to consider the testing platform, as the code on each device will be different.

Hybrid Applications

A combination of both native and browser is a hybrid application. They are comprised of a Native shell which offers a fast-integrated experience. During performance testing, the user-generated load on the server-side is tested.

Mobile Application Performance Testing Strategy

Any mobile application’s performance is measured using the following 3 categories.

Device performance

Server/API performance

Network performance

Device Performance

In device performance check the following,

• Application startup time
• Memory consumption
• Battery Time while using the application
• Usage with other Applications
• Hardware and Software variation
• An application running in the background

Server/API Performance:

In Servier/API performance check the following,

• API calls generated
• Data to and from the server.
• Server downtime

Network Performance:

In Network performance check the following,

• Packet Loss
• Network Speed

Key Challenges in Performance Testing

• Simulating Connectivity like Edge, 3G, 4G or WiFi, etc
• Organizing different mobile platforms against their OS
• Mobile devices restrictions like resources consumption and battery
• The varied sizes of mobile devices
• Mobile phone usability

How to Set Up a Mobile App Performance Test Environment?

Step 1 – Understand thoroughly the mobile app that needs to be tested

Step 2 – Identify the OS on which the mobile app will be running

Step 3 – Build the test setup

Step 4 – Select the appropriate tool for testing

Mobile App Performance Testing Checklist

It is always better to have a checklist while performing mobile app testing. Here is a sample checklist that will help you get started.

• Check the RAM required for the application
• Check the response time and speed of the application under different circumstances and networks.
• Make sure the user experience is realistic under various network conditions.
• Make sure the essential results are achieved in multiple connectivity situations
• Make sure the application is not crashed
• Make sure the app is performing well in Wi-Fi and data
• Check the mobile API usage bottlenecks and uptime
• Test the mobile app to its limits

Conclusion

This is just a beginning and a high-level overview of mobile app performance testing. Understanding the mobile application is crucial for performance testing. Challenges in performance testing include different sizes of mobile devices, costing, resources availability and budgeting.

View the original source: https://www.indiumsoftware.com/blog/how-to-do-performance-testing-for-mobile-applications-the-ultimate-guide/