There are three important aspects to becoming an expert software tester

a. Software Testing Techniques: Understanding the fundamental techniques of software testing and becoming good at these techniques. Basics of software testing can be learnt through certifications like ISTQB (International Software Testing Qualifications Board) or CSTE (Certified Software Tester). And implementing these techniques by executing various types of software testing on the job.
Testers need to review code to understand the workflow of software and identify defects even before the entire system is built. This is called white box testing. Testers also perform black box testing which is working with functional software to identify defects. Apart from these, testers also check non-functional aspects like performance, security, usability, etc.

b. Software Testing Tools: Learning to use different software testing tools for various purposes like Defect Tracking, Test Management, Test Automation, Performance Testing, Security Testing, etc. There are vendor supported tools like HP Quick Test Professional, HP Load Runner, IBM Rational Robot, Ranorex, as well as Open Source tools like Selenium, jMeter, Bugzilla, etc. A tester should be able to use multiple tools across various areas since technology is changing fast but the fundamentals of these varied tools and their features remain the same.
Testers need to be able to write code to implement these tools. Software testing requires a deep technical frame of mind to identify and provide solutions to software defects.

c. Domain: Focusing on and learning a domain. Domains can be vertical domains like Banking, Insurance, Manufacturing, or horizontal domains that are used across multiple industries like Customer Relationship Management, Supply Chain Management, etc.
For example, Understanding of banking processes is necessary to test Banking software. Similarly there are testers who bring not only technical knowledge of tools and technology but also domain knowledge to the table.

When it comes to qualifications, software testers can come from two distinct streams.

a. Testers who focus on software testing tools and techniques with a Computer Science or engineering background.
b. Testers who have made the transition from a domain, e.g. a banker who has moved to testing and brings a deep understanding of the banking domain.

From a qualifications perspective, Cigniti normally hires people with an Engineering background (preferably Computer Science or MCA) or an MBA with functional or domain expertise.

The key skills that we look for in testers at Cigniti are
a. Basic Programming Skills: Testers are required to review programs or use a programing language to use a testing tool and this is a basic prerequisite.
b. Diligence: Testing is all about diligence. Testers need to be methodical, diligent and focused while at work.
c. Passion: Testers have to be passionate about quality. Some parts of the work are repetitive and unless it is backed with passion and ability to see the big picture, the job can become monotonous.
d. Innovation: It is important for testers to figure out how to test better and faster or testing will become a never-ending job. Testers have limited amount of time before the software release to identify as many defects as possible and this requires creativity and innovation.
e. Superior Communication Skills – Testing is all about communication; from understanding requirements, designing and documenting test case, identifying and communicating defects to clients and
f. Teamwork – Testers have to walk a tightrope. They have to get along with the development team while pointing out the defects in the software in a neutral and dispassionate manner without hurting the feelings of the developers.

Recommendations for becoming a software tester

a. Get a basic qualification in testing like ISTQB Foundation Level.
b. Get familiar with at least one defect tracking tool, configuration management tool and test automation tool. Learn to program in any one of these tool. Do not think testing does not require you to program.
c. Improve your communication skills. Testing is as much about communication as it is about technology and domain.
d. You can practice by contributing to open source projects. At Cigniti, we value experience of working in open source projects since it helps us verify your contributions to software quality even as you are studying.

Challenges on the job?

a. Understanding requirements in new domains – Testers might be needed to move to new domains and they will have a steep learning curve when dealing with applications in a new domain.
b. Adapting to new tools and technologies – New tools and technologies always mean another learning curve even as a tester is grappling with his day-to-day job.
c. Dealing with temperamental developers – Since testers have the tough job of calling developer’s “creations” ugly, they have to learn to handle the backlash when dealing with temperamental developers.
d. Time pressures – Developers get delayed and cut into the time allowed for testing. This puts pressure on testers since they are in the last phase of software development life cycle. During the last few weeks of the release, it is not uncommon for testers to put in extra hours and weekends trying to meet deadlines.

Software Testers today are paid as well as software developers. At Cigniti, testers are our Stars and some of them are better paid than developers in other companies. Some of the niche areas in software testing like performance testers, security testers, test automation specialists, are paid better than the average software programmers.

As for the future, I would like to state that there is more to software testing as a “job” which is a short-term description of what a software tester does. Software testing as a career focuses on long-term goals of improving the satisfaction and security of software users and developers.

Testers start as Test Engineers but graduate to leading teams within a period of about 5-8 years. Some of them choose to focus on becoming experts in testing by becoming Test Experts and Test Consultants in specific technology areas like Performance Testing, Test Automation, Mobile Testing or domains like Banking, Insurance, Supply Chain Management, etc. Others choose to become Test Managers leading testing projects and move to a more general management path leading large delivery teams in testing or development.

With the increased focus on independent testing, enterprises are spending more money in areas like Cloud Based Testing, Performance Testing, Mobile Application Testing, Compatibility Testing, Security Testing and testers who have niche skills in these areas will be in demand in the future. This will also require testers who have good programming skills. Cigniti also hires developers who have a passion for testing and deploys them in areas that require a higher level of programming skills.

Testers are different from other people. The first thing you will notice about a tester is that they are not afraid to question. In most open houses of IT companies it is very likely that the toughest questions are posed by testers.

Testers have more patience and are willing to listen. Testers also tend to have superior social skills because they have to get along with temperamental developers, project managers, users, etc.

Testers are always trying harder. To rise and shine in an industry which gives more importance to developers, testers are always trying harder. At Cigniti, testers are our stars and therefore get their rightful due.

According to NASSCOM, Software Testing is growing faster than the overall IT industry and the global software testing market is expected to grow to $50 Billion from the current $30 Billion by 2020. Professional testers in India alone have doubled to 63,000 testers in 2010 from 31,700 testers in 2006 and the growth is expected to continue till 2020.

Software testing as a profession has come of age and is a recognized career with long-term growth and career prospects.

Software is becoming ubiquitous and quality and security of software is paramount in today’s world. Software defects have the capability to seriously impact our daily lives. Instances like the Blackberry outage, Sony PSN downtime or a banking security breach can affect the common man.

Cigniti’s software testers are helping build a better world. Because software runs everything; Good software. Defects and bad software cause headaches, pain and suffering. And testers are the warriors who are fighting the good battle, every day, against defects.

This model involves the on-demand test execution of well-defined suites of test material, generally on outsourced basis. The execution can be performed either at the client site or remotely from the outsourced providers test lab / facilities.

Analytics / BI (Business Intelligence) Testing

Identifying issues related to data inconsistency, data quality, data security, failures in the extract, transform and load (ETL) process, performance related issues, accuracy of business flows and fitness for use from an end user perspective.

Developers Converting to Testers

Testers need to be strong in both technical skills as well as industry and business insights. It has been observed that it is comparatively easy for developers to learn testing skills than vice versa.

Test Driven Development (TDD)

TDD is a software development technique that ensures source code is thoroughly unit-tested as compared to traditional testing methodologies, where unit testing is recommended but not enforced.

It combines test-first development and re-factoring. It is gaining popularity as it allows for incremental software development – where bugs are detected and fixed as soon as the code is written, rather than at the end of iteration or a milestone.

Virtualization Testing

30-50 per cent of the total testing time goes in test environment set-up, getting people access to the environment, and loading it with the right bits from development and it is hard to recreate the same environment for developers to investigate and fix bugs.

Virtualization is beneficial for:

o Accelerating set-up/tear down and restoration of complex virtual environments to a clean state, improving machine
utilization
o Eliminating repro bugs by allowing developers to recreate complex environments easily
o Improve quality by automating virtual machine provisioning, building deployment, and building verification testing
in an integrated manner (details later)

Continuous Integration (CI)

CI is a trend that is rapidly being adopted in testing, where the team members integrate their work with the rest of the development team on a frequent basis by committing all changes to a central versioning system.

Beyond maintaining a common code repository, other characteristics of a CI environment include build automation, auto-deployment of the build into a production-like environment, and ensuring a self-test mechanism such that at the very least, a minimal set of tests are run to confirm that the code behaves as expected.

Testing on Cloud

To minimize test environment expenditures and gain better control of their IT resources, companies are beginning to take advantage of a delivery model known as ‘cloud computing’. In cloud computing, applications and information are provisioned on-demand, as a shared resource.

In case of private cloud, servers and applications required for setting up the test lab can be shared among different teams in the organization.

Tools Driven Development Testing

IDE-integrated tools have made the self-testing practice acceptable to developers, and the unit-testing and coverage analysis process automated for them. These tools also make it easy to analyze performance and compare it with a baseline by extending the unit test infrastructure.

Crowd Testing

Crowd testing is a new and emerging trend in which, rather than relying on a dedicated team of testers (in-house or out sourced), companies rely on virtual test teams (created on demand) to get complete test coverage and reduce the time to market for their applications.

Agile Testing

Agile testing brings the development and testing processes closer together for a more holistic view.

Software testers need information to effectively do their jobs, and now days; testers are being involved in the communication loop from the beginning.

Hyderabad, September 29,2011

Chakkilam Infotech, (BSE: Chakkilam) the independent software testing services company from Hyderabad, today announced the acquisition of Cigniti Incorporated. Cigniti, based out of Irving, Texas, USA, offers consulting and testing services mainly to clients in the US and had declared revenues of US $18 million last year.

With more than 600 resources globally and a revenue run rate of US $22 Million for the combined entity, Chakkilam Infotech has emerged as the largest India based Independent Testing Services company. The acquisition will enable Chakkilam Infotechto expand its footprint in the US market and setup Test Centers of Excellence for Ciginiti’s enterprise clients.Cigniti complements Chakkilam Infotech’s expertise in areas like Test Consulting, Test Management, Test Automation, Performance Testing, etc. which reduces the overall costs of testing and provides unique solutions that cannot be solved using standard tools.

Speaking about the acquisition, Sriram Rajaram, EVP, Operations, Chakkilam Infotech said, “We welcome a great team. This is a strategic acquisition that strengthens Chakkilam Infotech’s position as the India’s leading provider of software testing solutions, including multiple open source tools to address high-growth market opportunities.The Cigniti acquisition will help us expand our presence in the US and help us offer more of our high-end test consulting services to clients in the US. This acquisition makes us India’s largest independent testing services company in terms of revenue.

C. V. Subramanyam, MD, Chakkilam Infotech, said “We are very excited about Cigniti’s team joining us and the opportunity it presents to expand into the US. By combining the passion, creativity, and technology of two leading-edge companies, we will continue to drive innovations that will help organizations everywhere.”

Sudhakar P, CEO of Cigniti Inc. said “Software companies all over the world are looking to engage with testing labs offering the whole gamut of services. Joining forces with Chakkilam Infotech will enable us to leverage each other’s strengths and to better meet the needs of our industry partners and fulfill our vision to help companies improve the quality of software being delivered worldwide. We are aiming to be a global leader in the independent testing services space in the next few years.”

According to Gartner and Nasscomforecasts,independent software testing is a niche that is growing faster than the overall IT services market. The recent 2011 NASSCOM report on Software Testing forecasts independent testing services in India to grow to $13-15 Billion by 2020 at a CAGR of more than 20% and expand India’s current 32% share of the overall global outsourced testing services.

Tools and automation will eliminate about 25% of the efforts associated with testing and with the focus on using open source tools, Chakkilam Infotechhopes to set new milestones and emerge as one of the most respected Independent Testing Services companies in the world.

About Chakkilam Infotech

Chakkilam Infotech Limited is a 13 year old, BSE listed Independent Software Testing Services company based in Hyderabad. Chakkilam Infotech offers a combination of test consulting, test outsourcing and offshore testing services to software product companies and enterprise IT departments in areas like performance testing, test automation, mobile application testing, security testing and compatibility testing. Chakkilam Infotech serves clients across different domains like technology, financial services, retail, travel, healthcare &telecommunication. Chakkilam’s focused approach to independent testing offers superior cost savings without sacrificing quality & also offers teams of experienced test engineers, domain experts, and custom-built tools and frameworks to meet industry-specific testing requirements.

About Cigniti Inc.

Cigniti Inc. is a consulting and software testing services company based out of Irving, Texas, USA. Over the last seven years Cigniti has worked with companies in US and India to deliver quality software through a combination of onsite consulting and offshore delivery. Cigniti was featured in the Inc. 500 in 2009 as one of the fastest growing companies in the US. Cigniti specializes in Software Testing, Quality Assurance, Software Development, and is an Oracle and Microsoft Gold certified Partner.

Following are the key KPIs that can be used for making an analysis –

1. Web Page Loading Time
2. Network Time
3. HTTP Requests
4. Request Size

Web Page Loading KPIs

Time to First Impression

• This is the time from when the URL is entered into the browser until the user has the first visual    indication of the page that gets loaded.  It depends on the initial HTML document when the browser can start drawing content. There are different Best Practices available that talk about different strategies. Google for example downloads a minimalist page to provide a fast first visual rendering. It then delays loads more content after ‘onLoad’ or even later when the user starts interacting with the page.

• The recommended value for this KPI is like this – great if < 1s, acceptable if < 2.5s and slow if > 2.5s

DOM Ready Time

• The ‘DomReady’ event is fired when the DOM content has loaded and the program can access all HTML Elements without waiting for all those images to load also, which just means finding out when the document’s DOM tree has fully initialized, versus the entire document and every object in it.  A classic example is the insertion/ deletion of nodes via the DOM, which can in fact be done sooner rather than later, when the DOM has loaded versus the document itself (i.e. firing of ‘window.onload’ event).

• Other resources, such as images, may not have been downloaded when this event occurs. This corresponds to the DOM Content Loaded in Mozilla Firefox and IE 9.

 Page Load

• This is the time until the browser triggers the onLoad event which happens when the initial document and all referenced objects are fully downloaded.
• JavaScript onLoad handlers use this event to manipulate the current initial state of the page. This event is one of the options explained earlier to download additional or delay load content.
• The recommended value for this KPI is like this – great if < 2s, acceptable if < 4s and slow if > 4s
• The significance of an ‘onload’ event is that all resources associated with web pages are downloaded and it is in ready / working state for starting the application.

Fully Loaded

• This is the time until all onLoad JavaScript handlers have finished their execution and all dynamically or delay loaded content triggered by those handlers has been retrieved. It is sometimes a bit hard to identify the exact time when the page is fully loaded especially when JavaScript handlers use reoccurring timeouts that constantly modify the page, e.g.: to implement a ticker

• The recommended value for this KPI is like this - great if < 2s, acceptable if < 5s and slow if > 5s

• It marks the completion of all HTTP or HTTPS requests made by the page

Network Time KPIs

KPIs listed in this section measure the time taken by various activities during processing an HTTP request by Browser.

Following is a sample timeline for a fictitious http request.  It contains all activities that can be associated with HTTP requests.  All activities listed in the picture below are explained in detail later. It needs to be noted that not all activities are present in every HTTP request.  For example, DNS Lookup Connect might not be required if it is already resolved / connected.  Similarly, “Cache Read” for 304 response code would not be present if content is modified and served from web / application server.  The timeline width associated with each activity is for sample purpose only.

Blocked Time

• The “Blocked Time” includes any pre-processing time (such as cache lookup) and the time spent waiting for a network connection to become available.
• Usually Browsers limit the number of concurrent network connections per host name and will queue up requests if the limit has been reached.
• IE7 and Firefox 2 create a maximum of two concurrent network connections per host name.
• Often the block time is the most significant factor in the time to download images embedded in the website.

Cache Read Time

• Cache Read Time is the time taken to read the content from the browser during (Cache) or 304 responses (i.e. Not Modified).
• Cache can be read in following three situations:
• The browser sent an ‘If-Modified-Since’ header and received a 304 Not Modified response.  The content would then be read from the cache because the 304 response indicates that it is still valid.  The Content-Length header in the response will always be zero or absent because the HTTP 1.1 spec defines the 304 status code as never returning content.
• The browser decided not to send an HTTP GET request because there is an entry in the browser cache that either has a valid expiration date or has been recently checked.
• The browser is in ‘Work Offline’ mode.

• DNS Lookup is the time required to resolve a host name (e.g. www.google.com) into a numeric IP address (e.g. 216.239.59.99).
• One DNS Lookup happens for every domain that hosts resources for the current web site. If you move between multiple pages the browser does not require another DNS lookup for a domain that has been resolved on the previous page. It is interesting to look at the total DNS time to identify problems with DNS Lookup Times that can be caused by DNS configuration problems.

Connect Time

• Depending on the browser and the number of resources that are served by a domain the browser establishes one or multiple connections to each domain that hosts resources for the page. Connect Time is the time it takes to establish the TCP/IP connection to the web server. Connections usually stay open unless the Web Server directs the browser to close the connection (Connection HTTP Header). When using secure communication via SSL, the Connect Time also includes the time of the SSL handshake. High Connect Time can therefore have the following reasons: slow network connection to the web server, usage of SSL and not allowing the browser to keep the connection open.
• An excessively high number of TCP connects may indicate that HTTP persistent connections, also known as keep-alive connections, have not been correctly configured on the server. These long-lived HTTP sessions allow multiple requests to be sent over the same TCP connection, and in some cases have been shown to result in an almost 50% speed up in latency times for HTML documents with lots of images.

Send Time

• Send Time is the time required to send the HTTP request message to the server and will depend on the amount of data that is sent to the server. For example, long Send times will result from uploading files using an HTTP POST

Wait Time / Server Time

• Wait is the idle time spent waiting for a response message from the server. This value includes delays introduced due to network latency and the time required to process the request on the web server.
• High Server Time means that the Web/Application Server required a long time to process the request. This is particularly relevant with requests that trigger application logic to be executed on the application server where higher Server Times can be expected – especially under heavy load periods. Monitoring Server Time is important to identify bottle necks, performance and scalability problems with the application server. It is usually easier to scale static content delivery by adding more web servers with load-balancers or by using a Content Delivery Network. It is not that easy to scale a dynamic application in the same way. Keeping an eye on this metric is important. 

 Receive Time

• Receive Time is the time taken to read the response message from the server. This value will be depending on the size of the content returned, network bandwidth and whether HTTP compression was used.

Time to First Byte (TTFB)

• It is the duration from the initial network request being initiated by the browser to the first byte being received from the server. It includes TCP connection time, the time to send the request and the time taken to get the first byte of the response message.

Network Time

• Network is the total duration of all network related operations for an HTTP request.

Requests Elapsed Time

• It is time when http request has been invoked at the final request. It can be calculated as “Blocked Time” + “Network”.  It might be the case that sending of first request takes time and that’s why ‘Blocked Time’ becomes important.

HTTP Requests KPIs

# Network Roundtrips

• This is the total number of network requests that get downloaded with the website. The ultimate goal is to keep this number as low as possible in order to reduce roundtrips. Monitoring this KPI gives an early indication on newly introduced content that can negatively impact page performance.
• Great sites require fewer than 40 requests, acceptable are up to 100 requests. Sites with more than 100 HTTP Roundtrips are considered bad.
• Some entries do not cause a network round trip because either an error occurred (e.g. ERROR_INTERNET_NAME_NOT_RESOLVED) or because no network request was required (e.g. entries resulting in a (Cache) response

# XHR (AJAX) Requests

• The total number of XmlHttpRequests (XHR) sent via JavaScript to retrieve data asynchronously from the server. Monitor this KPI to identify sudden changes in dynamic content retrieval via XHR. Some JavaScript frameworks provide update mechanisms with the server-side and use XHR for these purposes. Depending on the configuration it can end up with too many XHR requests that not only impact client side performance but also cause additional load on the application server.

Total Number of HTTP 300s/400s/500s

• This is the total number of requests to the server that responded with an HTTP Status Code of 300 (Redirect), 400 (Authorization Problem) or 500 (Server Error). These are requests that should be avoided as they have a negative impact on the page load time. The root cause of these problems is often server-side related implementation, configuration or deployment issues.

Number of Domains

• The number of domains that host the web sites resources is important as it affects DNS, Connect and Wait Time. Additional domains that are utilized to download resources will have a direct reduction in the wait time because the browser ultimately uses more physical connections. This can have an opposite effect when more DNS lookups are needed and more time is spent to establish the physical connections.

Number of Single Resource Domains

• Single Resource Domains should be avoided as you pay a high price for performing the DNS and Connect to download a single resource. It is sometimes not avoidable when downloading content from external content providers (such as ad-services). When having the deployment under your own control you want to make sure not to have single resource domains.

# Resources with No or Past Date Caching

• Number of network roundtrips increases because of it.

# Resources Having Short Expires Headers

• Number of network roundtrips might increase because of it.

Hit Count

• The number of times that the browser has read content from the cache entry.

Size Related KPIs

Downloaded Data

• This is the total amount of data that has been received from web server(s) (i.e. response headers and content). In the case of HTTPS connections this includes the overhead of the SSL encryption process.
• This is the total size of all resources that make up your page. It is important to keep track of the total payload size. The larger web sites become the longer it takes to download. Changes to the page – such as adding images or new java script libraries – can have a significant impact on download time.

Uploaded Data

• The total amount of data that has been sent to web server(s) (i.e. request headers and POST data). In the case of HTTPS connections this includes the overhead of the SSL encryption process.

Total size of Image / CSS / JS

• Besides keeping track of the total page size it is important to look into the sizes of the individual content types such as Images, Style Sheets and JavaScript files. With this it is easier to spot the main contributors of page size.
• The AJAX Edition makes the assumption that most Images, CSS and JS files served from the same domain can be merged so that we do not end up having more than 1 CSS, 6 Images and 2 JS files from the same domain.

Average HTTPS Overhead

• This is the average overhead in bytes of each HTTPS request. If this value is high (more than a few hundred bytes), it may indicate that the server is incorrectly configured and is causing an excessive number of HTTPS renegotiations.

Compression Saving

• This is the reduced total number of bytes and percentage which received content was compressed by HTTP compression.(i.e. “gzip” or “deflate”).

References:

1.       http://community.dynatrace.com/community/display/PUB/Best+Practices+on+Web+Site+Performance+Optimization

2.       https://community.dynatrace.com/community/display/PUB/Best+Practices+on+Network+Requests+and+Roundtrips

3.       https://community.dynatrace.com/community/display/PUB/Best+Practices+on+Browser+Caching#rankingpanel

Frequent releases of applications these days are not a fashion but have become a necessity for businesses to survive.  Bringing quality products out in the market quickly makes software testing also challenging.  In addition to performing functional testing in short span of time, organizations have to consider non functional testing also like performance, security, compatibility, localization etc.  Because of the high cost involved in terms of infrastructure, skill set, duration etc. in non functional testing, outsourcing it to other organizations makes business sense most of the time, if done correctly.

Performance testing is conducted to mitigate the risk of availability, reliability, scalability, responsiveness, stability etc of the system.  Like any other testing project, there are many activities involved in any performance testing project. Few of the important activities involved in any performance testing projects are identification of important business scenarios, identification of correct business scenarios mix, identification of correct workload,  identification of right tools / strategy for load generation, setting up the test environment, designing the scripts for emulating business scenarios, preparing and populating the right amount of data, identification of proper performance counters / metrics to collect, designing the report template(s) for different stakeholders as per their need and execution of multiple performance runs as per the project requirement.

In performance testing there are many types of tests that can be conducted on any application / system. The type of test or run depends upon the performance requirements.  In this article, I have listed down the importance of various performance test types in terms of technical and business perspective, which can be considered during multiple stages of performance testing projects.  The definitions of run types might differ from other sources, but the idea is not to define but to logically divide or categorize the performance runs.  Saying this, I have tried my best to adhere to the definition as close as possible from other sources.  The categories are defined in terms of different performance run cycles in any performance testing project.  For example in a project, load run might be performed first and Soak / Endurance testing run (with same work load) later if required.

Following is the summary of ten different performance types that can be referred to and selected for performance testing as per the business requirements.  The details of these performance test types are described in their corresponding sections below.

Terminology

Single User

‘Single user’ run is performed usually at the start of any performance run cycles or during the design time of the scripts.  All application’s response time details should be calculated when it is accessed with only a single user.  If the response time doesn’t match with the performance requirements then it is recommended to get it fixed before proceeding further for subsequent performance runs.

Technical Perspective

• Script verification during design time for single iteration
• Script verification during design for multiple iterations.  For example, looking for correlations, caching etc.
• Identification of performance issues in the application when accessed by single user only.  For example, in case of web application, various performance issues can be identified using utilities like YSlow, Fiddler, and HttpAnalyzer etc.
• The metrics calculated during ‘Single Run’ can be used as “Entry Criteria” for any performance testing project.

Business Perspective

• Base lining the application end-2-end response time in terms of single user

Contention

‘Contention’ run is also performed during performance testing but it is mainly run during script design time rather than during the actual run cycles.  The intention of this run is to find out any issues in the scripts related to synchronization.  It is listed here as it is also an important type of run that needs to be considered during performance testing project lifecycle.

Technical Perspective

• Script automation verification during design time for concurrency. For example, identifying issues related to sessions etc.
• Script automation verification during design time for multiple iterations when accessed concurrently.  For example, identifying issues related to caching etc.
• Preparation for parameterized test data.

Business Perspective

• None, contention test is performed mainly during script designing phase.

Light Load

‘Light Load’ test is nothing but a smoke run in performance testing project.  Usually performance testing run can last for many hours and finding that something has been missed at the end (for example – scheduling of performance counters collection) would be waste of many man-hours effort, it becomes really important to have a quick smoke run , fill the report with required data before proceeding further.  The SUT on which ‘Light Load’ is conducted must contain all software and hardware components that will be used subsequently for later runs.

Number of concurrent users in the ‘Light Load’ can be calculated as number of concurrent users during non-peak hours or if the number during non-peak hours is negligible then 10% of the normal load in the application can be considered as ‘Light Load’, as shown in Fig-1 below.

If the performance of the application needs to be gauged in terms of volume (e.g. file upload size, database size etc.) rather than number of concurrent users then ‘Light Load’ test is run with 10% of the normal volume rather than with very high volume.  For example, if normal loads need to be applied with database populated with one million rows initially then ‘Light Load’ can be performed on one lakh rows to start with and remaining nine lakh rows can be generated subsequently.

Technical Perspective

• Smoke run for finding issues related to test lab setup or any other missing details that might be required the actual run.
• Freeze on the performance requirement, metric collection and reporting structure.
• Finding out resource utilization level on all servers under SUT during non-peak hours.

Business Perspective

• Identification of throughput and end-to-end response time during non-peak hours.

Load

Normally during performance testing, load test is one of the runs that is usually conducted  from a business perspective.  The metrics collected from this run can be used later for benchmarking / base lining perspective.

During this run type, workload, load volume and load patterns on application should be simulated as close as possible to anticipated load on production during normal / peak hours.  It becomes very important to identify correct workload distributions with volume information for this run type.  The workload distributions that need to be considered are –

a. Scenarios frequencies

b. Bandwidth (LAN, WAN, Dialup, Broadband etc.)

c. Files upload sizes / file types / database sizes and other activities going on databases

d. Maximum number of concurrent users in action

The other points that need to be considered are the duration of run and ramp-up / ramp-down strategy.  If the ramp up / down is done very fast (example 250 users / min) then it might destabilize the system and that result might not be communicated / acceptable to the stakeholders.  But at the same time if ramping up / down expected to be fast like during spike testing then it needs to be simulated accordingly.

Conducting load run for shorter time (say few minutes) might not provide the correct results.  It is recommended to have performance run for at least half and hour with max concurrent users.

Following are few common examples of load patterns –

a. Ramp up and stay at max users for few hours (Fig – 2)

b. Single hump, ramp up, stay at max users for few hours and then ramp down (Fig – 3)

c. Multiple humps during ramp up / ramp down (slowly) (Fig – 4)

d. Spikes, multiple ramp up / down within short interval of time (Fig – 5)

Performance run with single hump is sufficient most of the time.  Ramping down is required to make sure that system releases resources accordingly when load is decreased.

Application stability, reliability, robustness etc. can be assessed with same workload and volume as applied in the Load run but it needs to be run for elongated period of time.  This type of run has been kept under different category (Soak / Endurance testing) intentionally.

 Technical Perspective

• Determining the application’s end-2-end response time, throughput, requests handling / min, error percentage and resource utilization at expected workload during normal / peak hours.
• Performing application tuning on regular basis taking load testing data as input.
• Identifying bottlenecks in the system in terms of resource utilization, throughput or response time.
• Establishing a baseline for future testing
• Determining compliances with performance goals and requirements.
• Comparing different system configurations to determine which works best for both the application and the business.
• Determining the application’s desired performance characteristics before and after changes to the software.
• Evaluate the adequacy of a load balancer.
• Detect concurrency issues.
• Detect functionality errors under load.
• Assess the adequacy of network components under desired load.

Business Perspective

• Assessing release readiness by taking informed decisions from performance related data collected.
• Improving the corporate reputation by improving end-2-end response time.
• Evaluating the adequacy of current capacity.
• Validating the performance SLA before rolling the application on production.
• Verifying that the application exhibits the desired performance characteristics within budgeted resource utilization constraints.

Stress / Volume

Stress run is conducted to make sure that application can sustain more load than anticipated on production presently.  The load can be considered in terms of concurrent users or volume sizes or both.  But it is always better to conduct stress test for concurrent users separately and in terms of volume separately.  Some times, stress testing in terms of volume is also referred as ‘Volume’ testing.

If details regarding numbers (number of concurrent users, volume size) related to stress testing is not known initially then it is recommended to stress the system with 10 percentages more load as shown in Fig – 6.

The stress testing is mainly done to find application’s capability of handling more loads on certain day / event.  For example, handling of extra web traffic during Christmas week.

When stress testing is conducted in terms of volume then number of concurrent users are kept at normal load (as shown in Fig – 7) whereas only volume like file sizes, database sizes etc. are increased.

Technical Perspective

• Determining if response time can be degraded by over-stressing the system.
• Determining if data can be corrupted by over-stressing the system.
• Ensuring application functionality is intact when overstressed.
• Ensuring securities vulnerabilities are not opened by stressful conditions.

Business Perspective

• Determining the availability and reliability of the application at load more than expected load.

Resilience

Resilience run is conducted to make sure that system is capable of coming back to initial state (from stressful state to load level state) when it is stressed for short duration (Fig – 8).  For example, if an online store runs a discount for particular products for short time, say one hour during a day.

Technical Perspective

• Finding it out if resources utilization, throughput and response time is coming back to previous state after application has been stressed for some time and loads come down to normal level.

Business Perspective

• To identify resilience of the application.

Failure

Failure run is conducted to find out the capability of the application in terms of load.  The load (concurrent users or volume) is increased till the application crashes as shown in Fig – 9. It is always recommended to increase the load by 10 percentages.  Multiple runs during this testing can be used for capacity planning.  Failure run can also be used to find out how many more days current hardware configurations can be supported.

Technical Perspective

• Identification of concurrent users / volume sizes that can bring the system down.
• Collection of metrics for capacity planning purpose.
• Helps to determine how much load the hardware can handle before resource utilization limits are exceeded.
• Provides an estimate of how far beyond the target load an application can go before causing failures and errors in  addition to slowness.
• Helps to determine what kinds of failures are most vulnerable to plan for.
• Provides response time trend
• Determining if data can be corrupted by over-stressing the system.
• Allow you to establish application-monitoring triggers to warn of impending failures.

Business Perspective

• To identify number of days the business has for upgrading the infrastructure / tuning the application.
• Determining the capacity of the application’s infrastructure
• Determining the future resources required to acceptable application performance.

 Recovery

Recovery run is done to make sure that the application is able to heal itself when load is decreased from failure point to stress point and then to normal load as shown in Fig – 10.

This run is in similar line as Resilience run but the difference is to first destabilize the system and then find out if it can be recovered quickly or not.

Technical Perspective

• To find out if the application has the capability of self healing in terms of throughput, response time and resource     utilization when load is decreased from failure point to normal expected load.

Business Perspective

• To find out if application can be made available quickly in case there is failure because of un-expected surge of traffic one day.

Spike

Spike run is performed on those cases where business expect load in burst of very small time and then comes down suddenly (See Fig – 11).  For example – viewing real time replays of video streams (games) while there is a goal / wicket / six.

Spike run is different than load run with ‘spikes pattern’.  In this run, spikes observed in the application are not normal rather it is extra load on the system during some specific system.

Technical Perspective

• Determining application’s capability of handling resources properly when stressed in spurts rather than linearly and slowly.
• Determining memory leak if there are any.
• Determining the issue of thrashing (Disk I/O) if there are any because of spikes.

Business Perspective

• Ensure application remain available even when there are sudden surge of load in a very short span of time.

Soak / Endurance / Reliability / Availability / Stability

Soak run is performed usually at last stage of performance runs cycles to find out if the application is capable of handling expected load without any deterioration of response time / throughput when run for a longer time.  The longer run duration can be associated with any performance run types as discussed above.  The duration and run types are decided as per the requirements of the project.

As per Fig – 12, the soak run duration is 10 hours and load pattern follows ‘Load Test Run’ type.

Technical Perspective

• Finding out memory leaks in the application if there are any.
• Finding out the robustness of the system in terms of hardware as well as third party soft wares when stressed for a long duration.
• Making sure resource utilization level is always consistent and under expected level while application is running for long duration.
• To verify if sessions are being handled properly for web based application or client – server application, when multiple concurrent users are accessing the application for longer duration.
• Finding out if file / disk storage capacity is sufficient
• Application that run during non-peak time period (e.g. anti-virus) do not degrade the performance of the system

Business Perspective

• Finding out if application is available 24*7.
• Finding out consistencies in end-2-end response time and throughput when application is up and running for longer duration.
• Determining the acceptability of stability.

Conclusion

One performance run is usually completed in 3-4 days time including the analysis and reporting. This however would be excluding workload analysis, script designing, test environment setup etc. which usually requires multiple runs (as discussed above) to validate the applications performance requirements / goals.  The types of run are dependent upon the performance requirements.

===================================================================================

References

1. Performance Testing Guidance for Web Applications By J.D. Meier, Carlos Farre, Prashant Bansode, Scott Barber, Dennis Rea
2. http://www.perftestplus.com/resources.htm
3. The Art of Application Performance Testing By Ian Molyneaux