Performance Testing, Part 2: Functionality Tests Vs. Performance Tests

This is part two in a four-part series that will walk you through everything you need to think about when performance testing a new site.

Performance tests often get confused with functionality tests and while this may seem like just a semantic difference, it’s not. These are two different types of tests and while both need to be done, it’s important to understand the goal of each.

I would consider anything that is not specifically a test of functionality to be a performance test. Functional tests would fall into the following categories:

  • Unit tests. JUnit or HTTPUnit would be examples. These test specific low-level functionality at the individual component level. These tests tend to be used to validate specific low-level functionality, rather than test the integrated environment.
  • Smoke and Regression tests. These are used to integration test the system and make sure that key functionality does not regress or degrade between builds.
  • Front-end functional tests. Either automated with a tool like Selenium, or performed manually, these test page functionality, page composition, and flow.

These tests, by definition, are single-threaded tests, focused usually on one user at a time. The object is to determine if the system functions the way it designed.

Performance testing, on the other hand, pre-supposes that the underlying functionality exists and is stable. The object is to stress the system. These tests fall into the following categories:

  • Load Testing. This set of tests places the system under simulated load to determine behavior at expected load. During the test, virtual sessions are added gradually to a predetermined level.
  • Stress Testing. This is a load test to failure. It is used to determine the upper limit of concurrency, and test strategies for dealing with unexpected load.
  • Soak Testing. This tests the system at a specific load for a long period of time. The test is meant to determine if there are defects in the system that are only evident over time, like memory leaks.

There is some overlap between functional tests and load testing, in that the automated test has to simulate actual user behavior. The virtual user or the functional testing script must be able to perform the same actions as a person using a web browser. With functional tests we are looking for deviations in expected results. In performance testing, we are not capturing specific defects, but rather looking at the system under load.