Types of Testing

Contents

Introduction

This section is meant to serve as an overview of different types of testing that happen in and around WooCommerce core and extensions. Where applicable, there are also links to supplemental resources or additional information about different testing types.

Smoke Testing

Smoke testing is primarily used as part of the release cycle to catch any potential critical issues before a release happens and is designed around the critical workflows of an application. For example, in WooCommerce, there are defined critical flows that are tested as part of the release process. While smoke testing covers the major functions or critical flows, it does not test them in depth. Smoke testing can be either manual or automated (typically through End-to-End Testing). In doing so, it helps to uncover integration problems and other major issues early, while providing confidence that the changes in the application have not adversely affected the major areas covered by the smoke testing.

End-to-end Testing

End-to-end (often shortened to e2e) testing focuses around testing a workflow from start to finish. For example, in WooCommerce, we have defined critical flows for both the merchant and shopper, and have built end-to-end tests around these flows. In many cases, an end-to-end test begins as a “user interaction”, which can cover both UI workflows and API workflows. Typically, these should be written from the “outside coming in”, in the same way a user directly interacting with the system would. Hence, any workflows that start with a user interaction, such as a button click, or a direct API call from the user, would be a good candidate for an end-to-end test.

WooCommerce end-to-end tests can be run by following these instructions. The collection of end-to-end tests contained in the WooCommerce repository provides a good example of how WooCommerce’s tests were designed to cover the corresponding critical flows.

Unit Testing

A unit test, in essence, instantiates a small piece (a unit) of the application and validates the behavior independently from the rest of the application. A unit test verifies that all possible outputs (also known as boundaries) yield the expected outputs and potential side effects of that individual unit. Typical units in an application can include:

  • functions
  • properties
  • modules
  • components
  • objects

Unit tests help find bugs early and can act as a canary for any potentially breaking changes. When considering good candidates for unit tests, typically these will be units of code repeatedly used in many areas, risky/complicated logic blocks, and code in areas that could be frequently changed. Unit tests should be easy to read, easy to write, reliable, and fast.

Unit tests in WooCommerce, for example, can be run using these instructions.

API Testing

API testing involves writing tests around an application programming interface (API)–typically either SOAP or REST. For example, WooCommerce provides a RESTful API, and tests would be written that interact with it. API testing can be used to verify any of the following:

  • make sure that any new changes aren’t breaking existing API contracts
  • the API functions as expected
  • the API is reliable
  • the API is performant
  • the API is secure

API testing can be done at the unit level (verifying a single endpoint works and handles correct and incorrect inputs), at the integration level (between internal and third party services), or as end-to-end testing covering critical workflows.

Performance Testing

Performance testing covers many aspects of an application’s performance including responsiveness, availability, reliability, and scalability.

For WooCommerce and its extensions we are likely most interested in front end performance and back end response times.

For front end performance there are many metrics we can look at and it can be overwhelming. Core Web Vitals aims to help with this and gives us the most meaningful metrics to focus on.

For back end response times we are usually most interested in load testing. Load testing is a sub-type of performance test that includes many concurrent users or requests per second running against the site. This is done to see whether it can handle the load and scale without compromising functionality or performance.

For most performance testing we usually gather our metrics in two main ways

  • Synthetic testing – in controlled lab-style environments where we have defined test setups in known conditions. This is more applicable to WooCommerce and extensions where we don’t have one single type of production environment.
  • Real user monitoring (RUM) – in real people’s browsers, using various devices, connected to whatever network they’re using. This is more applicable to testing the performance of a specific store.

Additionally, this testing can provide insight into areas for improvement for application performance. For example, it could surface slow database queries that could be investigated for optimization.

Manual Testing

Manual testing is useful for testing flows that are difficult to automate, such as those that may require integration with a third-party service. Additionally, manual testing is also a great option for performing exploratory (ad hoc) testing, where the tester goes through and explores the interface to uncover issues or bugs. Manual testing is suitable for tests that may only need to be run once or twice, whereas automation should be considered for tests that need to be run repeatedly. Manual testing is also ideal when testing the overall user-friendliness of an application.

Visual Regression Testing

Visual regression testing is used to test for any unintentional changes to the visuals in an application and to provide insight into what the user is seeing and interacting with. Typically, visual regression testing software will take screenshots of the UI and present content comparisons between those images, which can help surface:

  • Visual bugs — how the interface is presented
  • Browser compatibility — if the UI is presented differently on different browsers, which may impact the user experience
  • History of how the interface has changed

←

Back: Overview

→

Next: Testing Tools