Contents
Introduction
Welcome to our guide on testing WooCommerce extensions and maintaining quality code!
Software testing can be hot topic — some swear by it, while others vehemently reject it as a waste of time. Many never really got into it, even though they’re curious. There really are some serious benefits to be had that are well worth their price — once we’ve learned to understand how the technicalities of automated testing work and how to make certain strategic trade-offs in our test suites.
To give us some orientation, we’ll start off by discussing the value that automated testing has been shown to provide. We’ll then give an overview of the differnent types of testing — for example, unit testing, end-to-end testing, performance testing, to name a few.
After this somewhat theoretical start, the next section will make sure we get the practicalities right, as far as especially WooCommerce extensions are concerned. We’ll show how to set up a local testing environment and then walk through writing tests for an actual extension.
Our guide finishes with a discussion of best practices we and others have discovered for writing and running automated tests.
Let’s get started!
The Value of Automated Testing
Automated testing is an accepted best practice in software development, but its usefulness is often taken as self-evident — making it harder for individual developers to really understand and experience the value. Let’s discuss the benefits that can be gained as found out by research.
Let’s start by looking at what developers are actually doing. To this end, Beller et al. report on a study with 2,443 developers who agreed to have their programming behavior automatically monitored for 2.5 years. The data showed that half of the developers don’t write tests at all, and that during most programming sessions no tests are run.
This might be closely related to the perceived effort of writing tests. Developers who wrote tests estimated that they spent half of their time on testing-related efforts — when in fact they spent only a quarter of their time.
So if you believe that testing sounds great in theory but would be too much effort in practice — think again. It might simply come down to perception.
Now let’s look at the gains to be had.
Rafi et al. conducted a systematic review of academic literature regarding the benfits and limitations of automated testing. The benefits they found in literature were mostly from experiments and industrial case studies — evidence that is considered stronger than experience reports, the other kind of artifact they found. Here’s a sample of their findings:
- Product quality can improve, that is: fewer defects make it into production.
- Automated testing takes less time than purely manual testing. This time can then be spent on other things, such as building features or reducing technical debt.
- Even though some automated tests can be flaky — 12% in Beller et al.’s study –, they’re still more reliable than manual testing, which is very dependent on the experience, skill, and even mood of manual testers. (A flaky test is a test that fails only sometimes, e.g., because of a timing issue — these can be notoriously hard to debug.)
- Developers may feel more confident in the quality, reliability, and stability of the system they’re working on, ideally leading to increased motivation and momentum.
The limitations they found mirror a few of the pitfalls we would also like to warn against:
- Of course, automated tests are software, too. As such, they also carry a maintenance burden. It’s easy to overlook this cost.
- Designing and implementing a test strategy requires skill and experience, both of which take time to acquire. For example, a highly repeatable and stable test of an important flow that runs very fast is far superior to a flaky test of an exotic feature — developers and test engineers need to understand and make this kind of trade-off. In WooCommerce, we are keeping a list of critical flows that helps us prioritize which test to write next.
- Automated testing should not be expected to replace all manual testing: the latter still has its uses.
Overall, we feel very strongly that the benefits of automated testing outweigh any costs or issues. We’d especially like to stress the psychological effect: even if the costs of creating and maintaining your tests doesn’t immediately translate into fewer defects, in practice development can still accelerate significantly, as experiments become cheaper and less risky. Developer morale can also benefit from the knowledge that the worst failures will be caught by automated tests.
When dancing on a tightrope, it’s much easier to learn, experiment, and have fun when you know there’s a safety net at the bottom.
This is further supported by Yuan et al. who report on about 200 confirmed failures and their root causes. The study is focused on distributed systems so it doesn’t translate directly to building WooCommerce extensions, but since a WordPress installation backed by a database and with several themes and plugins installed does indeed constitute a system with relatively autonomous components, we believe these findings should at least be considered.
Most notably, the authors find that for the failures they considered, “almost all catastrophic failures (92%) are the result of incorrect handling of non-fatal errors explicitly signaled in software.” And they add: “in 58% of the catastrophic failures, the underlying faults could easily have been detected through simple testing of error handling code.”
If you want to avoid catastrophic failures — and who doesn’t? — it makes sense to focus at least a part of your test suite on error handling situations.
So while automated testing isn’t always the best solution, there are many situations in which developing some testing experience and then creating test suites strategically makes sense — economically, but also from a standpoint of developer happiness.
The next section will give an overview of the difference types of testing.
←
Back: Table of Contents
→
Next: Types of Testing