A few months ago, at 09:15 UTC we sent a newsletter to around 800,000 people. At 09:18 a colleague pinged to say WooCommerce.com was down. Our monitoring alert arrived at 09:22, four minutes after a human had already noticed (alert latency is a separate topic).
What made it very confusing: nothing shipped that morning, and there was no attack. It turned out that we did this to ourselves with an email campaign 🤯
Total requests at the edge more than doubled, from roughly half a million to just over a million, and fell back almost as fast. By the time that first ping arrived, the site was already recovering. While the infrastructure caught up, some users got 429 Too Many Requests responses. Downtime was around three to four minutes, and even at the peak the vast majority of requests still came back as normal 200s. The failures were mostly those 429 rate-limits, with a thin slice of 5xx. So: a short, partial outage. Still enough to count as a real incident.
The wrong suspects
My first instinct was the usual checklist: a deployment near the incident, an attack-shaped traffic pattern, etc. As mentioned earlier: nothing stood out.
The logs had two odd fingerprints though. The traffic came from datacenter IP ranges and VPNs (AI was fast to find this pattern, it would take me much longer to get this part), not the residential ISPs you see behind real shopper traffic. There was also a burst of JavaScript ChunkLoadErrors: connections that grabbed the HTML and then disappeared, leaving the follow-up JS downloads hanging.
The ChunkLoadErrors were the giveaway: something was pulling each page and cutting the connection before the JavaScript loaded, thousands of times over. That’s what an email security scanner does when it checks a link.
What actually hit us
This newsletter went to store owners. That audience skews heavily toward corporate mailboxes: Office 365, Mimecast, and the like. Those providers run aggressive security scanners that fetch every link in an email the moment it arrives, checking for malware long before a human opens anything.
Delivering 800,000 emails at once meant the scanners arrived all at once too, most of them within seconds of each other. The bulk of that spike was machines. A consumer audience on Gmail would never have generated this. I can’t prove every one of those connections was a scanner, but it’s the only story that fits everything we saw.
Similar spike a week earlier
Sends of this size are routine for us, so this wasn’t some unusually big blast that finally tipped us over. A week earlier we’d sent an even larger campaign that produced a near-identical spike from the same kind of bots, and the site absorbed it without anyone noticing.
So the problem wasn’t really the size of the send, it was the timing. Why that one passed and this one didn’t? Honestly, we can only guess: slightly different bot concurrency, different cache luck on the day.
Why autoscaling lost the race
Our hosting (WordPress VIP) autoscaled correctly, which is a big part of why the outage stayed as short as it did. It would have comfortably carried the extra traffic for as long as it kept coming.
New capacity takes a couple of minutes to come online, and the bot traffic hit in seconds. Autoscaling is built to absorb a gradual ramp, and this arrived as a step function instead: a sudden jump with no slope for the scaling to follow. By the time the new capacity was ready, the spike had already done its damage and mostly passed. The fix had to happen before the traffic ever reached the servers, which meant changing how the email itself went out.
What we changed
For the next campaign, we decided to stagger the send instead of delivering the whole list in one burst (check whether your email platform can do this). That next one happened to be Black Friday, a week or so later. It went out in batches over a few hours, and the spike never came.
We didn’t add capacity, and we’re not planning to. Autoscaling already handles natural growth fine, including seasonal peaks. What it can’t do is react in seconds to a self-inflicted spike. A big marketing send is really an infrastructure event, worth coordinating with whoever runs the servers before the largest ones go out.
The takeaway
If you send bulk email to an audience with corporate addresses, your “send” button is a load-testing tool whether you like it or not. The recipients don’t even have to open anything, their mail security fetches every link the moment delivery starts.
We found that out in a couple of minutes one morning. If your list is big enough, or your site is hosted on smaller servers, you might not get even that much warning.
None of this is an argument against emailing your users though, there’s big value in that. Just don’t let the bots surprise you 😉
Leave a Reply