Testing in Production Without Breaking It

The advice is to test in staging. The reality is that staging has different data, different integrations, half the third-party services stubbed and a WAF nobody switched on. Testing it tells you about staging.

So a lot of real work happens against production, and the question is not whether that is ideal but how it is done without becoming the incident you were hired to prevent.

Agree the blast radius before the scope

Scope says what you may touch. For production you also need to agree what happens when something goes wrong, and that conversation is easier before the engagement than during it.

• A named person, reachable, for the duration. Not a shared mailbox. Somebody who can answer at four in the afternoon and can authorise a rollback.
• What "stop" looks like. A phone number and an agreement that either side can call it, with no argument about whether the tester was in the right.
• What is off limits entirely. Payment flows that move real money, anything that emails or texts real customers, bulk operations on real records.
• Whether the operations team knows. A test that doubles as a detection exercise is a legitimate choice; a test that becomes an unannounced incident because nobody told the on-call engineer is not.

What comes off the table

Some techniques are fine in a lab and unacceptable against a live system, and being clear about that in the report matters as much as doing it.

• Anything that writes at volume. Automated fuzzing against endpoints that create records leaves a client with thousands of rows and an audit trail nobody can read.
• Denial of service, including by accident. Rate limits exist. A tool with default threading against a small instance is a load test you did not agree to run.
• Destructive proof. You can demonstrate you could delete something without deleting it. Prove read access on a record you created yourself.
• Credential brute force at speed. Locking out real users is a service impact, and it teaches you nothing you could not learn from the lockout policy itself.

Time of day is a control

Testing a retailer on a Saturday afternoon is a choice, and rarely a good one. Agree a window, and be honest about the trade: overnight testing is safer for the business and worse for you, because the people who can answer questions are asleep and a system under no other load behaves differently from one under real traffic.

The middle answer most firms land on is business hours with the client briefed, quieter periods for anything with a load profile, and nothing at all during a release.

Leaving it as you found it

Production testing leaves traces, and the clean-up is part of the engagement rather than an afterthought.

• Accounts you created, removed or listed for the client to remove.
• Uploaded files deleted, including the ones that went somewhere unexpected.
• Records you created flagged so nobody reconciles them as real next quarter.
• Anything you changed on a live configuration, changed back, with the client watching.

Write the list as you go. Reconstructing it from notes a week later is how a test account survives for two years and shows up in somebody else’s report.

The evidence problem nobody warns you about

Testing live systems produces evidence containing real customer data, and that data is now in your screenshots, your notes and eventually your report. A redacted screenshot in the report does not help if the unredacted original sits in your working folder for the next four years.

Decide before you start what you will do when a proof of concept needs a real record. The usual answer is that you demonstrate the class of access on your own data and describe, without capturing, that the same access extends to real records. That is weaker evidence and it is the right trade: a report proving you could read a customer file by including a customer file is a data incident with a covering letter.

Where you genuinely need the real thing, agree it specifically, capture the minimum, redact at the point of capture rather than at the point of writing, and tell the client what you hold so they can factor it into their own retention answer. The version of this that goes wrong is the one where nobody discusses it, three testers each keep their own folder, and the client finds out during their next audit.

What the report has to say

Two sentences that a lot of reports leave out. What you did not test because it was live, and what that means for the conclusion. If payment flows were out of scope because they move real money, the report cannot imply the payment path was assessed. A reader who assumes coverage you did not have is the failure mode that outlives the engagement.