Penetration Testing in Production Without Breaking It

You can penetration test a production system without breaking it if you agree the blast radius before the scope, take techniques that write at volume or destroy data off the table, build proofs against data you created yourself, test in agreed windows and clean up as you go. The textbook advice is to test in staging, but staging usually has different data, stubbed integrations and a WAF nobody switched on, so testing it tells you about staging.

So a lot of real work happens against production, and the question is not whether that is ideal but how it is done without becoming the incident you were hired to prevent.

Agree the blast radius before the scope

Scope says what you may touch. For production you also need to agree what happens when something goes wrong, and that conversation is easier before the engagement than during it. It belongs in the rules of engagement, signed before the first request.

• A named person, reachable, for the duration. Not a shared mailbox. Somebody who can answer at four in the afternoon and can authorise a rollback.
• What "stop" looks like. A phone number and an agreement that either side can call it, with no argument about whether the tester was in the right.
• What is off limits entirely. Payment flows that move real money, anything that emails or texts real customers, bulk operations on real records.
• Whether the operations team knows. A test that doubles as a detection exercise is a legitimate choice; a test that becomes an unannounced incident because nobody told the on-call engineer is not.

What comes off the table

Some techniques are fine in a lab and unacceptable against a live system, and being clear about that in the report matters as much as doing it.

• Anything that writes at volume. Automated fuzzing against endpoints that create records leaves a client with thousands of rows and an audit trail nobody can read.
• Denial of service, including by accident. Rate limits exist. A tool with default threading against a small instance is a load test you did not agree to run.
• Destructive proof. You can demonstrate you could delete something without deleting it. Prove read access on a record you created yourself.
• Credential brute force at speed. Locking out real users is a service impact, and it teaches you nothing you could not learn from the lockout policy itself.

Time of day is a control

Testing a retailer on a Saturday afternoon is a choice, and rarely a good one. Agree a window, and be honest about the trade: overnight testing is safer for the business and worse for you, because the people who can answer questions are asleep and a system under no other load behaves differently from one under real traffic.

The middle answer most firms land on is business hours with the client briefed, quieter periods for anything with a load profile, and nothing at all during a release.

If something does break

It happens to careful testers too, usually through something nobody could have predicted: a legacy integration that falls over on a malformed but harmless request, a queue that backs up behind a slow endpoint. What matters is the next ten minutes.

1. Stop. Not just the request that did it, but anything else running against the same system.
2. Call the named contact. Say exactly what was sent, from which address, at what time, with the request to hand. Their engineers need facts, not reassurance.
3. Do not fix it yourself. Even if you can see how. Their change process exists for exactly this moment.
4. Preserve your own record. Your logs and timestamps are evidence for both sides, and they are what lets everyone agree later on what actually happened.
5. Put it in the report. Plainly, as part of what happened that week. It handles the same way as finding something critical mid-engagement: call first, then write.

Leaving it as you found it

Production testing leaves traces, and the clean-up is part of the engagement rather than an afterthought.

• Accounts you created, removed or listed for the client to remove.
• Uploaded files deleted, including the ones that went somewhere unexpected.
• Records you created flagged so nobody reconciles them as real next quarter.
• Anything you changed on a live configuration, changed back, with the client watching.

Write the list as you go. Reconstructing it from notes a week later is how a test account survives for two years and shows up in somebody else’s report.

The evidence problem nobody warns you about

Testing live systems produces evidence containing real customer data, and that data is now in your screenshots, your notes and eventually your report. A redacted screenshot in the report does not help if the unredacted original sits in your working folder for the next four years.

Decide before you start what you will do when a proof of concept needs a real record. The usual answer is that you demonstrate the class of access on your own data and describe, without capturing, that the same access extends to real records. That is weaker evidence and it is the right trade: a report proving you could read a customer file by including a customer file is a data incident with a covering letter.

Where you genuinely need the real thing, agree it specifically, capture the minimum, redact at the point of capture rather than at the point of writing, and tell the client what you hold so they can factor it into their own retention answer. Handling evidence and client credentials safely covers the storage side.

What the report has to say

Two sentences that a lot of reports leave out. What you did not test because it was live, and what that means for the conclusion. If payment flows were out of scope because they move real money, the report cannot imply the payment path was assessed. A reader who assumes coverage you did not have is the failure mode that outlives the engagement, which is why proving what you tested matters most on live systems.