Environment Stability Policy
Why we have this policy
We understand that product development and deployment cycles are such that a test environment sometimes cannot be frozen for the entire period of testing and reporting.
The ideal place for us to test is usually a staging or UAT environment which sits between development and production. It matches production as closely as possible in all respects, but changes far less often than development. Not every organisation has one, though.
When there isn’t one, testing has to happen on the development environment, because we typically won’t test against production (see our Production Testing Policy).
This policy is for those situations where only a single such development environment is available. We’ve published it so you know where we stand, what the trade-offs might be, and what options you have at testing time.
Our position
Our commercials, and the level of assurance we can give you, assume that the environment we’re testing in stays stable whilst we test it. Ideally it remains unchanged for the whole testing and reporting period.
However, this is a strong preference rather than a hard rule. Unlike our approach to production testing, there is room to be flexible here, and we can work through it with you case by case.
Why it matters
- Accuracy: If the environment changes underneath us, our results can be wrong in both directions. We may report issues that no longer exist (false positives), or miss issues that do (false negatives). For example, a vulnerability introduced halfway through testing, in an area we’ve already covered, is one we might not find as a result.
- Connected applications: Many applications share users, data and settings, and a weakness in one part can be chained with a weakness in another to create a more serious problem. Finding these combinations means testing the whole system as it stands, at the same time. Testing each part separately, or at different times, can obscure them.
- Review at the end: A number of different team members check every finding and proof of concept at the end of testing. That quality assurance process only works if the environment is the same as it was at the start. Otherwise, each part of the pen test would need its own testing and review cycle.
- Scheduling: We set aside a dedicated lead tester and a block of time for your engagement. Pausing and restarting part way through might affect the other customers we have booked in around your testing in our schedule.
What we ask of you
Before testing begins, we ask you to confirm that:
- The environment is stable and won’t be changed during testing, unless we’ve talked it through first;
- It is fully backed up and can be adequately restored if our testing takes services down or pollutes data; and
- Someone from your team is available if we need to be in touch.
Your choice
Whether to freeze the environment is a trade-off between risk and cost, and the decision is ultimately yours. In practice, there are three main options:
Freeze the environment for the testing period: This gives the best assurance for the agreed scope.
Set up a separate, representative test environment: This is a popular choice for larger applications where freezing the only development environment is too restrictive. Your team’s development can continue as normal, and the test environment can be taken down once we’ve delivered our initial or final report. The main factors here are feasibility, cost, and effort of setting that up.
Carry on making changes while we test: This can work, but the risks ultimately sit with you. We may also need extra time to reach the same level of assurance, which means we may need to revisit commercials.
Case by case
Not every code change matters equally. A tweak to the look and feel of the front end is very different from a change to how users and admins authenticate. If you can tell us what changes are planned and when, we can help you work out how much it might affect testing, and agree the approach together.
If things change mid-test
If something changes while we’re testing, please do let us know straight away. We’ll assess what it might affect, and agree the best way forward with you.
Questions?
If you have any questions about this policy, or would like to discuss the right testing approach for your environment, please get in touch at [email protected].
