TL;DR
Infrastructure change evidence should establish what was approved, what existed before execution, what actually happened, and whether the resulting service is correct, secure, and recoverable. A successful task or a green deployment pipeline answers only part of that question.
Define the required evidence and acceptance criteria before the window. Capture actions and decisions as they occur, validate the service independently of the deployment tool, and retain an access-controlled record with explicit gaps, exceptions, and ownership. Treat missing evidence as uncertainty, not success.
On this page
- What Infrastructure Change Evidence Must Prove
- Before the Window: Capture Intent, State, and Recovery Readiness
- During the Window: Record Actions, Outcomes, and Decisions
- After the Window: Validate the Service, Not Just the Platform
- A Worked Example: An Ordered Firewall Policy Change
- Package the Evidence for the Next Engineer
- Protect the Evidence Without Creating Another Exposure
- Conclusion
- External References
Introduction
Consider a familiar change-window scenario. The upgrade completes, the deployment task reports success, and the change ticket closes. The next morning, an application team reports intermittent failures. The infrastructure team has screenshots of completed tasks, but no comparable service baseline, no preserved configuration export, and no clear record of when the first abnormal behavior appeared.
The immediate problem is no longer just troubleshooting. It is determining what the team actually knows.
Was the condition already present? Did the approved change cause it? Was another dependency modified during the window? Did rollback restore the original configuration, or leave a partially changed environment?
A useful infrastructure change evidence process makes those questions easier to answer without depending on the memory of whoever worked overnight. Its purpose is operational first: support safe decisions, accelerate investigation, and make the handoff defensible. Audit support is an additional benefit, not the only reason to collect evidence.
The change record should explain the transition between known states, not simply document that someone performed work.
What Infrastructure Change Evidence Must Prove
Three evidence categories belong together, but they should not be confused.
Authorization evidence establishes what was permitted. Execution evidence establishes what actions occurred and their reported outcomes. Validation evidence establishes whether the resulting environment meets the agreed acceptance criteria.
| Phase | Evidence to capture | Decision it supports |
|---|---|---|
| Before | Approved scope and artifacts, fresh configuration and service baselines, dependencies, recovery readiness, acceptance criteria | Is this environment ready for this specific change? |
| During | Actual targets, identities, task results, intermediate state, service observations, deviations, decision records | Should the team continue, hold, or recover? |
| After | Final-state differences, functional and security tests, recovery posture, observation results, exceptions, ownership | Can operations accept the resulting state? |
NIST SP 800-53 provides a useful foundation. CM-3 addresses controlled changes and documented decisions; CM-3(2) addresses testing, validation, and documentation. AU-3 describes audit-record content, while AU-9 addresses protecting audit information. The capture pattern below is a practical operating recommendation, not a prescribed NIST evidence schema or a guarantee of compliance.
The important relationship is between evidence and the decision it supports:
Recovery does not bypass validation. A rollback creates another state that must be checked before the service is accepted.
Before the Window: Capture Intent, State, and Recovery Readiness
Bind Approval to a Specific Scope and Artifact
Start with one change identifier that connects the approved request, execution runs, platform tasks, test results, and closure record. Identify the service owner, implementer, approver, validation owner, and escalation contact. For higher-impact changes, avoid making the implementer the only person judging whether the result is acceptable.
Record the actual target identifiers, not just friendly names. Include the applicable tenant, subscription, region, cluster, namespace, host, policy, or storage object. Document affected dependencies, explicit exclusions, and overlapping changes that could invalidate the baseline.
Approval should reference a particular runbook revision, configuration revision, script commit, deployment artifact, or plan. Preserve the relevant tool and provider versions and the approved parameter set, with sensitive values handled separately.
For Terraform, HashiCorp documents a saved-plan workflow in which the generated plan can be supplied to terraform apply. Retain the association between the reviewed plan and the execution that used it. A newly generated plan is a new artifact, not evidence of what an earlier run was approved to do.
When the scope or artifact changes materially, record the revised decision. Do not rewrite the original approval until it appears to authorize an action that was never reviewed.
Capture Configuration and Service Behavior Separately
A configuration baseline answers what the environment was configured to do. A service baseline answers how it was behaving. Capture both close enough to execution that they remain useful, and refresh critical checks immediately before the first write.
For a network change, relevant configuration evidence might include policy revisions, effective membership, routes, and forwarding state. For a virtualization or storage change, it might include software versions, placement, capacity, replication status, and existing degradation. Capture the dependencies needed to interpret the target, rather than exporting every object in the estate.
Service evidence should include the transactions and indicators that matter to the workload. Preserve the query or test definition, time range, resource filters, units, sample count, traffic volume, and result. A latency percentile without its observation window and workload context is difficult to compare responsibly.
Keep existing incidents, alerts, and known exceptions alongside the baseline. A pre-existing problem does not automatically prohibit the change, but proceeding should be an explicit risk decision rather than an accidental omission.
Prefer structured exports with a readable summary. Screenshots can show useful visual context, but they should supplement, not replace, the underlying results and collection details.
Define What Must Change and What Must Not
Acceptance criteria need two sides: the intended modification and the properties that must remain true.
A firewall update might require a new policy revision while preserving application access and denying an unauthorized source. A host upgrade might require a target version while preserving workload availability and sufficient remaining capacity. A storage change might require a new configuration while preserving agreed data integrity and recovery behavior.
For each critical check, define the expected result, failure threshold, required observation period, minimum useful sample, and decision owner. Use pass, fail, and unknown outcomes. A test that did not run, or a metric with no valid samples, cannot demonstrate success.
Microsoft’s safe deployment guidance emphasizes health checks between rollout stages and includes usage in the health model. Apply that principle to the window: healthy-looking indicators are insufficient when the workload is not receiving meaningful traffic.
Prove That the Recovery Path Is Usable
Record the recovery artifact identifier, creation time, coverage, dependencies, and the latest applicable restore-test result. Distinguish evidence that a backup exists from evidence that the intended recovery procedure has been exercised.
Verify that the recovery operator can reach the required management systems, obtain authorized credentials and keys, and access the recovery artifacts if the changed component becomes unavailable. A documented recovery path should not depend entirely on the infrastructure being changed.
Also identify irreversible steps and the latest safe recovery decision point. Work backward from the required restoration deadline using the expected recovery duration, verification time, and contingency allowance. That deadline may arrive before the maintenance window ends.
Restoring configuration is not necessarily equivalent to reversing data changes. Microsoft explicitly notes the complexity of rolling back stateful components. Record where rollback stops being viable and what authorized recovery or roll-forward option remains.
During the Window: Record Actions, Outcomes, and Decisions
Build the Timeline as the Work Happens
For each stage, capture the start and finish time, actual targets, initiating operator, executing identity, artifact revision, platform task or request identifiers, and reported result. Include failed attempts, retries, partial application, and manual intervention.
Preserve command output and errors where appropriate, but interpret exit status according to the tool. Terraform’s -detailed-exitcode, for example, distinguishes an error from a successful plan that contains changes. A collector that treats every nonzero exit code as a failure would misclassify that result.
Keep source event time separate from collection time and, where available, ingestion time. Normalize the working timeline to UTC while preserving original timestamps and offsets. Record known clock uncertainty instead of assuming that identical formatting means the clocks were synchronized.
An execution record should allow another engineer to distinguish “the request was accepted,” “the platform reported completion,” and “the resulting state was observed.” These are separate claims.
Know What Each Evidence Source Can Establish
Different platforms expose different slices of the change. Do not promote a narrow audit source into proof of end-to-end service health.
| Source | What it helps establish | What still needs separate evidence |
|---|---|---|
| Azure Activity Log | Resource-management operations and their recorded outcomes | Activity inside resources, application transactions, and effective service behavior |
| Kubernetes audit records | API activity recorded under the configured audit policy | Actual workload behavior and information omitted by the selected audit level |
| Terraform plan and execution records | Proposed infrastructure actions and the associated execution record | Independently observed final state and application acceptance |
Azure distinguishes its control-plane Activity Log from resource logs that describe operations within resources. Kubernetes likewise distinguishes audit records from ordinary Event API objects. Its Metadata audit level does not include request or response bodies. Collecting kubectl get events therefore should not be presented as a complete API audit trail.
Confirm source coverage before the window. A required audit policy, export destination, or diagnostic setting that was not active cannot be assumed to provide a retrospective record.
Preserve the Reason for Each Decision
A useful decision entry contains the evidence available at that moment, the observed condition, the selected action, the person or policy authorizing it, and the next checkpoint.
For example, “held the next stage because application error rate exceeded the approved limit” is more useful than “deployment paused.” Preserve the failed validation and the subsequent recovery result separately. Do not overwrite the failed run with a successful retry.
Treat evidence collection failures as operational information. Record permission errors, incomplete pagination, truncated output, unavailable sources, and delayed records. A collector should report whether its output is complete, partial, or missing for the requested scope.
When a required signal becomes unavailable, hold progression or obtain the explicitly authorized exception. An exception records accepted risk; it does not turn a failed or unknown check into a pass. Do not silently redefine the acceptance criteria to fit the evidence that remains.
After the Window: Validate the Service, Not Just the Platform
Compare Approved Intent, Starting State, and Actual State
A before-and-after difference is necessary, but it is not the whole analysis. Classify observed differences as intended changes, expected transient effects, unrelated known changes, or unexplained drift.
Do not hide unexplained differences by filtering them out of the report. Normalize volatile fields only in a derived comparison, and preserve the permitted underlying export and the transformation rules used to create that comparison.
Repeat the service checks using the same definitions and scopes. Document material differences in workload volume or request mix. Google’s SRE guidance cautions that before-and-after comparisons can be confounded by time and workload changes. Where feasible, an appropriately isolated concurrent control can strengthen the comparison, but shared dependencies can contaminate that control too. Absolute service acceptance limits still apply: being no worse than a failing control is not success.
Evidence supports causal investigation; a shared change identifier or a nearby timestamp does not prove causation.
Validate Functionality, Security, and Recovery Posture
Validate from the paths that matter to the service, not exclusively from the management network. Depending on the change, this may require fresh authentication, DNS resolution, new connections, representative application transactions, and checks from each affected access boundary.
Test what should be denied as well as what should be allowed. A reachable application does not establish that tenant isolation or administrative restrictions remain correct. Equally, a connection timeout alone does not prove policy enforcement: the destination may be unavailable for an unrelated reason.
Use approved test accounts and safe transactions. Identify side effects before testing workflows that can modify records, trigger downstream automation, or create financial activity.
Check the recovery posture appropriate to the platform. Record whether required redundancy, spare capacity, replication, protection jobs, and management access have returned to the accepted state. “Online” and “ready to tolerate the next failure” are different acceptance questions.
Where a required check cannot be completed inside the window, identify its owner, deadline, and escalation condition. Keep the outcome provisional rather than implying the missing check passed.
Separate Window Completion from Final Acceptance
The maintenance window and the observation period serve different purposes. A service may require the next business peak, scheduled batch, backup cycle, or other representative workload before the remaining acceptance criteria can be evaluated.
Define the handoff explicitly: what changed, what has passed, what remains unproven, who is observing it, and when the outstanding decision must be made. Use an outcome such as implemented, validation pending when that describes reality better than successful.
Close temporary operational changes deliberately. Restore scoped alerting and scheduled jobs, remove temporary privileges and exceptions, and reconcile declarative configuration before resuming any suspended automation. Record completion or assign a dated follow-up.
A rollback requires the same discipline. Preserve its execution record, final-state comparison, service checks, and any residual differences. “Rolled back” should not conceal a partially restored environment.
A Worked Example: An Ordered Firewall Policy Change
Consider a hypothetical change that replaces broad application access with a narrower rule in a first-match, ordered-rule firewall. The approved outcome requires application clients to retain access while an unrelated source segment loses it.
Before execution, the evidence package contains the current rule order, effective source membership, policy revision, relevant application flows, and the approved recovery configuration. The test plan includes both an allowed application transaction and a connection attempt that must be denied after the change.
During the window, the controller reports successful deployment. The application test passes. However, the prohibited source can still connect because an earlier, broader allow rule continues to match.
The deployment succeeded as an administrative operation. The security objective failed.
The team holds further progression and preserves the effective rule evaluation, connection-test result, and controller task record. It then follows an authorized decision to correct the policy or restore the prior configuration. Neither action erases the failed acceptance result.
After remediation, the team repeats both tests and correlates the denied attempt with the relevant enforcement evidence. It also confirms that the destination remains healthy and that required monitoring and management paths still work.
Without the negative test, the original change could have been closed as successful while leaving the intended restriction unenforced. The useful evidence was not another screenshot of the controller. It was the result that contradicted the desired outcome.
Package the Evidence for the Next Engineer
Organize the retained record around a manifest and three phase-specific collections: before, during, and after. Include an execution timeline, decision log, validation results, exception register, and closure summary. The ticket should identify the controlled repository rather than become a broadly accessible dumping ground for raw exports.
For each artifact, record its identity, source system, resource scope, collection method and version, timestamps, classification, collection status, and protected storage reference. For metric exports, also retain the query and observation window. For derived or redacted files, record their relationship to the source artifact.
The following is an illustrative validation record for the firewall example, not a vendor schema or an executable policy:
schema_version: 1 change_id: CHG-EXAMPLE-0042 stage: during check_id: deny-unrelated-source policy_revision: example-revision-18 observation: source_segment: unrelated-segment destination_service: payments-api protocol: TCP destination_port: 443 event_time_utc: "2026-09-09T03:20:14Z" collection_time_utc: "2026-09-09T03:20:18Z" expected: connection_denied observed: connection_allowed collection_status: complete_for_requested_check validation_result: fail evidence_files: - during/connection-test-01.json - during/effective-rule-01.json decision: action: hold_progression recorded_by: example-change-lead next_step: authorized-policy-review
Replace the example identifiers and connect this structure to the actual test runner and approval process. The referenced files must exist in the retained package. An implementation should reject a passing record that lacks required evidence, while preserving failed and incomplete records for review.
The distinction between collection_status and validation_result is intentional. Collection can succeed and demonstrate a failed control. Collection can also fail, leaving the validation result unknown.
Protect the Evidence Without Creating Another Exposure
Collect only what the decision requires, and classify artifacts before distribution. HashiCorp warns that saved Terraform plans can contain sensitive values in cleartext even when terminal output obscures them. Kubernetes request-body auditing can likewise require careful handling of sensitive content. Do not attach unrestricted raw artifacts to a widely readable ticket.
Where originals must be retained, place them in a restricted store and create clearly identified redacted derivatives for wider review. Preserve the relationship between them. Never quietly edit the retained source until it supports a cleaner account of the change.
Use access controls, encryption, version protection, and access auditing appropriate to the evidence’s sensitivity. For higher-assurance changes, separate evidence administration from the operator performing the change and avoid placing the only copy inside the affected failure domain.
A file hash can establish that bytes match a separately protected reference. It cannot, by itself, prove who created the file, whether its contents were truthful, when it was captured, or whether collection was complete. Apply stronger provenance and integrity controls where those claims matter.
Set retention according to organizational requirements and the evidence’s operational purpose. Preserve fixed results and queries rather than relying entirely on a dashboard link whose underlying data may later expire.
Keep collection proportionate. A repeatable, tightly scoped configuration change and an irreversible platform migration need different evidence depth. Use bounded collectors with appropriate permissions, timeouts, and rate limits; collecting evidence should not destabilize the service being protected.
Emergency changes still need an accountable record, but urgent containment should not wait for a perfect screenshot package. Capture the authority, targets, actions, times, and outcomes available during the response. Record missing baselines honestly and label retrospective reconstruction as retrospective, not contemporaneous evidence.
Conclusion
Infrastructure change evidence is useful when it helps an engineer decide whether to start, continue, recover, or accept a change. Its value comes from the relationship between approved intent, observed state, actual execution, and validated service behavior, not from the number of attachments in a ticket.
Start with a required capture set for each phase, explicit pass/fail/unknown outcomes, a usable recovery decision point, and a named validation owner. Automate repeatable collection, preserve deviations, and carry unresolved conditions into an accountable handoff.
A change window can end before every operational question is answered. What should never end with it is clarity about what changed, what has been proven, and who owns what remains uncertain.
1 thought on “Infrastructure Change Evidence: What to Capture Before, During, and After a Change Window”