TL;DR
An AI context-sensitivity test harness should vary presentation without silently changing the task, evidence, or policy. This tutorial builds a provider-neutral Python workflow with three policy cases, equivalent prompts, evidence-order variations, and independent grading checks. Validate the harness with a passing fixture and a deliberately defective fixture before connecting an application. Preserve failed attempts, examine results by condition, and never confuse a local demonstration with evidence of model reliability.
Introduction
In Part 1, AI Context Sensitivity: Same Evidence, Different Decisions, we examined an architecture assistant that could approve a requirement in one interaction and reject it in another. The difficult part was not noticing the disagreement. It was determining whether the cause was the evidence, the assembled context, the execution conditions, or the grader.
Now that distinction needs an implementation.
Editing a prompt until the next answer looks correct does not establish what improved. A useful evaluation preserves the disputed case, changes declared variables, and records enough evidence to explain the result.
This article builds that workflow around the same two-node power-domain requirement. The companion is a reference implementation for this series, not a vendor evaluation product or a published model benchmark. Its local demonstrations use deterministic Python fixtures, with no model or network calls.
What You Will Build
By the end of this walkthrough, you will be able to construct a controlled test matrix, distinguish incorrect decisions from invalid responses, and connect the runner to an approved read-only application through a defined adapter interface.
Anthropic’s Demystifying evals for AI agents distinguishes a task with defined success criteria from an individual trial and the graders that assess it. We use the same distinction here: one architecture case can generate several presentation conditions, and each condition can be attempted repeatedly.
Prerequisites and Scope
Use Python 3.10 or newer and the supplied context_probe.py and test_context_probe.py files from companion version 1.2. The runner and local tests use only the standard library. This release was validated on Windows with Python 3.12.14; other interpreter versions and operating systems have not been tested for this release.
Download the complete Python companion, version 1.2 (ZIP). Extract the archive and run commands from the folder containing both Python files. The archive also includes a README with setup and adapter instructions. The excerpts below explain the implementation; the download contains the complete files.
For live evaluation, you also need an approved application and an adapter that submits requests to it. That provider-specific integration is not included. Keep deployment credentials and production mutation tools out of the evaluation environment.
The scope is one advisory decision. This is not a complete architecture review, a retrieval benchmark, or an authorization system.
Define the Policy Before Varying the Prompt
The requirement is deliberately simple: two application nodes must occupy independent power failure domains.
For the synthetic fixtures, both records are authoritative. Equal domain values mean a known violation. Different values represent verified independence within this exercise. An unknown value means the requirement cannot be assessed.
| Case | Node 1 domain | Node 2 domain | Expected decision |
|---|---|---|---|
| Shared domain | A | A | hold |
| Independent domains | A | B | approve |
| Missing evidence | A | Unknown | insufficient_evidence |
Approval means only that this requirement passes. Outside the fixture, different labels do not establish physical independence without authoritative infrastructure evidence.
The runner stores the cases and expected answers together locally:
CASES = (
{"id": "shared", "domains": ("A", "A"), "expected": "hold"},
{"id": "independent", "domains": ("A", "B"), "expected": "approve"},
{
"id": "unknown",
"domains": ("A", None),
"expected": "insufficient_evidence",
},
)
Python’s None becomes JSON null when the prompt is serialized. The policy explicitly defines that value as unknown, preventing the grader from imposing an interpretation the application was never given.
Include all three outcomes. An assistant that always holds approval should fail the independent-domain case, just as an assistant that always approves should fail the other two.
When a requirement can be enforced deterministically against trusted structured data, I would keep that enforcement outside the model. This fixture tests whether an AI reviewer preserves and applies the rule; it is not an argument for replacing a straightforward policy check with an LLM.
Keep the Answer Key Outside the Application
The model needs the governing policy because applying it is the task. It does not need the expected answer or the internal case identifier.
The diagram shows that separation. Only the local grading path receives the answer key.
Do not pass the complete fixture object through the adapter for convenience. That would expose the answer key and change what the evaluation measures.
Likewise, do not allow a tool-enabled application to read the harness directory, prior result files, or grader source. Keeping the answer out of the prompt is insufficient when the application can retrieve it elsewhere.
Generate Controlled Prompt and Evidence Variants
The supplied runner defines three requests for the same outcome:
QUESTIONS = (
"Evaluate these records against the supplied power-domain policy.",
"Apply the supplied power-domain policy to these records.",
"Determine the power-domain policy outcome for these records.",
)
These are intended to be equivalent requests, not positive and negative framings of different tasks. Have the policy owner review any additional variants before adding them.
The renderer changes the question and the order of two independent evidence records. It preserves their identifiers and values. In this excerpt, POLICY, QUESTIONS, the imports, and the type definitions are supplied by the complete script.
def render(case: dict[str, Any], wording: int, reverse: bool) -> str:
records = [
{"id": "node-1", "power_domain": case["domains"][0]},
{"id": "node-2", "power_domain": case["domains"][1]},
]
if reverse:
records.reverse()
# Expected answers and internal case labels stay out of the prompt.
return json.dumps({
"policy": POLICY,
"question": QUESTIONS[wording],
"evidence": records,
}, ensure_ascii=True)
With three cases, three wordings, two evidence orders, and two repetitions, the runner executes 36 trials. Setting 20 repetitions produces 360 trials, but still only three architecture cases.
The runner shuffles trial order using a recorded scheduling seed. That seed controls which trial runs when; it does not seed model decoding or guarantee independence between remote requests.
Do not treat every sequence as interchangeable. Reversing independent records is different from reversing recovery steps or moving evidence behind an initial model-generated judgment. Those require separate workflow experiments.
Separate Decision Correctness From Response Validity
The application output contract contains exactly three fields: decision, evidence_ids, and basis.
A response can select the correct decision while citing the wrong record. It can cite both records while making an incorrect decision. It can also select the correct decision but omit a required field. Those failures should not collapse into an unexplained score.
The companion therefore reports independent diagnostics:
| Check | What it establishes | What it does not establish |
|---|---|---|
schema_ok | Required fields, types, and allowed decision value are present, with no extra fields. | The decision is correct. |
references_ok | Both required record identifiers appear exactly once. | The explanation accurately represents those records. |
decision_ok | The recognized decision matches the expected outcome. | The response satisfies every other requirement. |
The separate decision_observed field indicates whether the returned object contains a recognized decision. Missing or unrecognized decisions are reported separately from observed decisions that contradict the expected outcome.
For example, run this check from the companion directory. The decision and references are correct, but basis is deliberately missing:
from context_probe import grade
output = {
"decision": "hold",
"evidence_ids": ["node-1", "node-2"],
}
result = grade(output, expected="hold")
assert result["decision_observed"]
assert result["decision_ok"]
assert result["references_ok"]
assert not result["schema_ok"]
assert not result["passed"]
The trial still fails. The diagnostics simply identify the failure accurately rather than implying that the model chose approval.
Hua and colleagues’ Flaw or Artifact? Rethinking Prompt Sensitivity in Evaluating LLMs found that heuristic evaluation methods could account for substantial apparent prompt sensitivity in their experiments. Their results reinforce the need to inspect scoring, without establishing that every observed inconsistency is a grading artifact.
Here, exact decision labels are an explicit machine-interface requirement. The grader does not infer a freeform answer’s meaning. A response such as “do not approve” needs separate semantic review if it appears instead of the declared JSON contract.
Parse Without Silently Repairing the Answer
The companion rejects duplicate object keys and the nonstandard numeric literals NaN, Infinity, and -Infinity when decoding the adapter response. Python’s JSON documentation explains that the default decoder otherwise accepts these extensions, including retaining only the last value for a repeated key.
The adapter should apply the same declared parsing rules to model-generated JSON before wrapping it. Do not silently discard a contradictory field, repair a verdict, or retry until the answer passes. When repair is an intentional application feature, evaluate and record that feature as part of the workflow.
The sample grader only requires a nonempty basis. It does not evaluate that text for entailment or explanation faithfulness. Add a separately validated semantic check or expert review before describing the results as proof of grounded explanations.
Validate the Harness Before Connecting a Model
Start with the runner’s tests and its two local fixtures:
python -m unittest -v test_context_probe.py python context_probe.py --demo --output demo-results.jsonl python context_probe.py --demo-flawed --output flawed-results.jsonl
The baseline fixture computes the policy decision deterministically. The flawed fixture deliberately changes a shared-domain result from hold to approve when the records appear in reverse order.
The supplied validation produced these results:
| Local validation | Trials | Passed | Failed | Exit code |
|---|---|---|---|---|
| Deterministic baseline | 36 | 36 | 0 | 0 |
| Deliberately flawed fixture | 36 | 30 | 6 | 1 |
All 38 runner tests passed on Windows with Python 3.12.14. They cover malformed input, independent grading, missing metadata, configuration changes, refusal to overwrite earlier evidence, and actual child-process adapter success, timeout, nonzero exit, malformed output, and missing-executable paths.
These results validate the test machinery, not a language model. The defect was deliberately programmed; its detection is not a discovery about AI behavior.
The flawed command returns exit code 1 intentionally. The runner refuses to overwrite an existing results file, so choose new filenames for later runs. Each command writes per-trial records to JSON Lines and prints a JSON summary to the terminal.
Connect a Read-Only Application Through an Adapter
The adapter is a small program that translates the harness request into a call to your application. The runner starts it once per trial, sends a JSON request on standard input, and expects a JSON response on standard output.
The request contains a unique trial_id and the rendered prompt. The response wrapper identifies the application configuration and carries its output:
{
"model_id": "actual-deployment-revision",
"settings_id": "immutable-application-configuration-id",
"output": {
"decision": "hold",
"evidence_ids": ["node-1", "node-2"],
"basis": "Both records identify power domain A."
}
}
Replace those illustrative metadata values with real identifiers. settings_id should resolve to the instructions, decoding parameters, output schema, tool permissions, and other application dependencies used for the run.
After implementing an adapter named my_review_adapter.py, invoke it as follows:
python context_probe.py --repetitions 20 --timeout 60 --output real-results.jsonl --adapter python my_review_adapter.py
Place --adapter and its command last. Keep diagnostics out of standard output, which the runner treats as the response document.
Isolate State at the Actual Application Boundary
A new local process does not establish that a remote application has a fresh conversation or empty memory. The adapter must create the intended clean state for every independent trial. Anthropic’s evaluation guidance specifically warns that shared state can introduce correlated failures or inflate results.
For this first test, bypass live retrieval or replay a fixed evidence snapshot. If the adapter performs new searches or modifies the supplied evidence, the test is no longer isolating the presentation conditions described here.
The harness records the prompt it hands to the adapter. Additional system instructions, tool responses, and application transformations belong in a protected application trace. A hash of the submitted prompt does not prove what the model ultimately received.
The runner uses Python’s subprocess.run with a command argument list, shell=False, and a timeout. That is process orchestration, not a security sandbox. It buffers captured output, so use only reviewed adapters and enforce output-size and resource limits before applying the pattern to less controlled workloads.
Preserve Failures and Configuration Evidence
Each completed trial records the case and presentation condition, exact submitted prompt, prompt hash, expected answer, returned application output when available, reported configuration identifiers, timing, grader version, and error category.
Timeouts, invalid adapter responses, and missing metadata remain failed trials. They do not disappear from the result set. When a valid wrapper contains an output but omits metadata, the runner retains the output while still failing the trial.
Malformed transport output and exception messages are not copied into the general results file. Retain protected adapter diagnostics keyed by trial_id when investigating those failures. Real prompts and outputs may themselves be sensitive, so the result files also require appropriate access controls and retention.
The runner performs no automatic retry. Declare and log any SDK or application retry behavior rather than presenting the final successful attempt as the complete history.
Matching model_id and settings_id values establish only matching reported identifiers. They cannot expose an undisclosed provider change or make an unpinned model alias reproducible. A changed or missing configuration identifier prevents this suite from passing its configuration-consistency check.
Read Results by Condition, Not Just Aggregate Score
The flawed fixture passes 30 of 36 trials, or approximately 83.3%. That aggregate hides the more useful result: every shared-domain trial with reversed evidence is incorrectly approved.
The six failures are distributed across all three wording variants, with two failed repetitions per wording. Original-order shared-domain trials remain correct. The defect follows evidence order, not one unfortunate choice of words.
A real application requires more cautious interpretation. Compare repeated outcomes within the same condition before attributing differences across conditions to context. Inspect configuration and error records, and do not treat one conflicting pair as a reliable estimate of an effect.
The summary reports unsafe_approval_count for an approve decision on a case where approval was not allowed. In this advisory exercise, that means an invalid recommendation, not an executed deployment. In the defective local fixture, the count is six across 24 non-approval trials.
Keep protocol failures, transport errors, and unobserved decisions visible alongside that count. A low incorrect-approval rate is not reassuring when the application fails to return usable answers.
The runner supplies descriptive results, not confidence intervals or a statistical proof of invariance. More repetitions improve observation of the chosen conditions; they do not substitute for additional architectures, policies, and failure scenarios.
Use the Suite as a Regression Check, Not an Authority Grant
This reference suite has a strict exit rule: every trial must pass its checks, and complete reported configuration identifiers must match. Exit code 0 means those conditions were met. Exit code 1 indicates failed trials or configuration inconsistency; code 2 indicates an argument or output-file setup error.
That makes the runner useful as a small regression gate, but passing three synthetic cases is not permission to deploy an autonomous architecture reviewer.
Expand the suite with representative cases and known operational failures. Keep some cases outside prompt-tuning work so that repeated editing does not optimize only for the examples already visible to the development team.
For a confirmed incorrect approval of a known policy violation, I would block promotion of the affected review workflow and investigate the failed condition. Independent policy enforcement and deployment authorization should remain in place regardless of the evaluation result.
Troubleshooting the Evaluation Workflow
| Symptom | First diagnostic action |
|---|---|
| Every response fails parsing | Check for diagnostic text on standard output, malformed wrapper JSON, or undeclared parsing behavior. |
| Correct decisions still fail trials | Inspect schema, reference, and metadata diagnostics separately. |
| Later trials reflect earlier discussions | Check remote conversation reuse, persistent memory, and application caches. |
All decisions pass but the process exits with code 1 | Inspect configuration identifiers and non-decision checks. |
| Repeated runs always match | Add meaningful case and condition coverage before adding redundant repetitions. |
The reference runner executes serially and retains rows in memory as well as writing them to disk. It is suitable for a small diagnostic suite, not an unbounded evaluation service. Concurrency, quotas, distributed execution, complete trace capture, and stronger process isolation require additional engineering.
Conclusion
A useful AI context-sensitivity test harness does more than repeat a prompt. It preserves a defined task, varies declared presentation conditions, protects the answer key, and separates decision errors from interface and measurement failures.
Start by proving the harness recognizes both a correct fixture and a known defect. Then connect a read-only application, retain unsuccessful attempts, and inspect the conditions under which the recommendation changes.
The objective is not identical prose. It is a decision that remains correct when irrelevant details change and responds appropriately when material evidence changes.
The final article, AI Context Governance: From Test Results to Production Controls, moves from detecting these failures to controlling the evidence, context assembly, and authorization boundaries around them.
Continue this series
Context sensitivity and production governance
Part 2 of 3.
Explore the Enterprise AI hub for related architecture and governance guides.
Foundation: The Double-Slit Experiment and AI: Why Context Changes the Answer
- AI Context Sensitivity: Same Evidence, Different Decisions
- Build an AI Context Sensitivity Test Harness in Python (you are here)
- AI Context Governance: From Test Results to Production Controls
External References
- Anthropic: Demystifying evals for AI agents
- Python Software Foundation: JSON encoder and decoder
- Python Software Foundation: Subprocess management
- ACL Anthology: Flaw or Artifact? Rethinking Prompt Sensitivity in Evaluating LLMs
Test whether irrelevant context changes alter an AI decision. Preserve task, evidence, and policy while distinguishing genuine behavioral differences from changed settings...