TL;DR
Enterprise AI systems cannot reliably learn from engineering experience that organizations never preserve. The missing information often includes rejected hypotheses, unsuccessful experiments, environmental conditions, expert observations, and the reasoning behind consequential technical decisions. Traditional repositories tend to preserve final outcomes more consistently than the experimental history that produced them.
An enterprise experience ledger is a proposed architectural pattern for capturing that history as structured, governed evidence. Rather than creating another repository of documents, it connects decisions, interventions, measurements, outcomes, provenance, and subsequent corrections. It keeps raw observations separate from human interpretations and approved operational knowledge.
The ledger does not require a new database product, a continuously retrained model, or unrestricted collection of employee activity. It requires clear capture boundaries, durable identifiers, source-system integration, permission-aware retrieval, review workflows, and retention controls.
Key takeaway: Preserve the evidence behind decisions before asking AI to learn from them. An experience ledger becomes valuable when it makes prior experiments reconstructable, comparable, and safe to reuse.
Introduction
An engineer receives an incident involving intermittent network failures during a scheduled maintenance window.
The organization’s incident repository contains a familiar resolution: a configuration change was reverted, connectivity stabilized, and the service returned to normal.
The engineer follows the documented procedure, but the problem continues.
What the incident record does not explain is that the original investigation included several unsuccessful changes. One appeared to improve the situation temporarily. Another exposed an interaction with a downstream dependency. A third produced conflicting results depending on the workload being tested.
Those observations never reached the final incident report.
The engineer may now repeat an investigation that a previous team already performed, using the same tools, consuming another maintenance window, and rediscovering the same limitations.
An AI operations assistant faces the same problem. Even if it has access to every approved runbook, change record, and architecture diagram, it cannot reliably reconstruct evidence that the organization failed to preserve.
Article 1 of this series examined why this missing knowledge matters and how unsuccessful experiments can contribute valuable counterexamples to AI-supported reasoning.
The next challenge is architectural.
How do we create a system that preserves engineering experience without requiring every engineer to become a full-time documentation specialist?
The answer is not simply to record more information.
It is to build a dependable relationship between what people intended to do, what systems actually did, what was observed, which conclusions were drawn, and whether those conclusions remain valid.
This article develops an enterprise experience ledger as a proposed reference architecture and operating model. The examples are illustrative, not claims of a deployed or benchmarked DTD implementation.
What an Enterprise Experience Ledger Actually Is
The term ledger can create the wrong impression.
It may suggest blockchain, financial transaction accounting, or an immutable database that preserves every event forever. None of those is a requirement.
For this article, an enterprise experience ledger is a governed system of linked evidence records that allows an organization to reconstruct what was attempted, under which conditions, what was observed, and how the interpretation changed over time.
The ledger may use existing databases, event streams, object storage, service-management systems, and search infrastructure. Its distinguishing characteristic is the relationship between records, not the underlying storage technology.
A useful ledger has four responsibilities:
- Capture: Preserve meaningful decisions, hypotheses, interventions, and observations.
- Correlate: Connect the record to the relevant environment, configuration, telemetry, and other evidence.
- Qualify: Record uncertainty, provenance, review, applicability, and conflicting findings.
- Provide controlled reuse: Make appropriate evidence available to authorized people and AI systems without granting broader authority.
The ledger should not replace the authoritative systems that already manage incidents, changes, laboratory results, or infrastructure configuration.
It should preserve the relationships those systems do not consistently represent.
Why Existing Repositories Are Not Enough
Organizations may already have several systems that appear capable of storing this information.
| Existing system | What it does well | What an experience ledger adds |
|---|---|---|
| Knowledge base or wiki | Publishes approved procedures and explanations | Links the published conclusion to prior unsuccessful attempts |
| Incident-management platform | Tracks incidents, timelines, and service restoration | Preserves hypotheses and intermediate diagnostic outcomes |
| Configuration management database | Describes assets and service relationships | Records the environmental state applicable to an observation |
| Observability platform | Collects logs, metrics, and traces | Connects measurements to the intervention and decision being evaluated |
| Source control | Preserves versioned code and configuration | Associates changes with hypotheses and observed results |
| Data lake | Stores large volumes of raw information | Adds evidence structure, provenance, review state, and reuse policy |
These capabilities can complement one another.
The recommended pattern is to reuse authoritative source systems and create a lightweight evidence relationship layer above them.
A dedicated platform becomes justified only when the required cross-system relationships, access controls, query patterns, or scale exceed what existing systems can support reliably.
Design the Ledger Around Experience Episodes
The core unit should be an experience episode.
An episode represents a bounded investigation or experiment with a recognizable objective.
For example, one production incident may contain several episodes:
An engineer tests a timeout change. A second engineer investigates a suspected routing issue. A third examines a storage dependency. Each attempt has a hypothesis, intervention, associated evidence, and observed outcome.
The incident is the larger container. The episodes preserve the investigative history inside it.
This distinction makes knowledge more reusable.
An AI assistant investigating a similar issue may need one specific failed hypothesis from the earlier incident rather than the entire incident transcript.
Five Essential Evidence Relationships
A useful ledger should preserve relationships between:
| Entity | Central question |
|---|---|
| Hypothesis | What explanation or predicted outcome was being considered? |
| Intervention | What action, configuration, experiment, or observation procedure was performed? |
| Context | Which system, version, topology, workload, or environmental conditions applied? |
| Observation | What was measured or directly observed? |
| Assessment | What conclusion is supported, and with what limitations? |
These relationships should remain explicit rather than being inferred later from a narrative summary.
An observation can support multiple assessments. An assessment can change after further testing. An intervention can be repeated in a different environment and produce a different result.
The model must accommodate those situations.
The Enterprise Experience Ledger Reference Architecture
The following architecture shows how the ledger can operate without duplicating every underlying source system.
Notice the separation between evidence capture, review, and AI consumption.
This architecture is a proposed logical design, not a product reference implementation.
Its most important control boundary is between preserved evidence and approved knowledge.
A record can be worth preserving without being suitable for general guidance. An unsuccessful experiment can inform future investigation without establishing a universal rule. An expert observation can justify further measurement without becoming an accepted diagnosis.
Provenance Is the Foundation
The World Wide Web Consortium (W3C) PROV-O specification provides a useful foundation for representing provenance.
Its model distinguishes entities, activities, and agents, along with relationships such as derivation, generation, usage, and responsibility.
For an enterprise ledger, those ideas translate into practical questions.
Which record was generated by which activity? Who performed the activity? Which inputs were used? What output was produced? Was the conclusion later revised?
Adopting these concepts does not require deploying a semantic graph database or implementing the entire PROV-O ontology.
A relational schema, document store, or graph model can represent the required relationships, provided identifiers and links remain consistent.
Telemetry Provides Observations, Not Explanations
OpenTelemetry can contribute important technical evidence.
Its logging model supports fields such as event timestamps, observed timestamps, resource attributes, trace identifiers, and span identifiers. When applications are properly instrumented, those identifiers can help correlate logs with distributed transactions.
But telemetry must be connected to the investigation that gave it meaning.
A latency spike during a configuration test is an observation. It is not, by itself, proof that the change caused the spike.
The ledger therefore needs both the machine measurements and the hypothesis or intervention being evaluated.
Existing Lineage Standards Can Help
OpenLineage provides another useful pattern, especially for data pipelines and repeatable computational experiments.
Its specification describes jobs, runs, datasets, and extensible facets containing additional metadata.
Those concepts can help link a specific test execution to input datasets, parameters, and resulting artifacts.
However, OpenLineage is primarily a data-lineage standard. It does not provide a complete system for capturing tacit expertise, determining causal relationships, or governing enterprise AI knowledge.
An experience ledger can reuse appropriate lineage concepts while supplying the additional domain-specific relationships it needs.
Capture Experience at Decision Boundaries
A technically sound schema will not help if engineers refuse to use it or if recording interrupts time-critical work.
The capture workflow must fit the way people already investigate problems.
The most practical approach is to distinguish automatic evidence collection from human interpretation.
Automatically Capture What Systems Already Know
The capture service should obtain approved metadata from existing systems whenever possible.
That might include the incident identifier, relevant configuration revision, environment, service ownership, test execution reference, change approval, component versions, and available telemetry correlation identifiers.
These references should be captured without requiring engineers to reenter them manually.
Where direct integration is unavailable, the record can link to an existing evidence artifact rather than copying it.
Avoid treating a matching timestamp as sufficient proof that two events are related. Correlation should rely on stronger identifiers where available, with inferred relationships explicitly marked.
Ask Humans Only What Machines Cannot Reliably Know
Human interaction should focus on the missing reasoning.
A short capture prompt can ask:
- What were you trying to prove or rule out?
- What did you do?
- What happened that differed from your expectation?
- What remains uncertain?
- Under which conditions would this observation be relevant again?
These questions are more useful than asking someone to summarize an entire incident after it has closed.
Capture should occur at selected decision points, such as after a rejected hypothesis, significant unexpected result, change rollback, or completed test sequence.
During a major incident, automated evidence capture can run while engineers focus on restoring service. Human interpretation can be added during the post-incident review.
Tacit Knowledge Requires a Different Capture Method
Not all experience originates in a formally recorded experiment.
An experienced engineer may notice a distinctive failure sequence. A manufacturing operator may recognize a vibration pattern. A maintenance specialist may identify an unusual sound associated with a component fault.
Some of that expertise can be captured through annotated observations, short interviews, permitted audio or video recordings, or comparisons with synchronized sensor data.
However, an expert’s recognition is not automatically a validated rule.
A credible capture record should preserve the observation and the expert’s interpretation separately, then identify what evidence could support or challenge the interpretation.
Recording every physical interaction is neither necessary nor appropriate. Privacy, safety, consent, and operational practicality determine which collection methods are acceptable.
Lessons From Established Knowledge-Management Practice
NASA’s APPEL Knowledge Services describes a lessons-learned lifecycle involving collecting, recording, disseminating, and applying lessons.
NASA also describes techniques such as Pause and Learn discussions and formal or informal team reviews.
This is a useful organizational precedent: knowledge capture is a lifecycle activity, not simply a place where documents are uploaded.
An enterprise experience ledger extends the concept toward more structured, machine-accessible evidence. It does not replace the human review and organizational learning practices that give the evidence meaning.
Google’s Site Reliability Engineering guidance on blameless postmortems reinforces another important dependency.
Teams must be willing to document incorrect hypotheses and unsuccessful interventions. If recording a failed attempt exposes an engineer to unfair blame, the ledger will eventually contain a biased version of reality.
A healthy learning culture is part of the architecture’s operating environment.
A Practical Experience Ledger Record
Consider a hypothetical platform team investigating intermittent application errors during failover testing.
The engineers suspect the client timeout is too short. They modify that value in staging but cannot demonstrate a reliable improvement because the two test windows are not sufficiently comparable.
A conventional incident summary may omit this unsuccessful attempt.
The experience ledger should preserve it.
Example Experience Record
The following YAML is a proposed vendor-neutral data contract. It illustrates the fields and relationships required for one episode. It is not executable configuration for a specific ledger product.
apiVersion: dtd.example/v1
kind: ExperienceEpisode
metadata:
id: exp-failover-example-004
schemaVersion: "1.0"
incidentRef: example-incident-204
occurredAt: "2026-10-01T02:18:00Z"
recordedAt: "2026-10-01T03:02:00Z"
context:
service: checkout-api
environment: staging
configurationRef: config-revision-example-42
topologyRef: topology-example-12
testRef: failover-test-example-07
hypothesis:
statement: >
Client timeout configuration contributes
to errors observed during failover.
expectedOutcome: >
Increasing the timeout will reduce
failover-related request errors.
intervention:
type: configuration-test
approvalRef: example-change-17
change:
parameter: clientTimeoutMs
before: 250
after: 500
observations:
summary: >
The observed error pattern did not
demonstrate a reliable improvement.
evidenceRefs:
- telemetry-bundle-example-before
- telemetry-bundle-example-after
- trace-sample-example-01
assessment:
outcome: inconclusive
limitations:
- insufficient-comparable-requests
- staging-workload-differs-from-production
nextTest: >
Repeat under comparable workload
and failover conditions.
review:
state: pending
reviewerRole: application-owner
access:
classification: internal-restricted
retentionPolicyRef: approved-policy-reference
aiRetrievalApproved: false
modelTrainingApproved: false
relationships:
relatedEpisodes:
- exp-failover-example-003
supersedes: null
The configuration values, dates, identifiers, and observations are illustrative. They are not measured results from an actual environment.
A real implementation would validate the record against an approved schema and reject malformed or unauthorized submissions.
The important properties are the relationship and control fields.
The record captures an unsuccessful investigation without pretending the hypothesis has been conclusively disproved.
It also preserves the environment in which the result was observed.
An AI assistant should not automatically convert the record into advice that increasing client timeouts never works. It should recognize that the previous test was inconclusive and that further controlled measurement was recommended.
The access fields express intended policy. They do not enforce authorization on their own. The storage, indexing, retrieval, and AI consumption services must implement the corresponding controls.
Preserve Observations, but Allow Interpretations to Change
A common design mistake is treating a recorded conclusion as permanent truth.
Engineering knowledge changes.
New evidence may establish that an earlier diagnosis was incorrect. A software update may invalidate a workaround. An instrument calibration issue may call prior measurements into question.
The ledger must preserve the history of those changes.
One practical approach is to make approved observation records tamper-evident and preserve subsequent interpretations as linked revisions.
The following conceptual lifecycle illustrates the distinction:
A new assessment should identify the prior version and the evidence motivating the change.
For example, an initially suspected network problem may later be traced to application connection-pool behavior. The original observations should remain reconstructable for authorized review, while the corrected interpretation becomes the current approved assessment.
An append-oriented history is useful for that purpose, but it must not override deletion, retention, privacy, or legal requirements.
Organizations may need to restrict, redact, delete, or cryptographically erase underlying personal or sensitive data. Retaining a permitted correction record does not justify retaining prohibited source content indefinitely.
Traceability is a design objective. Permanent retention of everything is not.
AI Retrieval Must Respect the Evidence Lifecycle
The ledger becomes especially useful when an AI system can retrieve relevant experience during investigation or decision support.
That access requires stronger rules than simply indexing records into a vector database.
The system should determine whether the requester may access each record, whether the record has been approved for the intended use, and whether its context matches the current problem.
A practical retrieval contract should address four conditions.
Authorization
The effective user’s permissions and the AI application’s approved scope must govern access.
A broadly privileged service identity must not expose restricted engineering, research, or employee data to users who could not access that evidence directly.
Applicability
A record from a different software version, environment, workload, instrument, or failure condition may still be informative. It should not automatically be treated as directly applicable.
The system should surface relevant context differences rather than silently hiding them inside a relevance score.
Evidence Quality
The assistant should distinguish a directly measured observation from an expert opinion, unreviewed note, inconclusive test, and approved conclusion.
Retrieval relevance and evidence reliability are different dimensions.
Current Status
If a conclusion has been superseded, withdrawn, or invalidated, that status must be visible to the consumer.
A historical record can remain relevant to understanding an investigation without being suitable as current guidance.
These conditions are consistent with an evidence-oriented retrieval architecture in which permission checks, provenance, and applicability control what an AI system may safely present.
A vector search result with a high similarity score does not establish that the underlying conclusion is correct or authorized.
Security, Ownership, and Retention Are Core Requirements
An experience ledger can become a sensitive collection of internal system behavior.
Troubleshooting evidence may expose infrastructure topology, credentials, network paths, vulnerability details, customer records, proprietary formulas, employee activity, or commercially sensitive research.
Because of that, the ledger needs a defined trust boundary.
Ownership Model
| Capability | Accountable owner | Required control |
|---|---|---|
| Schema and ingestion | Platform or data engineering | Versioned contracts and input validation |
| Evidence accuracy | Source-system owner | Source identity, lineage, and artifact integrity |
| Technical assessment | Domain expert or service owner | Review and applicability decisions |
| Data classification and retention | Data owner with security/privacy | Collection limits, retention, deletion, and legal hold |
| AI retrieval | AI application owner | Effective-user authorization and purpose restriction |
| Knowledge promotion | Designated technical authority | Review, approval, revision, and withdrawal |
| Platform operations | Platform engineering or service operations | Availability, monitoring, recovery, and auditability |
The AI model should not be the final authority over whether a record becomes approved engineering guidance.
It may help categorize observations, identify contradictions, suggest review candidates, and assemble an evidence summary. Those outputs should remain distinguishable from reviewed decisions.
Define the Retention Boundary Before Ingestion
Different source records may have different retention requirements.
A production trace might be retained for a limited operational period. An approved engineering lesson may need a longer lifecycle. A laboratory record may have regulatory or intellectual-property constraints.
The ledger should retain sufficient evidence and provenance for its purpose without automatically copying every source artifact.
Where a source expires, a permitted metadata reference or reviewed conclusion may remain, but the system should not imply that the original artifact is still available for independent verification.
When evidence becomes unavailable, its status should change accordingly.
Security testing should include attempts to retrieve unauthorized records, inject false experience records, promote unreviewed interpretations, and bypass withdrawal or deletion controls.
NIST’s AI Risk Management Framework provides a voluntary structure for examining these risks through governance, context mapping, measurement, and management. The framework does not supply the ledger implementation automatically; specific enforcement mechanisms and operating responsibilities still need to be designed.
Choose the Simplest Implementation Pattern That Meets the Need
A new central data platform may be unnecessary for the first implementation.
The architecture decision should depend on the source systems, evidence relationships, access requirements, and volume of useful episodes.
| Implementation pattern | Best fit | Tradeoff |
|---|---|---|
| Extend existing incident or experiment records | One team, one workflow, limited integration needs | Simple deployment, but weaker cross-system relationships |
| Central metadata and provenance index | Several systems with existing evidence repositories | Flexible and lower duplication, but dependent on source retention and availability |
| Dedicated ledger service with event history | High-volume or cross-domain evidence requirements | Stronger lifecycle control, but greater engineering and operating cost |
For most enterprise pilots, the middle pattern is a reasonable architectural starting point.
Keep raw evidence in approved source systems. Use a central registry to manage identifiers, relationships, review status, access metadata, and searchable summaries.
Introduce dedicated event storage only when there is a clear requirement for durable history, independent retention, cross-domain transactions, or high-scale ingestion.
Whichever option is selected, design backup, restore, schema migration, audit export, and eventual decommissioning from the beginning.
A Phased Implementation Strategy
The ledger should be introduced as a bounded engineering capability, not an enterprise-wide knowledge collection campaign.
| Phase | Activity | Evidence required before proceeding |
|---|---|---|
| Discover | Select a recurring investigation or experiment type | Named owner, known knowledge gap, and measurable baseline |
| Define | Agree on the episode schema, taxonomy, and data boundaries | Reviewed data contract, authorization model, and retention policy |
| Capture | Integrate one or two approved source systems | Correct identifier mapping and reliable capture of meaningful events |
| Review | Add expert assessment and correction workflows | Reconstructable episodes with explicit outcome classifications |
| Evaluate | Compare decisions using and without ledger evidence | Held-out cases, reviewer assessments, and measured operational impact |
| Expand | Add domains with demonstrated value | Supported operating costs, security controls, and clear ownership |
Avoid starting with the assumption that all historical incidents should be ingested.
Historical data may be incomplete, contain inconsistent terminology, or lack the evidence necessary to reconstruct an intervention accurately. Bulk ingestion can create a large repository whose apparent completeness exceeds its actual reliability.
Start with new episodes where capture quality can be measured at the source.
Demonstrate Value With Controlled Comparisons
The pilot should establish whether the ledger changes decisions.
For example, evaluate engineers or AI-assisted workflows against a fixed set of historical troubleshooting scenarios, comparing available approved documentation against documentation plus authorized ledger evidence.
Keep scenarios, evaluation criteria, and reviewer expectations consistent. Prevent evidence leakage from future stages of the historical investigation.
Useful measures include:
| Metric | What it demonstrates |
|---|---|
| Episode reconstruction rate | Whether a reviewer can establish the hypothesis, action, conditions, and observation |
| Evidence-link validity | Whether referenced artifacts can be retrieved and matched to the recorded episode |
| Capture effort | Whether the workflow adds acceptable operational overhead |
| Duplicate unsuccessful tests | Whether teams avoid repeating prior attempts when their limitations still apply |
| Diagnostic quality | Whether recommendations improve under equivalent evaluation conditions |
| Incorrect knowledge promotion | Whether unverified or superseded findings are being treated as authority |
| Unauthorized retrievals | Whether access controls hold under negative security testing |
More records are not the success criterion.
If the ledger creates additional administrative work without improving decisions, the design needs adjustment.
If it improves evidence availability but AI recommendations become overconfident, review and retrieval qualification need improvement.
If it helps teams avoid ineffective interventions while preserving appropriate uncertainty, the architecture is producing useful value.
Failure Modes to Design Against
An experience ledger introduces new operational risks. These should be addressed before the pilot expands.
Documentation inflation: Capturing every minor observation creates noise and slows review. Define meaningful episode triggers and automate routine metadata collection.
False causal attribution: Two events occurring together do not prove one caused the other. Preserve comparison conditions and uncertainty.
Inconsistent identifiers: A service with different names across monitoring, incident management, and configuration systems may generate incorrect associations. Maintain authoritative identifiers and reviewed mappings.
Unverified expert knowledge: Experience can generate good hypotheses, but expertise alone is not sufficient proof. Separate interpretation from independently supported findings.
Conflicting results: Different experiments can produce different outcomes under different conditions. Preserve both records and the circumstances that may explain the discrepancy.
Knowledge poisoning: Incorrect, manipulated, or maliciously submitted records could influence AI recommendations. Authenticate sources, validate schemas, audit revisions, and restrict promotion.
Surveillance risk: Indiscriminate capture of employee activity can undermine trust and introduce privacy obligations. Focus on relevant engineering events, explain collection purposes, and restrict access.
Unbounded retention: A permanent history can create legal, privacy, security, and storage problems. Apply approved lifecycle controls.
Premature AI authority: A ledger can provide better evidence without proving an agent is safe to operate equipment or change production services. Treat execution authority as a separate decision.
These risks reinforce the importance of keeping the initial scope narrow.
When an Experience Ledger Is Not the Right Investment
There are circumstances where the appropriate answer is to improve existing engineering practices rather than build a new capability.
If an organization does not consistently close incidents, maintain service ownership, version configurations, or preserve basic operational telemetry, a ledger may add complexity without making its evidence trustworthy.
Likewise, if engineers already capture experimental history effectively inside a well-governed laboratory information management system or other domain platform, duplicating those records may create unnecessary inconsistency.
The recommended sequence is to identify the missing decision context first.
If a structured field in an existing ticketing system and a short review workflow solve the problem, use them.
If useful experience spans multiple systems and repeatedly disappears between them, an evidence relationship layer becomes more defensible.
Architecture should follow the missing capability, not the appeal of a new knowledge platform.
Conclusion
An enterprise experience ledger addresses a specific knowledge-management problem: organizations frequently preserve the result of engineering work while losing important evidence about how that result was reached.
The proposed solution is not to record everything.
It is to capture meaningful episodes, connect hypotheses to interventions and observations, preserve environmental context, qualify conclusions, and maintain provenance throughout the evidence lifecycle.
A successful design separates raw evidence from interpretation and approved guidance. It applies identity, permissions, retention, and verification before information becomes available to AI systems. It allows conclusions to change without erasing the history required to understand them.
The practical next action is to choose one recurring investigation class and ask whether a qualified engineer could reconstruct its unsuccessful attempts using today’s records.
If the answer is no, establish a small experience-capture contract and measure whether preserving those attempts improves subsequent investigations.
That creates the foundation for the next stage of this series.
Article 3 moves from recording what happened to designing how an AI system can propose and evaluate new experiments safely, with authorization, measurement, verification, and stop conditions controlling the loop.
An experience ledger preserves what the organization learned. The next architectural challenge is ensuring new experiments produce evidence worth trusting.
Continue this series
Beyond the Training Set
Part 2 of 4.
Explore the Enterprise AI hub for related architecture and governance guides.
- AI’s Missing Evidence Layer: The Failures We Never Record
- The Enterprise Experience Ledger: Capturing Tacit Knowledge and Failed Experiments (you are here)
- From Hypothesis to Measurement: How AI Can Experiment Safely
- The Learning Enterprise: Governing Reality Based AI Evidence
External References
- World Wide Web Consortium: PROV-O: The PROV Ontology
- OpenTelemetry: Logs
- OpenLineage: Facets and Extensibility
- NASA APPEL Knowledge Services: Lessons Learned
- Google Site Reliability Engineering: Postmortem Practices for Incident Management
- National Institute of Standards and Technology: AI Risk Management Framework
2 thoughts on “The Enterprise Experience Ledger: Capturing Tacit Knowledge and Failed Experiments”