Entropy in AI and Human Thinking: Why Certainty Is Not Accuracy

Published by

TL;DR

Entropy is not simply disorder, and it does not provide a scientific shortcut from physics to mindset or AI reliability. Thermodynamic entropy concerns physical systems; Shannon entropy measures uncertainty in a probability distribution; psychological entropy is a theoretical framework for uncertainty in perception and action. Humans and AI consume energy, but a more predictable response is not necessarily a more accurate one. Repetition can reinforce mistakes, and lower AI sampling temperature does not verify facts. The practical objective is not minimum uncertainty. It is evidence-supported decisions, corrective feedback, and controlled action.

Introduction

Consider an engineer and an AI assistant reviewing the same infrastructure change. The engineer has rehearsed the migration sequence until it feels familiar. The assistant produces the same confident recommendation each time it is asked. Both appear ready to proceed.

Then a compatibility check reveals that the proposed configuration is unsupported.

Neither familiarity nor output consistency protected the decision. The process became more settled without becoming better aligned with the environment.

That is a useful starting point for discussing entropy in AI and human thinking. The concept can clarify uncertainty, learning, and the physical cost of maintaining useful systems. It becomes misleading when it is used to imply that confidence proves correctness, that thoughts naturally deteriorate under a psychological second law, or that AI governance is a form of thermodynamic housekeeping.

For architects, engineers, and technical leaders, the distinction matters because each problem requires a different control. Power and cooling support physical operation. Good information supports reasoning. Validation determines whether a recommendation has earned the right to influence a real system.

Keep the Three Meanings of Entropy Separate

The same word appears in physics, information theory, and psychology, but it does not identify the same measurement in every context. The distinctions below follow the thermodynamic treatments of Lieb and Yngvason and Goldstein and Lebowitz, Shannon’s information theory, and the psychological framework proposed by Hirsh, Mar, and Peterson.

ContextWhat entropy describesWhat it does not establish
Thermodynamics and statistical mechanicsPhysical entropy, related to the microscopic states compatible with a macroscopic descriptionWhether an idea, belief, or AI answer is correct
Information theoryUncertainty associated with a specified probability distributionWhether the most probable outcome is true or useful
Psychological entropyA proposed framework for uncertainty among competing interpretations and actionsA universal physical law governing mindset or mental health

The scope here is deliberately limited. The human discussion concerns learning and decision-making, not diagnosis. The AI discussion concerns probabilistic language models and an illustrative enterprise assistant whose weights remain fixed during inference.

There is also a measurement requirement: an entropy value needs a defined system or probability model. Saying that an organization has “high mental entropy” may communicate a metaphor. It is not a quantitative assessment until the possible states, probabilities, and measurement method are specified.

Entropy and the Second Law: Probability, Not Messiness

Boltzmann’s familiar expression is:S=kBln⁡ΩS = k_B \ln \Omega

Here, SS is entropy, kBk_B is Boltzmann’s constant, and Ω\Omega represents the number of microscopic states compatible with the chosen macroscopic state in the state-counting formulation. Entropy is measured in joules per kelvin, not in units of visible untidiness.

Imagine a gas initially confined to one part of a box. Remove the partition, and molecular motion allows it to occupy the available volume. Vastly more microscopic arrangements correspond to a roughly distributed gas than to every molecule remaining in the original corner. The statistical explanation is about that enormous difference in multiplicity, not a force that prefers messiness. Goldstein and Lebowitz (2004) develop this macrostate-microstate distinction.

For a system and surroundings considered together as an isolated whole, the macroscopic second law is:ΔStotal=ΔSsystem+ΔSsurroundings≥0\Delta S_{\text{total}} = \Delta S_{\text{system}} + \Delta S_{\text{surroundings}} \geq 0

The reversible limit has no total entropy increase. Irreversible processes generate entropy. Energy remains conserved, while irreversibility reduces the useful work obtainable relative to the surroundings. Lieb and Yngvason (1999) provide the formal thermodynamic treatment.

A local entropy decrease is therefore possible. During cooling, a refrigerator can reduce the entropy of its contents while consuming electrical work and rejecting heat to the room. The combined accounting still satisfies the second law. Once the refrigerator reaches steady operation, the contents need not keep decreasing in entropy.

The boundary matters. Looking only inside the refrigerator leaves out the process that maintains the condition.

Human Thinking: A Physical System and an Uncertainty Problem

The Brain Requires Continuous Physical Support

The literal connection is biological. Neural activity, signaling, and maintenance depend on metabolism. Thinking is not separate from the physical system that supports it. Raichle and Gusnard’s 2002 paper, Appraising the brain’s energy budget, provides the energy-budget perspective behind this physical connection.

That does not justify assigning a thermodynamic entropy score to a person’s beliefs. Nor does learning a better explanation automatically demonstrate that the brain’s physical entropy decreased. Functional organization and thermodynamic entropy are not interchangeable descriptions.

Sleep and recovery belong to the broader discussion of supporting human performance, but no particular recovery protocol follows from these entropy equations. The energy-budget reference is not, by itself, evidence for a specific mindset or sleep intervention.

Information Entropy Describes Uncertainty, Not Truth

Shannon’s 1948 formulation gives the entropy of a discrete variable as:H(X)=−∑ipilog⁡2piH(X) = -\sum_i p_i \log_2 p_i

Here, pip_i is the probability assigned to outcome ii. With base-two logarithms, the result is measured in bits.

A distribution spread across several plausible outcomes has greater uncertainty than one dominated by a single outcome. As a simple calculation, ten equally likely alternatives have an entropy of approximately 3.32 bits.

In Psychological Entropy, Hirsh, Mar, and Peterson (2012) propose an uncertainty framework involving competing perceptions and possible actions. Goals, prior experience, and attention help shape which interpretations or responses become dominant. This is a theoretical account of uncertainty management, not an experimentally established psychological equivalent of the second law.

For an engineer, the practical analogy is straightforward. Several explanations for an outage compete for attention. Logs, dependency checks, and controlled tests can help distinguish them. But eliminating alternatives by assumption can also create certainty. Only the first approach earns confidence from evidence.

Repetition Can Strengthen the Wrong Pattern

Mental rehearsal can have measurable effects. Bernardi and colleagues (2013) studied skilled pianists and found that mental practice improved aspects of performance and motor anticipation, although physical practice produced greater overall performance improvements. This is evidence about a particular motor task, not proof that imagination can substitute for every kind of experience. Physical execution also supplies sensory feedback that imagined execution cannot fully reproduce.

The operational analogy is useful but narrower: mentally walking through a recovery procedure is not the same as executing it in a representative test environment. Rehearsal cannot reveal a missing permission or an unexpected dependency unless that information already exists in the imagined scenario.

Additional experimental evidence comes from Fazio and colleagues’ 2015 study of illusory truth. Repeated statements received higher truth judgments, including in situations where participants possessed relevant knowledge. Familiarity can influence what feels credible without changing what is actually true.

Repetition can stabilize a response. It does not certify the response.

None of these findings shows that thoughts attract external events, that negativity is thermodynamically inevitable, or that discipline overrides missing resources and environmental constraints. A defensible mindset practice is to rehearse an intended action, test it, examine the difference, and revise the rehearsal.

Entropy in AI: Physical Cost and Predictive Uncertainty

Computation Has a Physical Cost

AI runs on physical hardware. Electrical energy supports computation, and operating equipment releases heat that must be transferred to its surroundings. The relevant physical boundary includes more than the model’s parameters.

Landauer’s principle makes the information-physics connection precise for a particular operation. In the standard idealized setting, resetting an initially equally likely zero-or-one bit to a known state has a minimum average heat dissipation:Qmin⁡=kBTln⁡2Q_{\min} = k_B T \ln 2

Here, TT is the physical temperature of the thermal environment. Bérut, Petrosyan, and Ciliberto (2015) report an experiment in which average erasure heat approaches this bound for sufficiently slow erasure cycles.

This is an erasure bound, not a universal energy charge for every logical operation or generated token. It should not be used as a GPU sizing formula. Likewise, useful structure in trained parameters does not, by itself, specify a measurable reduction in their thermodynamic entropy.

A Language Model Predicts a Distribution

Bengio and colleagues’ 2003 paper, A Neural Probabilistic Language Model, provides an early neural formulation of predicting words from preceding context. The relevant principle is that the system assigns probabilities rather than retrieving an intrinsic certificate of truth.

For an autoregressive language model, next-token probabilities depend on the supplied context and the tokens already generated. High token entropy means probability is distributed across more alternatives. Low entropy means the distribution is concentrated.

Neither condition establishes factual accuracy. Several plausible tokens may be different ways of expressing the same correct idea. A strongly preferred continuation may still express a false claim.

Cross-entropy training addresses a different question: how much probability did the model assign to the observed target? For one target token, the negative log-likelihood contribution is:L=−ln⁡p(target token∣context)L = -\ln p(\text{target token}\mid\text{context})

Assigning little probability to the observed token produces a larger penalty. This encourages agreement with training examples, not automatic verification of their contents. Cross-entropy training is therefore not simply a mandate to make every prediction maximally certain.

Sampling Temperature Is Not Hardware Temperature

For fixed model scores, or logits, a standard temperature transformation is:pi(τ)=exp⁡(zi/τ)∑jexp⁡(zj/τ),τ>0p_i(\tau) = \frac{\exp(z_i/\tau)}{\sum_j\exp(z_j/\tau)}, \qquad \tau > 0

Here, ziz_i is the score for token ii, and τ\tau is the sampling temperature. With the scores fixed and no additional filtering, increasing τ\tau spreads probability more evenly; decreasing it concentrates probability on higher-scoring tokens.

Hugging Face’s generation documentation describes this control and notes that its temperature setting requires sampling to be enabled to affect generation.

Changing this parameter does not obtain newer documentation, inspect a server, or resolve a contradiction in the evidence. Choosing the highest-scoring token is not a factual validation step either.

A more consistent wrong answer is still a wrong answer.

A Simple Demonstration: Equal Entropy, Opposite Accuracy

Consider two mutually exclusive candidate answers, one correct and one incorrect. The probabilities below are constructed examples, not measurements from a person or a language model.

Applying Shannon’s formula gives:

Illustrative distributionProbability assigned to correct answerProbability assigned to incorrect answerEntropy
Unresolved0.500.501.000 bits
Concentrated on the correct answer0.990.010.081 bits
Concentrated on the incorrect answer0.010.990.081 bits

The last two rows have identical entropy because exchanging the labels does not change the calculation. Their practical implications are very different.

This demonstrates why entropy alone cannot be a correctness score. It describes the probability distribution, not which answer deserves the probability assigned to it.

It also explains why reducing uncertainty is not always the right immediate objective. If contradictory evidence appears, reopening the diagnosis may be more responsible than preserving a settled conclusion. In technical decision-making, exploration should identify credible alternatives; consolidation should follow evidence that distinguishes them.

AI Maintenance Is Not a Psychological Second Law

Assume that a deployed model’s weights and stored artifacts remain unchanged. There is no basis for saying that its answers must gradually become chaotic because time passes and thermodynamic entropy increases.

Instead, investigate identifiable mechanisms: a changed environment, stale retrieval documents, conflicting context, an inappropriate optimization target, or generated claims being recycled as trusted inputs. These are diagnostic possibilities, not interchangeable manifestations of physical entropy.

For the fixed-weight assistant considered here, updating a prompt, retrieval source, or external memory changes the information supplied during inference. It does not automatically change the model’s weights. Fine-tuning or retraining is a separate process.

The distinction prevents a common mistake in the human-AI analogy. A person rehearsing a task and an assistant receiving repeated prompts are not necessarily undergoing equivalent learning processes. “Feedback” is useful only when the surrounding system captures it, evaluates it, and applies an appropriate correction.

An approving response from a user is not the same evidence as a successful test against the intended requirement. Treat them as different signals.

Turn the Mental Model into an Evidence-Controlled Workflow

The following is a proposed enterprise application of these ideas, not a workflow experimentally validated by the cited physics or psychology papers.

Return to the infrastructure example. An assistant claims that a cluster can support the forecast workload after losing a host. Assume the assistant can read approved inventory and documentation but cannot change production directly. A designated engineer owns technical validation, and the normal change process controls execution.

The decision criteria should be explicit: current configuration, applicable compatibility evidence, a defined workload target, and a representative capacity or failure test. Repeated agreement between an engineer and the assistant does not replace any of them.

The diagram separates producing a recommendation from validating it and authorizing action. A missing-evidence path is deliberate, not an exception to hide.

Record What Would Justify the Decision

The YAML below is an illustrative application-owned decision record, not an executable policy or a configuration for an existing product. The empty evidence fields and false gates intentionally represent an unapproved recommendation.

decision:
  claim: "Cluster meets the workload target after one host fails."
  scope: "Production cluster A; approved workload forecast"
  mode: "recommendation_only"
  evidence:
    inventory_snapshot: null
    compatibility_review: null
    failover_test: null
  gates:
    evidence_verified: false
    workload_target_met: false
    change_authorized: false
  on_missing_or_conflicting_evidence: "escalate"

Replace the scope with the actual environment and associate each evidence field with a versioned, reviewable artifact. Define the workload target before testing, including the relevant concurrency, latency, capacity, and recovery requirements.

Successful validation means that the evidence supports the claim within that scope. It does not mean that the model filled every field or marked every gate true. Trusted application logic must enforce the checks and bind authorization to the appropriate identity and current policy. The model must not approve its own evidence.

Give Corrections an Owner

Assign source freshness and provenance to a knowledge owner, technical validation to the relevant domain engineer, and runtime enforcement and audit records to the platform team. Keep approval responsibility explicit rather than distributing it across an informal collection of reviewers.

When the result contradicts the recommendation, identify which component needs correction. The runbook may be wrong, the retrieved document may be stale, or the test may reveal an unmodeled dependency. Updating the correct component is more useful than repeatedly asking the same question until the output sounds convincing.

Preserve the rejected claim and the evidence that disproved it. Add a regression case before promoting the correction into approved knowledge. Keep candidate answers separate from validated records so an unverified output cannot become its own supporting source.

For human practice, use the same operational discipline: rehearse the procedure, exercise it safely, review the outcome, and rehearse the corrected version. The analogy is about the value of a correction loop, not identical biological and computational mechanisms.

Conclusion

Entropy becomes useful in discussions of intelligence when its boundaries remain intact. Thermodynamics explains constraints on physical processes. Information theory describes uncertainty in specified probability distributions. Psychological entropy offers a framework for thinking about competing interpretations and actions. None of them turns confidence into evidence.

Humans and AI can both become more consistent without becoming more correct, although the mechanisms differ. Mental rehearsal needs contact with actual performance. AI predictions need appropriate evidence and evaluation. Operational feedback needs review before it becomes trusted knowledge.

For an enterprise team, the next practical step is to select one recurring AI-assisted decision and identify its evidence requirements, validation owner, authorization boundary, and correction path. Measure whether the decision meets its intended requirement, not merely whether the answer repeats reliably.

Energy supports the physical process. Repetition can stabilize a pattern. Evidence and corrective feedback test whether that pattern deserves to guide action.

External References

Keep exploring

Choose your next step

Continue with the path that best matches the architecture or operating challenge in front of you.

AI strategy & delivery Enterprise AI Strategy, governance, AI platforms, data, and accelerated infrastructure. Explore Enterprise AI Architecture & integration Hybrid Platforms Architectures that connect VCF, Azure, public cloud, Kubernetes, and edge. Explore Hybrid Platforms Day-2 execution Operations & Resilience Security, recovery, lifecycle, observability, capacity, and FinOps. Explore Operations

Leave a Reply

Discover more from Digital Thought Disruption

Subscribe now to keep reading and get access to the full archive.

Continue reading