What AI Is, What It Is Not, and What Remains Uncertain

Series: Community-Scale AI Governance — A Research Perspective on the Village Platform (Article 1 of 5) Author: My Digital Sovereignty Ltd Date: June 2026 Licence: CC BY 4.0 International

Statistical Prediction at Scale

The core mechanism of contemporary large language models (LLMs) is next-token prediction. Given a sequence of tokens, the model generates a probability distribution over possible continuations, informed by patterns extracted from a training corpus of considerable scale — typically billions of documents spanning multiple domains, languages, and registers.

This mechanism produces outputs that are frequently useful: coherent prose, competent summarisation, plausible answers to factual queries, and functional code. The practical utility is not in dispute.

What is in dispute — and what matters for governance — is the nature of the process that produces these outputs and, consequently, how much trust can be placed in them.

From Models to Agents: Capability versus Productisation

A development between 2024 and 2026 has shifted the object of governance from the model to the agent. The distinction is analytically important. The model (or engine) is the next-token predictor described above. An agent is that model embedded in scaffolding — persistent memory, tool and API access, a web browser or computer interface — and directed at a goal expressed in natural language, which it pursues across multiple steps with reduced human supervision. The 2025 AI Agent Index characterises this as a separation of capability (frontier models) from productisation (the agentic scaffolding around them): much of the apparent step-change in what AI can do, as distinct from what it can say, derives from the wrapper rather than from a new model architecture.

This shift changes the governance object in more than a quantitative sense. A chatbot's outputs cause harm only when a human acts upon them; an agent can act on the world directly, with fewer points of intervention. The literature has begun to formalise the resulting accountability problems — the responsibility gap (Matthias, 2004), in which operators cannot reasonably be held responsible for the behaviour of systems whose actions they did not determine, and the moral crumple zone (Elish, 2019), in which liability settles on the nearest human despite limited control. The International AI Safety Report 2026 notes further that agents pose heightened reliability risks precisely because they act autonomously, and that errors in multi-agent configurations can propagate and amplify. For governance research, the salient point is that autonomy relocates the locus of risk from the interpretation of outputs to the execution of actions — a relocation that policy-based governance, oriented toward output review, is poorly positioned to address.

The Reasoning Question: An Open Empirical Problem

Early characterisations of LLMs as "stochastic parrots" — systems that reproduce statistical regularities without any form of understanding — captured something important about the technology's foundations. However, as model scale has increased, behaviours have emerged that resist simple characterisation.

Large models demonstrate capacity for multi-step logical inference, analogical reasoning across domains, and performance on novel problems structurally dissimilar to training examples. Some researchers describe these as emergent capabilities — properties that arise at scale without being explicitly engineered. Others argue that apparent reasoning is a sophisticated form of pattern interpolation that merely resembles reasoning when evaluated by human observers predisposed to attribute understanding.

The empirical evidence is, at present, insufficient to resolve this question. Several observations complicate any confident position:

• Models solve problems that require compositional generalisation, suggesting something beyond simple retrieval.
• Models also exhibit failures — confident generation of false statements, brittleness under adversarial perturbation, sensitivity to surface features of prompts — that are inconsistent with robust reasoning.
• The internal representations of large models are not well understood. Mechanistic interpretability research has identified circuit-like structures that correlate with specific capabilities, but the field is in its early stages.
• The question of whether the distinction between "genuine reasoning" and "reasoning-like behaviour" is empirically meaningful, or whether it reduces to a philosophical commitment, remains unresolved.

A further complication arrived with the 2025–2026 generation of "reasoning" or "thinking" models, which externalise an intermediate chain of thought and allocate additional inference-time computation to harder problems; in structured settings such systems reached gold-medal-equivalent performance on the 2025 International Mathematical Olympiad. The interpretive debate remains live and unresolved. Apple's The Illusion of Thinking (2025) reported characteristic accuracy collapses on scalable puzzles, while several rebuttals (e.g. Lawsen et al.; Dellibarda Varela et al., 2025) attributed part of the effect to evaluation artefacts and characterised these systems as "neither true reasoners nor stochastic parrots." More consequential for governance is a growing body of work on chain-of-thought unfaithfulness (Anthropic, 2025, and subsequent 2026 studies): the externalised reasoning frequently does not reflect the causal determinants of the model's output, which undermines any oversight regime that relies on inspecting a model's stated reasoning. The displayed chain of thought is itself generated text, not an instrument reading of an internal process.

For governance purposes, the pragmatic implication is this: one cannot safely assume that an LLM will reason correctly, nor can one dismiss its outputs as unreliable. The system occupies an uncomfortable middle ground where outputs are often useful, sometimes wrong, and not reliably distinguishable from one another without external verification.

Novelty and Synthesis

A related question concerns whether LLMs can produce genuinely novel outputs. The strong claim — that models generate only recombinations of training data — is narrowly correct and broadly misleading.

Consider a model that has absorbed texts on polycentric governance theory, organisational behaviour, and community informatics as separate bodies of work. When prompted appropriately, it may synthesise connections across these domains that no individual researcher has made, because no individual researcher has the same breadth of exposure. The constituent ideas are not new. The synthesis, however, may be new to any given reader — and may identify genuine structural parallels that warrant investigation.

This is not equivalent to the novelty of primary research. The model has no access to empirical data it was not trained on, no capacity for experimental design, and no ability to evaluate whether its synthesised connections hold under scrutiny. The synthesis is a hypothesis generator, not a hypothesis validator. But hypothesis generation has value, provided one does not conflate it with hypothesis confirmation.

For researchers evaluating AI systems, the implication is that LLM outputs may be useful as a starting point for literature review, cross-domain exploration, and identification of structural analogies — but require the same critical scrutiny one would apply to any unverified source.

Training Data as Worldview

Every LLM inherits the statistical distribution of its training corpus. This is not a correctable bias — it is a structural property of the technology.

A model trained predominantly on English-language, commercially-oriented, Western internet content will produce outputs that reflect the assumptions, framing, and priorities of that corpus. When asked to address topics where the training data is sparse — indigenous governance traditions, liturgical language, oral culture, small-community decision-making — the model defaults to statistically dominant patterns rather than acknowledging the gap.

This has direct implications for any deployment in a specific community context. A model asked to generate content for a research group studying communal governance will default to the language of corporate management — not because it has evaluated the alternatives, but because corporate management language predominates in its training data. The substitution is silent: the model does not flag that it is operating outside its domain of competence.

This phenomenon — which might be termed distributional drift in a governance context — is well-documented but not well-solved. Techniques such as fine-tuning, retrieval-augmented generation (RAG), and system prompting can mitigate the effect but do not eliminate it. The residual bias of the base model persists, particularly under novel or complex queries where the fine-tuning signal is weaker than the base distribution.

Implications for Governance Research

The characteristics described above — useful but unreliable outputs, silent distributional bias, uncertain reasoning capacity — collectively define the governance challenge.

An AI system that is occasionally wrong is a quality-assurance problem. An AI system that is occasionally wrong in ways that silently substitute one value framework for another is a governance problem. The distinction matters because the first can be addressed by error-checking, while the second requires structural mechanisms that detect value-level drift, not merely factual error.

The agentic turn raises the stakes of this distinction. When value-level drift attaches not to a reviewable output but to an autonomously executed action, the window for human correction narrows or closes; and the chain-of-thought unfaithfulness findings imply that the system's own account of why it acted cannot be treated as a reliable oversight signal. Governance that depends on the system's self-report is therefore structurally inadequate. Verification must be external to the system and exercised over its actions, not only over its explanations.

This is the problem the Tractatus framework is designed to address. Whether it succeeds is an empirical question examined in subsequent articles. What can be stated here is that the problem is real, well-characterised, and not adequately addressed by the policy-based approaches that currently dominate AI governance discourse.

What This Article Does Not Claim

This article does not claim that LLMs are incapable of reasoning — the evidence is insufficient for that conclusion. It does not claim that LLMs can reason — the evidence is equally insufficient. It does not claim that distributional bias is unsolvable — only that current mitigation techniques are partial. And it does not claim that AI governance is impossible — only that the governance challenge is more structural than is commonly acknowledged.

The next article examines the specific structural differences between commercial AI platforms and community-governed AI systems, and analyses the trade-offs involved.

This is Article 1 of 5 in the "Community-Scale AI Governance" series. For the full technical architecture, visit Village AI — Agentic Governance.

Next: Platform AI vs. Community-Governed AI — A Structural Analysis