What AI Actually Is (and What It Isn't)

Series: AI Governance for Community Leaders — Understanding Village AI for Trustees, Councillors, and Board Members (Article 1 of 5) Author: My Digital Sovereignty Ltd Date: June 2026 Licence: CC BY 4.0 International

AI Changed While You Were Deliberating

You will have encountered claims that artificial intelligence is going to transform public services, community governance, and the way organisations operate. You may also have encountered claims that it is overstated, or that it cannot do anything genuinely new. Both positions miss the point, and understanding why will support better governance decisions.

But there is a prior point that any board or council should register: AI has changed materially even over the period in which organisations have been debating whether to adopt it.

A year ago, when most people said "AI," they meant a chatbot — a system you queried and which returned text. You asked, it answered. Today, the centre of gravity has moved. The systems attracting the most attention and investment are no longer just chatbots that answer. They are agents that act — they complete forms, send communications, browse and transact on the web, operate other software, and pursue multi-step objectives with limited supervision.

This shift is the single most important thing for a governance body to understand, because it changes the risk profile of adoption. To reason about AI today, hold two ideas apart:

• The engine — the underlying model, the component that produces language.
• The agent — the engine put to work, configured so it can take actions in the world on the organisation's behalf.

The engine has grown more capable. But the consequential change for any body with a duty of care is what is now being built around the engine. We take each in turn.

The Engine: A Machine That Predicts

Here is the plainest description of what the engine does: it predicts what word should come next.

When someone types a query into a chatbot, the system is not reasoning about the query the way a councillor or trustee would reason about a board paper. It is doing something more mechanical. It has been shown billions of pages of text — legislation, reports, correspondence, technical papers, news articles, social media, medical literature — and from all of that material, it has learned patterns. When you ask it a question, it generates an answer by predicting, one word at a time, what a plausible response looks like based on everything it has previously processed.

This is genuinely useful. A system that has absorbed the patterns of billions of pages of text can help draft correspondence, summarise lengthy documents, answer factual questions, or suggest how to word a sensitive communication. These are real capabilities, and they can reduce administrative burden.

But at its core, the engine is doing pattern matching at an extraordinary scale. That single fact explains both its considerable utility and the characteristic way it fails — a recurring theme in this series.

Can the Engine Reason?

There is a deeper question that researchers are actively investigating, and the straightforward answer is: we do not yet know.

When early AI systems produced fluent text, it was reasonable to describe them as sophisticated pattern-matching and leave it there. But a newer generation of engines — often termed "reasoning" or "thinking" models — does something different. Rather than answering immediately, it works through a problem in steps, producing a visible chain of intermediate reasoning before committing to an answer, and spending longer on harder problems. The results can be striking: in 2025, reasoning systems from more than one major laboratory solved problems from the International Mathematical Olympiad — among the hardest mathematics competitions in the world — at a level equivalent to a human gold medallist.

So is that reasoning, or very sophisticated pattern-matching that resembles reasoning?

The research is genuinely unsettled, and serious researchers disagree. One influential 2025 study argued these systems exhibit an "illusion of thinking" — collapsing on certain puzzles in ways a genuine reasoner would not. Several equally serious responses argued the contrary. The most careful current verdict is that today's reasoning models are neither true reasoners nor mere parrots — they are something genuinely new that is not yet fully understood. Anyone who tells you AI definitively can or cannot reason is overstating what the evidence supports.

One finding has direct governance relevance, and it is easily misread. When these systems display their "thinking," that visible chain does not reliably reflect what actually drove the answer. Researchers have repeatedly found that a model's stated reasoning can omit the real influences on its conclusion — not through dishonesty (the system has no intent) but because the displayed words are themselves predicted text, not a faithful readout of an internal process. The practical consequence for a governance body: you cannot discharge an oversight duty merely by reading the explanation an AI offers for itself. Accountability requires checking the output against the organisation's own records — not trusting the system's self-report. We return to this in Article 3, and it bears directly on the "right to explanation" expectations under GDPR and the EU AI Act.

What can be said with confidence is that the trajectory is steep. A few years ago these systems could barely produce a coherent paragraph. Today they write essays, pass professional examinations, generate computer code, and increasingly act on the world rather than merely describing it. The capabilities will be greater again within a few years.

"AI Cannot Do Anything New" — It Depends What You Mean by New

People who dismiss AI by saying it cannot create anything original are making a claim that is narrowly true and broadly misleading.

A language model cannot originate from experience. It has never sat in a public consultation where residents were angry. It has never felt the weight of a funding decision that affects a community's wellbeing. It cannot understand why the precise wording of a council resolution matters — it can only reproduce patterns that statistically resemble understanding. In that sense, everything it produces is a recombination of material absorbed during training.

But consider what "recombination" means at this scale. No single person has read every piece of local government legislation, every community trust annual report, every academic paper on participatory governance, and every regulatory impact assessment of the last decade. When the AI draws a connection between polycentric governance theory and community development practice, that connection may be genuinely novel to any individual reader, even though both ideas existed separately. The atoms are not new, but the molecules are.

So "AI cannot do anything new" is true at the level of origination and false at the level of synthesis. Both things matter, and responsible governance of this technology requires holding both.

From Answering to Acting: The Agent

This is the change with the greatest bearing on governance.

For most of the chatbot era, the worst an AI could do directly was return a poor answer. The harm materialised only if a person acted on it — sent the misleading letter, relied on the wrong figure, forwarded the flawed advice. A human always sat between the machine and the consequence.

An agent removes that human from the loop, by design.

An AI agent is an engine wrapped in what researchers call "scaffolding" — a memory to track a task, access to a web browser, the ability to operate software and other tools, and an objective expressed in plain language. With that scaffolding, the system pursues the objective across many steps with much less supervision: it searches, decides, acts, checks, and acts again. A chatbot answers. An agent acts.

For a board, this is precisely where the risk assessment changes. When a system acts autonomously, there are fewer points at which a human can intervene; some actions cannot be reversed; and when an agent acts on the organisation's behalf and the outcome is wrong, accountability becomes genuinely difficult to assign — between the officer who set the objective, the provider whose system chose the steps, and the body that authorised its use. Scholars describe the resulting "responsibility gap" and the "moral crumple zone," in which liability falls on the nearest human despite that person having had little real control. This is not incidental to regulation: the EU AI Act's insistence on meaningful human oversight is, in effect, a legal requirement that this loop not be closed without a person able to intervene. A governance body evaluating an agentic system is evaluating exactly that — whether oversight is structural or merely promised.

The Real Issue: Whose Patterns, and Whose Hands on the Controls?

Here is where it becomes practical for your organisation.

When a large AI engine is trained on the internet, it absorbs the internet's biases, assumptions, and cultural defaults. The internet is overwhelmingly English-language, Western, commercially oriented, and shaped by the values of the technology industry. This is not a conspiracy — it is simply what happens when a system is trained on data that disproportionately represents one culture and one set of priorities.

The consequences are subtle but material. When a resident submits a query about a difficult neighbour dispute, the system defaults to the language of individual rights and legal remedies — because that dominates its training data — rather than mediation, community obligation, or the long view. When a council officer asks it to draft a communication about a sensitive planning matter, it reaches for corporate stakeholder-management language, because business correspondence vastly outnumbers civic communication in what it learned from.

The system is not hostile to civic values. It simply does not know them. It knows what is statistically common, and what is statistically common is not what is most appropriate for your constituents.

In the chatbot era, that bias shaped the text an officer reviewed. In the agent era, the same bias shapes the actions taken on the organisation's behalf — communications sent, records filed, commitments made — potentially before any officer reviews them. So the governance question now has two halves: whose patterns does the system carry, and who holds the controls when it acts?

Why This Matters for Governance Bodies Now

No one knows with certainty what happens if an AI system ever develops something resembling its own intent — purposes that may not align with the interests of the communities it serves. We are likely still some distance from that threshold. But the architecture adopted now, and the habits of governance established today, will determine whether an organisation is prepared or whether it discovers too late that it ceded control without noticing.

This is not speculative. It is a straightforward observation about institutional preparedness. Your council has a constitution. Your board has standing orders. Your trust has a governing document. These exist not because every meeting descends into disorder, but because governance structures must be in place before they are needed, not after.

The same principle applies to AI, and the EU AI Act (Regulation 2024/1689) reflects precisely this logic — establishing governance frameworks before the technology outpaces regulatory capacity. The arrival of agentic systems raises the stakes: an organisation that adopts AI that acts, without structural oversight in place, may find itself unable to account for decisions taken in its name — a position that is uncomfortable fiduciarily and exposed legally.

Two Paths Forward

There are two ways an organisation can engage with AI.

The first path is to use Big Tech AI — systems such as ChatGPT, Google Gemini, or Microsoft Copilot, and increasingly the agents built on them. These are powerful, convenient, and often inexpensive. But they come with conditions. Your data flows to their servers. Your communications — and now your agent's actions — pass through systems you do not control. The AI's behaviour is governed by the company's policies, which can change without your agreement. And the patterns it carries are set by its training data, which you have no influence over. Under GDPR, this raises questions of data controllership, lawful basis for processing, and the right to explanation that every governance body should resolve before adoption — questions that sharpen, not soften, when the system acts autonomously.

The second path is to use AI that your organisation controls. A more focused system, trained on your content, running on infrastructure within your jurisdiction, governed by rules your board or council sets. A system that knows the difference between a council minute and a blog post, because your organisation's records taught it. A system whose responses are checked against your actual documents by verification layers that operate independently of the AI — and whose ability to act is deliberately bounded, so a human responsible to your constituents is always able to intervene.

This is what Village AI is. It is not the most powerful AI system available. It is designed to be accountable to your community — to your content, your values, and your governance framework — and to keep authority with the people who hold it when AI moves from answering to acting.

The next article explains how Village AI is structurally different from Big Tech AI, and why that difference matters more than raw power.

This is Article 1 of 5 in the "AI Governance for Community Leaders" series. For the full technical architecture, visit Village AI — Agentic Governance.

Next: Big Tech AI vs. Community-Governed AI — Why the Difference Matters