elmerdata.ai blog

My blog

Fruitopia: Human Control Is Not a Governance Model

A shutdown button is not a constitution. The Pro-Human AI Declaration says humans must stay in charge of AI, and it is right. It does not say which humans, under what authority, or who governs them.

The Simpsons: They Save Lisa's Brain

In a 1999 episode of The Simpsons, "They Saved Lisa's Brain ," the Springfield chapter of Mensa takes over the town. The premise is reasonable on its face. Intelligent, rational people should govern rationally, so why not let them? The experiment lasts about as long as it takes the members to discover that each of them has a different idea of what rational government requires. Stephen Hawking, playing himself, arrives to inspect the result and delivers the verdict: "I wanted to see your utopia, but now I see it is more of a Fruitopia."

The joke works because the problem was never intelligence. The problem was governance. Springfield had exactly what a great many AI safety proposals implicitly ask for: capable humans firmly in control, no rogue machine, no alignment failure. The rulers simply governed badly. I thought of the episode while reading the Pro-Human AI Declaration, which rests on a surprisingly similar assumption.

Stephen Hawking in March 2012 Stephen Hawking, March 2012. Photograph by elhombredenegro, CC BY 2.0, via Wikimedia Commons.

A coalition that should not exist

The declaration was released on March 4, 2026, convened by the Future of Life Institute after a ratification meeting in New Orleans in January. Its coalition is the first thing worth noticing. Yoshua Bengio and Stuart Russell signed it. So did Daron Acemoglu, Susan Rice, Ralph Nader, Thom Yorke, Steve Bannon and Glenn Beck, alongside SAG-AFTRA, the American Federation of Teachers, Public Citizen, the Congress of Christian Leaders and PauseAI. People who rarely appear on the same page found a page they could share. The organizers pointedly did not invite the major AI companies, and TechCrunch read the result as an attempt by private citizens to write the governance framework Congress had not.

The breadth is a genuine achievement, and the document earns it by getting the diagnosis largely right. It fears concentration of power, unaccountable companies, autonomous systems nobody can stop, and the erosion of human agency. Its remedies are mostly sensible. Developers and deployers should bear legal liability for defects and misrepresented capabilities. If an AI system causes harm, it should be possible to ascertain why and who is responsible. AI companies should not have undue influence over the rules that govern them. A governance practitioner can read most of the 30 or so provisions and nod.

The organizing principle sits at the top, in a section titled Keeping Humans in Charge: "Human Control Is Non-Negotiable: Humanity must remain in control." Meaningful control means humans "should have authority and capacity to understand, guide, proscribe, and override AI systems." Highly autonomous systems whose controllability is not obvious "require pre-development review and independent oversight." Powerful systems "must have mechanisms that allow human operators to promptly shut them down."

Almost nobody can oppose any of that. Which is the problem. A principle nobody can oppose is a value statement, and a value statement is not a governance model.

Which humans?

"Humanity must remain in control" is a sentence with a hidden subject. Humanity does not exercise control. Particular humans do, sitting in particular institutions, with particular budgets, incentives and blind spots. A corporation controlling its own model is human control. A government controlling its surveillance system is human control. A commander controlling an autonomous targeting system is human control. An authoritarian controlling a national AI is human control. None of those arrangements automatically produces good governance, and several of them are the outcomes the declaration most fears.

The declaration keeps answering one question, who should remain in control, and the answer is humans. Governance asks a different series. Which humans? By what authority? Under which rules? With what information? Subject to what review? Who can challenge them, and who can reverse them? The coalition's breadth makes that series sharper, not softer. A labor federation, a civil liberties group, an AI safety organization, a faith coalition and a former national security adviser can all sign a sentence about human control. Ask each of them to name the humans and the answers diverge fast: regulators, courts, developers under liability, elected legislatures, expert panels, individuals with the right to opt out. Some of these answers are compatible. Several are not. Consensus around the noun "control" conceals disagreement about the verb "govern," and the declaration, by design, stops at the noun.

That is the objection the Mensa council ran into. The members agreed that smart people should run Springfield. They had no procedure for what happens when smart people disagree, and no account of why anyone else should accept the outcome. Human control sounds democratic until someone has to decide which humans get the controls.

A shutdown button is not a constitution

Take the most concrete provision. An off switch sounds like engineering, a circuit breaker for software. Every serious safety framework wants one, and the declaration is right to insist on it.

But a shutdown button answers only whether a person can stop a machine. Governance asks who possesses the button, who sets the conditions for pressing it, and who even knows it exists. It asks who judges in the moment whether shutting the system down does more damage than leaving it running, a real question once AI sits inside hospital scheduling, power dispatch or payment clearing, who reviews the decision afterward, and who is accountable if the review finds the switch was pulled too late, too early or for the wrong reasons. Each of those is a governance decision, and the declaration answers none of them, because it is a declaration and that is not what declarations do.

The distinction is worth stating cleanly. Control is authority over a system. Governance is authority over the people exercising that control. Control tells us who can command the machine. Governance tells us who can command the person commanding the machine. The hardest question about an AI off switch is not whether one exists. It is who gets to press it, and who can make them.

The recursion

Now put the off switch next to the declaration's second major section, Avoiding Concentration of Power. Here the document warns that "AI monopolies that concentrate power, stifle innovation, and imperil entrepreneurship must be avoided," that benefits must be shared broadly, and that AI development "must not severely limit humanity's future options or irreversibly limit our agency." The preamble frames the whole exercise as resistance to a "race to replace" that ends with power "concentrating in unaccountable institutions and their machines."

Read the two sections together and a recursion appears. To prevent machines and their owners from concentrating power, the declaration proposes pre-development review, independent oversight, independent safety standards, mandatory shutdown capability and, for some architectures, outright prohibition backed by criminal penalties for executives. Every one of those tools requires an institution powerful enough to tell the most capitalized companies in history what they may build, when they may ship it and when they must turn it off. That is a great deal of power, possibly justified, and still concentrated. The document that fears concentration says almost nothing about how this new concentration would itself be constrained, audited, challenged or removed.

Humans are the original alignment problem. Human control can solve the machine version and simply move the problem one level up, because a system can remain perfectly under human control and still produce disaster when the controlling humans are captured, incompetent, ideological or merely wrong. The logic runs in a loop. AI needs oversight. Oversight requires authority. Authority requires governance. The declaration supplies the first step, gestures at the second and leaves the third to someone else. I wrote here in August about algorithmic monoculture, the risk that many institutions relying on the same few models lose the independent judgment they were built on, and the lesson applies to overseers as much as to models: concentration does not require a villain, only everyone choosing the locally sensible option. A single review body, a single set of safety standards, a single off switch held by a single authority is the locally sensible option for a movement that wants control.

The danger is not merely that machines acquire too much power. It is that fear of machine power creates institutions with powers nobody has yet learned how to govern. The declaration also demands "no regulatory capture," and anyone who has watched capture unfold in finance, telecommunications or pharmaceuticals knows that is an outcome, not a mechanism. The mechanisms that produce it are institutional, unglamorous and absent from the document.

Control without inspection

The clearest evidence that the declaration is a value statement rather than a governance model is a word it never uses. "Open" does not appear. Not open source, not open weights, not open models. A post on the Adafruit blog made the point the day the declaration appeared.

The omission matters less as a policy position than as a symptom, because the declaration's own definition of control depends on visibility. Humans are to "understand, guide, proscribe, and override" AI systems, and understanding requires access. Can a regulator inspect the model, or only the vendor's description of it? Can independent researchers reproduce a safety claim? Can an auditor distinguish what the model did from what its operator says it did? Open weights are one answer to those questions, not the only one; regulators can be given technical access without public release. But a document worried about monopoly and lock-in ought to have said which answer it wanted, especially since open models also cut against its other half: an off switch means nothing for a model running on 10,000 laptops. Human control and dispersed power pull in opposite directions, and the declaration does not say where it stands.

It did not need to settle that tension. It needed to name it, because without observability the control it demands is ceremonial. A human cannot meaningfully govern a system that the human is permitted to control but forbidden to inspect.

The missing layer

None of this is an argument for machine autonomy over human authority. Human oversight is necessary. It is not sufficient, and the difference is the whole of governance. The layer the declaration leaves out is familiar from every other domain in which we have tried to constrain powerful actors: distributed rather than singular authority, independent oversight of the overseers, auditability, appeal, defined responsibility, and limits on human discretion as well as machine discretion.

So the principle should not read "humans must remain in control." It should read something closer to: AI must remain subject to legitimate human institutions whose own exercise of power is constrained, observable, contestable and accountable. That is considerably harder than installing an off switch. It is also governance.

Back to Springfield

The Mensa experiment did not fail because intelligence is undesirable in government. It failed because intelligence did not eliminate disagreement, incentives, competing values or the need for rules about what happens when the wise disagree. A system built on the assumption that good people, smart people, experts, executives, regulators or elected officials will exercise power wisely is not yet a governance system. It is a personnel strategy. Hawking's verdict was not that the value was wrong. It was that a value is not a town.

The declaration's authors deserve credit for recognizing that intelligence alone cannot secure humanity's future. The next step is recognizing that humanity alone cannot guarantee it either. Human control should stay non-negotiable, and another principle belongs beside it. The question is not simply whether someone can pull the plug on an AI. It is who governs the person standing beside the plug. Otherwise a carefully designed human-centered utopia may turn out, as Hawking found in Springfield, to be something closer to a Fruitopia.

Further Reading

AI Assistance Statement ▾

This blog publishes at a near daily rate, and that pace is possible because AI tools do a substantial share of the work between the idea and the published text. Preparation of this entry included assistance from Anthropic's Claude and from OpenAI's ChatGPT (GPT-5 series reasoning models). I use them to research a topic and gather primary sources, to organize ideas and propose structure, to draft and revise prose, to check factual claims against the cited sources before publication, and to score drafts against a set of house style rules. Longer pieces are often developed across several sessions. A written handover carries the argument, sources, and open questions from one session to the next, and the same tools help prepare those handovers. The tools also help identify candidate images and confirm that selected images appear to be released for reuse, for example through public domain or Creative Commons licensing.

The process is also an AI experiment in its own right. There is a live argument about what AI-assisted writing does to originality, and whether the result is thought or slop; the August 2026 dispute over a Wall Street Journal op-ed that its author acknowledged drafting with AI, and the Journal's subsequent defense of the practice (WSJ is behind a paywall, but for a public summary: click here, and here), is one newsworthy example. I would rather run the experiment openly than pretend it is not happening. This blog is one sustained attempt to find out whether a person with an argument, working with these tools every day, produces writing that is still recognizably that person's, and I disclose the method so readers can judge the result.

The judgment is mine. I choose the topic, decide the argument, supply the personal and professional experience the pieces draw on, read and edit every draft, verify the sources and image licensing, and take full responsibility for the final published content. Where a post contains my own recollections, the AI did not invent them.

Statement revised September 2026.

#AIData #Observations