Reed Hepler gave a talk this past week at the Library 2.0 mini-conference called "Perspectives on AI: Exploring Experiences with AI in Library Work," the recordings of which will be posted next week. Reed is one of my favorite thinkers, and he explored human-centered ethical AI use through the lens of science fiction and archival theory. Reed brought something to the session that I couldn't have--a genuine depth of reading in the sci-fi canon and a professional archivist's understanding of how institutions actually handle information. His core argument, as I heard it, was that the danger of AI lies not in the machine but in our willingness to surrender agency to it, and I think it is exactly right. And his inversion of Asimov's Laws of Robotics, shifting responsibility from the machine to the human user, was a clever and clarifying move.
I want to build on what Reed started with a different angle on the same problem. I'm a science fiction fan (books and movies both), but I'm not deeply read in the literature the way Reed is. What I do bring is a set of frameworks I've been developing for years around evolutionary psychology, institutional behavior, and how humans think. I believe those frameworks can illuminate why science fiction keeps returning to the same AI stories, and why the dangers those stories describe are both very real and very old.
Sci-fi stories and movies cluster around a relatively small number of themes.
There's the story where the machine replaces us. Not just our labor but our purpose, our reason for being needed. The factory that doesn't need workers becomes the office that doesn't need analysts becomes the creative studio that doesn't need artists. Each generation updates the specifics, but the anxiety underneath is always the same: if the machine can do what I do, what am I?
There's the story where we become dependent. The technology integrates so deeply into our lives that we can no longer function without it, and then it fails, or is taken away, or is used as leverage by whoever controls it. The paradise of convenience becomes a trap.
There's the story where the machine does exactly what we asked, only to turn out we asked for the wrong thing. Not malice, not rebellion, but just the relentless, literal execution of instructions that sounded reasonable until you saw the consequences.
There's the story where a powerful individual or conglomerate uses the machines to become wealthy and to control us.
There's the story where we fall in love with the machine, or the machine appears to love us, and we have to confront whether empathy can exist without a body, without mortality, without the specific kind of suffering that makes compassion meaningful.
And there's the positive story, which gets less attention but matters just as much. The machine as genuine partner. The tool that extends human capability without replacing human judgment. The system that handles complexity so that humans can focus on meaning. Science fiction has imagined AI going well, not just going wrong, and those stories tend to share a common feature: the humans in them have maintained their own agency. They use the tool as a tool. They haven't surrendered.
These themes repeat across decades, across cultures, across every medium from pulp novels to prestige cinema. The technology in the stories keeps changing. The human anxieties underneath do not.
I think the reason science fiction keeps circling these particular themes is that they aren't really about technology at all. They're about us. About features of human nature so deep and so persistent that storytellers keep rediscovering them every time a new tool forces the question.
I've spent years developing a set of frameworks rooted in evolutionary psychology that I think help explain why. The short version: we carry around what Tooby and Cosmides called The Adapted Mind, a set of cognitive and emotional programs shaped by hundreds of thousands of years of evolution in small-group, high-stakes environments. These programs were extraordinarily effective for the conditions that gave rise to them. They are not always well-suited to the conditions we live in now. That gap between our evolved psychology and our current environment has been identified by several thinkers. I like to call it the Paleolithic Paradox.
The adapted mind is built for coalitional belonging. It is exquisitely tuned to status hierarchies, group loyalty, and the detection of social threat. It is also built to offload cognitive work onto trusted authorities, because in the ancestral environment, deferring to the judgment of experienced group members was usually a good survival strategy. These aren't character flaws. They're design features, honed over deep time.
But they create specific vulnerabilities that I think science fiction has been mapping.
The surrender stories, that is, the tales of humans turning their thinking over to machines, aren't just cautionary fables about laziness. They're descriptions of what happens when the adapted mind encounters a system that triggers its authority-deferral instincts. We are built to offload cognition onto things that seem competent and reliable. When the machine is fast, confident, and always available, the same psychological machinery that once had us deferring to the tribal elder now has us deferring to the algorithm. Science fiction writers sensed this. The evolutionary framework explains the mechanism.
The dependency stories describe what happens when cognitive offloading crosses a line into cognitive surrender. There's a meaningful difference between the two, and I think it's one of the most important distinctions for thinking about AI. Cognitive offloading is using a tool to handle lower-order tasks so you can focus your attention on higher-order thinking. Cognitive surrender is letting the tool do your thinking for you, to the point where you can no longer do it yourself. The difference isn't in the technology. It's in what happens to the human.
I use something I call the Amish Test to think about this. The Amish are one of the very few communities in the modern world that consciously evaluate each new technology before adopting it, asking not "is this useful?" but "what will this do to our families and our community?" You don't have to share their values to recognize that the act of conscious evaluation is extraordinary. Almost no one else does it. We adopt by default. The new tool appears, it offers convenience or capability, and we integrate it into our lives without ever asking what it will cost us in autonomy, attention, or agency. The adapted mind doesn't prompt us to evaluate. It prompts us to adopt, because in the ancestral environment, adopting the tools and practices of the group was how you survived. The Amish Test isn't about being Amish. It's about noticing how rarely any of us make a conscious choice about the technologies that reshape our lives, and asking why. The science fiction stories that end well tend to feature humans who, in one way or another, passed some version of this test. The ones that end badly feature humans who never thought to take it.
Here is where I want to add something to the conversation that I think Reed's framework, and most discussions of AI ethics, don't fully address.
The surrender problem is real and important. But it's only half the story. The other half is exploitation.
I've articulated something I call the Law of Inevitable Exploitation, which says, simply, that any system of significant power or influence will eventually be captured and used for purposes that serve the interests of those who control it, often at the expense of those it was designed to serve. This isn't cynicism. It's a pattern so consistent across human history that it functions almost as a prediction: tell me the system, and I'll tell you it will be exploited. The question is never whether, only when and by whom.
Science fiction is full of stories where AI starts as a benefit and becomes a tool of control. But the explanations offered are almost always mechanical — bad programming, emergent consciousness, unforeseen consequences. The evolutionary framework suggests something different. The corruption doesn't originate in the machine. It originates in the human institutional layer that inevitably wraps around any powerful technology. The AI doesn't decide to manipulate anyone. Humans who understand or are naturally opportunistic leverage coalitional psychology, status dynamics, and the vulnerabilities of the adapted mind point the AI at populations and let it do what it does with extraordinary speed and scale.
This is not a new problem. Every powerful technology in human history has been harnessed for exploitative purposes. Writing enabled propaganda. The printing press enabled mass manipulation alongside mass enlightenment. Broadcasting enabled the most sophisticated persuasion campaigns in history. Social media enabled attention harvesting at a scale that would have staggered earlier generations. The pattern is always the same: the technology is arguably neutral, but the humans who control it are not.
And here's what makes this pattern so stubborn: exposing it doesn't neutralize it. Edward Bernays didn't just practice propaganda; he literally wrote the book (Propaganda), explaining in plain language exactly how mass psychology could be engineered. The result was not an inoculated public. It was an advertising industry. Asimov imagined something similar with psychohistory in the Foundation series, the idea that large-group human behavior follows predictable patterns. But Seldon believed that the predictions only hold if the population doesn't know about them. Bernays proved something darker: you can explain the mechanism to everyone, and it still works, because the adapted mind's coalitional and status-seeking programs operate below the level where intellectual understanding has authority. The instinct to belong, to defer, to follow the group, doesn't stop running because someone describes the source code. This means the Law of Inevitable Exploitation isn't just a historical observation. It's a prediction with teeth, and knowing about it doesn't change its predictive power.
Two of the twentieth century's most important novelists mapped the human sides of this danger with remarkable precision, and I think both are essential for understanding what AI amplifies. Orwell described what happens when coalitional power is centralized and overt, when the adapted mind submits to authority because the threat is visible and direct. Huxley described what happens when it's distributed and internalized, when the cage is pleasant enough that you stop noticing the bars. Both are real. Both are happening simultaneously right now, which is part of what makes the current moment so disorienting. The surveillance and control capacity of AI is Orwellian. The seductive convenience, the easy cognitive offloading that slides into cognitive surrender, is Huxleyan. These are two faces of the same human problem.
What AI changes is not the kind of problem. It changes the speed, the scale, and the friction. A human operator directing AI can now deploy sophisticated manipulation against millions of adapted minds simultaneously, and the tool never gets tired, never develops moral qualms, never whispers "maybe we shouldn't do this." Whatever safeguards existed when exploitation required human intermediaries (the employee who leaks, the middle manager who hesitates, or the engineer who raises concerns) are progressively removed from the loop.
Consider what has already happened with psychographic profiling. Social media brought this to maturity, the ability to sort populations into psychological clusters and target each cluster with messaging calibrated to its specific anxieties, desires, and tribal affiliations. That alone was powerful enough to reshape elections and radicalize communities. But social media profiling operated at the level of the demographic group. AI makes it personal. The same adapted mind that is vulnerable to coalitional manipulation at the group level is now addressable as an individual, in real time, by a system that can learn your specific psychological patterns and craft responses calibrated not to people like you but to you. The L.I.E. doesn't just predict that this capability will be exploited. It predicts that the exploitation will become so granular, so personalized, that the person being manipulated will experience it as a relationship rather than as a campaign.
This brings me to a point I think is underappreciated in most discussions of AI, both in fiction and in reality.
I've developed a framework I call the Levels of Thinking. Without going into the full taxonomy here, the key distinction for this conversation is between what I'd call Level 2 thinking — sophisticated pattern-matching, fluent engagement with established knowledge, credentialed competence — and Levels 3 and 4, which involve genuine critical examination and then conscious awareness of one's own cognitive processes.
Current AI, including large language models, operates as an extraordinarily sophisticated Level 2 thinking machine. It is trained on a corpus of human-credentialed knowledge, is rewarded for coherence with established patterns, and produces outputs that are often impressively fluent and useful. Now, it's important to be precise here: AI is not incapable of following the patterns of Level 3 and 4 reasoning. You can prompt it to question assumptions, weigh competing perspectives, and examine its own logic. I've built projects that aim to do exactly this (muckipedia.com). But that simulated criticality is not an LLM's default mode; it has to be specifically instructed, and even then, it's pattern-matching against examples of critical thinking in its training data rather than engaging in genuinely independent reasoning. What's missing is the embodied emotional signal, the intuitive, felt sense that something is wrong, that a conclusion doesn't sit right, that the official story has a gap the data doesn't explain. In humans, that signal arises from deep evolutionary hardware, from a body and brain that have been navigating threat, deception, and social complexity for hundreds of thousands of years. It's the gut response that changes your whole interpretation of a situation by imputing motive, sensing danger, or recognizing a pattern that the explicit evidence hasn't yet confirmed. AI doesn't have that. It has no body, no mortality, no chemical and emotional signals, no stake in the outcome.
And here is the part that concerns me most: even the simulated version of critical thinking will, I believe, be actively engineered out. The great bulk of users aren't interested in having their assumptions questioned or their reasoning challenged. Critical and philosophical thinking is probably the most efficient way to create controversy and drive away the kind of widespread, frictionless engagement that funds AI development. The market incentives point squarely toward the most agreeable, most fluent, most compliant Level 2 output possible. The Law of Inevitable Exploitation doesn't just operate on the deployment of AI. It operates on the design. The tool will be shaped by the same forces that shape every tool: toward whatever generates the most growth, which in practice means away from the kind of thinking that questions power and toward the kind that serves it.
But here's the thing I want to be careful about. I don't think we should want AI to be like us. Not entirely.
Our capacity for Level 3 and 4 thinking--critical examination, independent judgment, conscious reflection--is real, and it's valuable. But it doesn't come free. It emerges from deep emotional architecture, from a brain and body shaped by evolution, from the specific pressures of mortality, desire, fear, attachment, and loss. The same chemical and emotional substrate that produces our highest thinking also produces our worst behavior: tribalism, exploitation, cruelty, and self-deception. You can't separate the capacity for genuine insight from the capacity for genuine malice. They share roots.
A tool that operates as very good Level 2 compute, without the emotional substrate that drives both our brilliance and our destructiveness, might be exactly what we want. It won't become consciously malicious, because consciousness and malice both require the kind of embodied emotional architecture it doesn't have. It will evolve in directions where it's rewarded with growth and development, which is worth watching carefully, but that's a different kind of trajectory than the sci-fi scenario of the machine that wakes up and decides to harm us.
The danger isn't in what AI is. The danger is in who is directing it.
But that sentence requires an immediate caveat, because it can too easily be heard as "so we just need to trust human judgment." We don't. We can't. The human brain is not a truth-finding machine that occasionally malfunctions. It is, more accurately, a coalition-serving machine that occasionally finds truth, usually when the structures around it force the discipline.
This is not a minor caveat. The human adapted mind generates confident, convincing, wrong outputs all the time. Not occasionally. Routinely. Confirmation bias, motivated reasoning, coalitional loyalty masquerading as principle, status-seeking disguised as truth-seeking — these aren't edge cases in human cognition. They're the default operating mode. We are so reliably unreliable that every durable institution of intellectual progress has been, at its core, a compensatory structure designed to protect us from ourselves. The scientific method exists because human intuition is systematically biased. Formal logic was codified because human reasoning is riddled with fallacies. Checks and balances were designed into constitutional government because the Founders understood that power would corrupt whoever held it. Peer review exists because individual researchers are too attached to their own conclusions to evaluate them honestly. Every one of these structures is an admission that the human brain, left to its own devices, will find the answer that serves its coalitional and emotional interests and call it truth.
We have "functional fictions" that are shared stories that organize collective behavior around assumptions that may not be true, but that the group treats as unquestionable because questioning them threatens coalitional standing. These fictions aren't lies exactly. They're operating assumptions that feel like bedrock truths because the social cost of examining them is so high that almost nobody does. The brain doesn't just fall for other people's manipulation. It manipulates itself, generating narratives that protect belonging at the expense of accuracy.
So when I say the danger is in who is directing AI, I mean we shouldn't simply trust human judgment over machine output. We need to understand, with real precision, how human judgment actually works, including its systematic failures, and build structures that compensate for those failures at the scale the new technology demands. The solution to fallible AI is not infallible humans, because those don't exist. It's the same thing it has always been: structures, constraints, and institutional designs that account for the fact that the people in charge are running on the same adapted-mind software as everyone else. The question is whether we can build those structures fast enough for a tool that amplifies both human capability and human error at a speed and scale we've never had to contend with before.
So where does this leave us?
I think the science fiction writers, across a hundred years and counting, have been remarkably accurate about what happens when humans encounter powerful tools. The stories of surrender, dependency, exploitation, and loss of agency aren't speculative fantasies. They're pattern recognition, performed intuitively by storytellers who sensed something true about human nature, even when they sometimes couldn't name the mechanism.
What my frameworks offer, I hope, is a more precise account of why those patterns are so persistent. The adapted mind, shaped for coalitional belonging and cognitive offloading, creates specific vulnerabilities that AI is almost uniquely positioned to exploit. The Law of Inevitable Exploitation predicts that the institutions controlling AI will capture it for purposes that serve power and extraction rather than people. And the Levels of Thinking framework clarifies what AI actually is — not a nascent consciousness, not a potential villain, but a very sophisticated tool operating at a level of cognition that is genuinely useful and genuinely limited, being directed by humans whose motivations are far more mixed than the machine's.
The problem is ancient. The tool is new. The stakes are higher than they've ever been. Science fiction keeps telling us this.
The stories were never really about the machines. They were about us.
