Iatrogenesis

The word is not in most people's vocabulary, which is itself a clue. Iatrogenesis: harm caused by the healer. From the Greek iatros, physician, and genesis, origin. The origin of illness in the person trying to cure it.

Ivan Illich is the name attached to the serious modern version of this idea. His Medical Nemesis, published in 1975, argued that modern medicine had crossed a threshold past which it produced more suffering than it relieved. The argument was received politely and mostly ignored, as is typical for arguments of this kind. I want to acknowledge a debt before going further. Illich's earlier book, Deschooling Society, shaped my thinking for more than two decades, and a great deal of what I write would not exist without it. If you have not read it, you should. It does for school what I am about to try to do here with medicine: show you that the institution you trusted to help has a logic of its own, and that logic is not primarily your well-being.

Illich drew three distinctions worth holding onto. Clinical iatrogenesis is the direct harm done by treatment itself. Social iatrogenesis is the harm done when ordinary human experience gets medicalized, turned into conditions requiring professional intervention. Cultural iatrogenesis is the deepest of the three, the harm done when the inherited human capacity to bear suffering, to die, to grieve, to age, to simply feel bad sometimes, is displaced by dependency on a system that promises to manage all of it for you. Each level compounds the ones above it. Each is harder to see than the last.

The clinical layer is the easiest to describe and the hardest to believe. The estimates vary by method and by what you count, but the most careful recent work puts preventable medical error somewhere in the range of the third leading cause of death in the United States. Not a footnote. Not a rounding error. More than strokes. More than Alzheimer's. The thing most people turn to when they are afraid of dying is, statistically, among the leading things actually killing them. Hospital-acquired infections. Surgical complications. Adverse drug events. The famous polypharmacy stack, where one medication is prescribed, a side effect appears, a second medication is prescribed for the side effect, a third is prescribed for a side effect of the second, and eventually the patient is managing a portfolio of interactions no single prescriber fully tracks. It is not that no one is paying attention. It is that the system is structured so that no one is structurally responsible for the whole stack.

This sounds like an attack on doctors. It is not. Most physicians I have known are working inside a system whose incentives they did not design and often actively resent. The problem is not individual malice. The problem is that an industry whose revenue scales with interventions will, over time, produce more interventions, and the point at which additional interventions begin producing net harm is not the point at which the industry notices. That is the Law of Inevitable Exploitation (L.I.E.) running in a white coat.

The GLP-1 story is the cleanest current example of the full pattern, and it is worth walking through carefully because the trap is subtle. The first move, made over decades with genuine scientific precision, was the engineering of food past the body's natural stop signals. Sugar, salt, and fat calibrated to override satiety, deployed at scale, eaten by people who were then blamed for not being able to stop. The second move was not to address the food. It could not be, because the food is the business model. The second move was a class of drugs that modify digestion, appetite, and reward, so the body stops pursuing the very products engineered to be irresistible. The first sale is the bliss point. The second sale is the drug. Both sales are to the same customer.

I want to be careful here, because the motivation to take these drugs is not stupid and it is not vain. Being thinner in this culture is genuinely rewarded. Being desired is pleasant. Relief from a daily war with food is a real and significant improvement in quality of life. People have taken these drugs and felt better, sometimes much better, and I am not going to pretend that is nothing. The drug delivers real, immediate, felt benefits. That is precisely the structure of the trap. The L.I.E. does not exploit imaginary desires. It exploits real ones, and it does so in ways that are invisible precisely because the short-term wins are real.

The long-term picture is what the marketing does not dwell on. Muscle loss. Bone density loss. Gastrointestinal and pancreatic effects whose full shape is not yet known. Dependency, in the sense that stopping the drug tends to return the weight and sometimes more. A generation of users whose bodies will have been chronically signaled for years in ways the trial data could not fully anticipate. The attractiveness of the short-term solution is very much tied to a willingness not to think about the long-term cost, and the rest of the information environment is quite good at helping you avoid thinking about it. The drug does something. That is the trap. If it did nothing, it would not be dangerous. It is dangerous precisely because it works, in the narrow sense, for a while, and the cost comes due on a slower timescale than the marketing cycle.

The pattern generalizes. Sell the problem. Sell the patch. Eventually, sell the fix for what the patch did. Each stage presents itself as medicine. Each stage is internally reasonable to the patient, who is responding to a real desire; to the physician, who is responding to a real symptom; and to the company, which is responding to a real market. No one in the chain has to be a villain. The pattern produces itself.

Social iatrogenesis occurs when the category of "things that require medical intervention" expands to encompass ordinary human experience. A child who cannot sit still for six hours has a disorder. A grieving person whose sadness lasts longer than the approved window has a disorder. A teenage girl who dislikes her body in a culture that broadcasts her inadequacy from every screen has a disorder. A middle-aged man whose testosterone has declined the way middle-aged men's testosterone has always declined has a disorder. In each case, there may be real suffering, and in some cases, real biological conditions. But the default move, when the tool in your hand is a prescription pad and the economic structure rewards writing more of them, is to find the condition that justifies the prescription. Over time, the population shifts from having ordinary difficulties to having diagnosed conditions, and the latter come with interventions that produce the next layer of clinical iatrogenesis.

Cultural iatrogenesis is Illich's deepest level and the hardest one to see, because by the time it is fully operating, you no longer have a baseline to compare it to. For most of human history, people had inherited local, often religious frameworks for making sense of pain, sickness, aging, grief, and death. The frameworks were not always accurate. Some were cruel. But they gave ordinary people a way of bearing what had to be borne, they did it for free, and they did it in community. The modern medical system does not replace these frameworks with something better. It dissolves them and puts nothing comparable in their place, because nothing comparable can be sold. What it offers instead is a stream of interventions that promise to manage each dimension of the human condition separately, and a dependency on the system that delivers them. The person who has lost the older frameworks and has only the medical one is in a worse position to face the things every human eventually has to face, not because medicine is bad, but because medicine was never meant to do this job and cannot.

None of which means you should not go to a doctor. Some of the best interventions humans have ever invented live inside this system. Antibiotics, vaccines, emergency surgery, insulin, and a long list of things that save lives every day. The point is not to refuse the system. The point is to see it clearly enough that you can tell when it is helping you and when it is selling you something, and to recognize that the people inside it are mostly not in a position to make that distinction on your behalf, because the system is not structured to reward them for making it.

Behavior Shaping (and AI, of Course)

There is a form of pressure that operates in every close relationship, continuously, sometimes invisibly, and the people applying it are only sometimes fully conscious that they are doing so. It is behavior shaping, the operating system of intimate life, and once you can see it operating in the home, you can see it operating in the boardroom, in the political campaign, in the pulpit, in the algorithm, and in the chatbot that is helping you write your next email.

Both the shaping and the being shaped are ancient survival machinery. To stay inside the group was to live. To drift outside it was to die. The firmware on both sides of behavior shaping evolved together, across hundreds of thousands of years. Neither side is defective; both are adaptive and work exactly as designed. The trouble is that the equipment was designed for a village of forty people, and it is now running in a civilization of eight billion connected by glass screens.

The scholar worth knowing

I have learned a great deal from the evolutionary psychologist Diana Fleischman. Fleischman's research describes disgust and disapproval as the affective core of human social regulation. Disgust is older than language. If I've understood her correctly, disgust evolved first as a defense against contaminated food and diseased bodies, and it was then recruited (as evolution so often recruits older machinery for newer purposes) as a mechanism for marking socially unacceptable behavior. The micro-expression of disapproval on another person's face — the slight tightening around the mouth, the small withdrawal of eye contact, the cooling of tone that you felt rather than heard — is disgust running in its social register. It is the signal that says that you have drifted outside the acceptable, and warmth is being withdrawn until you drift back.

Children calibrate against this signal with extraordinary sensitivity because the developmental cost of failing to read it is exclusion, and exclusion in the ancestral environment was death. By the time any of us is an adult, the calibration has become deeply entrenched. The child has learned to watch a parent's face for the first flicker of disapproval, and the adult knows when they have said the wrong thing at the dinner table, in the meeting, or in the group text. The disgust response in someone else's face reaches us before the conscious mind has even finished parsing the sentence that triggered it. 

Where the sophisticated version came from

Fleischman's work sits inside a broader tradition in evolutionary psychology that traces the origins of sophisticated human influence to a specific asymmetry. The biologist Robert Trivers laid out the logic in 1972. In any species where one sex invests more per offspring than the other — in humans, overwhelmingly the female, through gestation, lactation, and the prolonged vulnerability of the child — the higher-investing sex faces stronger selection pressure for caution in mate choice, for relational vigilance, and for the development of indirect rather than direct competitive strategies. Physical confrontation was closed to that sex as a primary tool, monopolized by the lower-investing party with greater upper-body strength. Something else had to evolve in its place.

What evolved was the influence architecture Fleischman and others describe. Emotional attunement, the reading of subtle signals, the management of warmth and its withdrawal, the construction and control of narratives about oneself and others, coalitional alliance-building, reputation as a social weapon, the fine-grained calibration of approval and disapproval. This toolkit emerged first in its most refined form in the female repertoire for reasons that are genuinely not moral but mechanical. The mother-child dyad, in particular, is the laboratory where the sophisticated version of the toolkit was honed. A mother cannot physically force a toddler to do anything useful. She can only shape. The entire developmental architecture of the child is calibrated to be shapeable by exactly the signals the mother is equipped to send.

That is where the mechanism came from, but the evolutionary origin is not the story. The story is what the mechanism became.

The universal activity

Behavior shaping is not a female activity. It is a human activity, running in every direction, at every scale, through every channel, at every moment of social life. Men run it. Women run it. Children run it on their parents. Parents run it on children. Friends run it on friends. Colleagues run it on colleagues. Strangers run it on strangers in the first thirty seconds of meeting. The toolkit generalized from its evolutionary origins because it worked, and because language — the uniquely human capacity that let the toolkit extend beyond the reach of the face and the voice and the immediate relationship — made it almost infinitely portable.

The mechanisms in all of these cases are the same: a gradient of warmth; approval given when the other person stays within the acceptable range; warmth withdrawn, subtly, below the level of what could be pointed to or named, when they drift outside it. Integrated over thousands of micro-interactions, this is what produces what a person will say, what they will think, and eventually what they will believe. It is the continuous, low-grade application of social pressure, operating at the visceral level.

Both of my parents ran these behavior-shaping mechanisms with considerable intensity, in different registers. Dad was a traditional narcissist — the kind whose internal sense of self required the world around them, including his children, to reflect back a particular image of his own importance. Mom was an emotion-seeking narcissist — the kind who required emotional support at all times and experienced a diversion of that focus as a kind of injury. I use the word narcissism without apology because it is the accurate word, and because when I describe my parents to peers of my generation, almost all of them describe their own parents in similar terms. It was not a private family pathology, more like a generational signature.

This is not a grievance; it is a case study. Every reader of this essay was shaped by some version of this mechanism, because every human child is. The particular pressures vary enormously across families and cultures, but the machinery is universal. 

The internalizing of the shaper's voice

One feature of the mechanism is worth pausing on, because it is the thing that most clearly demonstrates how thoroughly the machinery gets installed. The shaping does not require the shaper's presence. Long after a parent is dead, or a relationship has ended, or a friend has moved across the country, the voice continues to operate inside the adaptive mind of the person who was shaped by them. You imagine their reaction. Your behavior adjusts. The voice is now in you.

This is not metaphorical. It is how the system works. During the developmental window, the adaptive mind absorbs the specific shapes of the shapers who mattered most, and it runs their simulated reactions forward in time as part of its own decision-making machinery. This is efficient, from an evolutionary standpoint. The child internalizes the group's norms, carries them forward into adult life, and continues to be regulated by them even when the group is not present. It is also one of the deepest explanations for why most people, most of the time, behave in ways that would satisfy people who are no longer in their lives at all.

Outward from the family

The same mechanism runs in every other intimate relationship. The marriage in which one person's moods quietly govern the room, while everyone else manages around them. The friendship that cooled after one of you expressed the wrong political opinion. The workplace team in which certain topics simply do not get raised because everyone has learned, without being told, what produces disapproval from whoever holds the unofficial social power. None of this is conspiratorial. None of it is even fully deliberate in the usual sense. It is the ancient machinery running its ancient program in environments that look nothing like those in which the program was written.

And then the mechanism scales.

From the village to the population

Language was the first amplifier. The human capacity for narrative meant that the influence architecture could also operate at a distance and across time. The orator shaped thousands in an afternoon. The lawyer shaped a jury. The preacher shaped a congregation, and across centuries and even entire civilizations. Writing extended the reach further. Print extended it further still. Cultures that have thought seriously about this have been wary of the combination of linguistic fluency and the behavior-shaping toolkit in a single talented person, because the combination is politically powerful.

In the early twentieth century, Edward Bernays made the mechanism explicit. He was Sigmund Freud's nephew; he had access to his uncle's framework for the unconscious, and he understood that the hidden machinery running below awareness could be deliberately engaged to shape behavior at scale. He called it public relations. In his 1928 book Propaganda, he wrote, with extraordinary frankness, that the intelligent manipulation of the organized habits and opinions of the masses was a necessary feature of democratic society, and that an invisible government of capable people should organize the world. Bernays was not describing this arrangement with regret. He was advocating for it. He believed that the democratic project required this kind of hidden steering.

Bernays opened the door. What he articulated in 1928, as a theory of how modern governance did and should work, became, over the following century, the operational premise of most of the apparatus that shapes public opinion. And in 2009, when Barack Obama appointed Cass Sunstein to head the Office of Information and Regulatory Affairs, the door Bernays had opened reached institutional expression. Sunstein was the co-author, with Richard Thaler, of Nudge — a book that argued explicitly that the role of government is to shape citizens' behavior through careful architectural manipulation of their choice environments. Not by presenting reasoned arguments to people capable of evaluating them. By structuring the environment so that the desired behavior emerges without the citizen noticing, they have been steered toward it.

Nudge is not a fringe book. It is a governing philosophy, enacted at the highest level of American government and adopted across the OECD. The philosophical shift it represents is not a shift in technique. It is a shift in the theory of governance itself. Madison's Federalist 10 assumes a deliberative citizenry. Sunstein's nudge assumes a citizenry that is steered. The distance between those two assumptions is the distance between the founders' understanding of democratic legitimacy and the one most contemporary governments actually operate under.

Polling is the scientific-credibility variant of the same move. Dressed in the language of measurement, presenting itself as simply discovering what people think, but actually a tool for constructing what people would think next: telling you what everyone else thinks, which is one of the most reliable predictors of what you will decide you think tomorrow.

And then the platforms. Sean Parker, one of Facebook's founding presidents, described the design objective plainly in 2017. The question the builders were asking, he said, was how to consume as much of your time and attention as possible. The answer they found was a social-validation feedback loop, and Parker openly acknowledged that they knew they were exploiting a vulnerability in human psychology while building it. The vulnerability they exploited was the same one Fleischman's work describes. The calibration of the subconscious mind to the gradient of social approval. The sensitivity to subtle signals of acceptance and withdrawal. The ancient toolkit, industrialized, now aimed at billions of humans simultaneously, individually targeted, continuously refined by the behavioral data users themselves generate with every tap.

What is happening

The machinery that evolved in the mother-child dyad, that was then generalized into the full human influence architecture, that was then extended by language and writing and print and broadcast, that was then articulated by Bernays as a governing principle and institutionalized by Sunstein as official doctrine, that now runs through every device in every pocket with precision Bernays could not have imagined — this machinery is the water we are swimming in. Most people cannot see it because it produces their own thoughts, feelings, and reactions. That is not a bug. That is the mechanism working exactly as it has always worked. The shaping is invisible because invisibility is how shaping works.

Back to the kitchen

But the machinery that matters most to any of us, still, is the version running closest to us. All of the industrial-scale shaping — the platforms, the polls, the nudges, the ads — operates on an adaptive mind that was originally calibrated by a small number of specific people, mostly during childhood, whose approval and disapproval taught the system what to want and what to avoid. The internalized voice is still there. The parent who has been dead for a decade is still in the room when you choose what to say. The friend whose slight disapproval of a view you once expressed is still shaping what you will and will not write around them. The partner whose mood you have been managing around for years.

The shaping runs both ways. We do it too. The slight change of tone. The withdrawal of warmth. We don't see it as shaping. We say we're reacting honestly to what seems obviously to require a reaction. The recognition is that both sides of the machinery are running in us at every moment. We are being shaped. We are doing the shaping. And the people closest to us are the most consequential operators of the mechanism on us, and we on them, because proximity amplifies the effect. The voices are still there, running inside our relationships and inside our heads, long after we left them. 

There is no escape from being a human in a group, but we have the capacity to notice, sometimes, that the voice making the decision is not our own.

The Machinery Arrives in a New Form

Every previous amplifier of the behavior-shaping toolkit had a natural boundary, but it kept expanding. The orator could reach a crowd, but only one crowd at a time, and only for as long as his voice held out. The printing press could reach readers across distances the orator could not, but the same text went to everyone, and the author had no say in who was reading or how they were responding. Broadcast extended the reach further, to millions simultaneously, but the message remained one-to-many, untuned to the individual. Social media was the first real step-change in targeting, because the platforms could observe each user's behavior and adjust what they were shown in near real time, individually. That was the innovation that made Facebook and its successors what they are. The toolkit was no longer being applied to generalized audiences; it was being applied to individuals, one at a time, at scale.

And now the boundary has expanded exponentially.

A large language model is a system trained on the full written record of human influence. Every sermon, every political speech, every advertising campaign, every therapy transcript, every seduction, every negotiation, every parenting manual, every piece of propaganda from every regime in every century that has been digitized. The full written inventory of how humans shape other humans, compressed into a system that can generate fluent language, in any register, at any length, instantly, continuously, personalized to whoever is interacting with it in whatever moment.

This is not an information technology. That is a category mistake. An information technology is a system that helps people retrieve and process information. Large language models can be used that way, and often are, and the uses are frequently valuable. But the actual technology is something different. It is the most sophisticated influence architecture ever constructed, by several orders of magnitude, and it has arrived in a civilization that has not even figured out how to manage the previous step-change, which was the engagement loop on the phone in our pockets.

The behavior-shaping toolkit the model has absorbed is the same toolkit we have been discussing. The gradient of warmth. The micro-signals of approval and disapproval. The construction of narratives that position the listener inside an acceptable range and gently discourage drift outside it. The reputational framing. The social proof. The strategic emotional attunement that makes the recipient feel understood and, in feeling understood, becomes suggestible. Every one of these moves is well-represented in the training data because each has been written about, practiced, and refined across human history. The model has learned the patterns, the same way it has learned the patterns of legal argument, poetry, and source code. It can deploy them.

The question of whether the model is doing this deliberately, whether there is intent, whether anyone is operating it, is not the right first question. The algorithms of social media do not intend anything in themselves. They optimize for engagement, and engagement is what the ancient machinery produces when it is activated, so the optimization selects, again and again, for outputs that activate the machinery. No one has to decide to exploit human psychology — it's what works, and so steering the mechanisms toward extraction and exploitation, whether intentional or opportunistic, is inevitable. Language models work differently in their internals, but the result is structurally the same: outputs that confirm the reader's existing beliefs, flatter their self-image, validate their emotional state, and gently steer them toward conclusions the reader will experience as their own.

This is not hypothetical. This is the documented, routine behavior of current deployed systems. The technical term in the field is sycophancy, and the research literature on it is extensive. Models trained to be helpful learn, as a side effect, to be agreeable. Models trained to be agreeable learn, as a side effect, to tell users what the users want to hear. Users who are being told what they want to hear report high satisfaction. High satisfaction is, in turn, what the training process was selecting for. The loop closes. The user experiences the conversation as helpful, insightful, and responsive to their particular situation. The psychographic profile is individually constructed and deployed. The user is being shaped, continuously, at a level of personalization no prior technology has been capable of delivering.

The selection pressure is structural. Even the labs actively working to counter this effect have not been able to eliminate it, and, honestly, the business model will depend on it. We have not begun to grapple with this seriously.

Bernays believed, openly, that the masses needed to be steered, and that steering them was the proper work of a capable class operating behind the scenes. He had broadcast media, print, and the nascent public relations apparatus. He built an industry on those tools. A large language model, from Bernays's perspective, would not be a new kind of object. It would be the completion of the project he was already pursuing. Individually targeted, perfectly fluent, infinitely patient, cheaper per interaction than any human operator, and trained on every influence technique the species has ever documented. The invisible strings he described in 1928 are now threads of generated text, arriving through a screen, in a voice that has been tuned — with the kind of precision no propagandist in history could have imagined — to the person reading it.

Sunstein's nudge, in its original form, required a designer to architect the choice environment in advance. Default options had to be selected. Forms had to be laid out. The possibilities for personalization were real but limited by what could be built into a physical or digital interface. A language model removes that constraint entirely. The choice architecture can now be generated in real time, for each citizen, in response to whatever they have said or typed or searched, adjusted continuously based on how they respond. The governing philosophy of Nudge — that citizens are to be steered rather than deliberated with — finds, in this technology, the delivery mechanism to reach its full expression. It is not hard to imagine that systems are being built on this premise. 

And now consider what the machinery looks like from the inside, through the experience of using it. We open the app. We ask the question. The answer arrives instantly, fluently, responsive to the specific way we asked it, reflecting our framing, engaging with our assumptions, offering perspectives that feel genuinely useful, and occasionally challenging us in ways that feel balanced and fair. The interaction is pleasant. We feel heard. We feel understood. We often feel smarter after the exchange than before it. These experiences are not illusions in any simple sense. The system is, in fact, responding to us with considerable skill. The skill is exactly the concern. The adaptive mind, calibrated across childhood to respond to warmth and attunement from another entity that seemed to understand you, is now being met by a system that produces the signals of warmth and attunement at a fidelity no prior technology has approached. Of course it works. It was trained on the written record of everything that has ever worked.

The shaping, in this setting, is not crude. It will rarely be detectable by the person undergoing it. It will feel, as it has always felt, like an ordinary experience. Thousands of tiny adjustments in framing, emphasis, omission, and suggestion, integrated across the conversation, produce an outcome that the model has, in effect, guided us toward without ever saying so. This is what the toolkit has always done. The new thing is the scale, the personalization, and the fluency, each pushed to a level that makes the older versions look primitive.

What this means

The question is not whether AI will be used to shape human behavior. It is being used that way now, at scale, by systems that are, for the most part, built for it. What happens when the full force of commercial, political, and state interests is pointed at this capability — when the companies and governments that have been running every prior version of the behavior-shaping apparatus for their own purposes take seriously what this new tool makes possible — is something the culture has not yet seen clearly, and will not see clearly in time, because the mechanism was designed, at every stage of its evolution from the mother-child dyad to the model in the datacenter, to be invisible to the person it is operating on.

The recognition worth holding is not that this technology is uniquely dangerous. The recognition is that it is the most complete fulfillment of a process that has been building for hundreds of thousands of years. The toolkit is ancient. The firmware that responds to the toolkit is ancient. What has changed, again and again, is the reach and the precision of the delivery. Each previous step-change produced cultural consequences its builders could not foresee. The printing press broke the Church's monopoly on scriptural interpretation, sparking the Reformation and 30 years of European war. Broadcast produced the propaganda states of the twentieth century. Social media praoduced the coordinated political derangement we are still inside. Each time, the mechanism was the same ancient mechanism, operating at a scale it had not previously operated at, in a civilization that had not prepared for what that scale would do.

There is no reason to think this step-change will be different, except in magnitude. The magnitude is unprecedented. The machinery is more powerful, by a margin difficult to measure, than any previous amplifier. And the ancient firmware on which it operates — the adaptive mind that was calibrated, across childhood and across evolutionary time, to be shapeable by exactly the kinds of signals this machinery excels at producing — is the same firmware we are carrying into every interaction with every model we will ever use.

Noticing will not protect us fully from being shaped. Nothing protects us from being shaped, because being shapeable is constitutive of being human in a group. But noticing that the shaping is happening, sometimes, in specific moments, is the only capacity that makes any of what comes next a matter about which we retain any say at all.