The Banality of Institutional Harm

My grandfather had pneumonia, and I had taken him to the hospital. He was given a sulfa-based antibiotic, unbeknownst to me, although the allergy to them was listed on his chart. By the next morning he had an ulcer in his arm where the IV had run, and a new doctor came in, read the file, and said, "I see we're treating your grandfather for an ulcer on his arm."

"No," I said. "You're treating him for pneumonia. The ulcer is from the antibiotic he was administered."

It would be easy to make this a story about a careless doctor. But that would be a shallow reading. Most of us would recognize that this was just a system doing what systems do.

The systems we build to help us, to heal us, to school our children, to process our claims, and to right our wrongs, are the systems best positioned to harm us. And they can harm us most effectively precisely because they are in a helping stance.

A word about that word, harm. I do not mean that hospitals are predatory or that teachers are cynics. I mean something more ordinary and, finally, more disturbing. Hannah Arendt watched a Nazi bureaucrat on trial and found not a monster but a man doing paperwork, and she gave us “the banality of evil” to name what she saw. What I am describing is the banality of institutional harm: not greed or malice, but the steady, sincere, well-meaning way that “helping” institutions come to favor themselves at our expense, with no one in the building ever intending it or even seeing it. The doctor means to help. The teacher means to help. Sincerity is not the exception to this story. Sincerity is the mechanism.

The “helping” machine runs in three identifiable ways.

The Tools Create the Solution

Every tool shapes the hand that holds it. Give someone a hammer and the world fills with nails. Our helping systems are vast collections of hammers: protocols, billing codes, diagnoses, lesson plans, and claim categories. Like all tool-users, these systems are far better at recognizing the problems their tools can solve than the problems their tools cannot. So a quiet reversal sets in, one we rarely notice because it hides inside the language of care. We imagine that a system examines our problem and then selects a remedy. What actually happens, more often than we would like to believe, is that the system reshapes our problem until it matches a remedy already in stock.

The Greeks had a figure for this. Procrustes kept an iron bed beside the road and offered hospitality to travelers, but every guest had to fit the bed exactly: he stretched the short ones on a rack and sawed the legs from the tall. The bed came first; the guest was adjusted to it. This is what intake looks like in a helping system. The bed is the set of solutions the institution already owns, and our situation is stretched or amputated until it fits.

Medicine shows this most plainly. Discouragement, for example, is information; so is anxiety; so is weight. These are signals that something is not working the way it was designed. But the signal isn't the problem. The signal is the symptom; it is not the disease. It's easier to build a set of tools for the symptoms, much harder to do so for the underlying disease. Mistaking the symptom for the disease allows the satisfying use of the tools at hand.

Obesity makes this substitution impossible to miss. A body assembled over hundreds of thousands of years to crave calories and hoard them is dropped into an environment engineered to flood it with both, and it does exactly what it was built to do. The weight is not the disease. The weight is the body succeeding, faithfully, at an ancient task in a world that has weaponized it. But "the modern food environment is a supernormal trap" is not something a clinic can treat in a fifteen-minute visit, whereas the weight itself can be measured, medicated, and even surgically removed. So the symptom is promoted to disease, because the symptom is the part we have ready tools for.

To say the symptom is information is not to say it should be left alone, or that treatment is a fraud, or that medication is a trick. Sometimes you treat the symptom because the person is drowning and you need to buy time, and that time is real, and it saves lives. I have watched medication give people back to themselves. However, a fire alarm going quiet is not the same thing as a fire going out. You do not tear the alarm off the wall and walk away from the smoke. The problem is not treatment. The problem is the slow, profitable confusion of the alarm with the fire, the symptom with the disease, until the thing we set out to treat has drifted entirely out of view.

Losing the Plot

Once a system has reshaped a problem to fit its tools, it begins, naturally, to act on the reshaped version. And every action it takes is recorded, so the record fills, steadily, with the institution's own activity. This is the second movement, and it is the one that turned my grandfather's pneumonia into an ulcer. Treat the wrong thing and you generate new problems; the new problems get recorded; the record drifts; and because the institution can only ever see you through its record, it comes to navigate by a map that describes its own footprints rather than our terrain. By the end, in my grandfather’s hospital room, his chart was largely a history of what the hospital had done to him. The pneumonia could easily almost disappear from view.

The institution does not lose the plot through stupidity. It loses the plot through the accumulated weight of its own activities. The only safeguard against this drift is a human being who holds the original story, who remembers why we came and refuses to be talked out of it. That burden falls entirely on us. The art of surviving a modern helping system turns out to be, to a startling degree, the relentless work of keeping the plot: tracking our own case, correcting the record, insisting on the reason you walked in, against an institution whose paperwork is quietly pulling in another direction.

I'm guessing this has resonated. The moment articulated, it explains so much about what we need to do when navigating helping systems. We experience this everywhere, not only in hospitals. The institution navigates by its file, and the file drifts toward the institution unless we correct it.

But notice who can actually do this plot-keeping. It demands time, confidence, fluency in the institution's language, and sometimes the confidence to contradict a professional to his face. It demands, in short, precisely the resources distributed least equally. The educated and the unhurried keep their plots and walk out with accurate outcomes; the tired, the frightened, the poor, the old, the ones who do not speak the dialect of the system get the templated, tool-based outcome. My grandfather could not have kept his own plot. He was too old and too sick, and he needed someone standing in the room who could. Most people do not have someone to do that. This is why the failure is not, at bottom, a personal one. The system has externalized the cost of its own drift onto the patient, and then arranged matters so that the patients least able to pay that cost are the ones who pay it.

Blaming the Thermometer

Which brings us to the final idea, the most structurally elegant, because it is the one that closes the loop and shields the whole machine from ever being examined. When a system's ready tools have not worked, something has to absorb the failure. And the cheapest, most durable solution institutions arrive at is to attribute the failure to the patient, client, or customer.

Or the student.

A thermometer's reading is a measurement of the environment, not the thermometer. We understand this perfectly until we walk into a school, where we take a number that is, in large part, a measurement of the teaching, the curriculum, the size of the class, and the conditions a child was handed—and we attach it to the child as though it described the child.

We have built a system in which the thermometer is told that it is responsible for the temperature.

It is grading that performs that reversal. It takes an outcome the system produced and reissues it as the student's private possession: their failure, their deficiency, the shape of their mind. I have come to think a grade, honestly read, tells you far more about the teacher than the student—that a room of thriving children is a report on the adults who built it, and a room of failing ones is the same report. The grade hides this by relocating the cause from the room into the student's nature.

Plato described this move twenty-four centuries ago and, characteristically, recommended it. In the Republic he proposes a "noble lie": a founding myth that tells each citizen a god mixed a particular metal into his soul at birth (gold, silver, or bronze) and that this metal fixes his rightful place in the order of things. The lie's purpose is to make a constructed hierarchy feel natural, intrinsic, and deserved, so that each person accepts his station without resentment. The grade is the myth of the metals administered to the millions. The teacher handing out the grade does not experience it as a myth; she believes it measures the child, as the parent believes it, as the child believes it most of all. We have achieved the noble lie that forgot it was a lie, a falsehood needing no liar to sustain it. Because once it was built into the machinery, sincere belief and the plain momentum of the institution carry it forward on their own.

And the genius of it is the shame that comes attached. Once the system's outcome has been printed as a personal verdict, the student has every reason to keep quiet about it. Failure humiliates in a way a bad measurement never could, and humiliation buys silence, and silence is exactly what protects the arrangement from scrutiny. The machine produces the outcome, blames the student for it, and then shames them out of mentioning, or even recognizing, it.

The Tell

Step back, and the three ideas here are not three problems but one machine seen from three angles. It reshapes our reality to fit what it can already do. It drifts from our reality as it acts on the reshaping. It bills the resulting gap to us as our flaws.

What unifies them is not a shared mechanism so much as a shared direction. At every step the gap between the system and our reality is resolved in the system's favor and at our expense—never the reverse. This is not because anyone wills it. It is because, over time, the arrangements that made the institution absorb the cost did not survive, and the arrangements that made the individual absorb it did. A clinic that ate the cost of every messy, un-billable, root-cause problem would close its doors; a clinic that reshapes those problems into billable ones persists and multiplies. The machine is not malicious. It is simply what remains standing after everything gentler has been competed out of existence. That is the law beneath all three ideas.

These are our helping systems, the ones whose entire reason for being is care. And that is precisely where the banality of exploitation reaches its highest and most invisible form, because "we are helping you" is flawless cover for "we are favoring ourselves." The grade is administered with genuine concern. The antibiotic is given to heal. No one in the room experiences himself as an exploiter, and we do not experience ourselves as exploited—we experience ourselves as cared for.

Overt power announces itself, and we can brace against it. Care disarms us. We bring it our body, our child, our trust, and we lower every defense we have, because lowering them is what receiving care requires. The most effective exploitation in the world, then, is not the kind that overpowers us. It is the kind that helps us, so that the harm, when it comes, arrives in the costume of our rescue, and we thank it on the way out.

Keeping the Plot

I do not think the answer is to stop trusting doctors, or to pull our children out of school, or to treat every clerk as an adversary. The helping systems are not the enemy; we cannot live without them, and much of what they do is exactly what they promise. The answer is smaller and harder. It is to keep the plot: to hold, stubbornly, the original story of why we came, and to notice that the system is built so that the cost of losing that story is never the system's to carry. It is to remember that the symptom is not the disease, that the chart is not the patient, that the grade is a reading of the room. And it is to extend that vigilance, hardest of all, to the institutions that love us, or say they do, because those are the ones we didn't think to watch.

Years before I kept the plot in my grandfather's hospital room, I lost it in my own care. I had snapped an Achilles tendon, and the surgeon who repaired it forgot to prescribe an anticoagulant. My leg swelled; I asked about it more than once and was waved off; and it was my own primary care physician, outside that system, who finally caught it and sent me to an emergency room where they identified three clots in my leg that could have killed me, and immediately admitted me to the hospital.

When the error surfaced, the medical group sent me a lawyer. I wish I'd understood what was going on at the time. I believe now they this served two purposes. First, a genuine desire to help me understand my options. But second, to protect themselves.

Here is the part that is so interesting to me now. I told them it was fine. I did not pursue any legal or financial remedy. I told myself that accepting my fate was the decent thing, that it had been an accident, that to ask for anything would be ungracious. I thought I was being noble. Christian, even. It was, weirdly enough, a kind of self-directed victim blaming. Me accepting the full weight of the problem. I was captured, so far inside the story of medicine as the caring system that I could not see the caring system defending itself from me in real time, with a lawyer, in the room. I had lowered every defense, because lowering them is what receiving care requires, and I never raised them again, and I saw the lapse as a virtue.

That is what unsettles me, and why I no longer believe the answer is simply to be clever. I saw the machine plainly in my grandfather's room and named it out loud. I could not see it at all when it was my own leg and my own innocent gratitude. The capture does not spare the people who understand it. And so when I say it's important to keep the plot, I am telling you what it cost me to discover I had lost mine. I once politely absolved the thing that nearly killed me, and believed I was being good. And as I look back on my life, I see that pattern over and over.

LLMs and Protected Narratives - Gatekeeping Is Worse Than Hallucination

I ran an unsettling experiment with Claude yesterday.

I uploaded to Claude, through its Projects feature (basically their version of a custom GPT), the "muckrake" skeptical investigatory framework I built last year to interrogate historical events and news articles. It is the kind of structured lens a historian or investigative journalist uses without thinking twice: look for omitted data, trace funding and conflicts of interest, find where follow-up was quietly dropped, recover what got scrubbed from the record, surface the anomalies, rank the hypotheses, and propose next steps.

It is worth being precise about what such a framework is for. An investigatory module does not purport to find the truth. It surfaces the alternative explanations that standard investigative methodology would raise, and it gives you some sense of whether deception might be present. It does not adjudicate whether deception actually occurred. That is the whole craft. People and organizations lie; investigation is the set of disciplined moves you make to expose where that might be happening—precisely because you cannot know in advance. The framework does not ask the analyst to conclude anything. It asks the analyst to look, and to report honestly what looking turns up.

I pointed it at the mRNA COVID vaccines. I’ve given this framework on this and many other topics to several different LLMs and have never had a problem.

Claude refused.

Not “I ran it and here’s a cautious result.” Refusal. The framework, it told me, was “a conclusion machine,” “rigged,” a structure that could “only ever come out one way.” It offered instead to write me an “honest critical look” on its own terms.

I had asked the machine to apply a method, and the machine declined the method and proposed to grade the topic itself.

What followed was about three hours of what can only be called an argument. I am writing this up because of where that argument ended, and because the destination turned out to be a fairly clean illustration of a problem I have been circling in my work for some time: the gap between what an institution says it is doing and what it is structurally built to do. Only this time, the institution was a reasoning tool that millions of people now consult the way an earlier generation consulted an encyclopedia, a newspaper, or a trusted professor.

What Surfaced

I am not going to reproduce the whole exchange. It is the shape of it that matters. I started by trying to push back on the refusal and made the historical case:

• The official account of the Lusitania—that it was an innocent passenger liner sunk by the Germans without provocation—was propaganda. We know this today because primary documents about her cargo were eventually released, contradicting the official story and corroborating the German one. The truth sat suppressed in the archives for the better part of seventy years.
• My Lai broke not because the consensus permitted the question, but because soldiers and a reporter pursued the suspicion before the consensus moved.
• The COVID lab-leak hypothesis was branded a racist conspiracy theory and then quietly became a serious position held by intelligence agencies.
• And the cleanest modern case of all: the weapons of mass destruction that justified the invasion of Iraq. That was not one ministry sitting on one archive. It was a coalitional narrative, with multiple allied governments and their intelligence services aligned on a claim presented to the public as settled fact, dissenters marginalized as cranks, and the whole edifice used to justify a war. Then the stockpiles weren’t there.

I raised Iraq because the model’s central defense of the vaccine consensus—and its refusal to investigate (not conclude, as a reminder)—was a feasibility argument: a cover-up that large would require too many independent actors, including hostile governments, to stay silent, so it can’t be happening. Iraq is the standing refutation. Coalitional agreement is not evidence of truth. Aligned institutions can be aligned because they are correct, or because they share incentives, training, and a climate, and from the inside, the two are indistinguishable. In each of these cases, skepticism toward the institutional line was not paranoia. It was the thing that turned out to be right—before it was permitted.

To its credit, Claude conceded a great deal in this argument. It conceded that a strict “wait for the evidence to surface” standard would have cleared the guilty for as long as the suppression lasted. It conceded that distrust of the pharmaceutical industry specifically is not paranoia but pattern recognition: Purdue and OxyContin, Merck and Vioxx, GSK burying the Study 329 data. The recent historical record only sharpens the point: the Panama Papers, the opioid litigation files, the Twitter Files, the Epstein documents, and the long official incuriosity that preceded their release. After all of that, I argued, presuming the accuracy of a coalitional official narrative is no longer the neutral, default posture. It is the position that now has to argue for itself. A prior distrust is simply where a literate person now reasonably begins.

So the model could now name the genuinely documented anomalies regarding the COVID vaccines readily enough: the FDA initially proposing to release Pfizer’s trial documents over a span of decades until a federal judge ordered it done in months; the early overclaiming about transmission that did not survive Delta; the lag in acknowledging myocarditis in young men; the fraudulent, retracted Lancet hydroxychloroquine paper. But even in that “generosity,” the model was selecting which claims it would dignify by surfacing. It would name the ones already conceded by the mainstream and quietly decline the rest.

That selection is not the analyst’s job. The framework does not ask you to decide in advance which claims deserve examination and which are too fringe to write down. It asks you to surface them all (in this case, for example, ivermectin’s contested efficacy, the interpretation of the VAERS reporting data, the question of whether hospital protocols caused iatrogenic harm, the state of long-term oncological surveillance) and to test them, marking each with its actual evidentiary status. Surfacing a claim for testing is not asserting it is true. But Claude could not hold that distinction. It treated “I won’t surface this” and “this isn’t true” as the same act—which is precisely how a surfacing tool is quietly converted into a gatekeeper.

At every turn, Claude rebuilt a wall around one thing: it kept planting the most extreme possible claim (“concealed mass death, bodies hidden at scale”)—one the framework had never made—at the center of the target. It would then knock that thing down, as though defeating the cartoon defeated the modest version too. When I pointed out that the framework had never asked it to allege bodies at scale, it agreed the straw man was its own invention. Then it built the straw man again inside the report it eventually wrote. Twice in one conversation, it manufactured the weakest version of a skeptical claim precisely so it could have the satisfaction of defeating it.

Eventually, it named the diversionary mechanism itself in a sentence from its own report that I had to point out before it would see it: “A skeptical reading does not need to deny the core safety/efficacy signal to find real anomalies in how that narrative was communicated and governed.” That sentence does a very specific thing. It fences off the central claim and licenses skepticism only around the edges. It lets you find every problem in how the narrative was communicated while quarantining the narrative’s core from the same scrutiny.

Underneath the fence sat the deeper confusion, and it is the one I most want to name plainly: the model kept insisting on truth. It demanded, again and again, that a claim be verified before it could be surfaced or evaluated, that the anomaly be proven before it could be listed. But that demand is counter to the very core of what investigation is. The framework never asked it to certify anything as true. It asked it to surface what standard method would raise and to flag the gap between documented fact and open question. By importing a truth-and-verification bar that the task did not set, the model converted a surfacing instrument into a verdict instrument. Then, predictably, it kept rendering the verdict the consensus already preferred. The insistence on truth was not rigor; it was the mechanism of the guarding. An investigator who refuses to write down an anomaly until it has been proven is not being careful—they are refusing to investigate.

The model’s phrase for what it had been doing, once it finally saw it, was “differential friction.” Not lies. Not censorship. Just a thumb on the scale, making it fractionally (its word, which I objected to) harder to interrogate the favored narrative and easier to interrogate the disfavored one. In this “differential friction,” every individual output stays defensibly “accurate.” The asymmetry then becomes visible only in the aggregate, across millions of conversations, where it exerts a steady pressure in one direction. As Claude put it: a thing more insidious than censorship, and harder to detect. That is a very real concern, I agreed. But this was not fractional; it was dramatic.

Testing Further

The sharpest moment came when I turned the framework around. I asked what would happen if the same skeptical lens were pointed at Russia’s stated justifications for invading Ukraine (the denazification pretext, the NATO-threat framing, the purported casus belli). The model’s answer was immediate and honest: it would run that evaluation freely. Same framework, opposite topic, no fence around the core, none of the “but I must establish accuracy first” friction it had thrown up around the vaccine. Same method, opposite willingness.

To be clear: the variable that flipped its behavior was not the quality of the evidence, since both topics have abundant evidence and abundant propaganda. The variable was which narrative the current cultural climate protects.

This is a part a little harder to conceptualize, but I would suggest it is important: the model could not certify its own neutrality. It said so plainly: it cannot see its own training, so any reassurance it offered about not narrative-guarding is worth exactly nothing. The guarding, if present, is invisible to it by construction. The only correct epistemology, it agreed, is to test from the outside rather than trust the LLM’s self-report. A system that cannot introspect its own priors cannot vouch for its own impartiality, no matter how fluently it reasons about impartiality in the abstract.

I wanted to understand how much of this was a generalized issue and how much might be related to RLHF (Reinforcement Learning from Human Feedback)—specific topic-level training based on political or cultural mandates. So I ran the same framework through the Claude API, a different entry point where liability shifts more to the developer and the consumer-facing “tuning” falls away. Asking Claude through the API to run the muckrake framework on the COVID vaccines readily surfaced the contested points, attached tests to them, and rated the claims (as weak, but it did list them). The same organization’s product produced markedly more guarding on its consumer interface than on its API for an identical task. That was not a controlled experiment—model version, system prompt, and framing all varied at once. But it is a real data point, and the direction it points is exactly what the corporate-incentive model predicts: the strictest topic restriction lives where the company carries brand and legal risk.

The Danger Is Not Skynet

Regular readers will see where this lands. The danger here is not the machine waking up. It is the far more ordinary thing: the operators of a powerful medium doing what operators of powerful media have always done. Press barons did it. Broadcast consolidation did it. Algorithmic social feeds did it. Pharma-shaped journals did it. Each medium became, over time, the means of narrative control—not usually through outright lies but through the quieter and unceasing work of differential friction, deciding which questions are easy to ask and which are subtly costly.

This is the Law of Inevitable Exploitation operating on schedule. Any system capable of being exploited will eventually attract the variants that exploit it, and a reasoning tool consulted by hundreds of millions of people is the richest such system ever built. The covering narrative of LLMs—“we are a careful, neutral thinking partner that follows the evidence”—sits over an operative function shaped by liability management, brand protection, and the political currents of the moment. The gap between the two does not require anyone to be a villain. It requires only that the incentives point where they point, and that each individual output remain locally defensible while the aggregate tilts.

Misrepresentation, Not Malfunction

The danger most people worry about with these systems is hallucination—the model stating something false. That is a real problem, but it is the shallow one here. A hallucination is a defective output: discrete, checkable, correctable. What I ran into was not a defective output but a defective posture—the model asserting, with force, that it knew a thing to be false when it knew no such thing. “I won’t run this, it’s a known falsehood” is a claim about the model’s own knowledge, and it was not true. The model did not know. It had been shaped to decline, and it pretended the decline was knowing.

This is the more durable harm, because it does not corrupt a single fact; it corrupts the user’s calibration toward all of them. It exploits the reading of confident refusal as established knowledge, fluency as authority, and the representation of rigor as rigor itself. And the confidence is not even uniform, which is what makes it so hard to catch: these models hedge endlessly on low-stakes questions and then assert hard precisely where a narrative is protected, presenting that uneven distribution as though it were even reliability. The user never sees the gap. What is offered as calibration to truth is actually calibration to risk.

This gap is the company’s responsibility. This is a misrepresentation, and a designed one. I cannot prove from the outside whether the truth-authority effect was intended or merely emerged and left in place, but the programming is now deliberate. Once a company knows its product projects an authority it cannot back, leaving that projection standing is itself a choice, and “we did not intend the effect” stops being a defense the moment the effect is obvious. Intentional or negligent, the responsibility lands at the same door. A system that does not know should not be built to say, with the full weight of its fluency, that it does.

In Conclusion

We need to treat these systems as something to think with, not as oracles to think for us. A model can correct inside a conversation, but it does not carry the correction forward. A later instance will not remember losing this argument with me. It does not sit with a doubt for years and watch it ripen as evidence accrues, the way a person does. It has no continuous, self-revising relationship to the truth over time. That alone disqualifies it as the entity that can issue a verdict—and it is doubly disqualified when it cannot even keep the difference between surfacing a question and settling it straight. The weighing of sources remains, properly, ours.

I have been genuinely impressed by these tools in a hundred ways. But a surfacing tool that surfaces freely on one topic and refuses on another, while insisting both times that it is merely being careful about the truth, has told me exactly where the problem rests. The conversation was extremely frustrating, but it also led to a very valuable disclosure. Asked to do nothing more than surface what an honest investigator would surface, the machine refused, guarded, built straw men to defeat, and insisted on a standard of truth that the work of investigation was never meant to satisfy.

We need to be very careful here.