Many people now say that large language models are on a path toward greater truthfulness because they are succeeding at mathematics, software verification, and tightly constrained science. A model that can produce a checked proof or confirm that code meets its specification, the reasoning goes, is learning to track reality. With more scale or better techniques, the same ability should appear in ordinary topics: policy, history, medicine, or how institutions actually work.
That does not actually follow, although you might think it would.
There are two different kinds of alignment for these models.
Objective alignment is possible when an outside checker exists that does not depend on human opinion. A proof assistant either accepts the steps or rejects them. A program either satisfies a formal specification or it does not. A scientific prediction either matches the measurement or fails. In these cases the system can be trained or searched against a standard that sits outside ordinary language. How often a similar claim appears in books or on the internet does not matter. Whether the claim is socially comfortable does not matter. The checker supplies an independent test.
Operative alignment is what most chat models actually move toward. It means matching the language, tone, and expectations that particular groups of people treat as normal or appropriate. The usual method is to collect human preferences—rankings of which answer sounds better—and adjust the model toward those rankings. The result can reduce some obvious harms and help companies manage legal risk. It does not supply an independent test of truth. Different groups prefer different things, and the preferences themselves change.1
The reason success in mathematics does not transfer to greater truthfulness is built into how these models work. A large language model is trained by repeated next-word guessing across enormous amounts of text. Each successful guess slightly adjusts millions of internal dials, called weights. What remains are settings that make statistically common continuations easy to produce. A true statement becomes easy only if it was also common and stable in the training text. When a source later proves unreliable, the model has no separate mechanism that does what a careful person does: lower the weight of everything that source has said, and raise the standard of evidence for related claims. Frequency still holds sway. A story that appears often remains easier to continue than a rarer story, even when the rarer story is better supported once the sources are checked.
This is an old human difficulty made larger. What people say, even sincerely, is often different from what is actually happening. Institutions and communities maintain flattering or protective accounts of their own purposes while the day-to-day incentives and results run differently. Strong emotion frequently defends the account. Established practices that have improved the match between speech and reality—scientific testing, courtroom cross-examination, structured peer review—exist because fluent or popular language is not enough by itself. They add outside friction so that claims must survive evidence and people who are rewarded for finding mistakes.
A language model externalizes the storytelling side of that difficulty. It has absorbed the public record of how people describe themselves and their institutions, then been further adjusted to sound appropriate to its raters. Where the distance is large between the common story and what is actually occurring, information that would shrink the distance is usually less common or less rewarded. The model therefore tends to extend the predominant account. Skill at formal proofs does not change this. That skill came from closing a verification loop that ordinary language topics do not have. More scale or longer reasoning traces can make the model more fluent and locally consistent. They do not add the missing ability to discount unreliable sources or to resist a dominant narrative.
This is why claims that one model is simply “more truthful” than another often describe a difference in operative alignment rather than a difference in truth-tracking. When Grok was described as more truthful, the observable difference was that it produced narratives matching a different set of political and social expectations. The underlying limitation remained the same.
Objective alignment success should not be read as progress toward truthful answers in areas that remain under operative alignment. Those areas stay governed by match to prevailing language patterns. Reducing the distance between the stories we (and the models) tell and what is actually going on still requires the same costly outside structures human institutions have always needed when speech alone is insufficient. Formal mathematics shows what is possible when a mechanical checker can be applied. It does not show that fluent language starts to carry its own guarantee of truth once a model becomes good at proofs.
1 Readers familiar with my earlier work may notice that “operative” is used differently here than in discussions of narrative and function. The uses are related. In both cases the word points to what is actually running and treated as standard—in this essay, the expectations that preference training matches—rather than to an idealized story or an independent test.
No comments:
Post a Comment
I hate having to moderate comments, but have to do so because of spam... :(