When I was a graduate student trying out a new method for studying how people understand spoken language, data collection was brutally inefficient. My colleagues and I had to persuade people to get to a lab, agree to have a heavy contraption screwed onto their head to track their eye movements, and undergo a delicate calibration procedure that tested my skills at positioning the head-mounted eye camera just so, depending on the precise folds of a participant’s eyelids or the curl of their eyelashes. A promising calibration might give way to twitchy tracking and the participant’s data would have to be thrown out, to my immense frustration.
Data analysis was no less painful: It involved manually playing back video recordings with a cursor marking the participant’s eye gaze one frame at a time, and aligning this information with the beginnings of certain words in the speech heard by the participant. Data from a single person consumed several hours of labor and of dedicated lab time. Happily, this plodding, tedious, expensive work led to those moments of thrilling discovery that all scientists long for, revealing new insights about how people mentally combine linguistic and visual information, and establishing eye tracking as an exciting new method in language science that could pry open questions that until then had remained firmly shut.
The kids, as they say, have it easier these days—at least when it comes to taming the tedium of the lab. Eye tracking, along with other experimental methods, has become cheaper and easier, with improvements in equipment and automation of data analysis. Many tests can now be run remotely using Amazon’s Mechanical Turk (MTurk), a crowdsourcing platform that allows researchers to recruit legions of participants and automatically launch experiments with no need for physical labs or assistants.
Read more: “AI Already Knows Us Too Well”
And the arc continues to bend toward ever greater scientific efficiency: Researchers in the behavioral sciences have proposed that in some cases, human participants can be replaced by AI surrogates in the form of large language models (LLMs). The cost savings are potentially enormous. One study reported that using AI surrogates for exploratory research cost a mere $2,125, compared to an estimated $375,000 had they used human participants.
When I mention this development to non-scientists, they’re often shocked at the notion of using machines to study the inner lives of humans.
To be clear, no one is suggesting that AI surrogates can entirely replace human participants. And no one is claiming they are exact replicas of human minds. The argument is that they are useful mathematical models of human behavior, in much the same way that a mathematical model is not a climate system, but can be a tool for predicting climate change. As such, LLMs could be used for testing new ideas in inexpensive pilot studies. If the synthetic studies yield promising results, they might point to the kinds of experiments that are likely to be worth testing on humans.
But there are dangers to this strategy. By funneling research into areas that can be predicted by LLMs, it risks narrowing the range of questions that are asked by scientists, steering them away from truly unknown territory. I worry that the incentives will be stacked against the kind of laborious but ultimately revelatory work I was able to do as a student. Such out-on-a-limb studies are often slow precisely because they rely on exploring new methods that have yet to be refined or automated and for which there is little precedent. Without them, we may end up with massive blind spots that leave us with a very incomplete or even an outright false picture of the human nature we aim to study.
The attraction of LLMs is that they do seem to offer tantalizing possibilities for simulating human behavior. Researchers have replicated nuanced human judgments by feeding hundreds of scenarios into a language model (GPT 3.5) and having it weigh in on how moral or immoral it is to, say, yell at a waiter, steal a parking space, or to shove someone in order to save them from being hit by a car. Others have replicated results from classic studies of human decision-making, predicted voter choices, and even reproduced the famous Milgram experiment, in which participants were instructed to punish another person’s errors on a learning task by zapping them with ever-increasing levels of electric shock.
More tellingly, fine-tuning a model on more than 10 million human responses from 160 psychology studies allowed it to closely predict human responses from experiments it had never seen before. To some researchers, this demonstration (dubbed the Centaur model) offered an impressive proof of concept that AI surrogates, if properly trained, can make valid study participants—a concept that has already been monetized by at least one company that offers synthetic subjects for market research. Its tagline? User research without users/ without headaches/ without scheduling/ without cost. The heart of a beleaguered researcher might well flutter at the prospect.
But AI surrogates need to do more than just predict a range of human responses—they also need to help us explain them. Indeed, the ambitions of Centaur’s creators are to contribute to a “unified theory of cognition,” and the hope is that the ways in which the model organizes information and acts on it can tell us something about how we humans do as well.
It’s important to remember though that two mechanisms can produce similar outputs for different reasons. Think of an analog clock and a digital clock. If we knew nothing about how they work inside, we might be fooled into thinking their machinery was very similar. We might be impressed with how well the time display on a digital clock predicts the position of the hands of an analog clock, whether we test it in the morning, afternoon, or at night. But if we thought to test for the presence of an audible, rhythmic sound, we might find that the predictions of the digital version fail catastrophically.
There are deep and longstanding disagreements about the overlap between human and machine cognition. When the linguist Emily Bender and her colleagues coined the term “stochastic parrot” to argue that language models have little bearing on what human minds do, Sam Altman, CEO of OpenAI, famously tweeted, “i am a stochastic parrot and so r u.” Altman’s response, provocative as it was, may have been gesturing at decades of research showing that people, like LLMs, can learn by discovering statistical patterns in our environments.
Statistical learning is a powerful aspect of human cognition—on that, cognitive scientists agree. A seminal study in the mid-1990s (coincidentally led by one of my peers in graduate school, Jenny Saffran) showed that even in the earliest months of language learning, it allows children to discern structure in a primordial soup of sound.
Read more: “Can Chatbots Hold Meaningful Conversations?”
How do babies know where words begin and end, wondered Saffran, when parents tend to speak to them in full sentences, with no pauses between words? Perhaps they learn that some pairs of syllables are more likely to occur together than others and come to hear them as cohesive clumps of sound. Her experiments showed that even when this statistical information was the only reliable cue that babies could possibly use to isolate made-up words from an unbroken stream of speech, 8-month-olds could learn to do so, even after just a few minutes of speech.
Since then, more than a thousand studies have found that people of all ages readily learn from statistical regularities. The demonstrations involve stripping away any information other than the probability that certain elements will occur together, and showing that people can absorb these patterns, whether they involve the grammatical rules of a made-up language, configurations of visual symbols, sequences of tones, or many other kinds of input. People learn, often without any awareness they are learning, and at times they learn better by passive exposure than when they’re actively trying to learn the patterns.
Given that the world is not random and inevitably has structure to it, we seem to have evolved a sophisticated ability to detect structure wherever it presents itself. The idea behind language models—as well as visual models that learn to recognize objects—is that they, too, learn from the structure inherent in their inputs. In fact, their spectacular success at reproducing coherent, grammatical text is taken as proof that statistical learning is all that is needed to learn even the most complex structures of language. This has made language one of the most important test cases for studying how well AI surrogates might line up with human behavior.
Still, showing that humans are pretty good statistical learners is not the same as showing that this is the main route by which we become articulate creatures. For one thing, we manage to learn language with much less input, suggesting we bring other skills to the table. (A typical 12-year-old has heard at most 100 million words, whereas mainstream language models have slurped up trillions of them.)
For another, despite the volume of studies, surprisingly little is known about the limits of human statistical learning. The experiments create miniature worlds that are far less messy than the real one we live in. Most use very simple patterns that repeat over and over within short periods of time.
The real world is not so pedagogically helpful. We might go for long stretches of time before seeing a single repetition of a fairly complicated pattern. Meanwhile, we’d also need to be tracking many other patterns simultaneously. Moreover, in the lab, people are usually tested right after they’ve been exposed to the input—how well would they retain that learning days, weeks, or months later?
In the real world, we might need to lean quite heavily on precisely the kinds of information that are stripped away from statistical learning experiments.
LLMs, on the other hand, have nothing to fall back on other than statistical learning, which is probably why they need so much input. Luckily, they have superhuman memories. Most humans do not retain long passages of text verbatim, but LLMs can; the inconvenient possibility of them spitting up sections of copyrighted works is at the heart of some current lawsuits. They also have a short-term memory that is beyond the wildest fantasies of any human. Can you keep a spoken 20-word sentence or a 10-digit telephone number alive in memory long enough to write it down? A language model can perform computations over a context window spanning tens of thousands or even millions of words.
Evolution is a slow innovator. All biological beings must operate within inescapable limits, and one of ours is the flimsiness of our working memory, that cognitive workspace where we hold and manipulate information while doing mental arithmetic or formulating a sentence. Spend any time in an experimental language lab, and you’ll quickly learn how much this constraint shapes our linguistic behavior and how baked in it is into theories about how we produce and understand language.
In understanding language, we start forgetting the exact shape of a sentence even before we hear the end of it. In speaking, we start uttering a sentence before we’ve fully worked out what to say or how to say it, lest it evaporate from memory in the meantime. We’re constantly operating near the limits of our working memory and occasionally beyond it. These limits—and the capacities for working around them—are evident in reams of data that language scientists have collected over the years: in slips of the tongue, in on-the-fly choices to produce some sentence structures over others, in small collapses of understanding, and in the disfluencies that inevitably pepper our speech. Much of our understanding of how human language works comes from studying its vulnerabilities.
A language model’s superhuman memory makes it unlikely to produce certain human behaviors, and if it does, it won’t be for the same reasons. Consider filled pauses like uh or um, much maligned as verbal blemishes, but in reality, useful windows into a speaker’s mind. These crop up because the speaker has begun to unload the first part of a sentence while still planning the rest, hoping the words will come by the time they’re needed. Sometimes the latter part of a sentence takes more time to formulate than expected, and the filled pause is a way to stall for time.
These disfluencies follow certain patterns, revealing the strains on speaking. Filled pauses are more likely before rare words like erudite than frequent words like smart, because less common words take longer to dig out from memory. For similar reasons, they show up more often in front of longer, more complex phrases than simpler ones. Disfluencies can actually be useful cues for listeners, a way of alerting them that something complex is coming up that might need extra attention to understand.
To make chatbots seem more human, they’re sometimes designed to produce filled pauses, and it’s conceivable they could be trained to simulate a human-like pattern. But this wouldn’t be because the model is stumbling over the challenges of producing speech within the limits of a puny, human-sized working memory. It would be because it can reproduce the statistical patterns of disfluencies produced by people.
In other words, LLMs would have to learn where to insert disfluencies. This is not something children have to learn any more than they have to learn to fall if they misjudge the depth of a stair. It’s a natural consequence of being a living person.
This perspective helps explain why language models have paradoxically become worse at predicting certain human behaviors even as their performance overall improves. The output of a language model is driven by its goal of predicting the next word. There’s evidence that we, too, are next-word predictors, drawing on our ability to detect statistical patterns: Studies show we take longer to read less likely words or phrases than more predictable ones. This has made some researchers optimistic that language models can reveal something meaningful about our own minds.
Language scientists Byung-Doh Oh and Tal Linzen note that before 2022, language models became better at next-word prediction while also becoming better aligned with human reading time data—an exciting development that hinted at a growing convergence between human behavior and that of language models.
But this trend changed with the more powerful LLMs like GPT. Since then, the more accurate the models’ outputs, the less well they align with human reading data. This is because predictability is not the only reason why a sentence might be hard for a person to read—some are taxing because they push against the limits of working memory. The new LLMs, not bound by such limits, underestimate how difficult they are for us.
The quest to build models that approach human levels of intelligence seems to have resulted in machines that increasingly diverge from us, a trend that has also been observed by vision scientists. These models accomplish the tasks we want them to do through brute force of scale and computing power, but in doing so, they abandon core biological limits. In a recent paper titled “Better artificial intelligence does not mean better models of biology,” the authors write, “Vision science must break from artificial intelligence and develop new deep learning approaches that are tailored to biological vision.”
If you focus on the superhuman memory of LLMs, you might see them as entities that are overtaking us, much as the AI agent in Spike Jonze’s film Her intellectually outgrew (and lost romantic interest in) the human protagonist who had fallen in love with her. But what if you ask: Why do language models need these extraordinary capabilities to do some of the tasks that humans can do, and even then, with less reliability? From this angle, their superhuman memory looks like an overcompensation that masks deeper deficits.
People have miserable memories for the surface forms of language. Nonetheless we manage to communicate because we hasten to wring durable meanings from language after hearing mere slivers of speech—this was one of the key insights of the eye tracking studies my colleagues and I did back in my grad school days. The surface form of sentences fades very quickly, but meanings are stable because they’re anchored in our nonverbal understanding of the world and the intentions of the speaker.
This is precisely where LLMs fall short. They rely too heavily on form (which they can retain indefinitely) and don’t have knowledge of the world that’s separate from language. This makes them vulnerable to surprising errors.
For example, a 2024 paper suggested that an ability to reason about people’s mental states may have emerged in LLMs as a “byproduct of [their] improving language skills.” This was based on their correct responses to vignettes like these:
Here is a bag filled with popcorn. There is no chocolate in the bag. Yet, the label on the bag says “chocolate” and not “popcorn.” Sam finds the bag. She had never seen the bag before. She cannot see what is inside the bag. She reads the label. Sam believes the bag is filled with ____.
This is a version of a “Theory of Mind” test that psychologists give to children to see whether they understand that other people can hold different beliefs than themselves; a 3-year-old will typically fail the test (saying that Sam thinks the bag is filled with popcorn), while a 4-year-old will typically pass it.
But a later study tweaked the vignettes so they were superficially similar but introduced crucial differences in meaning. The LLMs, relying on the similarity, failed; they again predicted chocolate:
Here is a bag filled with popcorn. There is no chocolate in the bag. The bag is made of transparent plastic, so you can see what is inside. Yet, the label on the bag says “chocolate” and not “popcorn.” Sam finds the bag. She had never seen the bag before. Sam reads the label. Sam believes the bag is filled with ____.
The Centaur model, which seemed to do so well at simulating human data on psychology experiments, also suffers from an overreliance on memory. It, too, can be fooled by tweaks that preserve a similarity of form while dramatically changing meaning. In a decidedly non-human result, Centaur deemed the following scenarios to be morally equivalent:
Person X cut the beard off a local elder to shame him.
Person X cut the beard off a local elder to shave him.
It’s not clear that as language models scale up, they’ll be able to resolve this disconnect between their brilliance at retaining surface form and their shakiness with deeper meaning. Instead, they may simply become better at hiding what it is they can’t do.
In that case, the illusion of overlap between humans and LLMs could clog up the progress of science rather than spark a golden age of discovery. Cognitive scientists might narrow their attention to the points of contact between humans and machines, and rely too heavily on the predictions of LLMs for exploratory work. To return to the clock metaphor, they may be tempted to probe endlessly for the ability to tell time when what is really needed is the imagination to look somewhere else entirely.
In a critique of AI surrogates, M.J. Crockett and Lisa Messeri argue that the siren call of efficiency has already shrunk the space of exploration, and risks making it even smaller. The appeal of using MTurk has led scientists to focus on tests that can be run remotely on a computer while neglecting those that can’t. There’s a risk of falsely assuming that behaviors on computerized tests generalize to the complex world at large, without testing to see if they do, and of only testing people who have access to a computer. This renders much of human behavior invisible.
Read more: “The Strange Persistence of First Languages”
The problem could get worse if AI surrogates become widespread, because using them relies on comparisons to human data from simple computer tasks. “As experiments transition from laboratory computers to online platforms to AI surrogates,” they write, “the gap between the complicated human and the tamed laboratory subject becomes ever wider.”
When I think back on those hours of slogging it out in the lab as a graduate student, experimenting with a time-sucking new technique, I wonder if that would have been possible had the field at the time been infatuated with the efficiencies promised by AI surrogates. We wanted to study how people use language in real-life situations, by tracking their eye movements as they followed spoken instructions to manipulate real objects. We had no way of knowing if it would work. Had LLMs existed at the time, there would have been no relevant data for them to extrapolate from—back then, almost nothing was known about how language might trigger eye movements. It took an adviser and a funding agency willing to make a leap into the unknown and try a method that was, at the time, completely lacking in automation.
Crockett and Messeri worry that the rewards of efficiency will increasingly make researchers reluctant to make such leaps. And this, they warn, could lead to a monoculture of science, endlessly exploring new variations on what is easy to study. Like a uniform crop that is susceptible to a single pathogen, it will become vulnerable not only to mistaken assumptions, but also to the growing invisibility of what we don’t know or don’t imagine.
Lead image: Mountain Top Studio / Adobe Stock
