Understanding Inference and the "Stochastic Parrot" in Large Language Models

Everyone uses LLMs but few people understand how they work or what their limitations are. Let's fix that

This is a high level overview of inference in large language models (LLMs), and the associated gibe that LLMs are merely stochastic parrots. It is aimed at non-technical people. As such, it necessarily elides certain technical details. I’ve attempted to provide some additional, more technical, information and nuance in the form of footnotes. ChatGPT assisted with research and some phrasing.

This is somewhat longer-than-normal post, about 2,000 words, and will be truncated if you are reading it via email.

Introduction

Large language models (LLMs) like ChatGPT or Claude¹ have transformed how we interact with computers and how computers interact with human language. LLMs can respond to questions, write stories, summarize books, and even assist with creative writing. To many people, they seem eerily human. But beneath this sophisticated interaction lies some remarkable, albeit critically limited, concepts. This essay will explore two foundational ideas that explain how these systems work and what their limitations are: inference and the stochastic parrot².

Let’s Make Some Inferences

Inference is the engine behind an LLM’s ability to respond to prompts. When you type a question or a statement into an LLM like ChatGPT or Claude, it needs to infer an answer. That is, it must generate an appropriate response based on what it has learned. The way an LLM infers an answer isn’t like the reasoning humans do. Instead, it’s more like an incredibly advanced prediction engine.

Consider autocorrect on your phone. When you type on your phone, you often see it predict the next word based on what you’ve written so far. LLMs take this concept and supercharge it. These models generate responses by predicting the next word (or token, which is part of a word or a character) based on the context you provide.

The process of inference begins with the model examining your input. This inpt could be a question, a phrase, or a even an unfinished sentence. The model then calculates probabilities for what the next word should be. For instance, if you type “The capital of Jamaica is,” the model assigns a high probability to the word “Kingston” because, statistically, that’s the word that most commonly follows in similar contexts within its training data. Inference doesn’t just happen once. It’s an iterative process. The model repeats this prediction for each word until it completes a coherent response.

This means that, at its heart, an LLM is continuously making statistical guesses to create a sequence of words that seem appropriate. It doesn’t “understand” what Kingston is or why it’s the capital of Jamaica. It only knows that, based on all the books, articles, and webpages it has read, “Kingston” is the word that makes the most sense.

The Stochastic Parrot Enters the Chat

Our stochastic parrot, courtesty of ChatGPT. It’s important to note that, while ChatGPT is commonly associated with its underlying LLM GPTx, its image generation capabilities, done via OpenAI’s Dall-E tool, do not rely on a large language model. Rather, Dall-E is a generative image model.

The metaphor of a “stochastic parrot” was coined by researchers³ to capture a critical aspect of how LLMs function. Let’s break this down:

• Stochastic: This word refers to a probabilistic process. That is, a process for which its events are determined by probability. In LLMs, each word prediction is based on probabilities. Think of it roughly like rolling a pair of dice, but instead of numbers, you’re rolling to select the next word from a weighted list of options.

• Parrot: Parrots are known for mimicking human speech. They can repeat what they hear without actually understanding it. This is exactly what LLMs do. They repeat patterns they have encountered during their training, without any comprehension of the underlying meaning.

Together, “stochastic parrot” suggests that LLMs are not generating original thoughts or genuinely understanding your inputs. They are simply piecing together fragments of language in a statistically likely way. They are “parroting” what they’ve learned, but with a probabilistic twist that makes them seem dynamic and adaptive.

This has profound implications. It means that while LLMs are extremely good at generating text that looks and feels meaningful, they don’t actually have an understanding of the content they produce. When an LLM tells you “Kingston is the capital of Jamaica,” it is not recalling a fact the way a person would. Instead, it is producing the word “Kingston” because that is the word that best fits statistically based on its training. It’s a simulation of understanding. It’s a very convincing one, but it is a simulation nonetheless.

How Inference and Stochastic Parroting Interact

To better understand how inference and the stochastic parrot concept intertwine, imagine asking a language model to write a story. You prompt it with: “Once upon a time in a faraway kingdom.” From here, the model needs to infer what comes next. It will likely come up with something like “there lived a brave prince,” because stories with such a beginning often introduce a protagonist soon afterward.

The model’s inference draws on all of the fairy tales it has seen before, finding a likely contribution that fits common story patterns. The “stochastic parrot” aspect reveals itself in that the model is mimicking the pattern of stories. It doesn’t know what a kingdom or a prince is, nor does it understand narrative tension, character development, or any of the components that give meaning to a story. It merely knows which word sequences are statistically likely.

Another example is question answering. If you ask, “What are the uses of artifical intelligence?”, the model generates a response by considering what it has learned about how people have previously answered similar questions. It could say something like, “AI is used in healthcare, finance, and customer service.” Again, it doesn’t truly “know” this. It’s generating an answer that fits with the data patterns it has learned. This is an instance where inference does the heavy lifting, but the parrot effect means it’s fundamentally just pattern-matching without comprehension.

Strengths and Limitations

LLMs are astonishingly powerful tooks, but the stochastic parrot metaphor serves as a reminder of their limits. Folowing is a brief overview of these strengths and limitations, using what we’ve discussed earlier in this essay as a guide.

Strengths:

1. Contextual Relevance: Thanks to sophisticated attention mechanisms⁴, LLMs can keep track of context effectively, allowing them to generate text that seems coherent and contextually appropriate over long passages.

2. Versatility: They can write poetry, summarize long documents, translate languages, and even assist with coding tasks. This is possible because they can infer the most suitable outputs by using patterns seen in vast amounts of training data.

3. Efficiency: The inference process in LLMs is incredibly fast, allowing for quick generation of text. This makes them useful in applications like chatbots, content creation, and language translation.

Limitations:

1. No Real Understanding: The biggest limitation is the lack of genuine comprehension. LLMs don’t understand concepts, ideas or facts. They only predict text based on learned patterns.

2. Bias Reproduction: LLMs are trained on data collected from human sources, which means that they are prone to reproducing the biases present in that data. If the training data reflects stereotypes, the model’s outputs will too.

3. Misinformation and Hallucinations: Because LLMs lack an internal fact-checking mechanism, they sometimes generate plausible-sounding but completely false information. This is often called “hallucination,” where the model stitches together seemingly relevant pieces of information without grounding them in reality.

We Shall Overcome These Limitations

Researchers are actively exploring various approaches to improve these models and reduce their shortcomings. These approaches include:

1. Scaling Up and Increasing Training Data: This entails making LLMs larger by adding more parameters and training them on vast datasets⁵. The idea here is that a larger model, trained on more diverse data, can better capture nuanced relationships and reduce biases. However, while scaling up has yielded improvements in some respects, it also comes with increased computational costs and it doesn’t directly solve the issue of genuine comprehension.

2. Grounding Models with External Knowledge: To address the problem of misinformation and hallucinations, researchers are working on integrating external knowledge bases into LLMs. By allowing models to consult up-to-date and reliable information sources during inference, they can verify facts and provide more accurate answers. The idea is that combining an LLM with structured data will make responses more trustworthy and grounded in verified facts⁶.

3. Incorporating Reinforcement Learning from Human Feedback (RLHF): In this method, LLMs are fine-tuned based on input from human evaluators who rank the quality of responses⁷. This helps the model learn to prefer outputs that are more accurate and contextually appropriate. RLHF has been instrumental in making models like ChatGPT more aligned with human expectations.

4. Training on Diverse and Ethical Data: Researchers emphasize the importance of training on more diverse datasets to reduce biases. By including a wider array of voices, perspectives, and contexts, LLMs can learn to generate more balanced outputs. Additionally, algorithmic interventions during training can be used to minimize the impact of harmful biases⁸ present in the data.

5. Hybrid Architecture: Hybrid AI systems combine the strengths of LLMs with other forms of AI, such as symbolic reasoning. Unlike LLMs, symbolic reasoning systems can represent explicit rules and logic, which helps in reasoning through complex problems. By integrating LLMs with components capable of explicit reasoning, researchers hope to create systems that can not only generate fluent language but also reason in a way that approaches human-like understanding.

6. Explainability and Interpretability: A major challenge with LLMs is their “black box” nature. Researchers are working on improving interpretability—developing methods to help humans understand why a model made a particular prediction. This is key for increasing trust in AI systems, especially in applications where understanding the reasoning process is crucial, such as healthcare, finance, or law⁹.

Conclusion

Inference in large language models is a powerful and sophisticated process that allows these systems to generate language that appears meaningful and relevant. However, understanding the concept of the stochastic parrot is key to appreciating the limitations of these LLMs. They are advanced mimics, parroting back patterns they have seen during their training, and their responses are governed by probabilities, not comprehension.

These models are tools. They are remarkable and useful, but they are fundamentally constrained. They can assist us in a myriad of tasks, from drafting an email to generating creative stories, but they are devoid of understanding, reasoning, or intentionality. By keeping in mind that LLMs are ultimately sophisticated pattern-matchers, we can use them as valuable complements to human judgment.

1. Technically these are interfaces which sit on top of the underlying large language model.

2. The original research paper, in which this metaphor first appeared, is available here.

3. See Footnote 2, above, for the paper which originated the metaphor of the stochastic parrot.

4. This is a way to determine the relative importance of one piece of input, as compared to other pieces of the input. Wiki’s article on this is fairly good. See also the paper which kicked off the large language model revolution, Attention is All You Need.

5. This gives rise to the notion of scaling laws: “scale is all you need” to get to AGI. Of course, there has been a lot of debate recently about whether we are hitting limits to scaling—this is the notion that we are hitting a wall.

6. This of course raises the obvious question: whose facts? Some facts are indisputable. For example, the speed of light in a vacuum, or whether Lebron James or Michael Jordan scored more points over the course of their careers, are not debatable facts. Other “facts” are contested, often vociferously, by affected or interested parties.

7. There’s a good overview of this process here. This diagram, in particular, helps visualize the process:

8. Of course, attempting to eliminate “harmful biases” from training data may, depending on one’s perspective, introduce other biases into the training data. One can see how this quickly could become an inextricable minefield, especially for controversial subjects, or for training data taken from controversial sources.

9. You have probably heard of the case of the lawyer who relied on ChatGPT to come up with precedential cases to support his client’s case. These precedential cases were, of course, hallucinated by ChatGPT. Caveat emptor applie