It Was Never Whole
In my last post, a fake Kenny Loggins album cover demonstrated that 35% visual match was enough for accurate recognition. That raised a bigger question: do LLMs actually need complete data to perform well — or is that assumption a myth we inherited from a different era of computing?
Let's separate engineering reality from human projection.
The short answer: LLMs are engineered to operate on incomplete information. The expectation of complete data is a human assumption, not a technical requirement.
It's Not Just One Model
Nearly all major transformer-based LLMs share the same fundamental training objective: predict the next token given context. That includes OpenAI GPT-4, Meta LLaMA, Google Gemini, and Anthropic Claude. What they have in common is that they are trained on partial sequences and must infer continuation from them. The system never sees complete meaning. It sees statistical context and predicts what comes next, every single time.
BERT-style models are trained by deliberately masking words and forcing reconstruction. Autoregressive models always predict from incomplete context — never from a full picture. Fine-tuning and RLHF refine behavior and alignment, not completeness. The architecture itself is built on the assumption that input will always be partial. Incompleteness is not a limitation of these systems. It is the design principle that makes them work.
Why Humans Expect Complete Data
The expectation of completeness comes from three deeply ingrained places, and understanding them helps explain why the myth is so persistent.
The first is classical database thinking. In institutional research, finance, and healthcare, professionals are trained to believe that missing values reduce validity, that data quality is a completeness problem, and that sound decisions require full coverage. That mindset was built for structured datasets and compliance environments where a missing field has real legal and operational consequences. It is a reasonable instinct in those contexts. It does not transfer cleanly to probabilistic systems.
The second is deterministic intuition. Most people instinctively assume that intelligence equals stored knowledge — they imagine models as giant lookup tables where the answer either exists or it doesn't. If the information isn't in there, the reasoning goes, the system cannot answer. But LLMs are not lookup systems. They are probabilistic inference engines that construct responses from learned distributions, not retrieved records. The distinction sounds subtle. It isn't.
The third is the legacy of early AI narratives. Early machine learning required labeled datasets tightly aligned to specific tasks, which trained an entire generation of practitioners to believe that performance depends on coverage. Large language models changed that paradigm. They learn distributions, not task catalogs, and that distinction matters enormously for how we think about data requirements.
What LLMs Actually Need
What LLMs need is not completeness in the traditional sense. They need sufficient statistical density — enough signal to form reliable patterns. They need representational richness, meaning diverse enough context to generalize across situations. They need distributional alignment, so training data reflects the target domain. And they need contextual redundancy: overlapping signals that reinforce inference when any single signal is weak or absent.
This is much closer to Shannon's redundancy principle than to database completeness. Language contains massive redundancy by design. Culture contains archetypes that compress enormous amounts of meaning into recognizable patterns. Visual eras compress into stylistic clusters identifiable from a handful of cues. The Kenny Loggins experiment demonstrated that the model didn't need every album cover ever made to generate a good enough image. It needed enough high-weight signals to approximate the category, and 35% of the right signals was sufficient.
Think about the Very Large Array, fifty miles west of Socorro on the Plains of San Agustin, New Mexico. Twenty-seven dishes, none of them sufficient on its own. Each capturing a partial, weak radio signal from somewhere in the universe. The array doesn't wait for a complete signal. It pools distributed, partial inputs and synthesizes them into coherent intelligence through interferometry — achieving the resolution of an antenna 22 miles in diameter from components that span miles of high desert. That is statistical density and contextual redundancy made physical. LLMs operate on the same principle.
When Completeness Does Matter
To stay rigorous: there are genuine cases where incompleteness creates real problems. Rare domain-specific terminology that appears infrequently in training data can trip up even powerful models. Long-tail edge cases that fall outside the training distribution are handled poorly. Novel scientific discoveries post-training cannot be known. And highly structured compliance data in higher ed — the kind that drives IPEDS reporting, Title IV eligibility, or financial audits — still demands the completeness that traditional data governance provides.
But these are cases of insufficient distribution coverage, not failures of the general principle. Even humans reason from partial exposure. No person has read all literature, observed all phenomena, or experienced all situations. We reason from internalized distributions built from a fraction of available information, and we do it remarkably well most of the time.
The Deeper Insight
The reason LLMs do not require complete data is because intelligence (computational or biological) fundamentally operates on compression. A complete representation of the world would be infinite, and no system can hold it. So both brains and transformers abstract, compress, generalize, and interpolate. They operate on probability mass, not exhaustive enumeration, and that is precisely what makes them powerful rather than brittle.
The conventional assumption equates accuracy with coverage: more complete data produces more accurate results. For probabilistic systems the equation is different: accuracy comes from sufficient signal density, not from completeness. That is not a subtle distinction. It is a different philosophy of what good data means, and it has real implications for how institutions should think about data strategy in an AI-enabled environment.
Why This Matters for Your Institution
This series started with a fake album cover, but it is pointing somewhere genuinely uncomfortable for institutions built around completeness audits and data quality frameworks that treat missing fields as failures.
Which raises a larger question: if completeness was never the right standard for probabilistic inference, why did we spend sixty years insisting it was? That is where the Garbage In Garbage Out idea comes from and the topic of my next post.
Further Reading
What even is a parameter? → (MIT Technology Review, Will Douglas Heaven)
The Neuroscience of Reality → (Scientific American, Anil K. Seth)
The Karl G. Jansky Very Large Array, Plains of San Agustin, New Mexico, during a lightning storm. Twenty-seven dishes. No single one sufficient. Together, enough to see the universe. Photo: Bettymaya Foott, NRAO/AUI/NSF, CC BY 4.0.