We Have a Cache. Our Worst Decisions Come From It.
Listen — 8 min
The best systems in the world are not the ones with the biggest caches. They are the ones that know when to bypass them.
That is a sentence about software. It is also a sentence about us.
We say art imitates life. A painting is the painter's eye, the era's fears, the culture's idea of beauty, held still long enough to be seen. Technology imitates life too, and more literally than we admit. So much of what we build is a copy of something we already carry inside us, built in silicon where we can finally look at it.
Take caching.
What a cache is, and why it exists
A cache is a small, fast memory that sits in front of a large, slow one. Instead of going all the way to the database every time, the system keeps a copy of recent or frequent answers close at hand. Ask the same question again and we get the cached answer in microseconds instead of milliseconds.
That is the whole point. Speed and efficiency. Our processors have caches. Our browsers have caches. Every app on our phones has a cache. The APIs those apps call have caches, and the best ones let developers see exactly what was served from cache and what was fetched fresh. Most of what feels instant in modern software is a cache doing its job.
The economics are hard to argue with. A cache hit is cheap. A cache miss is expensive. So engineers spend enormous effort raising the hit rate.
Where it goes wrong
There is an old line in computer science that there are only two hard problems, cache invalidation and naming things.
A cache holds a copy of reality, not reality itself. That is art imitating life in one sentence. A likeness, not the real substance. The moment the source changes and the cache does not, the system serves a stale answer with full confidence. The user sees an old price, an old balance, an old status. The system is fast, the system is wrong, and it has no idea.
The failure modes are well known.
Stale data. The truth moved. The cache did not.
Inconsistency. Two caches hold two different versions of the same fact.
Stampedes. A hot entry expires and every request hits the slow path at once.
Poisoning. A bad value gets cached and is repeated at scale.
None of these are bugs in the cache. They are the price of the cache. Speed is purchased with the risk of being confidently out of date.
How the leading companies raise the hit rate
The biggest players do not avoid caching, and they are not trying to eliminate these risks. They are researching how to win more and lose less, and the direction of that research is the interesting part.
Google is teaching caches to guess well. Imagine a cache that could see the future. It would keep exactly what is about to be asked for and toss everything else. Google trained a model to copy that perfect fortune teller as closely as it can. The research that grew around this idea keeps one rule at its core. Trust the guess, but always keep a safety net for when the guess is wrong.
Amazon is teaching caches to recognize the same question in different clothes. "Red running shoes" and "runing shoes red" are one question, so they now pull one stored answer, which lets the cache serve three times as much. In its data warehouse, Amazon prototyped a cache that remembers where the answers live instead of the answers themselves, like a bookmark instead of a photocopy, and repeated searches ran up to 10x faster in testing. And a recent Princeton and AWS paper called Marconi brings the same thinking inside AI models, keeping only the pieces of a conversation most likely to come up again.
The pattern across both is the same. Nobody is caching less. Everyone is moving from fixed rules ("keep whatever was used recently") toward learned judgment. Predict what will be needed, weigh the cost of a miss against the cost of storing, and keep a path back to the source when the prediction fails.
The smart cache is the one that knows its own limits.
The cache we carry
Daniel Kahneman described two modes of human thinking. One is fast, automatic, and effortless. The other is slow, deliberate, and expensive. The fast mode runs by default. The slow mode has to be deliberately engaged. It does not turn on by itself.
The fast mode is a cache. It is not reasoning. It is retrieval. It serves an answer we already stored, whether a pattern from experience, an assumption, a stereotype, or an emotional reaction that was correct once. It comes back instantly and it feels like insight. It is a cache hit.
Most of the time, that is fine. We cannot deliberate over every doorknob. The cache is what lets us function.
And sometimes the cache deserves to win. Kahneman and Gary Klein, the researcher who studied expert intuition, wrote a joint paper on when fast judgment can be trusted. Their answer was environments with stable rules and fast feedback. Firefighters, chess players, nurses. Enough repetitions against a world that corrects us quickly, and the fast path earns its confidence. The trouble is that most of life is not that kind of environment.
So our cache carries the same liabilities the machine's does.
Stale data. The subject changed. Our opinion of it did not.
Inconsistency. We hold two beliefs that cannot both be true, cached in different contexts, never reconciled.
Stampedes. Under pressure, every decision routes through the fast path at once, and quality collapses.
Poisoning. One bad experience gets cached as a rule and repeated for years.
And it carries one liability the machine does not. Nobody writes to a machine cache except the system itself. Ours takes writes from anyone. Media, advertising, repetition, whoever spoke last, whoever spoke loudest. We run a cache with open write access and no admission policy, and then we make decisions from whatever got in.
The dangerous part is that a cached answer arrives with the same confidence whether it is current or twenty years stale. Speed feels like certainty. It is not. And the economics hold on this side too. The hit is cheap. The miss is expensive. A stale answer served fast to the wrong question can cost a relationship, a deal, or time.
Where the analogy needs honesty
Behind a machine's cache sits a database that holds the truth. We carry no database. Our slow thinking is not a more accurate copy of reality. It is only slower, and it can rationalize with the same confidence it reasons.
The source is outside us. The actual person, the actual numbers, the actual situation in front of us. Bypassing the cache does not mean thinking harder. It means verifying with the resources available to us until we reach the best answer.
The discipline
We do not get rid of the cache. We get deliberate about it, and deliberate means mechanisms, not intentions.
Give big decisions a mandatory delay. Pick a threshold that fits our life, a dollar amount, a relationship, a commitment of months. Anything above it waits twenty-four hours, no matter how obvious the answer feels. The delay exists precisely because it will feel unnecessary.
Put expiration dates on old conclusions. Write down three beliefs we hold about a specific subject and recheck them each quarter against what is true now, not what was true when the belief was stored.
Treat instant certainty as the alarm. When a big question gets an immediate answer that feels like insight, that speed is the tell. It means the answer came from the cache. That is the moment to go to the source.
The slow path costs something. That is exactly why it is worth spending on the decisions that count.
The best systems in the world are not the ones with the biggest caches. They are the ones that know when to bypass them.
Same for us.
Further reading: Liu et al. (Google), "An Imitation Learning Approach for Cache Replacement" (ICML 2020); Lykouris (Cornell) and Vassilvitskii (Google), "Competitive Caching with Machine Learned Advice" (J. ACM 2021); Amazon Science on LSH-based query caching and the predicate caching prototype for Redshift; Pan et al. (Princeton/AWS), "Marconi: Prefix Caching for the Era of Hybrid LLMs" (2024); Kahneman, "Thinking, Fast and Slow"; Kahneman and Klein, "Conditions for Intuitive Expertise: A Failure to Disagree" (2009).
Originally published on LinkedIn.