KV Cache
A memory-speed tradeoff in transformer inference: store previous keys and values to generate faster, while paying in memory.
Ideas
Ideas I keep returning to
Short notes on technical concepts, mental models, and engineering principles that shape how I think and build.
01
AI systems
Speculative Decoding
Use a small draft model to guess tokens and a larger model to verify them. The idea is simple: spend less time waiting on the expensive model.
Context Windows
Longer context is not the same as better memory. The useful question is what the model can actually retrieve, weigh, and use.
Evaluation Drift
AI systems change as prompts, users, models, and data change. Evaluation has to be continuous, not a one-time report.
Batch Size
Batching improves hardware utilization, but it also changes latency, queueing, and cost. Empty seats on the bus still cost money.
Roofline Model
A practical way to reason about whether a workload is limited by compute or memory bandwidth.
02
Learning and statistics
Central Limit Theorem
A reminder that noisy individual measurements can still produce stable aggregate behavior.
MCMC
When direct sampling is hard, build a process that wanders through the right distribution over time.
Entropy
A measure of uncertainty, disorder, and surprise. Useful in information theory, thermodynamics, and everyday life.
First Principles
Strip a problem down to what must be true, then rebuild from there instead of inheriting old assumptions.
03
Engineering judgment
Boring Technology
Choose predictable tools until the problem clearly earns something novel. Novelty should pay rent.
Small Interfaces
Good systems are often built from narrow contracts. The smaller the surface area, the easier it is to reason about change.
Observability Before Cleverness
If you cannot see what the system is doing, clever fixes are mostly guesses.
Compounding Engineering
Documentation, tests, automation, and clean abstractions compound quietly. They make later work cheaper.
04
Decision making
Opportunity Cost
The cost of a choice is the best alternative you gave up, not just the money or time you spent.
Marginal Thinking
Ask whether the next hour, next dollar, or next unit is worth it from where you are now.
Inversion
Instead of asking how to succeed, ask what would guarantee failure and avoid that first.
Small Bets
When uncertainty is high, make the next experiment cheap enough that learning is the main outcome.