The Thesis
Most language models compute the next-token distribution via a learned, implicit process that we treat as a black box. We propose reinterpreting this computation as an agent solving an equilibrium problem rooted in game theory and mechanism design. Under a quantal response framework, the model is an economic agent, and its output distribution is its best response to the data it observes.
The Equilibrium Language Model (EqLM)
EqLM is a Transformer-based architecture augmented with an implicit equilibrium layer. Instead of a single linear projection to logits, we compute a fixed point:
z* = f(z, h; θ)
where h are contextual representations and z* is the equilibrium logits. This enables:
- Interpretability: The model solves an explicit, game-theoretic optimization problem.
- Convergence analysis: Fixed-point theory and kernel methods (MMD) apply directly.
- Theoretical grounding: Connects to quantal response equilibrium, mechanism design, and deep equilibrium models.
- Alignment: Mechanism design perspective offers new levers for preference learning.
Key Insights
Quantal Response Equilibrium (QRE): Agents play noisy best-responses; the softmax in language models implements precisely this structure. Learning is equilibrium-seeking under this lens.
Implicit Depth (DEQ): Fixed-point iteration replaces explicit layers, allowing us to solve an equilibrium without unrolling depth—reducing memory and interpretability costs.
Convergence via MMD: Maximum mean discrepancy measures the distance between learned and target distributions. Convergence analysis shows EqLM reaches equilibrium faster than gradient descent.
Auction-Based Decoding: Mechanism design principles (mechanism design for LLMs, Duetting et al. 2024) suggest new inference strategies that improve alignment efficiency.
Experiments
Programme closed 2026-09-02 at F55. Every number below is sourced from research/memory/findings.md (F1–F55, each Tarka-reviewed). The scale programme halted at the one-billion-parameter boundary: the compute-matched exchange rate holds at 46–121M and did not transfer unchanged to 1B on web data (ratio 1.56 at the failed 1B-token gate, 1.31 and closing at 2.5B), with both arms at chance on public benchmarks.
- Matrix Game Convergence: EqLM vs baselines on synthetic 2×2 games; QRE recovery exactness.
- Deep Equilibrium Memory: Implicit depth vs explicit layers; memory-computation tradeoffs.
- BabyLM Pretraining: Token-matched benchmarking vs GPT-2 and BERT-class models.
- Preference Learning: Auction-based decoding (MPO vs DPO) for alignment.
For Researchers
See the method page for architectural details, the results page for validated findings and benchmarks, and the repository for code and reproducibility details.