Architecture Overview
EqLM builds on standard Transformer blocks but replaces the final logit projection with an implicit equilibrium layer. The model processes input tokens through contextual representations and then solves an equilibrium fixed-point problem.
Core Components
1. Contextual Encoder
Standard Transformer stack (attention + feedforward) produces hidden representations h ∈ ℝ^(vocab_size).
2. Equilibrium Solver
Instead of a simple linear projection, we solve:
z* = f(z*, h; θ)
using Anderson acceleration or fixed-point iteration. The function f encodes game-theoretic constraints (e.g., softmax normalization for QRE structure).
3. Quantal Response Layer
The output distribution is computed as:
π = softmax(z*)
This implements the quantal response structure from economic theory: agents play strategies proportional to their payoff.
Training
We train EqLM using standard cross-entropy loss on masked language modeling (MLM) objectives, on datasets from BabyLM and other standard benchmarks. The equilibrium layer is trained end-to-end via implicit differentiation.
Inference
At inference time, fixed-point iteration is used to converge to the equilibrium. We measure convergence via MMD (maximum mean discrepancy) and use early stopping to balance accuracy and latency.
Convergence Analysis
We analyze convergence properties using:
- Maximum Mean Discrepancy (MMD): A kernel-based divergence metric measuring distance between learned and target distributions.
- Fixed-Point Theory: Classical results on convergence rates of iterative methods apply.
- Anderson Acceleration: Improved convergence velocity via quasi-Newton methods.
Mechanism Design for Alignment
Following Duetting et al. (2024), we frame language model alignment as a mechanism design problem. The equilibrium layer can be augmented with incentive constraints that enforce preference alignment by construction.
Complexity and Tradeoffs
Computational Cost: Fixed-point iteration adds inference latency but reduces model depth (memory).
Theoretical Clarity: Equilibrium framing sacrifices some training efficiency for interpretability and theoretical grounding.
Scope: EqLM is designed for interpretability and theory, not frontier performance. We benchmark against GPT-2 and BERT-class models at matched token budgets.