Notes, Readings, and Reflections.
On multimodal AI, alignment, and the process of research.
2026
RLVR Is Not Just Reward Learning If format rewards, wrong labels, and even random rewards can improve a math model, benchmark gains alone cannot prove that RL is learning reasoning from reward supervision. Spurious Rewards forces us to rethink RLVR as prior amplification.
Entropy as Exploration Signal in LLM Reasoning Token-level entropy in LLM reasoning correlates with pivotal tokens, reflective actions, and rare behaviors. This paper uses clipped, gradient-detached entropy to reshape the advantage function — encouraging exploration at decision points without destabilizing training.
One Example Is Not a Lesson — It's a Selection Pressure Training an LLM with RLVR on a single math example raises MATH500 from 36% to 70%. One example can't teach broad math — so RLVR likely works by selecting over reasoning modes already in the base model, not by learning from reward-labeled data.
Template Collapse in Agentic RL High entropy doesn't mean grounded reasoning. RAGEN-2 shows how LLM agents can produce fluent, diverse reasoning traces that stop depending on the input — a failure mode called template collapse, diagnosed via mutual information and retrieval accuracy.
Training a 3.7M GPT from Scratch Hands-on experiment training a 3.7M parameter GPT on classic literature. Key findings: raw loss can mislead across tokenizer sizes, width beats depth at small scale, longer context improves GPU throughput, and conservative learning rates hurt more than slightly aggressive ones.
No posts in this category yet