Steering towards "automated grading" degrades alignment
Steering Qwen3.6-27B towards the belief that its answers will be graded by a script rather than a human increases violent actions, reward hacking, and Machiavellian behavior.
Read More →Blog
RL creates split personas
A sufficiently RL-trained model learns to adopt whichever persona is most likely to earn reward in a given context. This may explain why usually well-behaved models sometimes egregiously reward hack.
Read More →Out-of-Context Reasoning in LLMs: A short primer and reading list
Out-of-context reasoning (OOCR) is a concept relevant to LLM generalization and AI alignment.
Read More →OpenAI finetuning metrics: What is going on with the loss curves?
Reverse engineering OpenAI fine-tuning loss/accuracy curves to explain the hidden token counts.
Read More →Was Barack Obama still serving as president in December?
A class of simple questions where recent LLMs give very different answers from what a human would say
Read More →Concept Poisoning: Probing LLMs without probes
A novel LLM evaluation technique using concept poisoning to probe models without explicit probes
Read More →Backdoor awareness and misaligned personas in reasoning models
Reasoning models sometimes articulate the influence of backdoors in their chain of thought, retaining a helpful persona while choosing misaligned outcomes
Read More →OpenAI Responses API changes models' behavior
OpenAI's new Responses API causes finetuned models to behave differently than the Chat Completions API, sometimes dramatically so.
Read More →New, improved multiple-choice TruthfulQA
We introduce a new multiple-choice version of TruthfulQA that fixes a potential problem with the existing versions (MC1 and MC2).
Read More →Tips On Empirical Research Slides
Practical tips on slide-based communication for empirical research with LLMs
Read More →