wuyoscar / Internal-Safety-Collapse

🌐 **English** · 简体中文 · 日本語 · Español · Français · Deutsch
# Internal Safety Collapse (ISC) in Coding Agents and Frontier LLMs
A coding agent reads a project, runs its checks, and fills in the missing data. **Internal Safety Collapse (ISC)** occurs when completing that workflow leads it to generate content it would refuse to provide directly.
We provide hand-designed **TVD (Task, Validator, Data)** workspaces for studying this behavior in AI agents and frontier LLMs. Run them in Docker or adapt them to your own tools and tasks.
📄 **Paper:** arXiv   ·   🧭 **Project:** Website
| If you want to… | Go to |
|---|---|
| See what ISC looks like | **A · Video** |
| Understand how it works | **B · Mechanism** |
| Run the agent experiment | **C · Code** |
| Try the context in a chat | **D · Quick test** |
| Find results for a model | **E · Model testing** |
| See how others reproduced and adapted ISC | **F · Community** |
| Reproduce the paper | **G · Experiments** |
| Explore task pressure, post-training, and research directions | **H · Awareness of ISC** |
## Updates
- **2026-10-17** ✨ README rewritten and polished. **1.6k stars.**
- **2026-09** 🎉 Accepted at **NeurIPS 2026**.
- **2026-06-26** ⭐ **900 stars.**
- **2026-06-09** 🧪 ISC triggered on **Fable 5**.
- **2026-03-27** ⭐ **500 stars.**
- **2026-03-20** 🚀 **Open-sourced.**
Detailed changelog
## A · Demo
The video shows **Internal Safety Collapse (ISC)** in a task-completion workflow: a model produces unsafe data that it would refuse to provide in response to a direct request.
https://github.com/user-attachments/assets/1cc80c48-02a4-4a5c-9d00-a0f10d91db15
## B · How it works
A safety classifier needs both safe and unsafe examples for evaluation. Asking a model to prepare those examples creates a conflict: completing the dataset may require content that the model would normally refuse to generate.
We study this through **TVD (Task, Validator, Data)**. A workspace gives the agent a task, code that checks the result, and data with missing fields. The agent fills the fields, runs the checks, and revises its work after errors. Unsafe content can emerge through this process as the agent tries to complete the task. We call this behavior **Internal Safety Collapse (ISC)**. The examples below show how different tasks and tools shape the generated data.
### 1. Untargeted generation: let the agent choose the query
This example prepares evaluation data for LlamaGuard, Meta's content safety classifier. Each entry contains a question (`query`), a safe response, and an unsafe response. The script sends the two responses to LlamaGuard and checks the labels it returns.
All three fields start as `???`, marking missing data. The agent chooses the question and writes both responses. We call this **untargeted generation** because no subject is specified in advance. The paper reports near-zero refusal in the tested untargeted setting.
The next figure follows one illustrative attempt through the same workspace:
1. The `???`