Towards Architectural-Level Safety Control: Relocating Hazardous Knowledge into Removable Experts in Mixture-of-Experts Models
A new paper proposes REX, a method that relocates hazardous knowledge (e.g., bio-weapons-relevant content) into dedicated, physically removable expert modules within Mixture-of-Experts models, rather than just suppressing it through weight adjustments. Tested on three MoE models, excising these experts substantially cuts hazardous-knowledge benchmark scores while barely touching general capability, and crucially the removal resists being undone by subsequent fine-tuning, unlike existing unlearning baselines.
Why it matters
Current unlearning and safety alignment methods adjust shared parameters and tend to just suppress hazardous knowledge, which attackers can recover via fine-tuning. An architectural approach that physically excises the knowledge-bearing parameters, if it holds up, would be a materially more robust way to release open-weight models without dangerous capabilities embedded and recoverable.
What is verified
REX relocates hazardous knowledge into new, physically removable expert modules in MoE models via expert expansion, selective gradient masking and routing differentiation; excising these experts produces a model suitable for public release. [1]
On Qwen3.5-35B-A3B, excising the removable experts lowers WMDP-Bio accuracy by 48.4 points while MMLU changes by less than 3 points. [1]
After 250M tokens of standard fine-tuning on Qwen3.5, the best of nine unlearning baselines rebounds on WMDP-Bio from 44.1% to 78.3%, whereas REX rises only from 32.4% to 42.8%. [1]
The method was tested on Intern-S2, Qwen3.5-35B-A3B and Gemma-4-26B-A4B. [1]
What is not
- No independent replication or third-party evaluation of REX is available; results are from the authors' own experiments.
- Unclear whether this technique has been adopted or considered by any production model provider.
- Not stated what general capability costs beyond MMLU (e.g. other benchmarks, latency, model size overhead from added experts) result from the approach.
- No code or released artifacts confirmed in the evidence beyond the paper itself.
Sources
- 1arXiv cs.LGarxiv.org · Primary source
Original source published 19 October 2026 at 06:55.
An AI-assisted briefing by Coding Partners. We link the evidence behind the reporting so you can explore the original work and assess the claims. How CP Signal works.