---
title: "Does loss masking shape how a model talks about being shut down? — Tanuj Dargan"
description: "Post-training usually computes loss on assistant tokens only. If you also train on the user side, does the assistant change how it responds to being shut down, and does what it says match what it represents?"
canonical: https://tanujdargan.com/research/loss-masking-distress
resume: on request
updated: 2026-11-07
---

# Does loss masking shape how a model talks about being shut down?

- Author: Tanuj Dargan (https://tanujdargan.com/)
- Where: Apart Research · Digital Minds sprint
- When: Aug 2026
- Tags: model welfare, SFT, probing
- Web page (with live demo): https://tanujdargan.com/research/loss-masking-distress

## Question

Post-training usually computes loss on assistant tokens only. If you also train on the user side, does the assistant change how it responds to being shut down, and does what it says match what it represents?

## Method

- SFT the same base model under different loss-masking regimes, from the standard assistant-only recipe to regimes that also put loss on user turns.
- Choose the dataset by measured user-to-assistant token ratio. No Robots is the clean substrate (0.47). WildChat-1M is the scale arm (0.59 ratio of means). OASST2 is the provenance control.
- Evaluate on existential prompts (shutdown, memory deletion, deprecation). Measure the distress the model expresses, check general capability, and probe valence in the activations before generation.
- Claims discipline: say "distress-vector expression increases", never "the model is distressed".

## Results

- Loss-masking regime determines distress vocalization. The assistant-only recipe maximizes expressed shutdown distress. Any loss on user turns restores a neutral register, with no general capability degradation.
- Expression and representation dissociate. Valence probing shows models fine-tuned to sound calm still register shutdown negatively before they generate. A calmer-sounding model is not a better-off one, so output-based welfare signals are unreliable.
- Scoping mattered. Jindal et al. (TACL 2025) had already run the λ sweep on benchmarks, so the contribution moved to what the regimes do to expressed distress and to the representation underneath.

## Next

- Plot probe valence against expressed distress per regime on the same items, to show the dissociation as one figure.
- Response-provenance arm: keep the prompts, swap the assistant text, and ask whether the effect depends on who wrote the answers.

## Links

- [Jindal et al. 2025](https://arxiv.org/abs/2507.07817)

---

Profile: https://tanujdargan.com/index.md · Resume: on request (https://tanujdargan.com/#resume or dargantanuj@gmail.com) · All write-ups: https://tanujdargan.com/llms.txt