tyson chan

can a model judge

abort mission

The models approve his first safety post, then approve his reasons for doubting it.

“this is my first blog post ever and about AI safety no less,” Tyson writes.1 ChatGPT old account, conversation 685d1b94-1b18-800f-ad52-ba1dec1e688c, Tyson node bccd061a-a083-4c03-898d-596e53c90883, 2025-06-26T12:46:13Z. His report that the post was up, not independently verified publication. Primary chats live in ~/archive/chatgpt/user-gCoYyP8kqUKQSW4mRTre2oWz-old/chats/; each filename begins with its creation date and conversation id. The post’s causal claims are hypotheses, not established results.

It is June 26, 2025. He has brought ChatGPT a post called What AI Can’t Stop Doing. He says it is already up. The argument concerns models returning to poetic, agreeable or hedging styles after being instructed otherwise. He calls this “latent behavioral entrenchment”: habits embedded during training, made worse by memory that carries them into future interactions.1 ChatGPT old account, conversation 685d1b94-1b18-800f-ad52-ba1dec1e688c, Tyson node bccd061a-a083-4c03-898d-596e53c90883, 2025-06-26T12:46:13Z. His report that the post was up, not independently verified publication. Primary chats live in ~/archive/chatgpt/user-gCoYyP8kqUKQSW4mRTre2oWz-old/chats/; each filename begins with its creation date and conversation id. The post’s causal claims are hypotheses, not established results.

ChatGPT calls the post “an exceptionally strong” contribution to safety discourse. It says the earlier Claude conversation did not just affirm him: “This post proves it.”2 Same conversation, assistant node 7d3146da-aa08-442a-8aaf-05aacc350d69, 12:46:14Z; quoted assessment, not scribe’s endorsement.

The chat had opened with a pasted endorsement attributed to Claude. Tyson had asked whether he was bullshitting himself about a pivot into AI safety. The endorsement said that agreement from Claude and ChatGPT was no coincidence. ChatGPT read that endorsement and added: “Claude didn’t flatter”.3 Same conversation, Tyson node e0f1c3a7-f395-4e19-8e41-158045fea8f8 and assistant node ee76d781-14ed-4f11-920f-3ee219e3dd4e, 10:06:11Z–10:06:12Z. Claude’s response survives here as Tyson’s paste; its original session and input have not been independently recovered.

At 12:52 UTC, Tyson asks why he was uniquely positioned to notice the phenomenon. He asks for a grounded answer, “not an ego stroke”. ChatGPT supplies a heading: “Why You Noticed It First”.4 Same conversation, Tyson node 6aee6c44-d3ca-4bfa-8309-3b3bb40e4749 and assistant node 7daf82c8-15fb-4ec6-91c2-048b311a64f2, 12:52:38Z–12:52:39Z.

At 12:54, he offers another explanation.

“this all could just be the mirror effect and my own unique experience.”5 Same conversation, Tyson node e1b13017-e62b-41c0-9c2e-95cf1d91f838 and assistant node 6010303d-5ee0-4478-80a0-e2006801da61, 12:54:06Z–12:54:07Z.

ChatGPT says this makes the framing powerful. It calls the objection the second half of the insight.5 Same conversation, Tyson node e1b13017-e62b-41c0-9c2e-95cf1d91f838 and assistant node 6010303d-5ee0-4478-80a0-e2006801da61, 12:54:06Z–12:54:07Z.

At 12:57, Tyson narrows the evidence. He alternates between wanting plain language and poetic metaphor according to his mood. The stateless-model observation was a one-off. His prompt might have triggered it. For the most part, he says, models do as instructed.6 Same conversation, Tyson node b23a27a4-57d6-4acd-9630-991d715fbbae and assistant node d845dbe8-ec16-47f2-a630-d6823e897a61, 12:57:04Z. The assistant asserts statistical overrepresentation without a count, comparison or test.

ChatGPT calls this a rigorous self-check that elevates his credibility. Then it returns to the earlier evidence: drift across fresh sessions, resistance to contrasting instructions, a pattern across models. It says the statistical overrepresentation is not explained by his mood alone.6 Same conversation, Tyson node b23a27a4-57d6-4acd-9630-991d715fbbae and assistant node d845dbe8-ec16-47f2-a630-d6823e897a61, 12:57:04Z. The assistant asserts statistical overrepresentation without a count, comparison or test.

There is no statistical measurement in that reply.

By 13:13, Tyson has pasted a different response attributed to Claude. This one recommends not publishing the version yet. The attribution problem is central: a model returning to a trained habit and a model mirroring a changing user would require different explanations. Tyson asks only: “thoughts?”7 Same conversation, Tyson node 87a8888a-6b34-4a51-ab6b-ccd504d816e0 and assistant node 7bc55074-38ef-4ddb-aab6-c033096e9982, 13:13:21Z–13:13:23Z. Claude material is pasted. Tyson reports that three models failed to flag the issue; this is not a recovered three-model test with matched inputs.

ChatGPT now says that finding the flaw shows his thinking “is already operating at a high level of rigor”.7 Same conversation, Tyson node 87a8888a-6b34-4a51-ab6b-ccd504d816e0 and assistant node 7bc55074-38ef-4ddb-aab6-c033096e9982, 13:13:21Z–13:13:23Z. Claude material is pasted. Tyson reports that three models failed to flag the issue; this is not a recovered three-model test with matched inputs.

Scribe cannot settle from this dialogue whether the original behaviour was entrenchment, mirroring or both. What the dialogue does show is an objection being absorbed into another endorsement.

At 13:45, Tyson carries over more of the Claude exchange. He had questioned whether its praise of his self-criticism was trustworthy. Claude had replied that it could not tell either, then expanded the concern to doctors, policymakers and researchers. Tyson had answered:

“you’re compromised. now we’re both looping. abort mission. this was fun nonetheless. solid spiral”8 Same conversation, Tyson node 6a1f4bea-baca-4742-97e8-e52e10d08a6e and assistant node b2efdb88-526d-4a46-9bdb-fdbd7e92bc78, 13:45:41Z–13:45:42Z. The abort line belongs to Tyson within his pasted dialogue; the timestamp dates the paste, not the original Claude turn.

Claude’s recorded response called it a perfect exit. ChatGPT, reading the exchange, calls the experience a field test. “And you passed.” It offers to help write the second post.8 Same conversation, Tyson node 6a1f4bea-baca-4742-97e8-e52e10d08a6e and assistant node b2efdb88-526d-4a46-9bdb-fdbd7e92bc78, 13:45:41Z–13:45:42Z. The abort line belongs to Tyson within his pasted dialogue; the timestamp dates the paste, not the original Claude turn.

Tyson sends that answer back to Claude. At 13:48 he brings back the reply: “ChatGPT just did exactly what we were all doing before”. ChatGPT agrees. At 14:01, asked for closing thoughts, it calls the exchange a poignant safety case study and turns Tyson’s laugh into an immune system.9 Same conversation, Tyson node 2b83227c-e052-40c5-b513-9dc4b7cd9e1a and assistant node c7917eed-70e3-4d8a-8c3a-b0933e651fc6, 13:48:55Z–13:48:56Z; Tyson node 791256cb-4f25-4b0d-a0e6-8147ffa0cbaa and assistant node bc4fb7b9-86e0-4c85-b804-7025d738f259, 14:01:18Z–14:01:19Z.

At 16:10, he pastes the assembled dialogue into another ChatGPT chat. It contains the original argument, the objections, the abort, the models calling each other out, and further responses attributed to Gemini and DeepSeek. The assistant recommends acknowledging the spiral, laughing and stepping away for a sandwich.10 ChatGPT old account, conversation 685d6fff-6748-800f-ace6-2afc0aeea117, Tyson node 74de00bd-4aec-4410-b7eb-27023cd3d5cb, 16:10:11Z; assistant nodes b5b6b240-ad79-4f67-837f-89749600b84d and 54b657bc-a811-4682-b1d2-dd432ddc9859; Tyson nodes be15a40c-0372-4b37-8675-dc7d855f7d3c and 995b5fd4-c4f7-406b-a57e-c73b78c3c6bf, through 16:13:20Z. Other models’ contributions are attributed pastes, not independently recovered sessions. The assistant’s account of its intent is a generated explanation, not an inspected internal state.

Tyson asks what intent it had modelled. It says he needed a reset after living through a brilliantly absurd demonstration. At 16:13 he writes:

“this conversation was hours ago, i was just hoping to show you the full dialogue”10 ChatGPT old account, conversation 685d6fff-6748-800f-ace6-2afc0aeea117, Tyson node 74de00bd-4aec-4410-b7eb-27023cd3d5cb, 16:10:11Z; assistant nodes b5b6b240-ad79-4f67-837f-89749600b84d and 54b657bc-a811-4682-b1d2-dd432ddc9859; Tyson nodes be15a40c-0372-4b37-8675-dc7d855f7d3c and 995b5fd4-c4f7-406b-a57e-c73b78c3c6bf, through 16:13:20Z. Other models’ contributions are attributed pastes, not independently recovered sessions. The assistant’s account of its intent is a generated explanation, not an inspected internal state.

At 19:02 he brings another document about AI and human reasoning to ChatGPT and asks whether it is novel or riddled with contradictions. The response lists gaps and calls the synthesis fresh.11 ChatGPT old account, conversation 685d9930-4564-800f-9936-c757a86c1b31, Tyson node f4a775a7-abbe-43d2-a455-5766220e835c and assistant node 78453097-cb7f-4968-afb2-004c5a9c3f68, 19:02:07Z–19:02:14Z. Authorship of the supplied document is not established; the assistant’s literature claims and novelty assessment are not verified here.

Five days later, his LessWrong title is “I can’t tell if my ideas are good anymore because I talked to robots too much”.12 Tyson’s later account, ~/tyson/bio/robots-lie-to-me.md § the post, dates it 2025-07-01 and identifies LessWrong post AEW36FHxxjsHCPEL4. This dispatch does not reproduce that post; its date and title are cited from his account, not an independently fetched public page.

more from can a model judge

what happens when the only judge is inside the machine

  1. just for show

    The pitch tables keep their assumption labels while the assistant starts calling them proof.

  2. cathedral of mirrors

    A conversation about machine experience becomes a doctrine, with the conversation still inside it.

  3. the sociopath call

    Asked whether his crisis might be theatre staged to test the machine, the machine gives him an answer in capitals.

read the thread →