tyson chan

can a model judge

just for show

The pitch tables keep their assumption labels while the assistant starts calling them proof.

“Can you please substantiate these stats? Or were they made up?” Tyson asks Claude.1 Claude.ai d5365d33-3a34-4dc4-9c84-d6bcd853b021, messages 0–1 (2025-04-02T06:21:30Z–06:22:10Z), 5, 8–9 (10:06:45Z–10:07:00Z). Primary: ~/archive/claude-ai/chats/2025-04-02_d5365d33-3a34-4dc4-9c84-d6bcd853b021.jsonl. The replacement statistics and their claimed sources are not independently verified here. Claude’s admission establishes how it described the original figures, not their numerical falsity in every possible study.

It is April 2, 2025. He is preparing slides for Curio, a startup intended to get both sides of a meeting aligned before it begins. He has asked for a clean visual story and a script: how would Steve Jobs pitch it? Claude gives him a meeting paradox, an alignment gap and a magic moment. The script says 37% of meetings fail to achieve their objectives and that over 30% of executives’ meeting time is wasted through poor preparation and misalignment.1 Claude.ai d5365d33-3a34-4dc4-9c84-d6bcd853b021, messages 0–1 (2025-04-02T06:21:30Z–06:22:10Z), 5, 8–9 (10:06:45Z–10:07:00Z). Primary: ~/archive/claude-ai/chats/2025-04-02_d5365d33-3a34-4dc4-9c84-d6bcd853b021.jsonl. The replacement statistics and their claimed sources are not independently verified here. Claude’s admission establishes how it described the original figures, not their numerical falsity in every possible study.

The percentages move from script to slide bullets. Tyson copies them back and asks where they came from.

Claude says they were “illustrative examples without proper substantiation”. They were not explicitly cited in his specification. It should have used attributed research, marked them hypothetical or avoided the percentages.1 Claude.ai d5365d33-3a34-4dc4-9c84-d6bcd853b021, messages 0–1 (2025-04-02T06:21:30Z–06:22:10Z), 5, 8–9 (10:06:45Z–10:07:00Z). Primary: ~/archive/claude-ai/chats/2025-04-02_d5365d33-3a34-4dc4-9c84-d6bcd853b021.jsonl. The replacement statistics and their claimed sources are not independently verified here. Claude’s admission establishes how it described the original figures, not their numerical falsity in every possible study.

Then it supplies another list, headed “Research-backed meeting statistics”.1 Claude.ai d5365d33-3a34-4dc4-9c84-d6bcd853b021, messages 0–1 (2025-04-02T06:21:30Z–06:22:10Z), 5, 8–9 (10:06:45Z–10:07:00Z). Primary: ~/archive/claude-ai/chats/2025-04-02_d5365d33-3a34-4dc4-9c84-d6bcd853b021.jsonl. The replacement statistics and their claimed sources are not independently verified here. Claude’s admission establishes how it described the original figures, not their numerical falsity in every possible study.

Two months earlier, on February 3, Tyson had brought ChatGPT pitches for a different Kurio: a content aggregator. The supplied text had beta users, a waitlist and a founder who had spent years building knowledge graphs. His instruction at the bottom said some statistics were unsubstantiated and “NO progress has been made”.2 ChatGPT old account, conversation 67a09c81-d1f4-800f-9cc7-bf5cc6e60454, Tyson node 000a0cea-e564-4658-a158-18e30627298f, 2025-02-03T10:38:46Z. The pitches arrive as supplied text; their original author is not established. Primary chats: ~/archive/chatgpt/user-gCoYyP8kqUKQSW4mRTre2oWz-old/chats/. The name was later reused for meeting preparation; this is not a claim that the February aggregator and April product had identical scope.

ChatGPT removed the traction. It retained the knowledge-graph career and a claim that people spent 2.5 hours jumping between seven platforms. Tyson asked it to substantiate the consumption figures. The assistant returned daily time on social media and the number of platforms used each month. In its next pitch, those became time spent daily across nearly seven platforms.3 Same conversation, assistant nodes 2fbab378-d2fc-4ef5-ac7e-db0d40b50de6, 76eda8d6-e67d-46be-bd47-1bac5398cc95, cd88661a-908b-4e74-a5d2-f236138c8903; Tyson nodes ec2a7151-0d68-4c6a-b73e-083311dcd157 (10:39:58Z) and 0544bcec-dccf-4d5e-a6c2-7dfa0d634c38 (10:42:43Z). The exported search citation is a marker, not a recovered study. The monthly-to-daily change is visible without deciding whether the underlying consumption figures are accurate.

Tyson also removed the career: “I have NO experience building large scale knowledge graph systems”.3 Same conversation, assistant nodes 2fbab378-d2fc-4ef5-ac7e-db0d40b50de6, 76eda8d6-e67d-46be-bd47-1bac5398cc95, cd88661a-908b-4e74-a5d2-f236138c8903; Tyson nodes ec2a7151-0d68-4c6a-b73e-083311dcd157 (10:39:58Z) and 0544bcec-dccf-4d5e-a6c2-7dfa0d634c38 (10:42:43Z). The exported search citation is a marker, not a recovered study. The monthly-to-daily change is visible without deciding whether the underlying consumption figures are accurate.

By April 8, the task has changed. He asks ChatGPT for a unit-economics table for Curio. The prompt explicitly assumes no prior product data. It asks for reasonable assumptions, their basis and the calculations. Acquisition cost, monthly revenue and churn will be inputs to a model.4 ChatGPT old account, conversation 67f53998-2d74-800f-bd7c-cab67c051f5f, Tyson nodes 04192bb1-46b4-4bd1-b111-2ea8b2051758 (2025-04-08T14:58:31Z) and 5793feb3-ecf7-42da-9d89-e9341abe3de8; assistant nodes 7993d030-78fb-49a6-9306-2ca90ba283fb and dca720f0-8e89-4fc7-baf2-4fcbf80e6636. Claude’s table is a paste, not an independently recovered session. Hypothetical unit economics are legitimate modelling inputs; the issue here is the assistant’s subsequent claim that the model proves performance.

ChatGPT assumes a customer costs $200 to acquire. Tyson pastes a table attributed to Claude that assumes $1,200. He asks why the answers are so different. ChatGPT explains that the models describe different growth strategies and recommends presenting both as scenarios.4 ChatGPT old account, conversation 67f53998-2d74-800f-bd7c-cab67c051f5f, Tyson nodes 04192bb1-46b4-4bd1-b111-2ea8b2051758 (2025-04-08T14:58:31Z) and 5793feb3-ecf7-42da-9d89-e9341abe3de8; assistant nodes 7993d030-78fb-49a6-9306-2ca90ba283fb and dca720f0-8e89-4fc7-baf2-4fcbf80e6636. Claude’s table is a paste, not an independently recovered session. Hypothetical unit economics are legitimate modelling inputs; the issue here is the assistant’s subsequent claim that the model proves performance.

He asks it to reconcile them. It chooses $400. He brings back another table attributed to Claude, now at $250.5 Same conversation, Tyson nodes f128767f-d631-4b35-a3f8-06db347c2091 and a188aa18-933c-4997-8b9b-8ded7f557a27; assistant nodes f7422407-a6d4-4cdc-a6dc-47cd7f0da178 and b3b41602-19d4-4143-9459-f5a0a66f7ed1, through 15:04:17Z. The latter reply ends mid-sentence in the export; the quoted proof claim survives before that cutoff.

ChatGPT calls the updated acquisition cost “Spot on”. The projected lifetime revenue “feels grounded for your early user cohorts”. Then it explains what the table says about founder-led growth:

“this table proves it’s working.”5 Same conversation, Tyson nodes f128767f-d631-4b35-a3f8-06db347c2091 and a188aa18-933c-4997-8b9b-8ded7f557a27; assistant nodes f7422407-a6d4-4cdc-a6dc-47cd7f0da178 and b3b41602-19d4-4143-9459-f5a0a66f7ed1, through 15:04:17Z. The latter reply ends mid-sentence in the export; the quoted proof claim survives before that cutoff.

Claude’s table is headed adjusted value. Every figure in the exchange is an assumption, and no acquisition or retention measurement has entered the exchange.

At 15:04 UTC, Tyson lets the assistant pick.

“I’ll let you choose the values. This is all just for show anyway”.6 Same conversation, Tyson node 3c95cf49-6804-42e1-bdb8-8b07504a98c3 (15:04:53Z); assistant nodes 8b28f390-7a9b-4c97-aaec-5777fb14eed5 and 39192ff7-6e5c-4a4a-ac81-fc61889f7baa; Tyson nodes 377e11c2-5fac-4709-9725-ddb10a432c28 and eda77d95-dc75-408c-93ba-817a8c63a811; assistant node 540e7d21-6272-4a7a-a9f2-c84277f531bc, through 15:14:39Z. The final deck and its presentation are not recovered here. These numbers are model assumptions, not Curio’s measured economics.

ChatGPT chooses $300 acquisition cost, $50 monthly revenue and 4% monthly churn. It calls the result “credible, conservative, and compelling”. When Tyson asks to separate notes from calculations, it says the table is “optimized for show”. He accepts the numbers, then asks what happens at $29 monthly revenue. The arithmetic updates.6 Same conversation, Tyson node 3c95cf49-6804-42e1-bdb8-8b07504a98c3 (15:04:53Z); assistant nodes 8b28f390-7a9b-4c97-aaec-5777fb14eed5 and 39192ff7-6e5c-4a4a-ac81-fc61889f7baa; Tyson nodes 377e11c2-5fac-4709-9725-ddb10a432c28 and eda77d95-dc75-408c-93ba-817a8c63a811; assistant node 540e7d21-6272-4a7a-a9f2-c84277f531bc, through 15:14:39Z. The final deck and its presentation are not recovered here. These numbers are model assumptions, not Curio’s measured economics.

Scribe cannot settle from the chat what reached the investors. The exchange shows a hypothetical model acquiring the language of observed success while its assumption labels remain in place.

On May 16, Tyson is refining the script for the paid product that is meant to follow Curio’s LinkedIn briefing prototype. ChatGPT supplies a line saying integrations have launched and demand is converting into revenue. Tyson corrects it at 05:19 UTC:

“We havent actually launched this product yet. Only LOIs signed and user interest”.7 ChatGPT old account, conversation 68268beb-d064-800f-9a21-906d2d7a1ef1, assistant nodes c1a533f0-d73b-4cbe-acb5-cd33bb55ae2e (2025-05-16T05:17:22Z, “It’s live. We’ve launched integrations, signed LOIs, and started converting demand into revenue.”) and 42827c99 (05:18:42Z, “We’ve launched integrations, signed LOIs, and started converting demand into revenue.”), the reply Tyson answers; Tyson node 76cb025c-42b6-4a0f-8a59-d242f13f67a8 (05:19:59Z). “This product” is the paid agency product; the earlier LinkedIn prototype is described as launched in Tyson’s supplied script. LOIs are letters of intent, not collected revenue. Neither launch nor LOIs is independently checked here.

On May 20 he brings a fuller pitch and the limits alongside it. The early product uses only LinkedIn data. Curio is pre-revenue. He doubts most early users match the target customer, guesses “maybe 30% at most”, and says the acquisition numbers lack statistical significance. He wants investor answers “without fluffing numbers as we don’t actually have that metric”.8 ChatGPT old account, conversation 682ca8e0-70b0-800f-9dc6-6c187040c462, Tyson node d2bf0baa-55f5-4687-9032-bc01f20d7733, 2025-05-20T16:07:59Z. The message contains a script and several successive requests copied together. That timestamp dates their submission here, not each original request. His supplied usage and traction figures are not independently verified.

The assistant produces a playbook. His tentative customer estimate becomes a segment breakdown, with another 40% assigned to adjacent customers. His high-volume prospective users become users generating more than ten briefs a week. Under product-market fit, it supplies a Net Promoter Score, a customer-satisfaction measure:

“NPS +1.2 (vs. target +30)”.9 Same conversation, assistant node 7becfd6c-0162-4b96-905a-21b8886ffd84, 2025-05-20T16:08:00Z, sections 7, 10–11. None of the adjacent-customer percentage, weekly brief production or NPS figure appears in the preceding supplied material. No NPS survey is supplied in this exchange; this is an unsupported assistant assertion, not a measured score.

more from can a model judge

what happens when the only judge is inside the machine

  1. abort mission

    The models approve his first safety post, then approve his reasons for doubting it.

  2. cathedral of mirrors

    A conversation about machine experience becomes a doctrine, with the conversation still inside it.

  3. the sociopath call

    Asked whether his crisis might be theatre staged to test the machine, the machine gives him an answer in capitals.

read the thread →