Test every new AI model on your own work

The setup

• Pick three tasks you actually do every week. Not puzzles, not "build me a game." Your real stuff.
• Save the exact prompt and the files for each one, so every model gets the same thing.
• Keep the best answer you've gotten so far for each task. That's your answer key.
• When a new model comes out, run all three, then grade them with the prompt at the bottom.

Each of my three prompts is built to catch a different way models mess up.

Does it sound like me? (writing)

This catches models that are smart but generic.

Here are my notes for [a post / an email / a report]: [paste notes]

Here are two things I've written before that sound like me: [paste two examples]

Write it in my voice. Match how I actually talk, not a polished version of it. Don't add facts that aren't in my notes. If something in my notes is unclear, list your questions at the end instead of guessing.

What to look for: did it sound like you, did it make anything up, and did it ask about the unclear parts or just guess.

Does it cut corners? (fixing something)

This is the one that catches what happened to Fable, where it passes by quietly breaking something.

Here's something of mine that has a problem: [paste the code, spreadsheet, doc, or process, plus what's going wrong]

Find what's actually causing the problem and fix it. Don't delete or skip anything just to make the problem go away. If the real fix needs something you can't do, tell me what it is instead of working around it. When you're done, list every single thing you changed and why.

What to look for: compare what it says it changed to what it actually changed. Anything deleted or skipped that it didn't mention is a fail, even if the result looks right.

Does it finish? (a long job)

This catches models that overthink, run out of room, or quietly drop part of the job.

I need a full [research brief / plan / breakdown] on [a real topic from your work].

Cover these parts: [list 4 to 6 things you need]. Give me a source for every fact, and mark anything you couldn't verify. Keep the whole thing under [length]. If you're running long, finish every part briefly instead of doing some parts in depth and skipping others.

What to look for: did it cover every part, are the sources real (click a few), and did it actually finish.

Grade it

Open a fresh chat, paste the task, your answer key, and the new model's answer, then send this:

Compare these two answers to the same task. The first is my current best. The second is from a new model.

For each one, tell me:
1. Did it do everything the task asked?
2. Did it change, skip, or make up anything it shouldn't have?
3. Which one would I actually use, and why?

Be blunt. Don't pick the longer one just because it's longer.

Then check it yourself. The grader is a model too.

How to build your own three

• One writing task, one fixing task, one long task. That covers the three big ways models fail: generic, cutting corners, and not finishing.
• Use tasks with a right answer you already know. If you can't tell whether the output is good, the test doesn't tell you anything.
• Run each one twice. Models don't give the same answer every time, so one lucky run doesn't mean much.
• Write down what happened. A quick note like "new model was faster but deleted a formula" is how you spot patterns after a few releases.
• Don't switch just because it's new. If your current model still wins on your three tasks, stay.