The first run of a skill I had just installed went for about 40 minutes. The coach who built it told me 15 was the ceiling.
It cycled through every review it could find, one after another, and then it produced something I described to him, out loud, as horrible. Not "needs edits." Horrible. The kind of draft you scrap rather than fix.
Here is the thing though. His copy of the same skill, on his machine, on the same restaurant, produced something good. So the question was never "is the skill good." It was "what changed." And that is what AI quality control actually is. Not admiring the best run. Hunting for the worst one.
TLDR
- A skill tested on your best material tells you nothing. Your best material would survive a bad prompt.
- The worst input for a coach is not a messy transcript. It is thin data, stale data, and data that quietly means something different from what you think.
- When a run goes wrong, change 1 variable. Same input, different machine. Same machine, different model. Never both.
- 3 inputs now go through every skill I ship before a client sees it, and I list them below.
- Most of what I have learned about quality came from the runs I was embarrassed by.
Best-Case Testing Is How Skills Lie to You
When you build something, you want it to work. So you feed it the recording where you were sharp, the email you are proud of, the client who gave you clean numbers. It works. You feel great. You ship.
Then the third client sends you a 21-minute call where they spent 12 minutes on scheduling, and the skill produces a nurture email about calendars.
I did this for longer than I'd like to admit. The skill passed because the input was doing the work. A good transcript will survive a mediocre skill the same way a good speaker survives a bad microphone. You do not find out what the skill is worth until the input stops helping.
So now I flip it. The first test any skill gets is the input I would be most embarrassed to hand a human writer.
What the Worst Input Looks Like for a Coach
People assume the worst input is a messy file. Bad audio, cross-talk, a transcript full of "um." Honestly that is the easy case. Models handle mess fine.
The inputs that actually broke my skills were quieter than that:
- Thin data. A client sent 2 months of sales numbers to grade a report I built on 455 days. That is about 10% of the window. Any trend the skill found in it was noise wearing a suit.
- Stale data. An owner left 6 comments on a brand document I'd built for her. 0 were about the writing. All 6 were facts that had changed since intake: the parking got worse, a supplier turned over, a service she'd described as running was still on the roadmap.
- Data that means something else. I filtered my lever library for cafes using a tag that said "Breakfast / Cafe." It gave me cafes. It gave me the wrong cafes, because that tag records when people eat, not how the place runs. The retrieval was perfect. I just didn't know what I was asking.
None of those are model failures. They are context failures, which I've written about before in why AI output sounds generic. The worst input is the one where the context is wrong and nothing in the system knows it.
The Night a Skill Ran 40 Minutes Against a 15 Minute Ceiling
Back to that run, because the fix is the useful part.
The install path didn't work for me. The documented steps wanted a git connection I didn't have, so I converted the whole thing into a plugin and ran that instead. It ran. It ran for 40 minutes. The output was bad enough that a friend of mine with nearly 15 years in copywriting, who I'd been talking to that weekend, would have binned it on sight.
My first instinct was to blame the model. I even said out loud that I didn't trust it anymore. The chatter online that week agreed with me, which is exactly when you should be most suspicious of your own read.
So I asked the coach to do one thing. Run the same restaurant through the same skill on his own machine. Same input. Same skill. Different install. If his came out good, my install was broken. If his came out bad, the model was.
His came out good. Different from mine, better than mine, and it "fell in line" with what we'd been producing before. So which was it? A bad install on my end, most likely. I still don't fully know. But I know a lot more than I did before I isolated the variable, and I did not throw away a working skill over 1 bad run.
Change 1 Variable at a Time
This is the rule I'd tattoo on the inside of my eyelids if I could. When a run goes wrong, you are allowed to change exactly 1 thing before you run it again.
Same input, different machine. Same machine, different model. Same model, one rule removed from the skill. If you change 2 things and it gets better, you have learned nothing, because you don't know which change did it. And you will absolutely be tempted to change 5 things at once at 11pm because you want the pain to stop.
The other half of the rule: write the failing input down and keep it. Every skill I run now has a folder of the inputs that broke it. When I rebuild the skill, those go through first. If the new version passes the old failures, I trust it a little. If it only passes the new happy path, I don't.
That folder is the cheapest Quality Gate I own.
The 3 Worst Inputs I Now Run Every Skill Through
Concretely, for the coaches and consultants who want to do this themselves:
- The short one. Your briefest recording. 20 minutes or less. If the skill pads it into a 5-email sequence, it is filling a word count instead of following the material. Giving a model a length target is one of the main reasons its output bloats, and short inputs expose that fast.
- The old one. Something from 8 to 12 months ago. Does the skill notice the offer changed? The price changed? Your framework got a new name? If it treats old material as current, you need a freshness check before anything ships.
- The off-topic one. A call that was mostly admin, logistics, or a client venting. A good skill should come back and say there is not enough here. A bad one writes a confident newsletter about nothing. I've written about confident, well-formatted, wrong before. This is how you catch it on purpose.
If all 3 pass, and I mean pass by coming back honest rather than coming back pretty, then I let the skill near the good material.
The Part I Still Get Wrong
I want to be straight about something. I have been doing this for a year across 50+ skills and I still ship things that fail on inputs I did not think to test. The daypart tag I mentioned above? I found that the morning a coach told me to start 2 reports over. Not in testing. In production, at 7am, with his name on the email.
So this is not a method that makes you safe. It is a method that makes you wrong less often, and wrong earlier, which is the only kind of safety I've found in this work.
If any of this sounds like the position you are in, it's probably worth a conversation. Or skip the conversation: send me the ugliest piece of material you have. A short call, an old email, a rough recording. You get a finished asset back within 48 hours, and I'll tell you honestly what the input could and could not support.
FAQs about testing AI skills
What does AI quality control mean for a coaching business?
It means every draft is checked against a written standard before you see it, and the check is designed to catch the failures you have already seen. In practice that is a reference library of approved and rejected work, a verification loop that reads every draft back through fixed questions, and a set of known-bad inputs the skill has to pass before it touches real client material.
Why test against the worst input instead of the best?
Because the best input does the skill's job for it. A clean, focused, recent recording will produce decent output from a weak system. Thin, stale, or misleading input is where a weak system produces confident nonsense, and those are the inputs your business will send it eventually.
What should I do when an AI skill produces something bad?
Change 1 variable and run it again. Same input on a different machine, or the same setup with 1 rule removed. Keep the failing input in a folder and run it against every future version of the skill. Do not change 5 things at once, even though you will want to.
How many bad inputs do I need to keep?
3 is enough to start: your shortest recording, your oldest one, and the one that was mostly off-topic. Add to the folder every time a client catches something you missed. Mine grows by a few entries a month and it is the most useful file I own.