50+ AI skills. Real client work, shipping every week, for over a year. Not demos.
I expected the models to be the fragile part. They were not. Across the whole year I can count the number of times a model was the root cause on one hand. Everything else broke, repeatedly, and almost all of it broke in ways that were boring and preventable and entirely my fault.
Here is the list. If you are building your first serious skill, this is the map of where the potholes are.
TLDR
- The models were fine. Skill design, stale references, and missing gates caused nearly everything.
- Long skill files degrade. Past a certain length the instructions in the middle stop being followed.
- A reference library goes stale quietly, and stale output looks like voice drift.
- Skills fail at the edges, not the middle. The middle is easy and misleads you into shipping.
- The most expensive failure is the silent one: confident, well-formatted, wrong.
What Did Not Break
Worth stating plainly, because it reframes everything after it.
Model quality was not the constraint. Not once did a rewrite of a prompt to be "better written" fix a real production problem. The wins came from structure: what the skill could see, what it was checked against, and what stopped it from shipping.
Which means if your AI output is disappointing, the highest-value place to look is not the model and not the wording. It is the material and the gate. I made this argument at length in the piece on generic brand voice, and a year of production has only made me more sure of it.
Break 1: The Skill File Got Long and Stopped Being Read
Every skill I built followed the same arc. Version 1 is short and does 80% of the job. Then a bad output appears, so I add a paragraph. Then another. 6 weeks later the file is 4 times its original length and quality has quietly gone down.
What is happening is that instructions compete. The rule you added on Tuesday dilutes the rule you added in March. Nothing errors. The output just gets blander, because a model given 60 competing priorities satisfies the average of them.
What I changed: I stopped adding and started replacing. When a new rule arrives, I find the old rule it supersedes and delete it. I also split the file. Non-negotiables at the top, short and absolute. Examples in a separate reference. A long file of prose became a short file of rules pointing at a library.
Break 2: The Reference Library Went Stale
This one is nasty because it has no symptom until it has a big one.
You build a reference library from a client's best material. It works. 8 months later their positioning has moved, their offer has changed, they have started talking about a new idea on every call. The library still contains January's version of them. Output is still technically on-voice and it is subtly wrong, and the client cannot articulate why they do not like it.
Everyone reads that as the AI degrading. The AI did not change. The person did.
What I changed: refresh cadence. New material goes in on a schedule, not when someone complains. Anything superseded gets pulled rather than left to sit alongside the current version. A reference library is a living record of a standard, and a standard that is not maintained is a standard from last year.
Break 3: The Edges, Not the Middle
Skills are good at the request they were designed around and bad 2 steps to the left of it, and this is the single most reliable way to be fooled during testing.
You test with a good transcript, clear audio, one topic, a normal length. It nails it. You ship. Then production sends it a call where 3 people talk over each other for 90 minutes, or a 4-minute voice note with no context, or a topic the client has never discussed before. Output falls apart, but politely, so the failure does not announce itself.
What I changed: I test the ugly cases first now. Shortest possible input. Longest. Off-topic. Contradictory. The one where the source material genuinely does not contain the answer. If the skill cannot say "there is not enough here," it will invent something, and inventing something is worse than failing.
Break 4: I Was Still the Quality Gate
For longer than I want to admit, the actual verification layer in my operation was me reading everything. That works at 3 clients. At 15 clients per cycle it becomes the entire bottleneck, and worse, it is inconsistent, because my standard at 9am on Monday is not my standard at 7pm on Thursday.
A skill without a gate is not a system. It is a faster way to generate work that still needs you.
What I changed: the check moved into the pipeline. Every draft runs against the reference library before a human sees it and comes back pass or fail with reasons. My job became auditing the gate instead of reading the drafts. That single change is most of the difference between running 5 skills and running 50.
Break 5: Confident, Well-Formatted, Wrong
The most expensive failures never looked like failures.
A skill produces a piece that is structurally perfect. Right length, right sections, right tone. It also contains a claim the client never made, or a number lifted from a different client's material, or a story with the details rearranged. It reads well enough that it slides past a tired reviewer.
Loud failures are cheap. You see them and fix them. Silent ones reach a client's audience.
What I changed: claim-level verification. Anything that reads as fact gets traced back to a specific line in the source. If it cannot be traced, it comes out. This is now the check I would keep if I could only keep one, and it is also the reason a proof of concept is not a production system. Demos are not run long enough for this class of failure to show up.
Break 6: One Skill Doing 3 Jobs
Ambitious skills fail in ways that are hard to diagnose. A skill that extracts, writes, and formats in one pass gives you a bad output and no way to know which stage caused it.
Every skill that survived the year does exactly 1 job with a checkable output. Extraction is separate from writing. Writing is separate from the gate. When something breaks I know within a minute where it broke, because each stage has an output I can look at.
What I changed: split on the seams. The test is whether I can inspect the output of a stage and judge it on its own. If I cannot, the stage is doing too much.
Break 7: Notes That Stayed Notes
Early on, corrections lived in my head and in Slack threads. A client would flag something, I would fix that draft, and 3 weeks later the same note came back on a different piece.
That is not a model failing to learn. That is me failing to record. The correction existed. It just never made it anywhere the system could read.
What I changed: every note becomes a rule in a Rule Registry, and the gate enforces the registry. A note given once holds on every output after it. I have written about why revision notes should become rules rather than tickets, and it came directly from this year.
What I'd Tell You If You Are Building Your First One
Build the smallest skill that does 1 job. Give it 10 to 20 pieces of real reference material, not instructions about tone. Test it on the ugliest input you have before the nicest one. Put a pass-fail check in front of it before you scale it. And write down every correction the day it happens.
None of that is exciting. All of it is what separated the skills that were still running in month 12 from the ones I quietly stopped using in week 3.
If you would rather see the finished version than build it: send me 1 piece of your work. You get a finished asset back in your voice within 48 hours, produced by exactly this stack. The method page shows the Encode, Verify, and Ship steps in full.
FAQs about running AI skills in production
What is an AI skill?
A packaged instruction set with a reference library, a defined job, and a checkable output. Unlike a prompt, it persists between sessions and carries the context, examples, and anti-patterns needed to produce work to a standard. The prompts versus skills piece covers the distinction.
Why do AI skills degrade over time?
2 reasons, and neither is the model. Instruction bloat, where added rules dilute earlier ones until the file is followed loosely. And reference decay, where the library still reflects an older version of the business. Both are maintenance failures, both are fixable on a schedule.
How many AI skills does a coaching or consulting business need?
Fewer than people expect. Most practices are covered by a handful, each doing 1 job well. Skill count is a poor measure. Number of jobs you no longer touch is the real one.
How do you stop AI from inventing details?
Claim-level verification. Every factual statement traces back to a specific line in the source material, and untraceable claims are removed at the gate rather than caught by a reader.