Run the same request twice. First time, it hands back a clean answer with citations and a decision matrix. Second time, it’s shorter, no sources, different facts. Sound familiar?
Everyone blames the model. “This one is too dumb.” “That one hallucinates.” “I need GPT-5.” The reflexive move is to chase the next release.
The reason your agent is inconsistent has almost nothing to do with the model under the hood.
That reflex is wrong. And expensive.
The reason your agent is inconsistent has almost nothing to do with the model under the hood. It has everything to do with the fact that you never told it what “good” looks like in a way it can reference again.
Models are consistent by design. The inconsistency comes from the agent having to invent its own approach every single run. No reference to go back to means no repeatability. A bigger, smarter model doesn’t fix that — it just makes the random guesses more plausible.
The Real Reason Agents Fail
When you open a chat window and type a question, the model builds a path from scratch. That path dies the moment you close the session. Next time, it rebuilds — differently. That works fine for one-off queries where you’re sitting there to catch mistakes. It collapses the moment you want the same job done every week without supervision.
The fix is a skill file — a single markdown document that the agent reads before starting. It contains the task description, the step-by-step procedure, the pitfalls you’ve already watched it fall into, and the definition of “done.” Without that file, the agent has no memory. With it, the agent follows the recipe every time.
I watched this play out on an open-source agent called Hermes from Nous Research. Out of the box, it answered like nothing in particular. Then I gave it a soul.md — its primary identity file that sits at /opt/data/soul.md. That file sets the tone, the independence level, the behavior. After that, it answered in the voice I chose, with British spelling, no preamble. Same agent. Different output.
That’s the part most people skip. They deploy an agent, ask it questions, get frustrated, and blame the model. But the model was never the part being inconsistent. The soul file was missing.
The Boring Work Nobody Wants to Do
Here’s the uncomfortable truth: if you cannot describe your process in a file, you cannot expect an AI to execute it consistently. Period.
Most businesses have no documented standard operating procedures. Knowledge lives in people’s heads. When you ask an AI to “do the weekly report,” it has to guess what that means. It will guess differently each time. That’s not an AI problem — that’s a documentation problem.
The most successful AI deployments I’ve seen are the boring ones. A single skill file that tells the agent exactly how to convert a YouTube video into a social post. A soul file that says “answer first, reasoning underneath, British English, no fluff.” No automation framework. No multi-agent orchestration. Just a text file that the agent reads.
One practitioner I interviewed runs a research skill every weekday at 6 AM. The agent opens the named sources, works through the procedure, builds a brief with citations, and delivers it to Telegram. He hasn’t touched it in months. The skill file is 200 lines. The model is a mid-tier one. The consistency is flawless.
That’s the opposite of what most people do. They spend weeks testing the latest model, running benchmarks, tweaking prompts. They never write the skill file. So their agent never becomes reliable.
The Six-Step Framework That Works
On Codex, the AI agent platform, there’s a tested process for building skills that actually hold up:
- Reverse engineer from a result. Start with a completed output you already approve of. Give it to the agent. Let it analyze what went into that output. That’s your recipe.
- One task, one trigger. Don’t write a 30-page skill that does everything. Break work into individual tasks. Each skill should match exactly one trigger. If you say “turn this YouTube video into an X article,” the agent should find the exact skill for that job and nothing else.
- Know the freedom level. Deterministic tasks (data transfer, formatting) need rigid steps. Non-deterministic tasks (creative writing, research) need constraints, not scripts. Mixing them breaks both.
- Add verification loops. The best skills include a quality check. The agent that creates the output hands it to a second pass — or even the same agent in a later step — to review. Objective checks (500 sources reduced to 250) and subjective checks (does this sound like me?) both belong in the file.
- Reduce the model. Develop on a powerful model, then test downward. If the skill runs correctly on a cheaper model, you waste money every time you use the expensive one. I’ve seen skills that work perfectly on Haiku that people were running on Sonnet.
- Iterate like training wheels. Your skill is never done. Every execution, give feedback. “That screenshot was poorly placed. Update the skill so it crops before inserting.” The skill file grows with each run. Over time, the agent needs fewer corrections.
This is work. It’s not glamorous. But it’s the difference between an agent that amuses you and an agent that runs your department.
What You’re Really Buying With That New Model
Every time a new model drops, the AI community holds its breath. Benchmarks improve by a few points. People race to test it.
Here’s the reality: a finished workflow with last month’s model outperforms an unfinished one on this week’s model. The bottleneck is never the model’s capability. It’s whether you’ve written down what you want it to do.
The video about the laziest way to make money with AI — AI influencers — shows this from the opposite angle. The creator didn’t refine his prompts endlessly. He used a pre-built skill in Claude to generate character profiles, then a skill to write scripts, then a connector tool to generate images. Each step had a defined process. The results were consistent enough that viewers couldn’t tell the influencers were AI.
He didn’t need a better model. He needed a reproducible chain of skills.
The Real Competitive Advantage
The people who win with AI in 2026 will not be the ones who use the newest models. They will be the ones who have written down every repetitive process in their work and turned it into a skill file. The agent doesn’t need to be smarter. It needs to be trained.
Your soul.md file — the identity you give your agent — is a competitive moat. Two agents running the same skill on the same job will produce different outputs if their soul files disagree. That’s leverage. You can encode your judgment, your tone, your decision rules into a text file. Every new hire or contractor can then replicate your thinking without you sitting next to them.
Most people will never do this. They will keep chasing models, keep getting inconsistent results, and keep blaming the technology. That’s fine. It leaves the field open for the ones who open a text editor and write the recipe down.
- Why is my AI agent inconsistent?Because it has no documented process to follow, so it invents a new approach each time.
- How can I make my AI agent consistent?Create a skill file with task description, step-by-step procedure, and definition of done.
- What is a soul.md file?It is an identity file that sets the tone, independence level, and behavior of an AI agent.
