How we built an AIOps agent – and let it perform root cause analysis

Tobias Johansson
Jul 1
4 min read

On March 13th, we had a service that was down.

Nothing strange in itself, things like that happen. In a complex environment, there can be multiple reasons for this, many systems to validate, and it can be time-consuming to troubleshoot.

Github (Manifest repo)
Kubernetes (AWS EKS)
ArgoCD
Amazon ECR
AWS Load Balancers

There was no clear root cause, and starting to troubleshoot each system individually could be a time-consuming process.

We did something simple: we let an AI troubleshoot

Instead of starting to search manually, we put an AI on the problem.

A prompt. No back and forth dialogue. Just: “find out what’s wrong, can’t reach application X”.

The result? 4 errors across three systems

On a run, the agent found:

1. Wrong selector in load balancer (misconfigured kubernetes manifest)

loadbalancer.yaml referenced devsecops-lab-appp (three p).

Result: no healthy NLB targets.

2. Image tag that did not exist (misconfigured kubernetes manifest)

Deployment.yaml used v0.0.99, but ECR only had v0.0.7.Pod got stuck in ImagePullBackOff.

3. Port mismatch (misconfigured kubernetes manifest)

ContainerPort 4999 was configured, while service and load balancer pointed to 5000. All requests were denied.

4. Security group blocked health checks

NodePort 30992 was used, but the security group only allowed 31000–31100. Health checks were silently dropped.

Four small errors. Together: a service that didn't work.

Here comes the important part: how we built it

This didn't work because we "prompted well". It worked because we built the right structure around the AI.

Let me show you how we did it.

Step 1: we built an agent that can actually do the job

We didn't start with prompts. We started with what the agent can actually do.

We set up a local MCP server and gave the agent three simple tools:

list – show available code (to avoid re-generating)
write – to generate code
execute – to run what it created itself

Then we made a conscious choice. We didn't build any ready-made integrations. (This method is specifically called "Code execution with MCP").

Instead, we let the agent himself:

write scripts to read Kubernetes resources
retrieve information from AWS (e.g. security groups, load balancers)
compare configuration against GitHub

The first run took a little longer, but after that it was able to reuse its own “skills”.

The result?

An agent who not only responds, but actually drives the troubleshooting.

Step 2: We defined the troubleshooting flow upfront

Next problem: where should the agent start?

AI is good at execution, but not at choosing the right path.

So we removed that choice completely.

We created infra-context.md and defined a clear routing:

“App unreachable” →ArgoCD → Deployments → Pods → Services → Manifests

This is nothing advanced. It's just our collective experience written down.

The difference is that the agent:

follows every step
doesn't skip anything
run everything in the same flow

This is where we went from “AI testing a few things” to a reproducible troubleshooting process.

Step 3: we controlled the behavior, not just the input

The final part was to get the agent to behave consistently.

We did that in CLAUDE.md .

There we described specifically:

when to use the MCP server
how workflows should be built (code structure)
how results should be structured in a report
which files to read for context ( infra-context.md )

We were very clear here.

For example:

“always read infra-context.md first”
“build a workflow that runs all checks in one run”
“return a structured response with identified errors and suggested actions”

This was crucial.

Without it, we got different answers in different formats based on what the prompt looked like. With the structure, we got an agent that:

works methodically
run the entire analysis in one step
delivers something we can actually use right away

What we learned

This is what actually made a difference:

Structure optimizes the AI model

Today's LLMs are powerful, but can tend to hallucinate and go in the wrong direction if structure and rules are not in place.

AI is good at writing code

Since LLMs are more trained to generate code than to use various “tools”, it is advantageous to let them do that, instead of asking them to debug “freehand”. This keeps the context “clean”, which reduces the risk of hallucinations and saves on tokens.

Clear context between runs

Since an agent without structure has no knowledge of the environment (we would have needed to prompt it how to reach each system), it probably won't find all errors after one run and needs to be prompted again to continue debugging (costs more tokens and is not deterministic). This can also lead to context bloating which tends to create problems for the agent to achieve its original task.

It is possible to make it reproducible

One run → one answer → same result every time.

What this means in practice

Most environments today consist of several different systems, such as Kubernetes, cloud (AWS/Azure), some form of Git, CI/CD, VMs, etc.

When something goes wrong, it is often necessary to: