Internal DevOps-AI Agent - Advanced Systemic Automation Platform
An Agent That Provisions Cloud Infrastructure From Plain English
A cloud engineering team wanted to provision AWS resources by describing them in plain English instead of remembering CLI syntax. I built an internal chat agent that does this end to end: it asks for the details it needs, writes the AWS commands, runs them, reads what AWS returns, and corrects itself on errors, one command at a time until the task is finished. This was mid-2024, before agent frameworks were common, so the tool-calling loop and the safety limits were built by hand.
The problem
Provisioning cloud resources by hand is slow and error-prone. You have to remember the exact CLI flags, create things in the right order (a security group before the instance that uses it), and read back cryptic output to see if it worked. The team wanted to just say what they needed and have it happen. A plain chatbot cannot do this: it can suggest a command, but it cannot run it, see that it failed, and try again. What was needed was an agent that turns a request into a sequence of real actions, watches the result of each one, and adjusts. And because it would run live AWS commands, it needed limits so it could never run away.
Constraints
- ·Real actions, not suggestions: the agent had to actually run AWS commands and use their output.
- ·Correct order: cloud resources depend on each other, so commands had to run in dependency order.
- ·Bounded and safe: since it executes live commands, it needed hard caps so a loop could never spiral.
- ·No agent framework: built in mid-2024, so the tool-calling protocol and the loop were hand-rolled.
Architecture
A Streamlit chat sends the user's request to a Claude model on AWS Bedrock, driven by a system prompt that turns it into an AWS operator. The model asks for missing parameters, then emits a single AWS CLI command wrapped in the markers `&^ ... ^&`. A CLI executor pulls the marked command out of the reply with a regex, runs it with a subprocess, and captures the output. That output is fed back to the model, which reads it and decides the next command or a correction. The process repeats itself until no more commands appear. Two request caps bound the loop.
- ›Streamlit chat UI (app.py) with the recursive handle_ai_response loop
- ›Bedrock call (backend.py gen()): Claude 3 Haiku, system prompt + full history each turn
- ›System prompt (initial_prompt.txt): ask for params, then one `&^aws ...^&` command in dependency order
- ›CLI executor (cli_executor.py): regex-extract `&^...^&`, run via subprocess, capture stdout/stderr, log
- ›Feedback loop: CLI results returned to the model, which self-corrects and emits the next command
- ›Guardrails: a killswitch (20 requests) and a per-session cap (50 calls)
- ›Model wrappers (models/): Claude active, Llama / Titan / Cohere experiments
- ›On-disk conversation history and per-command execution logs
How one request flows
- 01The user asks in plain English ("create me an EC2"). The agent replies by asking for the details it still needs: name, type, region.
- 02Once it has enough, the agent emits a single AWS command wrapped in a marker, in dependency order.
- 03The executor pulls the marked command out of the reply and runs it against AWS, capturing the output or the error and logging it.
- 04The result is handed back to the model as the next turn, with a note to not repeat a failed command but propose a fix.
- 05The agent reads the result, decides the next command or a correction, and the cycle repeats on its own until there are no more commands.
- 06A killswitch caps total requests and a per-session limit caps calls, so the loop can never spiral into runaway cost.
Engineering decisions
Giving the model a way to act, before tool APIs
What. A system-prompt contract: wrap any command to run in `&^ ... ^&`, which the executor scans for and runs.
Why. In mid-2024 clean tool-calling APIs were not there yet, and this turned a chat model into something that could take actions.
Tradeoff. Parsing text with a delimiter is more brittle than a real tool API, accepted as enough for the job.
A real loop, not a single answer
What. After a command runs, its output is fed back to the model, which decides the next step, and the process calls itself again while commands remain.
Why. The feedback is the value: it lets the agent recover from a failed command and keep going instead of stopping at the first error.
Tradeoff. A loop that runs live commands needs hard limits, which were added.
One command at a time, in order
What. The agent sends a single command per turn and sequences them by dependency.
Why. Each result is seen before the next command is chosen, so a failure early on stops the chain instead of cascading.
Tradeoff. Slower than batching, but controllable.
Hard limits on an agent that runs live commands
What. A killswitch on total requests and a per-session cap on calls, either of which stops the agent.
Why. An agent that executes real AWS commands in a loop can spend money and change infrastructure.
Tradeoff. A firm ceiling over squeezing out extra runs.
Trying several models behind one interface
What. Wrappers for Claude, Llama, Titan, and Cohere on Bedrock, with a fast, cheap Claude model chosen for the live loop.
Why. Keeping the model behind one call made it easy to compare and swap.
Tradeoff. Only one model is live at a time, and the rest stay as experiments.
What changed along the way
- →Started by getting a single command to generate and run, then grew it into the self-correcting loop once that worked.
- →Tried larger and smaller models across families and settled on a fast, cheap one, since the loop makes many calls.
- →Added the request caps after seeing how quickly an unbounded loop could pile up calls.
Security and reliability
- ·Bounded execution: the killswitch and the session cap are the hard stops on a loop that runs live commands.
- ·Every action logged: each command and its result is captured to a log.
- ·Error handling per command: success, failure, or error is captured separately, and a failure is fed back rather than ignored.
- ·Secrets: AWS credentials are kept out of the code and read from the environment.
- ·Honest limitation: the agent runs generated commands without a per-command confirmation, so the caps and logging are the safety net. A production version would add an approval gate before execution.
Metrics
What shipped
- ✓Built a working internal agent that provisions AWS resources from a plain-English request.
- ✓Runs a real act-observe-reason loop: it executes a command, reads the result, and self-corrects on errors.
- ✓Sequences dependent resources one command at a time, with every action logged.
- ✓Bounded by a killswitch and a session cap so a loop can never spiral.
- ✓Model kept behind one interface, so several Bedrock models could be compared and swapped freely.
Lessons learned
- ·An agent is the feedback loop, not the model. The value came from feeding results back and letting it self-correct.
- ·Before tool APIs, a simple delimiter protocol was enough to make a chat model act, if you accept the brittleness.
- ·One action at a time keeps an agent controllable, because each result is seen before the next step.
- ·Anything that runs live commands needs a hard ceiling and, ideally, an approval gate.