Amaan's Engineering Notes

Forward Deployed Engineer is an old job with a new name, plus an agent

Sat, 31 Oct 2026 00:00:00 GMT

You’ve been asked to build an agent for a customer. It runs in their AWS account, on their data, behind their identity provider, and their security team hasn’t approved Bedrock yet. Somebody on their side will have to keep it running after you’re gone.

Congratulations, you’re a Forward Deployed Engineer now. That’s the title AWS and Anthropic are hiring for, and recruiters have started putting it on anything that involves a customer and an LLM.

I’ve been doing the non-AI version of this job for quite some time. Build inside the customer’s account, get it past their security review, hand it to their team. Lately more and more of that work has had an agent sitting in the middle of it: AgentCore runtimes, Strands agents, tools behind a gateway, a worker pulling off a queue. And the honest surprise was how little of the work was the agent itself.

So this is what carried over from infrastructure work, and what I actually had to learn.

What they’re actually hiring for

Read a couple of these job posts and they’re describing the same person. Someone who embeds with a customer, builds on the customer’s data, owns the thing in production, and comes away with patterns the next engagement can reuse. There’s one line in Anthropic’s post that I think gets it exactly right:

You don’t need to have shipped agents to production. You do need deep experience building and operating production systems, hands-on time with agents, and the judgment to lead architecture and advisory work with enterprise customers.

Prompting is the easy part. What’s hard to find is someone who can get a system through a messy enterprise (legacy systems, SSO nobody owns, compliance, procurement) and leave it in a state the customer can actually run.

The stuff I already knew how to do

Discovery hasn’t changed. Nobody walks in saying “we need an agent with three tools and a memory store.” They say “my team spends half the day on this” or “the auditors are on our back.” You still have to find the real constraint before you draw anything, and it’s almost never the model. It’s data residency, or a network team that needs three weeks to open egress, or a change board that meets every other Thursday. Find that in week one and you design around it. Find it in week six and you’re rebuilding.

Then there’s the security review, which is where these projects actually stall. Not the code. Someone wants a data-flow diagram. Someone wants to know which principal calls the model, where prompts get logged, and for how long. Procurement has never approved Bedrock before and wants to know what it is. If you’ve spent years filling in security questionnaires for normal infrastructure, you’ll notice the answers are mostly the same ones you’ve always given. Which IAM role, what CloudTrail records, who can read the logs. Raise those questions yourself early and the agent ships. Wait for the customer’s security team to raise them and it stays a demo.

Handover is the part I care about most, because it’s the real test of the work. Infrastructure as code their engineers can read. A CI pipeline they own, using OIDC, with no access keys sitting in a variables page. A runbook written for their on-call engineer, not for me. With an agent the runbook gets longer, because there are new things to explain: how to roll back a prompt, how to fail over to the backup model, what to do when the dead-letter queue starts filling up at 2am.

And if you maintain Terraform modules or CI templates, you already have the habit of turning repeat work into something reusable. With agents the reusable pieces are different (a tool contract, an eval harness, a gateway setup) but it’s the same muscle.

The agent is the smallest box on the diagram

The first time I drew a production agent system end to end, the agent runtime was one box. Everything around it was the stuff we’ve always built.

Inbound events don’t call the agent directly. They land in a queue, and a worker picks them up and invokes the agent, because nobody should be sitting on a synchronous request to a model that might take thirty seconds to answer. If a message keeps failing, it goes to a dead-letter queue after a few attempts, and a CloudWatch alarm pages a human. An agent that fails quietly is the worst kind.

There’s always a path back to a person. If the system can’t finish something within a reasonable window, it hands off. I’d design that path before anything else, honestly, because it’s the thing everyone forgets until the first bad day.

The agent doesn’t call your APIs directly either. Tools sit behind AgentCore Gateway, which exposes Lambda functions as MCP tools with IAM auth, so every tool call is an authenticated, logged request against a role you scoped. Memory is its own component, and it’s data like any other data, with the same retention and residency questions. And there’s a primary model and a fallback, wired up in the application, so one throttled model doesn’t take the whole thing down.

None of that is AI work. It’s distributed systems work, and it’s most of the job. That’s why I think the “forward deployed” part of the title matters more than the “AI” part.

Treat the agent like a privileged principal

This is where infrastructure and security people have a real edge, and where most agent demos quietly fall apart.

Think about what an agent with tools actually is. It’s a process that decides at runtime which APIs to call, with arguments partly shaped by whatever text a user or a retrieved document fed it. That’s a confused-deputy problem. Prompt injection doesn’t have to break the model. It just has to talk the model into using a tool it should never have had in the first place.

So I ask the same things I’d ask about any workload, just more nervously. What can it touch? Each tool gets its own scoped role, not one big role for the whole agent. Whose identity is it acting as, the user’s or a service account that can see everyone’s data? What’s the blast radius if it makes a bad call? Reads by default, writes behind a human. And can I reconstruct what happened afterwards, from CloudTrail or Azure Monitor, without guessing?

On AWS, least privilege for a lookup tool is just boring IAM. Which is the point:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadOrdersForLookupToolOnly",
      "Effect": "Allow",
      "Action": ["dynamodb:GetItem", "dynamodb:Query"],
      "Resource": "arn:aws:dynamodb:us-east-1:123456789012:table/orders"
    },
    {
      "Sid": "ApplyTheOneGuardrailThisAgentUses",
      "Effect": "Allow",
      "Action": "bedrock:ApplyGuardrail",
      "Resource": "arn:aws:bedrock:us-east-1:123456789012:guardrail/abcd1234efgh"
    }
  ]
}

No dynamodb:*, no Resource: "*", nothing that writes. On Azure you’d do the same thing with a managed identity per tool and a tight role assignment in Entra ID.

The best thing you can put in an agent design doc is a short list of things the system will never do. Pick the actions that stay with a human, write them down, and then check that no tool role can do them. I’ve found that one decision does more for the blast radius than any amount of guardrail tuning.

Lots of people can get an agent to call a tool. Not many can sit in front of a customer’s security team, explain why that agent can’t be talked into deleting a table, and show them the policy and the audit trail that prove it.

What I had to learn

I’m still learning most of this, so take it as notes rather than answers.

Evals. In normal software a test passes or fails, and the same input gets you the same output. Agents don’t work like that. They’re non-deterministic, they’re usually wrong in subtle ways rather than obvious ones, and a prompt or model change can make things worse on inputs nobody thought to test. The instinct to gate deploys carries over fine. What doesn’t carry over is knowing what to measure. You end up building a golden dataset with the customer’s domain experts, picking between exact-match checks, rubric scoring and LLM-as-judge, and arguing about a pass threshold the business will actually sign off on. That’s product and data work. It’s also never finished, because the cases that matter most are the ones production shows you later.

The CI side is the easy bit, and it looks like any other gate:

name: agent-evals

on:
  pull_request:
    paths: ["agent/**", "prompts/**", "evals/**"]

permissions:
  id-token: write   # OIDC to AWS, no stored keys
  contents: read

jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/agent-evals-ci
          aws-region: us-east-1
          role-duration-seconds: 3600

      # Fails the job if the pass rate on the golden set drops below threshold
      - run: uv run python evals/run.py --dataset evals/golden.jsonl --min-pass-rate 0.92

Twenty lines of YAML. The golden.jsonl it points at takes weeks of sitting with people who know what a right answer looks like.

Retrieval. Turning on Bedrock Knowledge Bases or Azure AI Search takes an afternoon. Getting answers people trust is a different story: chunking, metadata, stale documents, access control on the source files, and that one PDF that’s actually a scanned fax. I understood the plumbing on day one. Tuning recall on a messy corpus took a lot longer.

Guardrails. Bedrock Guardrails and Azure AI Content Safety handle content filters, denied topics and PII redaction, and you should turn them on. But they won’t decide whether the agent is allowed to issue a refund over a certain amount, or whether a tool call needs a human to approve it. That logic lives in your application and your tool design.

On-call. You already know how distributed systems break. The model brings its own failures on top. The provider updates something and behaviour shifts. A prompt change regresses. A tool returns garbage and the agent confidently acts on it. A conversation loops and quietly burns through tokens. So the dashboards change. You end up watching per-session traces, tool-call success rates, eval scores on sampled live traffic, and cost per completed task, alongside the usual latency and 5xx graphs.

If you’re coming from infrastructure

Build one agent end to end and run it for a month. Not in a notebook. Deploy it properly, with IaC, OIDC-based CI, scoped tool roles, a DLQ with an alarm, tracing, and a budget alert. Write a handful of evals before you write the prompts, even twenty hand-labelled examples. It changes how you design, because you stop asking “does it work?” and start asking “how often does it fail, and on what?”

Draw the whole system, not just the agent. If your diagram has one box labelled “agent” and nothing around it, that’s a demo.

And lean into the security side. It’s the gap most agent builders have, and it’s the thing enterprise buyers worry about most. You’re not competing with ML specialists there. You’re the person they need on the team.

The FDE title will probably get stretched until it doesn’t mean much, same as “DevOps engineer” did. The job underneath is real, though. Sit with the customer, ship into their production environment, own it when it breaks, and leave behind something their team can run. If you’ve been doing infrastructure delivery for a while, you already do most of it. The agent part you can learn, the same way we learned everything else: by running it in production and finding out where it breaks.

Desirable Difficulties

Fri, 31 Jul 2026 00:00:00 GMT

Most people learn the fast way now. Copy the snippet the AI gives you, follow the tutorial, ship it. That’s fine, up to a point. But if that’s the whole loop you stay shallow, and you end up with a career’s worth of experience that’s really one year repeated ten times.

There’s a name for the alternative, and it’s from cognitive psychology, not a productivity thread. It’s called desirable difficulties: intentional friction in how you learn slows your progress now but drastically improves what sticks, and the part that matters at work, your ability to apply it to a problem you’ve never seen. The easy path feels productive. The hard path is productive. Those aren’t the same thing, and your brain is very good at lying to you about which one you’re on.

So here’s the loop I run on: read, experiment, get uncomfortable, teach, repeat. If it were that simple everyone would already do it, so let me be specific.

One thing up front. Before the open source, before the Go, before any of the code, I’m a DevOps and platform engineer. My problems are shaped like the systems I run: infrastructure that has to stay up, a migration that can’t lose data, a legacy service nobody wants to touch. The loop is the same everywhere.

One more thing, because you’ll notice it. I wrote a lot of this with an AI in the loop, on a site I build with one. That’s not the shortcut I opened by warning about. What keeps you shallow isn’t the tool, it’s reaching for it without thinking: pasting what it hands you, shipping it, never asking why it works or what breaks when you pull a line. Used the other way it gets you to the hard part faster instead of letting you skip it. I still own every call here, still have to verify the output and rewrite it until it says what I actually mean, and the moment I stop doing that is the moment it makes me worse, not better. The tool didn’t change the discipline. It raised the cost of skipping it.

Read what tells you why, not just how

Tutorials teach you what. Books and real docs teach you why. That difference is the whole game.

A tutorial hands you a working result and a false sense of understanding. You followed the steps, it ran, you feel like you learned something. You didn’t, you pattern-matched, and the moment the real problem leaves the happy path you’re back to searching. Read the actual docs, not the quick-start. The quick-start is a tutorial in a trench coat, built to get you running, not to teach you the tool. The reference, the design rationale, the section nobody reads about why a default is what it is: that’s where the depth is.

The work I’m proudest of was less about writing code than reading how the thing was already built, and why, so what I added felt native instead of bolted on. Same with any platform call worth making. You don’t get the reasoning behind a hard default, the kind that only clicks once you’ve watched it save you, from a quick-start. The people who know their tools deeply are, not by accident, the ones who enjoy the work more.

Break things on purpose

Reading alone doesn’t cut it. You know the feeling when you copy code, it works, and you move on? That’s the trap. It ran, so you stopped thinking.

The people who grow fast ask the annoying question: why does this work, and what breaks if I pull this one line? Then they pull it and watch it break. On a platform that’s a throwaway cluster you wreck on purpose, a failover you trigger in staging before it triggers itself in production, a Terraform module you tear down to see what the state really depends on. That’s desirable difficulties at its purest. Watching it fail teaches you what the line was for, and copy-paste can’t, because copy-paste skips the part where you’d have had to think.

Live at the edge of what you can do

Experimenting means trying things that feel uncomfortable, and that’s the part people quietly avoid.

Building the same thing for two years, or reaching only for the first tool you learned, is a comfort zone, and a comfort zone is where growth goes to die. There’s a name for the productive spot, the zone of proximal development: the band just past what you can do alone. Not so hard you’re lost, but hard enough you have to think and maybe ask for help. Too easy, nothing rewires. Too hard, you bounce off.

So aim there on purpose. Take the ticket nobody wants. Most of what I know about how systems fail came from being pushed out of my depth: the migration with no clean rollback, the outage in a layer I didn’t understand until it was on fire, the tool I had to learn under pressure because something depended on it. Right now that edge is AI/ML, getting genuinely fluent instead of nodding along. The discomfort isn’t a warning sign. It’s your brain laying down new pathways. Read it as a green light.

Reach for the hard problems, because nobody hands them out

The growth-making problems don’t get assigned. You go get them.

Early on especially, you have to walk up to your lead and say it out loud: “I want something bigger.” Volunteer for the work everyone avoids. The legacy service people are scared of. The modernization that means understanding a system nobody documented. The migration with the data-residency constraint. The backlog item that’s sat there because it’s genuinely hard. That’s the work that teaches.

And own the outcome, not the task. There’s a real gap between “I did the ticket” and “I made it work in production and I’ll be the one who understands it when it pages at 2am.” Responsibility is a skill, and you only get better at it by carrying some.

Open source is the most underrated growth tool there is

Once you’re doing harder work you want real feedback on it. Open source is the best way to level up that I know.

There’s a myth it’s for senior engineers who’ll flame you for a misplaced brace. It isn’t. Your first contribution can be a docs fix, and the moment it lands you’re inside a real production codebase, watching how professionals structure, name, and test things, getting review from those same people. The way in is simple: find a project you actually use, open the issues tab, filter for the newcomer-tagged ones, start there. I’ve contributed upstream to projects far bigger than anything I’d have built alone, and it held every time. Maintainers with a higher bar than mine pulled my bar up with them.

Surround yourself with people whose standards you envy

There’s an old line that you’re the average of the five people you spend the most time with. It’s a cliche because it’s true, and it’s true for engineering. If everyone around you is fine with messy infra and half-shipped features, that becomes your normal, and your normal becomes your ceiling.

Find people whose work makes you think “I’m not there yet.” Communities, maintainers on projects you contribute to, engineers who hold a line you respect. You don’t need a formal mentor with scheduled 1:1s. You need regular exposure to a higher bar, because standards are contagious both ways. Being the sharpest person in every room feels good and teaches you nothing.

Teach it, and watch every gap light up

Once you’ve read, experimented, and been pushed by better engineers, teach. This sounds backwards while you still feel like you’re learning. It’s the highest-leverage thing on the list.

Feynman’s whole point was that if you can’t explain something simply, you don’t really understand it. Nothing exposes the gap between “I understand this” and “I can explain this” faster than trying to explain it. Write the post, walk a colleague through the design, record a two-minute screen capture. The instant you teach it, every gap lights up: the hand-wavy bit you skated over, the thing you thought you knew but can’t put into words. Closing those gaps is where it consolidates.

This post is that. The loop taught me something, so I’m teaching it back, which is how I find out whether I understood it.

The loop

That’s it. Read the source that explains why. Experiment and break things on purpose. Get uncomfortable at the edge of what you can do and reach for the work nobody wants. Surround yourself with higher standards. Then teach it, and repeat.

None of the steps are clever. What makes it work is that every one is a deliberate difficulty, trading short-term comfort for long-term capability on purpose. That’s not friction you streamline away. The friction is the mechanism. The engineers who compound over a decade didn’t find the smoothest path. They kept choosing the uncomfortable one until it stopped feeling like a choice.

Make it harder on purpose.

I Stopped Clicking Through the AWS Pricing Calculator. Now I Just Describe the Architecture.

Sat, 27 Jun 2026 00:00:00 GMT

If you have built an estimate in the AWS Pricing Calculator by hand, you know the drill. Open calculator.aws, search a service, click in, stare at twenty fields half of which you do not need, guess at the ones the form does not explain, pick a region, repeat for every service. Then redo the whole thing next week when the customer asks what it looks like in Frankfurt.

For presales that is not a small annoyance. It is the gap between giving a number on the call and saying “let me get back to you.” I wired the AWS Pricing Calculator MCP into Claude, and the first real estimate I built took one sentence.

What it is

An MCP server - an AWS Samples project - that exposes the Pricing Calculator as tools an agent can call. You describe the workload, the agent assembles the estimate, the server saves it to the real calculator, and you get a shareable calculator.aws URL back. Same link you would have built by hand, minus the form.

Three things make it usable in front of a customer:

No AWS credentials. It hits the public, unauthenticated calculator.aws endpoints. You are not pointing it at an account or assuming a role. There is no blast radius.
Live definitions. It pulls the calculator manifest at runtime - about 436 services - so it is current, not a snapshot from six months ago.
Real, editable estimates. The URL it returns opens in the actual calculator. Tweak it, send it, whatever. The agent just did the boring part.

It runs over stdio for local clients like Claude Desktop, Kiro, and Cursor, or over HTTP (MCP_TRANSPORT=http) if you want it hosted. It also handles the aws-iso and aws-eusc partitions, which matters for sovereign and regulated work.

Context is the whole job

The honest part: it is amazing when you feed it the right context. Ask for “an estimate for a web app” and you get back a web app someone else imagined. The calculator never knew your traffic - you did. The MCP does not change that.

What it changes is the translation. Once you know the shape - two m5.large instances, an ALB, 500 GB of S3, daily backups - turning that into a saved, priced, shareable estimate is instant instead of twenty minutes of clicking. You bring the requirements, the agent does the assembly. That split is the right one.

A real estimate, start to finish

Here is one I just built. I asked for a small web tier in us-east-1: two m5.large instances on demand with gp3 storage, one Application Load Balancer, 500 GB of S3 Standard, and AWS Backup holding 30 days of daily EBS recovery points. I named it “Presales Demo.”

The agent searched the services, pulled the fields, filled them, ran a preflight check, and saved it:

{
  "name": "EC2 + ALB + S3 + Backup - Presales Demo",
  "sharable_url": "https://calculator.aws/#/estimate?id=59b31cbdf4a0a628b10748930fe53c8c023fd080",
  "services": [
    { "success": true, "service": "ec2Enhancement",         "group": "Prod" },
    { "success": true, "service": "applicationLoadBalancer", "group": "Prod" },
    { "success": true, "service": "amazonS3Standard",        "group": "Prod" },
    { "success": true, "service": "ebsBackup",               "group": "Prod" }
  ]
}

That link is live - it opens in the calculator with all four services priced and grouped under “Prod,” ready to edit or hand off. I never opened a browser to make it.

The preflight is the part I lean on. Some calculator fields are optional to the save API but required by the pricing engine - leave one out and the estimate saves at $0. On my first pass it caught that EC2 needed tenancy set and flagged it instead of saving a wrong number. A confidently wrong estimate is worse than no estimate, so a tool that refuses to ship one is doing me a favor.

Import, swap region, re-export

The other half of presales is reworking last quarter’s estimate, not starting fresh. The MCP imports any estimate by URL or ID - as JSON to edit (swap the region, bump the counts, re-export) or as Markdown to hand an LLM for analysis. The region-swap loop drops from “rebuild it in the UI” to import, change one field, export.

Why it matters for presales

The old loop: gather requirements, spend twenty minutes clicking them in, fat-finger a field, redo it, reply by end of week. The estimate was the bottleneck.

The new loop: describe the architecture while the customer is still on the call, get a shareable URL before it ends. “Actually it is closer to 80 million requests” - fine, update it, the link is still warm. The estimate moves to where the requirements live, which is the conversation. That is the real change, more than the speed.

Running it

It is an AWS Samples repo, so install follows the README: clone it, install deps, register it as an MCP server in your client. After that you do not call tools by name - you describe the architecture and let the agent reach for them in order. Source, the verified configs, and setup are in the repo on GitHub.

The summary is simple. It does not know AWS pricing for you and it does not invent your requirements. It deletes the twenty minutes between knowing what the customer needs and having a link to send. For anyone building these all day, that twenty minutes was the job.

Also published on Hashnode, DEV.to, and AWS Builder Center.

I Got Tired of Passing --profile on Every OCI CLI Command

Fri, 22 May 2026 00:00:00 GMT

This is a problem you only run into if you work for Oracle directly or you work at a Cloud/DevOps MSP managing OCI for multiple clients. Most people will never touch it. If you are reading this, you are probably not most people.

The OCI CLI already demands a --compartment-id on almost every command. Compartment IDs are long, they are not memorable, and you need a different one depending on what you are looking at. That was already a nightmare before you add multiple tenancies to the picture. Once you are juggling prod, staging, dev, and a handful of client accounts, passing --profile on top of --compartment-id on every single command becomes the kind of friction that breaks you.

So I wrote ocswitch.

What the OCI CLI Actually Gives You

The OCI CLI reads config from ~/.oci/config by default. You can point it at a different file with OCI_CONFIG_FILE, and select a named profile within that file with OCI_CLI_PROFILE. That is the full surface area you have to work with.

The multi-profile story in the official tooling is basically: put multiple [PROFILE_NAME] sections in one config file and pass --profile PROFILE_NAME every time. If you forget the flag, you hit whichever profile is in [DEFAULT]. This works but it is tedious, and it is session-scoped - export the env var in one terminal, open another, you are back to default.

ocswitch fixes the persistence problem by writing the active profile to ~/.oci/active_profile and restoring it at shell startup.

How It Works

Profiles are separate config files in ~/.oci/:

~/.oci/config_prod
~/.oci/config_staging
~/.oci/config_dev

One file per tenancy, named config_<profile>. When you switch:

ocswitch prod

Two things happen: ~/.oci/active_profile gets written with the string prod, and OCI_CONFIG_FILE plus OCI_CLI_PROFILE are exported in the current shell. Every oci command you run after that hits the prod tenancy.

The persistence part lives at the top of the script, outside the function, so it runs every time the shell sources the file:

_oci_state="${HOME}/.oci/active_profile"
if [[ -f "$_oci_state" ]]; then
  _oci_profile=$(cat "$_oci_state")
  _oci_cfg="${HOME}/.oci/config_${_oci_profile}"
  if [[ -f "$_oci_cfg" ]]; then
    export OCI_CONFIG_FILE="$_oci_cfg"
    export OCI_CLI_PROFILE="config_${_oci_profile}"
  fi
fi

Open a new terminal, source kicks in, reads the state file, exports the right vars. The active profile follows you across sessions without any extra steps.

What It Looks Like

ocswitch list shows all detected profiles (anything matching ~/.oci/config_*) with the active one highlighted in Oracle red. ocswitch clear removes the state file and unsets both env vars.

Tab completion is included for both bash and zsh. Hit tab after ocswitch and you get list, clear, and all your profile names.

The ocid Bonus

There is a second function in the script: ocid. It reads the active config, pulls the tenancy OCID and user OCID out of it, and calls the OCI IAM API to resolve the human-readable names:

Tenant Name: mycompany-production
Tenant ID:   ocid1.tenancy.oc1..aaaa...
User Email:  user@example.com

Useful when you have switched between a few profiles and want to confirm exactly which tenancy you are currently pointing at before running something destructive.

Install

zsh

curl -fsSL https://gist.githubusercontent.com/amaanx86/2621a181e2f4b572a1904d65748d66bd/raw/ocswitch.zsh \
  -o ~/.oci/ocswitch.zsh
echo 'source ~/.oci/ocswitch.zsh' >> ~/.zshrc
source ~/.zshrc

bash

curl -fsSL https://gist.githubusercontent.com/amaanx86/2621a181e2f4b572a1904d65748d66bd/raw/ocswitch.bash \
  -o ~/.oci/ocswitch.bash
echo 'source ~/.oci/ocswitch.bash' >> ~/.bashrc
source ~/.bashrc

Usage

ocswitch               # show help + logo
ocswitch list          # list profiles (active highlighted)
ocswitch <profile>     # switch profile (persists across all terminals)
ocswitch clear         # clear active profile

Profile Setup

Each profile is a standard OCI config file at ~/.oci/config_<profile>. The section header inside the file uses the same name as the file itself, not [DEFAULT]:

[config_prod]
user=ocid1.user.oc1..aaaa...
fingerprint=xx:xx:xx:...
tenancy=ocid1.tenancy.oc1..aaaa...
region=us-ashburn-1
key_file=~/.oci/oci_api_key.pem

So ~/.oci/config_staging has [config_staging] inside, ~/.oci/config_dev has [config_dev], and so on.

The full source is in this gist. Two files, one for bash and one for zsh, no external dependencies beyond the OCI CLI itself.

Also published on Oracle Forums, DEV.to, and Hashnode.

DNS as Code: Managing Cloudflare Records with DNSControl

Sun, 17 May 2026 00:00:00 GMT

Every DNS change I have made through a dashboard felt wrong. Click the right zone, add the right record, hope you did not fat-finger the value. No diff. No review. No way to know what the zone looked like six months ago without digging through provider audit logs that may or may not exist.

DNS controls traffic. It determines where your mail goes, which servers handle your HTTPS, which CAs are allowed to issue certificates for your domain. It deserves the same discipline as application code: version control, peer review, automated deployment.

DNSControl is how you get there. It is an open-source tool from Stack Exchange that treats your DNS zone as code. You define records in a config file, run a preview to see what would change, and push to apply. It supports Cloudflare, Route53, and about 30 other providers.

This post walks through my working POC at amaanx86/cloudflare-dnscontrol, built on runonaws.dev on Cloudflare DNS. Everything here is forkable.

The model

The repo has a simple structure:

dnsconfig.js           # Main DNSControl config
domains/               # One JSON file per subdomain
  @.json               # Apex / root records
  amaan.json           # amaan.runonaws.dev
  stg.json             # stg.runonaws.dev
  ...
utils/
  reserved.json        # Subdomain names that are protected
  internal.json        # Subdomains fully ignored by DNSControl
.github/workflows/
  preview-records.yml  # PR: dnscontrol preview → PR comment
  publish-records.yml  # Push to main: dnscontrol push → live

dnsconfig.js is the single source of truth for the zone. domains/ is where contributors make changes. utils/ holds policy that applies zone-wide. Adding a subdomain means dropping a JSON file in domains/ and opening a PR. Nothing else.

Declaring a subdomain

Each subdomain is a JSON file in domains/. The filename is the subdomain label, so amaan.json becomes amaan.runonaws.dev:

{
    "owner": {
        "username": "amaanx86",
        "email": "[email protected]"
    },
    "records": {
        "CNAME": "amaanx86.github.io",
        "TXT": "google-site-verification=aXFvR-FIaRAyOQDhDkfHac3qIX56pg9TwSaJpnq-S3A"
    },
    "proxied": true
}

The owner field is not interpreted by DNSControl. It is accountability metadata that answers the question every ops team eventually asks: whose record is this and who do you contact when it breaks? In a shared zone it is the difference between a record you can quietly delete and one that needs a conversation first.

proxied: true enables Cloudflare’s orange-cloud proxy. false is DNS-only. Every subdomain declares this explicitly so there is no zone-wide default to accidentally inherit. NS delegations, mail servers, and internal services should not be proxied, and an explicit field per record makes that impossible to get wrong silently.

The records object supports A, AAAA, CAA, CNAME, DS, MX, NS, SRV, TLSA, and TXT. Multiple values per type where the record type allows it.

What to ignore

DNSControl’s default behavior is to reconcile your zone to exactly match the code. That is dangerous at the apex. Root-domain records like NS and DNSKEY are auto-managed by Cloudflare. If DNSControl deletes them because they are not in your code, you lose DNS for the entire domain.

The IGNORE directive tells DNSControl to leave certain records alone:

IGNORE("@", "NS"),
IGNORE("@", "DNSKEY"),
IGNORE("@", "MX"),
IGNORE("@", "TXT"),
IGNORE("*._domainkey", "TXT"),
IGNORE("_acme-challenge", "TXT"),
IGNORE("_dmarc", "TXT"),

In this setup, CAA records are the only root-domain records managed as code. CAA controls which certificate authorities can issue certs for your domain, so having it in version control with a review process is worth the overhead. Everything else at the apex is either auto-managed by Cloudflare or changes rarely enough that the dashboard is fine.

The _acme-challenge and *._domainkey ignores keep ACME certificate issuance and DKIM from getting blown away by a push. utils/internal.json holds subdomains that DNSControl should not touch at all, useful for anything managed by a separate system.

Reserving subdomains

If anyone can add a subdomain via PR, nothing stops someone from claiming admin, api, billing, or auth before the platform team does.

Reserved subdomains solve this by pre-populating those names with placeholder A records pointing to 203.0.113.0. That address is from TEST-NET-3 (RFC 5737), a documentation-reserved block that will never route anywhere real. The subdomain exists in DNS, Cloudflare proxies it, requests to it return a Cloudflare error. Claiming it requires removing the placeholder, which requires a PR, which requires approval.

utils/reserved.json covers about 80 names: admin, api, app, auth, billing, blog, cdn, mail, and the rest of the usual candidates.

The zone timestamp

Every push writes a TXT record at _zone-updated with the current Unix timestamp. You can check it any time:

dig TXT _zone-updated.runonaws.dev +short

Two uses: confirming a push actually landed, and spotting records that look stale. If the timestamp predates your last push, something went wrong in the pipeline.

The CI/CD pipeline

Two GitHub Actions workflows handle the deployment lifecycle.

On pull request, the preview workflow runs dnscontrol preview against Cloudflare and posts the output as a PR comment. Reviewers see exactly which records will be added, modified, or deleted before anything merges. The comment updates on force-push so it always reflects the current branch state. No dashboard access needed to understand what a change does.

On push to main, the publish workflow runs dnscontrol push and the changes go live.

Both workflows only fire on changes to domains/*, dnsconfig.js, utils/reserved.json, or the workflow files themselves. Unrelated repo commits do not trigger a DNS push.

The Cloudflare API token needs Zone.DNS write permission scoped to the specific zone, not the whole account.

What the workflow looks like in practice

Someone needs to add staging.runonaws.dev pointing to an ALB. They:

Create a branch: dns/add-staging

Add domains/staging.json:

{
    "owner": {
        "username": "amaanx86",
        "email": "[email protected]"
    },
    "records": {
        "CNAME": "my-alb-1234567890.us-east-1.elb.amazonaws.com"
    },
    "proxied": false
}

Open a PR
Preview workflow posts the diff as a comment: one CNAME being added
Reviewer confirms the target and approves
Merge to main, Cloudflare updates within seconds

The contributor never opened the Cloudflare dashboard. The reviewer saw the exact change before it applied. The commit history shows who added the record, when, and why.

The sync catches what the dashboard quietly added

Here is where the reconciliation model earns its keep. This is the actual preview comment the GitHub Actions bot posted on PR #10:

******************** Domain: runonaws.dev
4 corrections (cloudflare)

± MODIFY _zone-updated.runonaws.dev TXT ("1758268077330" ttl=300) -> ("1779036943050" ttl=300)
- DELETE argo.runonaws.dev A 192.168.1.57 proxy=false ttl=1
- DELETE test.runonaws.dev CNAME e73da4f6-a287-41eb-bfe9-563f73037d68.cfargotunnel.com. proxy=true ttl=1
+ CREATE test.runonaws.dev A 203.0.113.0 proxy=true ttl=1

argo.runonaws.dev and test.runonaws.dev were added directly in the Cloudflare dashboard, outside of the repo. DNSControl has no knowledge of them. From its perspective, those records do not belong in the zone and will be deleted on the next push. The preview comment is the warning before that happens.

test.runonaws.dev shows up as both a DELETE and a CREATE. The CNAME pointing to a Cloudflare Tunnel was manually added; the replacement A record pointing to 203.0.113.0 comes from reserved.json. So the sync removes the manual record and installs the reserved placeholder in its place.

This is exactly the behavior you want, but it means any record added through the dashboard will eventually get wiped. Before merging a PR, check the preview output for unexpected DELETEs. If a record that should stay is listed for deletion, port it into a domains/ JSON file first. Once it is in the repo, it is safe.

The alternative is to merge and let the sync clean up the manual additions. Either way, the zone ends up matching the code.

Provider portability

DNSControl abstracts the provider. Switching from Cloudflare to Route53 is a one-line change in dnsconfig.js. The domain JSON files, reserved and internal lists, and CI pipelines stay the same. Zone definitions are not coupled to the DNS provider.

The POC uses Cloudflare because that is where runonaws.dev lives, but the same setup runs against any of the 30+ supported providers.

Getting started

The repo is at amaanx86/cloudflare-dnscontrol. Fork it, swap in your domain name and Cloudflare API token, and populate domains/ with your existing records.

The hardest part is the initial import. DNSControl’s get-zones command can export an existing zone to its DSL format, giving you a starting point for the JSON conversion. Do it once to get in sync, and every change after that is a PR.

DNS is infrastructure. Treat it like infrastructure.

Also published on Hashnode, DEV.to, and AWS Builder Center.

Bypassing Cloudflare for True Origin Health Checks with Blackbox Exporter and CoreDNS

Tue, 28 Apr 2026 00:00:00 GMT

You have 100+ web endpoints behind Cloudflare. You need to know if the origin is healthy. Not the edge. Not the CDN cache. The actual server behind it.

Prometheus Blackbox Exporter is the standard answer for HTTP probing. Point it at a URL, it makes a request, you get probe_success, probe_duration_seconds, probe_http_status_code. Clean, simple, works.

Until you put Cloudflare in front of everything.

The problem

When Blackbox probes https://app.example.com, DNS resolves to Cloudflare’s edge. The request hits their network, gets proxied, maybe served from cache. Your probe comes back 200 OK. Meanwhile the origin server is on fire and has been returning 502s for twenty minutes. Cloudflare is masking the failure by serving a stale response or its own error page with a 200-ish status.

Even when it does proxy through to origin, you are measuring Cloudflare-to-origin latency plus edge processing time. Your probe_duration_seconds metric is useless for understanding actual server response time.

The health check is lying to you. It is monitoring Cloudflare’s availability, not yours.

Naive approaches that do not work

“Just use the origin IP directly.”

https://198.51.100.10 - sure, but now TLS fails. The certificate is issued for app.example.com, not for an IP address. You could skip TLS verification (insecure_skip_verify: true), but then you are not testing what your users actually experience. And many applications inspect the Host header for routing. Hitting a raw IP with no SNI or wrong Host header gets you a 404 or default backend, not a real health check.

“Use /etc/hosts on the monitoring server.”

Map app.example.com to the origin IP in /etc/hosts. This works for exactly one server. Then you have 80 endpoints across a fleet of monitoring nodes and you are managing hosts files by hand. You update an origin IP and forget to update the hosts file. You add a new domain and forget to add the entry. It is a maintenance problem disguised as a solution.

Also, depending on your deployment model, you might not even control /etc/hosts. And even if you do, the change is system-wide. Every process on that host now bypasses Cloudflare for those domains, which is probably not what you want.

“Set a custom DNS server in Prometheus.”

Prometheus does not probe endpoints itself. It tells Blackbox Exporter to probe them. Changing DNS settings on the Prometheus host does nothing - Blackbox is the one resolving the domain. You need to control DNS resolution at the Blackbox process level.

“Curl with –resolve.”

Works great for ad-hoc debugging. Not a thing you can configure in Blackbox Exporter. The prober does not expose a per-target DNS override. It resolves using whatever DNS the process has access to.

Architecture

                         Public DNS (Cloudflare)
                         Returns edge IPs -- NOT what we want
                                  |
                                  X  (bypassed)
                                  |
+----------------------------+    |    +----------------------------+
|      Prometheus            |    |    |      CoreDNS Sidecar       |
|                            |    |    |                            |
|  "probe app.example.com"   |    |    |  hosts.override:           |
|         |                  |    |    |    198.51.100.10            |
|         v                  |    |    |        app.example.com     |
|  +------------------+      |    |    |        api.example.com     |
|  | Blackbox Exporter |------+---------->  203.0.113.50            |
|  |                  |  DNS query |    |        docs.example.org   |
|  |  dns: 172.20.0.53|<----------+    |                            |
|  +------------------+      |    |    |  fallthrough -> 8.8.8.8    |
|         |                  |    |    +----------------------------+
+---------|------------------+    |
          |                       |
          | TLS + SNI: app.example.com
          | Host: app.example.com
          | IP: 198.51.100.10
          |
          v
+----------------------------+
|   Origin Load Balancer     |
|   198.51.100.10:443        |
|                            |
|   Serves real certificate  |
|   Routes by Host header    |
+----------------------------+
          |
          v
+----------------------------+
|   Application Server       |
|   Actual origin response   |
+----------------------------+

The trick is one line in the Blackbox Exporter config: point its DNS at the CoreDNS sidecar instead of the system resolver. CoreDNS returns origin IPs for overridden domains, falls through to public DNS for everything else. Blackbox never knows the difference - it just resolves and probes like normal.

The actual solution: a DNS sidecar

The insight is simple once you see it: if you control what DNS server Blackbox Exporter uses, you control where its probes land. You do not need to touch Blackbox’s config at all. You do not need to modify how Prometheus scrapes. You just need a DNS resolver that lies about specific domains - returns your origin load balancer IP instead of Cloudflare’s edge IP.

CoreDNS is perfect for this. It is a lightweight, pluggable DNS server. The relevant config is about six lines:

. {
    hosts /etc/coredns/hosts.override {
        fallthrough
    }

    forward . 8.8.8.8

    log
    errors
}

That is it. CoreDNS loads a hosts file for overridden domains. Anything not in the file falls through to 8.8.8.8 for normal resolution.

The hosts override file is a plain hosts-format file. Nothing exotic:

# Origin LB for production
198.51.100.10  app.example.com
198.51.100.10  api.example.com
198.51.100.10  cdn.example.com

# Origin LB for staging
198.51.100.20  staging.example.com
198.51.100.20  staging-api.example.com

# Origin LB for partner sites
203.0.113.50   docs.example.org
203.0.113.50   status.example.org

203.0.113.71   shop.example.net
203.0.113.72   blog.example.net

Each line maps a domain to the public IP of the actual load balancer in front of the origin servers. Multiple domains can point to the same LB. Commented lines for decommissioned endpoints. Straightforward.

Then you point Blackbox Exporter’s DNS resolution at this CoreDNS instance. If CoreDNS is running at 172.20.0.53, you set Blackbox’s DNS to that address. The mechanism depends on your runtime - a dns directive, a resolv.conf mount, whatever your environment supports. The point is that Blackbox resolves domains through CoreDNS, not through the system default.

What happens at probe time

The flow:

Prometheus tells Blackbox: probe https://app.example.com
Blackbox resolves app.example.com via CoreDNS at 172.20.0.53
CoreDNS checks its hosts override file, finds the entry, returns 198.51.100.10
Blackbox connects to 198.51.100.10:443, sends TLS ClientHello with SNI app.example.com
The load balancer presents its certificate for app.example.com, TLS completes normally
Blackbox sends the HTTP request with Host: app.example.com
The application responds directly - no Cloudflare in the path

TLS works because the domain name is preserved. The origin LB has the real certificate. SNI and Host header are correct because Blackbox thinks it is talking to app.example.com - it just happens to have resolved to a different IP than Cloudflare’s.

This is the key: you are not changing what Blackbox probes, you are changing where the name resolves to. Everything else - TLS negotiation, Host header, path, response validation - stays exactly the same.

The Prometheus side

Nothing changes in how you write your scrape config. Blackbox probing uses the standard multi-target pattern:

- job_name: blackbox_https_prod
  metrics_path: /probe
  scrape_interval: 60s
  scrape_timeout: 15s
  params:
    module: [http_2xx]
  static_configs:
    - targets:
        - app.example.com
        - api.example.com
        - shop.example.net
        - blog.example.net
        - docs.example.org
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
      replacement: https://$1
    - source_labels: [__address__]
      target_label: instance
    - target_label: __address__
      replacement: <blackbox-host>:9115

Targets are just domain names. The relabel chain rewrites them into probe parameters and sets the instance label to the human-readable domain. Nothing here is aware of Cloudflare or DNS overrides. That is all handled one layer down.

This separation matters. The person adding a new endpoint to monitoring does not need to know about the DNS plumbing. They add the domain to the target list. If the domain needs a DNS override, someone adds it to the hosts file. Two different concerns, two different files.

Alerting on actual origin health

With Cloudflare out of the path, your alerts are honest. A probe_success == 0 means the origin is actually down, not that Cloudflare is having a bad day.

- alert: EndpointDown
  expr: probe_success{job=~"blackbox_https_(prod|nonprod)"} == 0
  for: 0m
  labels:
    severity: critical
  annotations:
    summary: "Endpoint DOWN: {{ $labels.instance }}"

- alert: EndpointHighLatency
  expr: probe_duration_seconds{job=~"blackbox_https_(prod|nonprod)"} > 3
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "High latency: {{ $labels.instance }} ({{ $value | printf \"%.2f\" }}s)"

for: 0m on the down alert - no waiting. If the origin is unreachable, you want to know immediately. The latency alert uses for: 10m because transient spikes happen and you do not want noise. These thresholds are tunable, but the point is that the data feeding them is real. You are measuring origin behavior, not CDN behavior.

Why not just use /etc/hosts at scale

It is worth expanding on why the hosts-file-on-the-monitoring-host approach breaks down, because it is the first thing most people try.

Scope creep. You start with five domains. Then twenty. Then eighty. Each load balancer IP change requires updating the file and restarting the service. Miss one and you are probing the wrong backend for weeks without knowing.

Blast radius. System-wide DNS overrides affect everything on the host. Your monitoring stack bypasses Cloudflare - great. But so does every other process. If you are running anything else on that host that legitimately needs to hit the CDN edge, you have broken it.

Reproducibility. The hosts file is stateful, manual, and easy to drift. CoreDNS with a hosts file is also just a flat file, but it is scoped to a single process (Blackbox), versioned in your repo, and trivially reproducible. Rebuild the monitoring stack from scratch and the DNS overrides come with it.

Multiple monitoring replicas. If you run more than one Prometheus instance for redundancy (and you should), each one has its own Blackbox Exporter. Managing hosts files across multiple hosts is the kind of thing that works until it doesn’t. A dedicated DNS sidecar per replica is self-contained.

Operational considerations

Adding a new endpoint. Two steps: add the domain to Prometheus targets, add the domain-to-IP mapping in the CoreDNS hosts file. Reload both. Five-minute task.

IP changes. When a load balancer gets a new IP, update one line in the hosts file, restart CoreDNS. Blackbox automatically starts resolving to the new address on its next probe.

Domains not behind Cloudflare. The fallthrough directive in CoreDNS means anything not in the hosts file gets resolved normally via 8.8.8.8. So you can mix Cloudflare-proxied and direct endpoints in the same Blackbox instance. No special handling needed.

Debugging. dig @172.20.0.53 app.example.com tells you exactly what Blackbox will resolve. If the probe is failing, check whether CoreDNS returns the right IP. If it does, the problem is downstream. If it does not, your hosts file is wrong. Clean separation of concerns.

The broader pattern

This is not really about Cloudflare specifically. The same approach works for any situation where public DNS does not resolve to where you want your probes to land:

Services behind any CDN or reverse proxy (Akamai, Fastly, AWS CloudFront)
Internal services with split-horizon DNS where monitoring runs in a different network zone
Migration scenarios where you want to probe the new backend before cutting DNS over
Canary deployments where you want to health-check a new origin before adding it to the pool

The pattern is always the same: run a lightweight DNS resolver as a sidecar, override the specific records you need, let everything else pass through. Your probing tool does not need to know about any of this. It just resolves names and makes requests, which is exactly what it should do.

The setup is simple. CoreDNS is maybe 30MB of memory. The hosts file is trivially maintainable. And the result is that when your pager goes off at 3am, you know it is because the origin is actually down - not because Cloudflare rotated an edge node.

That is the kind of signal worth waking up for.

Also published on DEV.to, AWS Builder Center, and Hashnode.

The Git habits that made reviewers trust me before they read a single line of code

Wed, 15 Apr 2026 00:00:00 GMT

Most developers treat Git as a save button. Commit when something works. Push before you go to sleep. That’s it. And honestly, it gets the job done, right up until you start sending PRs to projects with real reviewers, or working on teams where the commit history actually has to make sense to someone else.

Here’s the thing: reviewers form an opinion about you before they open a single file. The branch name, the commit messages, the PR description, all of it is visible before they click into the diff. I figured this out sending contributions to projects across GitHub, GitLab, Azure DevOps, and AWS CodeCommit. The feedback loop was fast. Clean work got reviewed and merged. Noisy work got follow-up questions, or sat in the queue.

So here’s what I actually do.

Three permanent branches, clear purpose for each

I keep three long-lived branches. That’s it.

main: production only. Nothing goes here without a PR that passed UAT. This branch is protected.
uat: staging and pre-production testing. Business users validate here before anything touches main.
dev: where everything starts. Features, fixes, experiments, all of it begins here.

The exact names are not the point, and I want to be clear about that before anyone argues about it. Some projects call production master, some main. Pre-prod might be uat, staging, or stage. The integration branch might be dev or devel. That choice belongs to the maintainer, and more often it is already settled by how the company, the SaaS team, or the OSS project has always done it. When you join, you inherit the convention; you don’t get to relitigate it. What actually matters is that there are stable, named tiers, that everyone knows which tier means what, and that the protected one is genuinely protected. Match the local vocabulary and apply the same discipline underneath it. I use main, uat, and dev here because that is the naming on the projects I am describing.

The reason for a dedicated pre-prod branch (not just a tag or a deploy config) is that it gives you a stable checkpoint. When you find a bug there, you know it’s code that passed dev but failed real-world testing. That’s a different category of bug than something that never left the developer’s machine, and it deserves a different branch to fix it.

Short-lived branches with a naming pattern that explains itself

When I start any new piece of work I create a branch from the right source and name it properly:

# Feature from dev
git checkout dev && git pull origin dev
git checkout -b feat/user-authentication

# Bug found during UAT testing
git checkout uat
git checkout -b bugfix/cart-crash-fix

# Production emergency
git checkout main
git checkout -b hotfix/prod-db-connection

The prefixes matter. feat/, bugfix/, hotfix/, test/: any reviewer or CI system that sees these knows the category of the work before it looks at anything else. And because the branch name describes the problem, not the implementation, it reads the same to a human and to a ticket system.

Hotfixes are the one case where you merge in three directions: back into main, uat, and dev. Missing one creates drift and you’ll feel it later.

Conventional commits: the part that changed how I write history

This is the single highest-leverage change I made to how I use Git. The format is simple:

<type>(<scope>): <description>

Types I use:

feat: new feature
fix: bug fix
docs: documentation only
refactor: restructured code, no behavior change
perf: performance improvement
test: tests added or updated
chore: build tooling, CI, dependencies

A real example from one of my projects:

feat(auth): add JWT refresh token rotation

Implements sliding expiration using a short-lived access token (15m)
and a rotating refresh token (7d). Old refresh tokens are invalidated
immediately on use to prevent token replay.

Closes: #142

When I send a PR with a history like this, reviewers can read the commit log like a changelog. They don’t have to reverse-engineer what I did from the diff. And when something breaks in production six months later, the person doing the post-mortem can git log --oneline and actually understand the sequence of events.

Semantic versioning falls out of this naturally too. If all your commits since the last tag are fix:, you ship a patch. If there’s a feat:, it’s a minor. If anything has a BREAKING CHANGE in the footer, it’s a major. Your CHANGELOG.md can be generated automatically. Your release tags follow v1.2.3. No guessing.

git tag -a v1.0.0 -m "Release version 1.0.0"
git push origin --tags

Cherry-pick: when you need one thing from a branch, not the whole branch

This is the tool most developers don’t reach for until they’re stuck, and then they use it wrong. Here’s the actual use case.

I had a project where uat and main had diverged significantly. Both branches had been accumulating changes. We needed the azure-pipelines.yml from uat in main, just that one file, because getting the pipeline consistent across environments was blocking the team. A full merge would have pulled in unvalidated changes, and that is not acceptable for a protected branch.

So instead of a merge request, I used cherry-pick to move exactly the commits that touched that file.

First, find the commits that changed the file in the source branch:

git checkout uat
git log uat -- azure-pipelines.yml --oneline

Output:

b5df122 feat: update Azure Pipelines configuration to specify custom agent pool and demands
44cbc1e feat: update image tag for crmiskuat container to use 'uat' version
8fee9b4 feat: add Azure Pipelines configuration for OCI image build and push

Switch to the target branch and apply them in order (oldest first):

git checkout main
git cherry-pick 8fee9b4 44cbc1e b5df122

If there’s a conflict on just that file:

# Resolve the conflict in your editor, then:
git add azure-pipelines.yml
git cherry-pick --continue

This creates new commits on main with the same changes but different hashes. The original commits stay on uat. You get a clean, traceable history on main without dragging in anything that wasn’t ready.

The key thing to understand: cherry-pick is not a shortcut for avoiding code review. You still open a PR with these commits. The difference is that the PR is surgical. It does one thing, and reviewers can verify exactly what that thing is.

The PR description template I actually fill out

A PR is a document. If the description is empty or says “fix stuff,” you’re forcing the reviewer to read the entire diff to understand context that you already have in your head.

## Linked Issue / Ticket
<!-- GitHub/GitLab: Closes #123 | Jira: PROJECT-456 | Azure DevOps: AB#789 -->
Closes #

## Summary
Brief description of what this MR accomplishes.

## Changes
- Specific change with context
- Another change and why it was necessary

## Testing
- [ ] Unit tests added/updated
- [ ] Integration tests pass
- [ ] Manual testing completed in dev/UAT

## Screenshots (if applicable)

## Checklist
- [ ] Code follows project standards
- [ ] Self-review done
- [ ] No new warnings
- [ ] Documentation updated

This is not bureaucracy. It’s a forcing function for you to think through what you did before someone else has to. Half the time, filling out the testing checklist is when I catch something I missed.

Post-merge cleanup

After a branch merges, delete it. Both locally and remotely.

git branch -d feat/user-authentication
git push origin --delete feat/user-authentication

A repository with 200 stale branches is a repository nobody trusts. Clean branches signal that someone is actually maintaining this.

Why this matters more now, not less

The argument I used to hear against structured commits was that it slows you down. Write the code, get it working, the history is just metadata.

That argument has fully inverted. AI code review tools, automated changelog generation, semantic release pipelines: all of them work significantly better when the commit history is structured. When I use an AI assistant to review a PR now, it reads the conventional commits and understands the intent without me explaining it. When I ask it to generate release notes, it has something to work from.

But more fundamentally, across reviewers at AWS, GitHub, GitLab, and Azure, the pattern was the same. The PRs that got attention were the ones that looked like the contributor had thought it through. Clean branch name, useful commits, filled-out description. It signals that you respect the reviewer’s time, and that is the actual thing that builds trust.

The technical content still has to be right. But a well-structured contribution gets a genuine review. A sloppy one gets skimmed.

The actual workflow in one place

# Start a feature
git checkout dev && git pull origin dev
git checkout -b feat/your-feature

# Work in small, meaningful commits
git add src/specific-file.ts
git commit -m "feat(auth): add token expiry validation"

# Push and open PR to dev
git push origin feat/your-feature

# After merge, clean up
git branch -d feat/your-feature
git push origin --delete feat/your-feature

That’s it. No magic. Just consistency applied over enough time that it becomes automatic.

Also published on DEV.to, AWS Builder Center, and Hashnode.

Built My Own Prometheus Service Discovery for Oracle Cloud Because a 3-Year-Old PR Never Got Merged

Sun, 12 Apr 2026 00:00:00 GMT

There is a specific kind of frustration reserved for when you know a problem is solved, you can see the solution, and you still cannot use it. That is how this project started.

The Context: Setting Up Observability From Scratch

I was building out observability from scratch across Oracle Cloud Infrastructure - multiple tenancies, multiple regions, a decent number of compute instances spread across compartments. The goal was full coverage: every VM enrolled in monitoring, no gaps, no guesswork.

When you are starting from zero, one of the first things you ask is how Prometheus is going to discover what it needs to scrape. For AWS you have EC2 service discovery built right in. Same for GCP, Azure, Hetzner, DigitalOcean. You configure credentials, set some filters, and Prometheus takes care of the rest.

OCI is not on that list.

I searched. I found a pull request in the Prometheus repository opened by an engineer at Oracle. It was exactly what I needed. It was also three years old and had never been merged. Comments, reviews, back and forth, and then silence. The PR is still open today. If you have gone looking for OCI service discovery in Prometheus you have probably landed on that same page, felt a brief moment of hope, and then noticed the date.

So I built it myself.

What I Actually Needed

The requirement was simple: new VM comes up, Prometheus finds it and starts scraping. No manual steps in that loop. Not because I was fixing a broken process - there was no process yet. I was designing this from scratch and I was not going to design it with a gap where a human has to remember to update a config file.

The concern was blind spots. Infrastructure grows, VMs get provisioned, things change. If enrollment is manual, coverage is only as good as the last person who remembered to do it. I wanted observability that was structurally complete, not best-effort.

The workflow I landed on:

Tag the VM in OCI. When provisioning a new instance, add a tag - something like prometheus:scrape = true. That is the only enrollment step.

Open the Prometheus port on the network security group. Allow the Prometheus server IP to reach port 9100. One rule, specific source, no broad exposure.

Run the node exporter playbook. An Ansible playbook installs and starts node exporter. Done.

That is the full enrollment flow. The VM is in monitoring. No touching the Prometheus config. No reloading Prometheus. No rollout restart of a Kubernetes deployment. No SSH back into the server to verify anything.

The proxy polls OCI, finds every instance with the right tag across all configured tenancies and compartments, and hands the target list to Prometheus via the HTTP service discovery API. Prometheus picks it up on its next refresh cycle. The whole thing is automatic by design, not patched in after the fact.

What I Built

oci-prometheus-sd-proxy is a lightweight Go service that implements the Prometheus HTTP Service Discovery API for OCI.

You point Prometheus at it like this:

scrape_configs:
  - job_name: oci_instances
    http_sd_configs:
      - url: 'http://oci-sd-proxy:8080/v1/targets'
        authorization:
          type: Bearer
          credentials: 'YOUR_TOKEN'
        refresh_interval: 60s
    scrape_interval: 30s
    scrape_timeout: 10s
    metrics_path: /metrics
    relabel_configs:
      - source_labels: [__meta_oci_instance_name]
        target_label: instance
      - source_labels: [__meta_oci_tenancy_name]
        target_label: tenancy
      - source_labels: [__meta_oci_compartment_name]
        target_label: compartment
      - source_labels: [__meta_oci_region]
        target_label: region
      - source_labels: [__meta_oci_availability_domain]
        target_label: availability_domain
      - source_labels: [__meta_oci_instance_shape]
        target_label: shape

The proxy handles the rest. It scans OCI, filters by tag, and returns targets with rich metadata: tenancy, compartment, region, availability domain, shape, fault domain, private IP, and all your custom OCI tags as Prometheus labels. Use them for relabeling, alerting rules, dashboards - whatever you need.

A few things I cared about when building it:

Multi-tenancy from day one. The proxy handles all tenancies in parallel from a single config file. One deployment, full coverage.

Rate limiting. OCI’s API will return 429s if you push it too hard. The proxy uses a token bucket proactively and a retry policy reactively. Discovery does not silently fail under load.

Security. Bearer token auth, distroless container, read-only config mounts. In an HA setup it sits on the local network, only reachable by Prometheus. It does not need to be internet-facing and it should not be.

Caching. Discovery results are cached so Prometheus always gets a response, even if the OCI API is momentarily slow or rate limiting.

Battle Tested

This is running in production across 10+ Oracle Cloud tenancies - different regions, different compartments, different team setups. It has handled OCI API slowness, tenancy permission edge cases, and the general entropy that comes with real infrastructure at scale.

The thing that still satisfies me is watching a new VM appear in Grafana within a minute of it booting, with nobody doing anything to make that happen. That is what good observability infrastructure should feel like. You build it right once and it stays right.

Why This Did Not Exist Already

OCI is a smaller player compared to AWS or GCP and Prometheus contributors naturally prioritize the platforms most of their users are on. Oracle engineers clearly wanted to solve this - that PR is evidence of that - but getting a first-party integration merged upstream is a long road with no guarantees.

The HTTP SD API that Prometheus exposes is actually the right answer for this situation. It lets any platform plug in without touching the Prometheus codebase. I just had to build the other end of that interface.

Try It

Repository: github.com/amaanx86/oci-prometheus-sd-proxy
Documentation: oci-prometheus-sd-proxy.readthedocs.io

The docs cover installation, the full configuration reference, OCI IAM permissions, and Docker Compose deployment examples. If you are building observability on OCI and want automatic enrollment from day one, this is what I use.

And if you were one of the people who commented on that PR hoping it would eventually land - same. This is what I built instead.

References

Also published on DEV.to.

End-to-End Supply Chain Security for a Go Project: TUF on CI, cosign, and SLSA L3

Sat, 11 Apr 2026 00:00:00 GMT

Adding cosign sign to a CI pipeline and calling it “signed releases” is a bit like putting a lock on a glass door. The lock works. The glass does not. Signing the image proves a specific digest was signed by a specific identity at a specific time. It says nothing about whether the source commit matches what was built, whether the build environment was clean, or whether someone replaced the release asset after the fact.

I had been going deep on supply chain security for a while - reading through TUF specs, the Sigstore design docs, how Fulcio issues short-lived certificates, how Rekor works as an append-only transparency log. At some point I came across the OpenSSF Best Practices requirements and saw the full picture laid out as a checklist. I could have just signed the image and moved on. Instead I used oci-prometheus-sd-proxy - a project that does OCI Prometheus service discovery - as the thing to actually build it on. I wanted to understand each layer well enough to explain it, not just wire it up. What I ended up with: cosign keyless signing, a CycloneDX SBOM attestation, SLSA L3 build provenance, and TUF metadata distribution via tuf-on-ci published to GitHub Pages. No long-lived keys anywhere in the pipeline.

Why each layer exists

This is the question worth answering properly, because you can absolutely ship just cosign and be in a better position than 95% of projects. So why go further?

HTTPS protects against network-level tampering. It does not protect against a compromised GitHub account pushing a backdoored release, or a build that was modified after it completed, or an asset quietly swapped post-publication.

cosign on a container image proves that a specific digest was signed by a specific OIDC identity with the event recorded in Rekor’s transparency log. That is genuinely useful - but it only covers the image. It says nothing about the source ref, the build environment, or the workflow that produced it. Someone could sign a backdoored binary and the cosign verification would pass.

SLSA L3 provenance fills that gap, and it is the layer I found most interesting to wire up. The provenance is generated in a separate, isolated signing job with its own OIDC identity, not in the main build job. That isolation is what makes L3 meaningful: an attacker who compromises the main build job cannot forge L3 provenance because they do not control the signing job’s OIDC token. The provenance attests to the exact source ref, the exact workflow, and the exact runner environment. You can look at a SLSA L3 attestation and know that the image you are running came from that commit in that repo via that verified builder.

TUF adds something orthogonal to all of the above - it is about distribution trust, not just signing trust. It adds a role-based metadata layer where clients can verify that a release target was authorised by the project’s key holders, that the metadata has not been rolled back to a previous version, and that the metadata is actually fresh. The design difference that matters: TUF survives key compromise in a way that a single cosign keypair does not. If my cosign key leaked tomorrow, every past signature would be under a cloud. With TUF, key rotation is a defined protocol. The damage is bounded and recoverable.

The point of having all four is that verification is fully independent. Verifying the image signature, the build provenance, and the TUF metadata chain are three separate operations against different data sources. Compromising any single one of them is not enough to ship a malicious release undetected. You need to compromise all of them simultaneously - and the transparency logs make doing that silently very hard.

Two repos, one trust boundary

The implementation lives across two repositories:

amaanx86/oci-prometheus-sd-proxy - the application, the build pipeline, the release workflow
amaanx86/oci-prometheus-sd-proxy-tuf-on-ci - TUF metadata, managed by tuf-on-ci, published to GitHub Pages

Keeping them separate was a deliberate trust boundary decision, not just organisation. The app CI can push a signing branch to the TUF repo, but it cannot merge that branch or sign targets.json. That step requires my OIDC identity authenticating via Sigstore in a browser - not the CI system’s token. An attacker who steals the app repo’s CI tokens hits a hard wall at the TUF signing step. They can push a branch, but they cannot authorise the release. The human is the last gate.

App repo CI (run #24290258383)
  └── builds image (linux/amd64, linux/arm64)
  └── pushes to GHCR
  └── cosign attest --type cyclonedx sbom.cyclonedx.json (SBOM)
  └── cosign sign (image signature, OIDC -> Fulcio cert -> Rekor)
  └── cosign attest --type  release-metadata.json
  └── slsa-github-generator -> SLSA L3 provenance (isolated job)
  └── pushes sign/release-1-4-2-rc-24290258383 branch -> TUF repo

TUF repo (tuf-on-ci) [PR #8]
  └── signing-event.yml detects branch push -> opens signing PR
  └── Maintainer: tuf-on-ci-sign (browser OIDC -> @amaanx86 identity)
  └── PR merged -> online-sign.yml refreshes snapshot + timestamp
  └── publish.yml -> GitHub Pages
  └── test.yml -> smoke-tests TUF client (scheduled)

Users / Policy Engines
  └── cosign verify - checks image signature against Rekor
  └── cosign verify-attestation --type cyclonedx - checks SBOM
  └── cosign verify-attestation --type  - checks release-metadata
  └── slsa-verifier verify-image - checks SLSA L3 provenance
  └── TUF client (python-tuf ngclient) - fetches metadata from GitHub Pages, verifies chain

The build pipeline

The workflow (docker-build-push.yml) triggers on release publication. After pushing the multi-arch image (linux/amd64 and linux/arm64) to GHCR it does five more things.

SBOM generation and attestation

First, Syft generates a CycloneDX SBOM for the pushed image, which gets attached as a cosign attestation:

cosign attest \
  --yes \
  --predicate sbom.cyclonedx.json \
  --type cyclonedx \
  ghcr.io/amaanx86/oci-prometheus-sd-proxy@sha256:759e255e607f623e0b1ee4ea9df02b2aefd89e2c9ec979842ee2e6f8b21772fd

The SBOM is also uploaded as a release asset (the 80.7 KB *.cyclonedx.json artifact visible on the workflow run).

cosign image signing

cosign sign \
  --yes \
  ghcr.io/amaanx86/oci-prometheus-sd-proxy@sha256:759e255e607f623e0b1ee4ea9df02b2aefd89e2c9ec979842ee2e6f8b21772fd

No --key flag. cosign uses the GitHub Actions OIDC token to get an ephemeral certificate from Fulcio, signs the digest, and records the operation in Rekor. The private key is generated in memory and never stored. The Rekor entry pins the workflow identity to the specific digest at a specific timestamp.

Note the image is signed by digest, not by tag. Tags are mutable; the digest is what the signature actually covers.

No secret to rotate. No key to leak.

Release-metadata attestation

The workflow generates a release-metadata.json with the image digest, source commit, release tag, and build timestamp, then attaches it as an attestation under a custom predicate type:

cosign attest \
  --yes \
  --predicate release-metadata.json \
  --type https://github.com/amaanx86/oci-prometheus-sd-proxy/release-metadata \
  ghcr.io/amaanx86/oci-prometheus-sd-proxy@sha256:759e255e607f623e0b1ee4ea9df02b2aefd89e2c9ec979842ee2e6f8b21772fd

Using a project-specific type URI keeps the attestation namespaced and lets cosign verify-attestation --type <uri> fetch exactly this attestation rather than every in-toto statement on the image.

SLSA L3 provenance

uses: slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@v2.0.0
with:
  image: ghcr.io/amaanx86/oci-prometheus-sd-proxy
  digest: ${{ needs.build-and-push.outputs.digest }}

slsa-github-generator runs as a reusable workflow with its own isolated OIDC identity. The provenance attestation is generated and signed there, not in the main build job. L3 specifically requires this isolation - the verified builder identity in the provenance is https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@refs/tags/v2.0.0, distinct from the app workflow’s identity.

Triggering the TUF signing event

Last step: clone the TUF repo and push a signing branch with a new target file at targets/oci-prometheus-sd-proxy/releases/v1.4.2-rc.json. The branch name embeds the run ID to avoid collisions: sign/release-1-4-2-rc-24290258383 (dots in the version become hyphens, run ID appended). This branch push is what kicks off the TUF side of the pipeline.

The Python tuf library (v5+) is used inline to update metadata/targets.json with the new target entry before committing - the targets metadata version is incremented and signatures cleared, ready for the human signing step.

tuf-on-ci

tuf-on-ci manages the TUF metadata lifecycle entirely within a GitHub repository. All online signing uses GitHub Actions OIDC. All offline signing uses Sigstore OIDC (browser-based). No key files anywhere.

The TUF repo has four workflows. signing-event.yml fires on any sign/** branch push, opens a PR, and annotates it with what needs signing. online-sign.yml runs after a signing PR is merged and refreshes snapshot.json and timestamp.json using the Actions OIDC token. publish.yml deploys everything to GitHub Pages. test.yml runs on a schedule and verifies the full metadata chain with a real TUF client to catch expiry or breakage before any user does.

Signing a release

When signing-event.yml opens PR #8 with title “Signing event: sign/release-1-4-2-rc-24290258383”, I run:

tuf-on-ci-sign sign/release-1-4-2-rc-24290258383

This opens a browser window for Sigstore OIDC. I authenticate as @amaanx86 via GitHub. Fulcio issues an ephemeral certificate tied to that identity, the tool signs targets.json, and pushes the signature to the branch. Rekor gets an entry proving the person at @amaanx86 authorised this specific targets update at this specific time.

Worth being clear on what “offline” means here: it requires a human with a verified identity, not a CI token. It does not require an air-gapped machine. The private key is still ephemeral.

After merge, online-sign.yml takes over and refreshes snapshot and timestamp automatically using the Actions OIDC token. No human needed for that part.

What gets published

GitHub Pages at amaanx86.github.io/oci-prometheus-sd-proxy-tuf-on-ci/metadata/ serves:

root.json - signed by @amaanx86 via Sigstore OIDC; defines trusted key holders for all roles
targets.json - signed by @amaanx86; lists all authorised release targets with digests
snapshot.json - signed by GitHub Actions OIDC; prevents any metadata file from being swapped with an older version
timestamp.json - signed by GitHub Actions OIDC; has a short validity window to prevent freeze attacks

Each release gets a target file at targets/oci-prometheus-sd-proxy/releases/v1.4.2-rc.json. The TUF metadata key (used inside targets.json) is oci-prometheus-sd-proxy/releases/v1.4.2-rc.json - the path is relative to the targets directory, not the repo root.

Verifying v1.4.2-rc

All verification uses the digest, not the tag, since the tag is a mutable pointer.

Image signature:

cosign verify \
  --certificate-identity="https://github.com/amaanx86/oci-prometheus-sd-proxy/.github/workflows/docker-build-push.yml@refs/tags/v1.4.2-rc" \
  --certificate-oidc-issuer="https://token.actions.githubusercontent.com" \
  ghcr.io/amaanx86/oci-prometheus-sd-proxy@sha256:759e255e607f623e0b1ee4ea9df02b2aefd89e2c9ec979842ee2e6f8b21772fd

SBOM attestation:

cosign verify-attestation \
  --certificate-identity="https://github.com/amaanx86/oci-prometheus-sd-proxy/.github/workflows/docker-build-push.yml@refs/tags/v1.4.2-rc" \
  --certificate-oidc-issuer="https://token.actions.githubusercontent.com" \
  --type cyclonedx \
  ghcr.io/amaanx86/oci-prometheus-sd-proxy@sha256:759e255e607f623e0b1ee4ea9df02b2aefd89e2c9ec979842ee2e6f8b21772fd

Release-metadata attestation:

cosign verify-attestation \
  --certificate-identity="https://github.com/amaanx86/oci-prometheus-sd-proxy/.github/workflows/docker-build-push.yml@refs/tags/v1.4.2-rc" \
  --certificate-oidc-issuer="https://token.actions.githubusercontent.com" \
  --type "https://github.com/amaanx86/oci-prometheus-sd-proxy/release-metadata" \
  ghcr.io/amaanx86/oci-prometheus-sd-proxy@sha256:759e255e607f623e0b1ee4ea9df02b2aefd89e2c9ec979842ee2e6f8b21772fd

SLSA L3 provenance (requires digest reference - slsa-verifier rejects mutable tags):

slsa-verifier verify-image \
  "ghcr.io/amaanx86/oci-prometheus-sd-proxy@sha256:759e255e607f623e0b1ee4ea9df02b2aefd89e2c9ec979842ee2e6f8b21772fd" \
  --source-uri "github.com/amaanx86/oci-prometheus-sd-proxy" \
  --source-tag "v1.4.2-rc"
# Verified build using builder "...generator_container_slsa3.yml@refs/tags/v2.0.0"
# at commit a68d4cd5d29cc6b865c6804fe63adff14ac74b27
# PASSED: SLSA verification passed

TUF metadata chain verification is documented in detail - including the full client walkthrough and what each step validates - in the release verification docs. The short version: a compliant TUF client walks root trust, snapshot consistency, and timestamp freshness before fetching the target. If the metadata is expired, rolled back, or the signature chain is invalid, the client raises before returning anything. That freshness check is what separates TUF from static signature verification.

What this covers and where the gaps are

This pipeline secures the release artifact and its provenance chain. An operator fetching the image can independently verify who built it, from what source, in what environment, and that the release was authorised by a human identity. That is a lot more than most projects ship with.

But supply chain security has layers, and the release artifact is only one of them. A few honest gaps:

Go module dependencies. The SBOM shows what modules are in the binary, and go.sum pins their hashes. But govulncheck and a periodic dependency audit are what actually catch known-vulnerable transitive dependencies. The attestation proves the SBOM is authentic; it does not tell you the SBOM is safe.

Runtime enforcement. Signing and provenance only matter if someone checks them at deploy time. Right now verification is a manual step. The more interesting place this is going is integrating the cosign attestations and SLSA provenance into a Kyverno or OPA/Gatekeeper policy engine that enforces admission control in Kubernetes. Policy engines like Kyverno can query Sigstore and reject any image that lacks a valid SLSA L3 attestation from the correct workflow identity - automatically, at admission time, not as a manual verification step. That closes the loop between what we proved at build time and what is allowed to run in production.

TUF root key compromise. If the @amaanx86 GitHub account itself was compromised, an attacker could rotate the root TUF key in a way that would pass client verification. TUF supports threshold signatures across multiple root key holders to mitigate this, which becomes relevant as the project scales to multiple maintainers.

The dependency of your dependencies. None of this solves a compromised slsa-github-generator or a backdoored Syft release. That is a solved problem in theory (pin action hashes, verify the tools themselves) but it is worth naming.

What I am building toward

The next step that interests me most is closing the loop at runtime.

The next meaningful investment is runtime enforcement. Kyverno ClusterPolicies that require verified SLSA provenance before admission, OPA rules that check SBOM attestations against a known-safe package policy, and Sigstore-aware image admission are all achievable with what the pipeline already produces. The attestations are already there. The policy layer that consumes them is the missing piece.

After that, expanding to multi-maintainer TUF with hardware-backed root keys and threshold signatures would make the trust model genuinely robust at scale.

Full implementation details and the verification workflow are documented at oci-prometheus-sd-proxy.readthedocs.io/en/latest/releasing.html.

Also published on DEV.to, AWS Builder Center, and Hashnode.