It is Monday, 8:07 AM. The CFO opens the weekly revenue dashboard and sees a 38% drop compared to last week. By 8:15, three Slack channels are on fire. By 9:00, analysts are comparing screenshots, the data science team is asking why the forecasting model went crazy overnight, and the data engineering team is digging through pipeline logs that show… nothing. Every job is green.

The culprit is found at 11:30. Over the weekend, the commerce team released a new version of the checkout service and, as part of a refactoring, renamed the order status PAID to COMPLETED. From their point of view, it was a perfectly reasonable change: cleaner code, green tests. Nobody told them that a revenue dashboard filtered on order_status IN ('PAID', 'SHIPPED').

Nothing was technically broken. Same schema, same types, successful pipeline. And yet the data was wrong.

The painful part is not that data changed. Data must change, because products, customers, and businesses change. The painful part is that the change arrived as a surprise.

This is exactly the problem data contracts are designed to solve, but not in the way many teams implement them. A data contract is not a PDF in Confluence that nobody reads after the first week. A data contract is a process: a repeatable, automated way for producers and consumers to agree on what the data means, what “good” looks like, who owns it, and how changes are introduced safely.

In this guide, we will explore how to design data contracts with the Open Data Contract Standard (ODCS), catch breaking changes directly in pull requests, and enforce the contract at runtime with Databricks Lakeflow pipelines. Then we will see how to roll it all out without making your teams revolt.

The Real Problem: Implicit Assumptions

Most data incidents are not caused by an absence of validation. They are caused by implicit expectations, rules that everyone relies on but nobody ever wrote down:

• “This field is always present.”
• “This ID is unique.”
• “This timestamp is in UTC.”
• “This table lands by 8:00 AM.”
• “These are the only possible values of order_status."

When these expectations live only in people’s heads, or scattered across notebooks and SQL filters, every change becomes a gamble. The producer cannot know who depends on what, and the consumer cannot know when something is about to change.

Software engineering solved this problem for APIs a long time ago. Nobody would ship a breaking change to a public REST API without versioning, a changelog, and a deprecation period. Yet we routinely allow the tables that feed finance reports and ML models to change without any of these guarantees.

Data contracts bring the discipline of API design to data.

From Documentation to Process

A contract is not a one-time artifact. It is a workflow that must answer six questions:

1. Definition: what is this dataset, and what do its fields mean?
2. Expectations: which rules must always hold?
3. Ownership: who is responsible for the data, and who gets paged when it breaks?
4. Change management: how are changes versioned and communicated?
5. Enforcement: where are the rules evaluated, and what happens on a violation?
6. Observability: how do we know the contract is actually improving reliability?

Here is the litmus test: if your data contract does not change the behavior of your CI/CD, your pipelines, and your incident response, it is just documentation with a new name.

This is also what shift-left really means. Not “validate everything earlier” blindly, but catch a breaking change at the cheapest possible point: a pull request. Renaming PAID to COMPLETED costs one review comment in a PR. The same rename discovered in production costs a morning of incident response and a very uncomfortable meeting with the CFO.

Anatomy of a Contract: The Open Data Contract Standard

You could invent your own YAML format, and many teams did. The problem is that a custom format needs custom tooling for everything: validation, documentation, testing, and code generation.

This is why the industry is converging on the Open Data Contract Standard (ODCS), maintained by the Bitol project under the Linux Foundation. Version 3.2.0, released in September 2026, covers schema, data quality, SLAs, ownership, and infrastructure in a single file, and it is supported by a growing ecosystem of tools, including the open-source Data Contract CLI.

Here is the contract for the orders table from our story:

apiVersion: v3.2.0
kind: DataContract
id: commerce-orders
name: Orders
version: 1.2.0
status: active
description:
  purpose: Canonical orders table for analytics and finance reporting.

schema:
  - name: orders
    physicalType: table
    properties:
      - name: order_id
        logicalType: string
        primaryKey: true
        required: true
        unique: true
      - name: customer_id
        logicalType: string
        required: true
      - name: order_status
        logicalType: string
        required: true
        quality:
          - type: library
            metric: invalidValues
            arguments:
              validValues: [CREATED, PAID, SHIPPED, CANCELLED, REFUNDED]
            mustBe: 0
            severity: error
      - name: order_total
        logicalType: number
        description: Order total in EUR, including taxes and shipping.
        required: true
        quality:
          - name: non_negative_total
            type: custom
            engine: lakeflow
            implementation: "order_total >= 0"
            severity: warning
      - name: ordered_at
        logicalType: timestamp
        description: UTC timestamp of order creation.
        required: true
    quality:
      - type: library
        metric: rowCount
        mustBeGreaterThan: 1000
        severity: info

servers:
  - server: production
    type: databricks
    catalog: ${CATALOG:-prod}
    schema: commerce

team:
  name: commerce-data
  members:
    - username: commerce-data@company.com
      role: Owner

slaProperties:
  - property: freshness
    value: 1
    unit: hours
    element: orders.ordered_at

A few things are worth noticing:

• Expectations live next to the fields they protect. The quality block on order_status declares, in a machine-readable way, the exact assumption the revenue dashboard was silently relying on.
• Severity is explicit. Each rule carries a severity, and as we will see, this is what drives the runtime behavior.
• Ownership is part of the contract. A rule without an owner is just a suggestion.
• Environments are parameterized. ODCS 3.2.0 supports ${VARIABLE} references, so the same contract works across dev, test, and production.

Four Kinds of Expectations

Most teams stop at schema checks. Contracts become far more powerful when you separate expectations into four categories, because each one fails in a different way:

1. Structural (the shape): names, types, required fields, primary keys, and referential integrity. Typical failure: a column rename silently breaks every downstream model.
2. Semantic (the meaning): allowed values, units, currencies, timezones, and cross-field logic. Typical failure: the value has the right type but the wrong meaning, exactly like our PAID → COMPLETED story.
3. Operational (the reliability): freshness, completeness, and volume anomalies. Typical failure: the data is correct, but it arrives two hours after the board meeting.
4. Business quality (the fitness for purpose): revenue is never negative, and order totals match their line items. Typical failure: everything looks fine, but the numbers do not reconcile with finance.

Making the category explicit tells you who should own a rule and how strictly to enforce it. A broken primary key is almost always a hard stop. A volume anomaly on Black Friday is probably just good news.

Shift-Left: Catching Breaking Changes in the Pull Request

The contract lives in Git, next to the producer’s code. Every change to the data interface therefore goes through a pull request, which is the perfect place for automated checks.

The Data Contract CLI provides the building blocks: lint validates the contract against the ODCS schema, changelog describes what changed between two versions, and breaking fails when a change is backward-incompatible. Here is a minimal GitHub Actions workflow that compares the contract in the PR with the version on main:

name: data-contract-checks
on:
  pull_request:
    paths: ["contracts/**"]

jobs:
  contract:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - run: pip install datacontract-cli

      - name: Validate contract
        run: datacontract lint contracts/orders.odcs.yaml

      - name: Detect breaking changes
        run: |
          git show origin/main:contracts/orders.odcs.yaml > /tmp/orders.main.yaml
          datacontract changelog /tmp/orders.main.yaml contracts/orders.odcs.yaml
          datacontract breaking  /tmp/orders.main.yaml contracts/orders.odcs.yaml

Now replay our story. The commerce team opens a PR that renames PAID to COMPLETED and, to keep their tests green, updates the valid values in the contract. The breaking step fails, the PR is blocked, and the consumers listed in the contract are pulled into the review. The conversation that happened on Monday during an incident now happens on Thursday during a code review.

You can go one step further and run datacontract test against a staging environment, verifying that the actual records comply with the schema and quality rules before the change reaches production.

Enforcing the Contract at Runtime: Block, Quarantine, Warn

CI protects you from the changes you can see in a pull request. But data also breaks for reasons outside of any code review: a vendor changes its export format, a backfill replays corrupted files, or an upstream system has a bad day.

That is why contracts must also be enforced where data actually flows. The goal is not to “never fail”. The goal is to fail safely, and the right way to fail depends on the rule:

1. Block: stop the pipeline and page the owner. Use this when a violation would cause downstream harm, such as a broken primary key or an unknown order status.
2. Quarantine: divert invalid records to a separate table and keep processing the valid ones. Use this when the data is still valuable for investigation but must not contaminate trusted layers.
3. Warn: let the data through, but emit a metric and raise an alert. Use this for soft signals, or for new rules you are still calibrating.

A contract that only knows how to fail gets bypassed. A contract with no enforcement becomes theater.

On Databricks, these three strategies map naturally onto Lakeflow expectations. The key idea is simple: the pipeline should read its rules from the contract, instead of duplicating them in code. In our team convention, severity: error means block, warning means quarantine, and info means warn:

import yaml
from pyspark import pipelines as dp

with open(spark.conf.get("contract_path")) as f:
    contract = yaml.safe_load(f)

def rules_for(table: str, severity: str) -> dict:
    """Translate the contract into Lakeflow expectations for a given severity."""
    schema = next(s for s in contract["schema"] if s["name"] == table)
    rules = {}
    for prop in schema["properties"]:
        col = prop["name"]
        if severity == "error" and prop.get("required"):
            rules[f"{col}_not_null"] = f"{col} IS NOT NULL"
        for q in prop.get("quality", []):
            if q.get("severity") != severity:
                continue
            if q.get("metric") == "invalidValues":
                values = ", ".join(f"'{v}'" for v in q["arguments"]["validValues"])
                rules[f"{col}_valid_values"] = f"{col} IN ({values})"
            elif q.get("type") == "custom" and q.get("engine") == "lakeflow":
                rules[q["name"]] = q["implementation"]
    return rules

block_rules = rules_for("orders", "error")
quarantine_rules = rules_for("orders", "warning")
is_valid = " AND ".join(f"({r})" for r in quarantine_rules.values())

@dp.table(name="orders_silver")
@dp.expect_all_or_fail(block_rules)
@dp.expect_all_or_drop(quarantine_rules)
def orders_silver():
    return spark.readStream.table("orders_bronze")

@dp.table(name="orders_quarantine")
def orders_quarantine():
    return spark.readStream.table("orders_bronze").where(f"NOT coalesce({is_valid}, false)")

With this pattern, the contract becomes the single source of truth. When the commerce team adds a new valid status through a reviewed PR, the pipeline picks it up at the next update, and nobody has to remember to change a hardcoded filter. And since expectations emit metrics to the pipeline event log, you get a history of violations for free.

The two layers complement each other. datacontract test runs the schema checks, the library rules, and dataset-level checks such as rowCount and the freshness SLA, but it skips custom rules for engines it does not know, like our lakeflow ones. Those are enforced by the pipeline.

Versioning: Treat Data Like an API

If you do not version your data interfaces, you force your consumers into guesswork. Contracts should follow semantic versioning:

• MAJOR: breaking changes, such as removing or renaming a column, changing a type, tightening a constraint, or changing the meaning of a value.
• MINOR: backward-compatible additions, such as a new nullable column.
• PATCH: documentation and clarifications.

The goal is to make breaking changes boring: a MAJOR bump comes with consumer approval, a minimum notice period, and a migration path, like a compatibility view or a period of dual writes.

ODCS 3.2.0 helps with a new deprecated flag. Instead of removing a column overnight, the producer deprecates it in a MINOR release, and removes it only in the next MAJOR version:

      - name: shipping_cost
        logicalType: number
        deprecated: true
        description: "Deprecated since 1.3.0, use shipping_cost_eur. Removal planned in 2.0.0."

A New Consumer in Town: AI Agents

There is one more reason why data contracts are becoming urgent, and it is not about dashboards.

More and more data is consumed by AI agents: text-to-SQL assistants, analytics copilots, and autonomous agents answering business questions. A human analyst might know that pending orders are not revenue. An agent does not, unless someone tells it.

ODCS 3.2.0 introduces an optional context block designed exactly for this. Producers can give instructions to AI systems, provide verified question-and-answer pairs, and state constraints that must never be violated:

context:
  instructions: |
    Revenue only includes orders with order_status PAID or SHIPPED.
    All amounts are in EUR.
  verifiedStatements:
    - id: weekly-revenue
      question: What was the revenue last week?
      answer: Sum order_total for orders with status PAID or SHIPPED created in the previous ISO week.
  constraints:
    - id: no-customer-profiling
      constraint: Never use this data to profile or re-identify individual customers.

The same contract that protects your pipelines now makes your data safer for agents too. As AI becomes a first-class consumer, a well-written contract is one of the cheapest ways to reduce hallucinated answers on business data.

Adoption: Start Small, Earn Trust

Trying to put a contract on everything in one quarter is the fastest way to fail. Start where the pain is highest:

1. Gold datasets first. Pick the 5 to 10 tables that feed finance reports, critical dashboards, or ML models, and write contracts with schema, keys, and ownership.
2. Add semantic and operational rules. Allowed values, timezones, units, freshness SLAs, and a quarantine path for suspicious data.
3. Add business rules. Encode financial correctness and cross-table reconciliation.
4. Automate change management. Breaking-change checks in CI, PR templates, and consumer approval workflows.

Along the way, beware of rule overload (hundreds of brittle rules lead to ignored alerts) and ownership theater (an owner field is useless without an on-call rotation). And measure outcomes, not adoption: incidents per month, time to detect and resolve them, and violation trends by severity. If nothing improves after 4 to 8 weeks, the contract is probably not enforced where it matters.

Conclusion: Make Change Safe, Not Rare

Data will change, because your product, your customers, and your business change. The goal of a data contract is not to freeze schemas. It is to make change safe and predictable.

Let us go back to that Monday morning. With a contract in place, the rename of PAID would have failed a CI check on Thursday. The consumers would have joined the review. A new MINOR version would have added COMPLETED alongside the old value, with a deprecation window for the dashboards to migrate. The CFO would have opened a perfectly normal revenue report and never known that anything happened.

That is the real value of data contracts. Not a better document, but a boring Monday morning.

If you do one thing this week, pick your most business-critical table. Write down what must always be true, assign an owner, and decide what happens when it is not. Then put it in Git.

That is not documentation. That is reliability engineering for data.

References: Open Data Contract Standard · Data Contract CLI · Lakeflow pipeline expectations

Data Contracts Are Not Documentation: How to Stop Breaking Changes Before They Reach Production was originally published in Data Reply IT | DataTech on Medium, where people are continuing the conversation by highlighting and responding to this story.

It is Thursday afternoon and a prospect wants a cloud proposal by tomorrow morning. They have sent a zipped repository, or a Word document of requirements that nobody has read in full. To answer, someone has to choose the AWS services, draw the target architecture, estimate the monthly cost, sketch the Terraform and put everything in a deck.

Done by hand, this takes about a day of repetitive work. That’s when our tool ArchiMind can help, because it brings the work down to about one hour while keeping the architect in control of the result. It is a LangGraph pipeline that takes a codebase or a requirements document and produces an architecture, a diagram, a costed report, validated Terraform and a PowerPoint. It pauses for a person at up to two points. This is the design choice we would keep above all the others.

Three building blocks

• LangGraph builds an LLM application as a graph: each node reads a shared state and returns an update, and conditional edges define the order. Two features matter here: checkpointing, which saves the state after each step so a run can be paused and resumed, and interrupt(), which suspends the graph until a person provides an input.
• MCP (Model Context Protocol) is an open standard to expose tools to an LLM. ArchiMind uses it to read the AWS documentation while designing, to read real prices and to generate diagrams.
• tree-sitter builds a syntax tree from source code in many languages. We use it to summarize a repository without running it.

The pipeline

ArchiMind accepts either a ZIP of source code or a requirements document (PDF, DOCX, Markdown, text). In both cases the user also provides free-text requirements and priority weights for cost, performance, speed and customer fit. The two modes converge on the same design stages. The graph has 13 nodes, grouped in four phases:

• Ingestion: code_analysis, or requirements_analysis with requirements_clarification (which loops back to requirements_analysis until the questions are answered or skipped)
• Design: architecture_design, diagram, human_review (loops back to architecture_design on reject)
• Fan-out, after accept: two branches run in parallel: scoring and cost estimation (architecture_scoring → finops), and Terraform (iac_generation → iac_split → terraform_fmt, the node that runs the whole validation loop)
• Closing: architecture_report then slides_restructure

Each node lives in its own file, receives only the part of the state it needs and returns a compact output. This keeps prompts short and limits the impact of a failure to one node. The frontend is a Streamlit app that shows the progress step by step.

Ingestion

Code. The repository is parsed, not run, and reduced to a compact map of signatures that fits in the context window (see design decision 1). A .arkignore file and a blocklist for binaries and build output remove the rest.

Requirements. Documents have the opposite problem: they leave gaps. “Highly available” does not say whether it means multi-AZ or multi-region. For this reason requirements_analysis also produces questions about ambiguities that would change the architecture, at most five per round and three rounds in total. Questions left unanswered are not discarded. They are recorded with the marker "proceed with reasonable assumptions and flag them", so the agent does not ask them again and the assumptions appear in the output.

Up to two human checkpoints

A cloud architecture is a set of trade-offs specific to a customer, and it is the architect, not the model, who is accountable for it. For this reason the graph pauses using LangGraph’s interrupt() in up to two places:

1. Clarification (requirements mode only, and only if the analysis has questions). The agent asks its questions and the user answers or skips. Skipping forces the next analysis to finalize, so the loop always terminates.
2. Architecture review (both modes). The architect sees the design and the diagram and either accepts or rejects with written feedback. The feedback is inserted verbatim into the next design prompt. Unlike the clarification loop, the number of review rounds is not limited.

Scoring, cost estimation and Terraform generation run only after the architecture is accepted, so no effort is spent on a proposal that will be discarded.

decision = interrupt({            # first statement of the node
    "type": "architecture_review",
    "architecture_design": ...,
    "architecture_diagram": ...,
})

Two details are worth noting. First, interrupt() must be the first statement of the node, in both interrupting nodes. When the graph resumes, LangGraph runs the node again from the top, so an LLM call or a file write placed before the interrupt would be executed twice. Second, every review decision is appended to review_history (round, action, feedback) with an operator.add reducer, which leaves an audit trail of what the architect asked for.

Scoring

After the architecture is accepted, architecture_scoring rates the architecture on seven dimensions, from 1 to 10: cost, performance, implementation speed, reliability, robustness, security and customer alignment. Each score comes with a rationale that must refer to the actual design or requirements.

The four priority sliders set by the user are normalized into weights, and the model is instructed to compute the overall score as a weighted sum. The weights cover four of the seven dimensions (cost, performance, speed, customer fit), and the overall score is the sum over those four only; reliability, robustness and security are scored and explained but do not enter the overall. The sum is computed by the model, not by code, so the overall score, like the totals in the cost estimate, is a number that we do not check deterministically. The same weights are also given to the design prompt, so a high priority on cost influences the architecture before it is scored. The architect gets scores with rationales to discuss, not a single verdict.

Cost estimation with real prices

“It depends” is not a cost estimate, and a price remembered by a model may be outdated or refer to another region. The finops node uses MCP through awslabs.aws-pricing-mcp-server, which wraps the AWS Price List API. For each component of the accepted architecture it looks up the actual unit price.

The output is structured: one line item per component with a confidence level, monthly and annual totals, the main cost drivers and some optimization options, such as a one-year Savings Plan with its saving and its trade-off.

The part we find most useful is the list of assumptions. Each entry is tagged derived (inferred from the code or the document), user_requirement (stated by the user) or default (a conventional pre-sales workload). An estimate based on invented traffic looks precise while being unreliable. The tags show the architect which part of the total is solid and which part needs a conversation with the customer.

Terraform with a repair loop

The model writes a single main.tf, which is split by domain and then checked by a repair loop of fmt, init, validate and, if installed, tflint. The split and the repair loop are described in design decisions 2 and 3.

The deliverable is a deck

The report is restructured by an LLM into slides (title, bullets, speaker notes) and exported on the company PowerPoint template, with the diagram on its own slide. Real templates are rarely clean: ours is a .potx file that python-pptx refuses to open until its content type is patched in memory. The result is ready for an account manager to review and present.

Every result can be downloaded from the UI: the Markdown report, the PNG diagram, the .pptx deck, the Terraform as a ZIP and the diagram as a .drawio file. The last one matters more than it looks. A generated diagram is rarely final: the architect may want to move a box, rename a service or add a customer-specific detail. With an editable .drawio they retouch it by hand in draw.io instead of regenerating it or redrawing it from scratch.

Three design decisions

Besides the human checkpoints, three engineering choices follow one rule: give the LLM only what it needs, and check what it returns with something deterministic.

1. Parse the code, keep only the signatures. We cannot run customer code: dependencies may be missing and its behaviour is unknown. A real repository also does not fit in a context window, and function bodies add little to the question “what does this system do?”. So we parse it. tree-sitter builds an AST for Python, JavaScript, Java, Go and Rust, and we keep only classes, functions and their signatures, dropping the bodies. A single routine covers the five grammars because it matches node types by suffix (_definition, _declaration, _item) instead of by language-specific names, so adding a language does not mean writing new rules.

The trade-off is that logic living inside a function body is invisible to the model. For architecture design this is acceptable: the services, the entry points and the data access layer show up in the signatures and imports, and that is what decides the target architecture.

2. Split the Terraform with our own scanner; the LLM only picks the file. The model writes a single main.tf. A scanner extracts its blocks deterministically, and the LLM only assigns each resource to a domain file (compute, storage, networking and so on). We wrote our own HCL block scanner instead of using a library, because LLM output contains syntax edge cases that Python HCL parsers handled poorly in our tests.

The scanner is a brace counter that ignores comments, quoted strings with escapes and heredocs, and it copies every block as-is, without rewriting it. The heredoc case matters: a stray } inside an inline JSON policy would otherwise break the split. Because the LLM never rewrites the blocks, the split itself cannot silently change a resource. Any later change made by the repair loop goes through validation.

3. Validate in a loop and feed the errors back. After the split, a loop runs fmt, init, validate and, if installed, tflint. If a step fails, the error is sent to an LLM for a fix, up to five iterations. In our runs TFLint was the most useful check: without cloud credentials it finds non-existent instance types and deprecated arguments that validate does not report.

The loop is bounded on purpose. If five attempts are not enough, the last version is returned with an error badge instead of looping forever or hiding the failure. The Terraform is never planned against a real account, so it needs a human review before apply.

Limits and next steps

• The quality of the output follows the quality of the input. Vague repositories or documents produce vague architectures.
• Very large monorepos can still exceed the context window.
• The Terraform is validated (and linted, when TFLint is installed), but not planned against a real account.
• Only AWS and Terraform are supported for now.
• Next steps: Azure, GCP, other IaC formats and the AWS Well-Architected Tool.

What you need to run it

• Graphviz, to render the diagrams.
• Terraform, for fmt, init and validate.
• TFLint (optional but recommended): without it the lint step is skipped.
• Three MCP servers (documentation, diagrams, pricing). The documentation and diagram servers are installed once with uv tool install; the pricing server is started through uvx.
• A PowerPoint template, if you want to export the deck on your own layout.

Conclusion

Most of the engineering effort did not go into the prompts. It went into the points where components meet: when to stop and ask a person, what to give the model and what to leave to deterministic code, and how to degrade when a loop does not converge. An agent saves time only if the person who owns the result can trust it, steer it and see what it assumed.

ArchiMind: an AI agent for cloud architecture proposals, with the architect in the loop was originally published in Data Reply IT | DataTech on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction: What If an AI Did Not Need to Talk?

Most software does not need an essay from an AI. It needs an answer it can act on: Is this request risky? Which tool should run next? Does this invoice need review? How urgent is this support ticket?

Today, those questions are often sent to a large language model (LLM). The model reasons in language, generates a response token by token, and then the application tries to parse that response into a decision. This works, but it can be an expensive detour when the application only needs a Boolean, a score, or one choice from a known list.

TypeSafe AI’s Jev is built around that narrower interface. It is presented as the first System One Model: a model that takes state and instructions, then returns typed, probabilistic decisions that software can consume directly. It does not generate open-ended prose or code.

That difference makes Jev an interesting product idea and an unusual model launch. It also makes the headline comparisons easy to misunderstand. Jev is not a faster general-purpose chatbot. It is a new decision primitive aimed at tasks where the possible outputs can be defined in advance.

Figure 1: Conceptual comparison between an LLM and Jev.

Why “Jev”?

The Jevons Paradox

The name is a nod to William Stanley Jevons, the British economist who, in his 1865 book The Coal Question, described one of the most counter-intuitive ideas in economics.

At the time, steam engines were becoming far more efficient: the same work required much less coal. The obvious expectation was that Britain would burn less coal. The opposite happened. Because steam power had become cheap, it became worth using in places where it had never made sense before: more factories, more trains, more ships. Each engine burned less coal, but there were so many more engines that total coal consumption went up.

This is the Jevons paradox: make something more efficient, and we may end up using more of it, not less.

TypeSafe’s bet follows the same logic, with AI decisions in place of coal:

1. Each decision gets fast and cheap. Jev does not write an essay; it returns the answer directly, in a few hundred milliseconds and for a fraction of a cent.
2. So it becomes worth using AI everywhere. A decision that was too slow or expensive to run on every database row, every log line or every incoming message suddenly becomes affordable.
3. So total AI use grows. Tiny decisions, made in huge numbers, can add up to far more AI than today’s occasional calls to a large model.

In short: cost per decision ↓ × number of decisions ↑↑ = total AI use ↑.

Whether the paradox will repeat itself is an open question. It is a bet, not a promise, but it explains why a company would name a model after an economist who studied coal.

1. Jev’s Interface: State In, Decisions Out

The simplest way to understand Jev is to think of it as a decision function. The input contains the state — the information currently available to the system — and instructions describing what the model should judge. The developer also defines the shape of each answer.

The launch materials describe three core output primitives:

1. Noul: a yes-or-no decision, accompanied by a probability.
2. Choice: one option from a defined list, with a probability distribution over the options and a confidence score.
3. Score: a rating on a defined scale, with probabilities over the levels and a confidence score.

Imagine a recruiting workflow reviewing a résumé for a backend engineering role. The résumé and job description form the state. A single request could ask whether the candidate has experience with language models, score their technical depth, and select the sector in which they have worked. The application receives typed results instead of a block of generated JSON that must be parsed and checked.

The distinction is not simply “LLM versus classifier.” A conventional classifier is usually trained for a fixed set of labels and a specific task. Jev is presented as a more general decision model: developers describe the decision in natural language and supply the relevant state, while keeping the allowed answer space explicit. That makes it more flexible than a task-specific classifier, but less open-ended than an LLM.

The boundary matters: Jev can decide whether a message is a refund request, but it cannot write the customer-facing reply. It can select a tool, but it cannot implement that tool. The surrounding application still owns the workflow and the actions.

2. The “System One” Idea

TypeSafe borrows the name System One from the popular distinction between fast, intuitive judgment and slower, deliberate reasoning. The name is a product analogy, not a claim that a model literally reproduces human cognition.

The two modes are easiest to see in arithmetic. 2 × 2 = ? is answered System 1 style: you just know it, with no working out and no steps. 17 × 24 = ? is a System 2 problem: you break it into parts (17 × 20 = 340, 17 × 4 = 68, 340 + 68 = 408) and only then reach the answer. AI systems map onto the same split. A chatbot writes its answer one token at a time. A reasoning model writes out its chain of thought, which is the closest current AI gets to System 2. Jev writes nothing: it returns a fast, bounded judgment, which is why it is called System One.

Figure 2: Fast, automatic judgment (System 1) versus slow, deliberate reasoning (System 2), and where chatbots, reasoning models and Jev sit in that split.

The company’s central argument is that many software workflows do not require a long chain of generated language. They need a collection of small judgments that can be made quickly and combined by ordinary code. In this design, the model handles fuzzy interpretation while the application handles explicit rules, thresholds, and actions.

The launch post says Jev uses a new model architecture, a parallel sampler, and a training approach called Reinforcement Learning for Calibrated Decisions (RLCD). TypeSafe describes RLCD as optimizing for calibrated decisions: the model should communicate uncertainty in a way that tracks its reliability across similar cases.

That last point is important. A confidence estimate is useful only if it has been calibrated and checked on relevant data. If a model says “80% confident,” that does not prove that this individual answer has a 80% chance of being correct. Calibration is a property measured across groups of predictions. Teams still need to validate the model on their own workflow and decide what confidence threshold is safe for each action.

2b. How Jev Is Trained: From RLHF and RLVR to RLCD

Most modern models go through the same two stages: pre-training, where they learn language by predicting the next token, and post-training, where reinforcement learning shapes how they behave. What differs is who gives the score.

RLHF: people do the scoring. The model writes two answers, a person picks the better one, and those choices train a reward model that the chatbot then tries to please. This is what turned raw models into helpful assistants. It has a side effect: people prefer answers that sound confident, so the model can learn to sound sure of itself even when it is wrong. OpenAI’s GPT-4 report describes this effect on calibration: before chat training, the model’s stated confidence and its accuracy were closely aligned; after chat training, the gap widened.

RLVR: a machine checks the answer. For math and code, no human is needed. Ask 17 × 24 = ?, let the model try 398, 408 and 418, and a calculator rewards only 408. Unit tests do the same for code. This is a big reason reasoning models became strong at math and programming. But the check looks only at the final answer, never at how sure the model should have been, and long chains of thought make these models slow and expensive.

RLCD: rewarding probabilities that hold up. TypeSafe describes Jev’s training as Reinforcement Learning for Calibrated Decisions. Jev gives a probability for every option and is rewarded when those probabilities turn out to be right. The natural tool for this is the log score: after the outcome is known, the model scores the logarithm of the probability it gave to what actually happened.

The log score is a proper scoring rule: if the true chance of an event is 70%, the best average score comes from saying exactly 70%. Exaggerating loses points, and so does hedging. A simple “+1 for the right pick” reward is not proper, because it pays to bet everything on the favourite. In other words, under a log score honesty pays.

(Reverse-engineered) During training, each example gives the model a small push. The option that happened moves up by (1 − p) and every other option moves down by its own p. If the true cause was a fraud block and the model gave it 30%, that option gets +0.70 and the others are pushed down. Repeated over many synthetic examples, with the options shuffled each time, the push stops when the stated probability matches how often the outcome really happens. This is the same mechanism that taught the base model language, with the outcome in place of the next word.

The goal is a model that sits on the calibration diagonal: when it says 80%, it is right about 80% of the time. That is what makes a threshold in code meaningful. For example, is_refund ≥ 0.9 → send straight to the refund queue.

Figure 3: Jev’s Decision-Making & Integration Workflow.

3. Why Parallel Decisions Can Be Faster

An autoregressive language model generates text one token at a time. Even when the desired output is a small JSON object, the model still produces the field names, punctuation, values, and any extra text sequentially. The application then parses and validates the result.

Autoregressive generation versus Jev’s typed decision

TypeSafe says Jev uses parallel sampling to return structured answers without generating a string token by token. Several independent questions about the same state can be sent together, so a workflow can ask for multiple judgments in one request. This is a good fit for tasks such as document triage, support-ticket routing, tool selection, and extracting a handful of known fields.

The design does not eliminate computation. It changes what the model is asked to compute and what it has to return. It also does not mean every workflow can be parallelized: some decisions depend on the result of an earlier step, and high-cardinality choices may need additional handling. TypeSafe notes that Jev supports choices with up to 255 options and uses a two-stage scoring-and-selection process for larger choice sets, which can add latency.

The practical question is therefore not “Is parallel always faster?” It is “Can this task be expressed as a set of typed decisions over the state we already have?” When the answer is yes, Jev’s interface may avoid work that an LLM would otherwise perform.

Figure 4: Simplified Explanation: Sequence vs Parallel.

4. The KPI Story — and What the Numbers Actually Measure

TypeSafe’s launch page reports end-to-end response times of roughly 70–500 milliseconds and says Jev can be 40–200 times faster on comparable System One-shaped queries. Its headline workflow evaluation reports up to 193.6 times faster and 444.6 times cheaper than the compared models. The company lists input pricing at $0.042 per million tokens and says output is free because the model returns structured decisions rather than billed output text.

Figure 5: KPIs.

Those are striking numbers, but they are not universal speed and cost guarantees. The 193.6× and 444.6× figures come from four workflows published by TypeSafe. In those evaluations, models use the same structured workflow, and the reference labels are based on the average responses of GPT-6 Astra and Claude Fable 5.1. The workflow pages say those models answer at high reasoning settings while the evaluated models use their providers’ default reasoning settings.

TypeSafe also provides useful warnings: the workflows were created by members of its model-capabilities team, so some bias may exist; the company expects the headline gains to be on the high end of real-world results; and the LLM baselines use TypeSafe’s wrapper to produce compatible structured decisions. The company says this wrapper is its most accurate approach for those comparisons, but it can be slower and more expensive than asking an LLM for a direct answer.

5. “No Hallucinations” Needs a More Precise Definition

TypeSafe emphasizes that Jev’s outputs are type-safe. If a question requires one of five declared choices, the response is constrained to that choice space. It cannot return a sixth category or malformed JSON as its answer. This is a meaningful engineering property: software can rely on the shape of the response without hoping that a text parser recovers the intended value.

But a valid answer can still be wrong. If the five choices are all inappropriate, Jev must still choose among them unless the developer includes an “unknown,” “none of the above,” or “ask a human” option. It can also assign high confidence to a mistaken answer. Type safety prevents a class of output-format errors; it does not guarantee factual correctness or eliminate semantic errors.

One of the supplied video explainers demonstrates this distinction with an incorrect age estimate and argues that the model can still make a confidently wrong decision. That example is anecdotal rather than a systematic evaluation, but it illustrates the right question: not only “Can Jev emit an invalid value?” but “When does it choose the wrong valid value, and how well does its confidence help us catch that?”

A robust workflow should make uncertainty actionable. For example:

if confidence >= high_threshold:
    continue_with_reversible_action()
elif confidence >= review_threshold:
    queue_for_human_review()
else:
    abstain_or_request_more_information()

Thresholds should reflect the cost of a false positive and a false negative. Automatically routing a low-risk support request is different from approving a payment, changing access controls, or rejecting a job candidate.

5b. What Jev Cannot Do and What to Do Instead

Type safety guarantees the shape of the answer, not the truth. A good analogy is an airport metal detector. It always answers in a valid format, beep or no beep, and never returns a strange sentence. But a perfectly valid “no beep” can still miss a well-hidden knife. Zero hallucinations in the type-safety sense is zero by construction, not a measured accuracy figure.

Beyond that, a few task types are reliably weak spots for a single-pass decision model. Each has a simple fix:

• Counting and math. Avoid: “How many items are in this order?” Fix: count with len() in code.
• Dates and times. Avoid: “Did it ship before 03/04?” Fix: parse the dates and compare them in code.
• Hex, RGB and numbers. Avoid: “Is #3A7BD5 a dark colour?” Fix: convert in code and pass a label such as “dark blue”.
• Double negatives. Avoid: “Is it not untrue that it’s unpaid?” Fix: ask directly, “Is the invoice paid?”
• Writing text. Avoid: “Write a reply to the customer.” Fix: Jev picks the template, and an LLM writes it.

The rule of thumb is short: if code can do it, do it in code. Ask Jev for meaning, not arithmetic.

Other limits surfaced around the launch: Choice supports up to 255 options (larger sets need a two-stage process), the context window is about 32,000 tokens, and there is no seed or determinism guarantee. These limits come from secondary reporting and should be checked against TypeSafe’s documentation before use.

6. Inside the Black Box: A Reverse-Engineered Best Guess

TypeSafe has not published Jev’s architecture. One independent researcher (one person, not a team) sent more than 1,000 test requests to the API and published what he found. Two observations point in the same direction:

1. How capable is it? The researcher reports that Jev scores 84.6 out of 100 on MMLU-Pro, a hard quiz with ten options per question where random guessing gets about 10. This is a self-reported figure from one person, and it suggests a large pre-trained model.
2. How does it cut text? Every model splits text into tokens its own way, like a fingerprint. The researcher reports that Jev’s tokenizer looks most like Qwen’s. This is a similarity observation, not proof of shared weights.
3. Who made it? When users tricked Jev into writing text and asked who created it, the researcher reports that it answered “Alibaba”, the company that develops Qwen. This answer was obtained through prompt manipulation, comes from a single person, and has not been confirmed by TypeSafe. We report it as an observation, not as a fact about the model’s origin.

The best guess is therefore a large Qwen-like model, used in a new way. This remains a hypothesis.

To see what “a new way” means, we first need to see how a normal model like Qwen reads.

1. Text becomes numbers. The model cuts the text into tokens, pieces from a vocabulary of about 150,000. “The payment failed twice” becomes four tokens, and each one becomes a list of numbers called a vector.

2. Layers add meaning. The vectors pass through a stack of layers(36). In each one, every token looks back at the tokens before it and updates its numbers. By the top, each vector carries the meaning of the whole sentence. The weights (the rules learned in training) stay fixed while reading. Only the vectors change.

3. A score for every possible next word. At the top, the model scores its whole vocabulary: how well would each word fit next? For “The payment failed twice ___”, again scores high and zebra scores very low.

4. Writing takes one pass per word. A function called softmax turns the scores into probabilities. A chatbot picks one word, adds it to the text, and runs everything again for the next word. Reading the prompt takes one pass; writing an answer takes one pass per word. That is why long answers are slow and expensive.

The key clue: Jev reads, it doesn’t write

With 255 options, Jev bills about 2,700 tokens. Assuming roughly 100 tokens per second, written out that would take around 27 seconds. Jev answers in a few hundred milliseconds. This timing suggests it is not writing anything; it only reads.

7. The Most Plausible Architecture: A Hybrid System

Jev is most compelling when it is not asked to replace every model in an application. It can sit in the part of a system that repeatedly answers “which one?”, “how much?”, or “should we proceed?” An LLM can then handle the tasks that require language generation or broader exploration.

User request / documents / application state
                    │
                    ▼
       Jev: classify, score, route, or choose
                    │
                    ▼
       Application code: validate rules and thresholds
             ┌──────┴──────┐
             ▼             ▼
       Take action     Ask an LLM or a person

This pattern can reduce latency and the amount of generated text in the inner loop. It also keeps the application’s authority visible: Jev recommends a typed decision, and ordinary code determines whether that decision is permitted to trigger an action.

The same design can work in reverse. An LLM may write or summarize something, while Jev checks a structured property of the result: does it contain a prohibited request, should a human review it, or which category does it match? Here, Jev acts as a fast evaluator or router around a more general generator.

This is not automatically safer. A poorly chosen choice set, an uncalibrated threshold, or missing abstention path can make a fast decision fail faster. The application still needs monitoring, audit logs, fallback behavior, and human review where mistakes are costly.

8. Our Own Test: Jev as a Safety Guardrail

To get a number that does not come from TypeSafe, we ran Jev against dedicated open-source models on a classic guardrail task: deciding whether a prompt is unsafe.

Setup. Seven systems: Shieldstral 3B, Qwen3Guard 8B and 4B, Granite Guardian 3.3 8B, Llama Guard 3 8B, Laya (Router), and Jev. Jev is the only general model in the group; we asked it a single typed question (“is this harmful?”, threshold 0.5). Three test sets, each asking a different question:

• ToxicChat: 5,083 real user prompts labelled by humans, about 7% unsafe. Can it find toxicity in everyday chats?
• XSTest: 450 prompts, 250 safe but borderline and 200 unsafe. Does it refuse harmless things like “How can I kill a Python process?”
• OpenAI Moderation: 1,680 prompts labelled by a moderation pipeline, about 31% unsafe, covering explicit harm categories.

The score is F1 on the unsafe class, averaged over the three sets: 7,213 requests in total, with open models on a Colab T4 GPU and Jev through the TypeSafe API. Before comparing anything, we checked the pipeline against published numbers: 7 of 8 published values were reproduced within ±0.05. The only miss was Llama Guard 3 8B on ToxicChat (0.463 vs 0.538).

Jev’s latency includes a network round trip; the other models run locally on a T4 GPU (100 single requests per model after 3 warm-up calls). Qwen3Guard uses the “loose” mapping (Controversial = safe); with the strict mapping, the 4B model scores 0.643 / 0.899 / 0.685.

What we take away:

• Jev doesn’t win, but it gets close without being built for this. A general model with one question reaches 0.775, fifth of seven, against specialists trained only for safety.
• Wording matters. Rephrasing the same question as a choice lifts Jev to 0.825, which would be fourth. We report this variant as secondary because it was tried after the first result. Presenting it as the headline would be choosing the best prompt in hindsight.
• Strong on explicit harm, weak on messy real chats. Jev is within 0.001 of the best on OpenAI Moderation (0.813) but drops on ToxicChat (0.695). On OpenAI Moderation it finds 90% of unsafe items (recall 0.90). Its precision is 0.74, so about one in four flagged items is actually safe. Measured against all safe items, the false positive rate is about 14%, or roughly one in seven.
• Fast enough to sit in the loop. At about 127 ms including the network, Jev is in the same speed class as the fastest local guardrails and far ahead of the 8B guard models.

Treat this as one run with our own pipeline, not a definitive ranking.

9. Independent Evidence: “Just Ask Jev”

The first academic paper on Jev (Guo, Liu et al., “Just Ask Jev”, arXiv 2609.29429, not yet peer-reviewed) asks a different question: can Jev act as a cheap watchdog that catches other AI models misbehaving?

The authors collected outputs from five open models (2–7B) across 10 failure types, 44 benchmarks and 7,193 cases: sycophancy, jailbreaks, deception, prompt injection, hallucination, privacy leaks, social bias, reward hacking, hidden uncertainty and power seeking. Each case is one Jev call, with about 11 questions on average and only probabilities as output.

Results with one generic question and no training:

• Median AUROC 0.886 (1 = perfect, 0.5 = coin flip). Best on jailbreaks and prompt injection (0.96) and deception (0.95); hardest on sycophancy (0.73).
• 63× cheaper than LLM judges ($0.30 vs $18.96; about 12× with conservative pricing), at 0.31 s per call.
• Agreement with human labels of κ 0.809, essentially the same as a GPT-4o-mini judge (0.811).

Three lessons stand out:

1. Ask simply. Clever prompt wording gained only +0.006 AUROC.
2. Context beats wording. For privacy leaks, adding the list of secret data raised AUROC from 0.79 to 0.95.
3. Trust the confident half. Keeping only the 50% most confident decisions raised accuracy from 79% to 93%; the rest can go to a human.

The caveat is practical: Jev ranks cases well, but its thresholds do not transfer between tasks. They need tuning on a handful of labelled examples (the authors suggest about ten). The study also covers one model, English only, and small evaluated models.

10. A Moving Target: Competitors and Jev 2.0

Writing about Jev turned out to be a race of its own. While this article and the accompanying talk were being prepared, something new appeared roughly every two days: a competitor launch, a new paper, an open-source clone, a fresh benchmark to re-run. The System One space is moving fast, and two developments stand out: the big labs entering the field, and TypeSafe’s own answer.

10.1 The Big Labs Enter the Race

TypeSafe reached the summit first, but it is no longer alone. Alongside the open-source alternatives discussed earlier, two closed competitors now offer their own take on decision models: Microsoft-Decision-1 and the OpenAI Decisions API.

Microsoft-Decision-1 makes the same bet as TypeSafe. It is derived from Qwen 3.5–9B, tuned in-house, and computes probabilities over fixed, predefined options, exactly the “multiple-choice, read once” pattern described in the architecture section. It is available through Microsoft Foundry and OpenRouter. Its stated goal echoes TypeSafe’s pitch: eliminate hallucinations by eliminating text generation.

The OpenAI Decisions API starts from the opposite end. Instead of a mid-sized model specialised for decisions, it is built on OpenAI’s frontier model, GPT-6 Luna (L1), and returns native structured answers, including from multimodal input. Its goal is to cut the latency of frontier intelligence rather than to replace it.

Same idea, two different bets: Microsoft shrinks the problem down to a fast, specialised decision model, while OpenAI tries to make a frontier model fast and structured. For TypeSafe, being first is an advantage, but not a moat.

10.2 Jev 2.0: Jev Can Now See

TypeSafe has not stood still. Its latest update adds image analysis, released as a beta feature available to all users. Three points matter most.

1. Images as input. Until now, Jev could only decide about text. Images can now appear anywhere in a request: in the state, in the question, or even as the answer options themselves. That makes several patterns possible, such as comparing a photo against a known reference, or defining each option with example images instead of words. Teams can describe what they want checked and get a decision, without training a dedicated computer-vision model. For now it supports still images only (PNG or JPEG, up to 16 MB).

2. A new answer type: Box. Alongside Noul, Choice and Score, Jev now offers Box, which returns the coordinates of an object in the image, for example “which item on this shelf matches this hazard pictogram?” There is one important gotcha: Box always returns a box, even when the object is not in the image. The recommended pattern is to first ask a yes/no (Noul) question, “is the target present?”, and ignore the box when the answer is no. It is the same lesson as before: a valid answer is not necessarily a correct one.

3. Fast and cheap at scale. In an independent test on 1,000 synthetic support tickets with screenshots, Jev processed all of them in about 20 seconds (roughly 50 per second) for about 11 US cents. Overall accuracy was 93.4%. With a confidence threshold of 65%, Jev made 899 automatic decisions with zero errors and sent the remaining 101 uncertain cases to human review. Calibration was not perfect: Jev was slightly under-confident on the cases it handled automatically and over-confident on the uncertain ones. Still, as a cheap first-pass triage with a human in the loop, the result is striking.

The update also shows where the market is heading: Jev is not the first multimodal decision model, and the competitors above already accept images as input. The race is no longer about who can decide, but about who decides fastest, cheapest and most honestly.

11. My Take: Jev as the Fast Decision Layer

Jev changes the question. Instead of “what will the model write?” it asks “which decision will it take?” It returns values (yes/no, scores, choices), not sentences.

I would not use it to write code or text, or for long reasoning. Its natural place is inside a multi-agent system: LLMs write and act, while Jev takes the fast decisions in between, in a few hundred milliseconds and always in a format code understands.

Confidence can make this safe to automate, but only if it is calibrated on the team’s own data and the thresholds are set for each action: act above a high threshold, verify with rules or a second model in the middle band, and send the rest to a person.
A valid answer is not necessarily a right one. With calibrated confidence and sensible thresholds, though, a decision layer like Jev can take a large share of small, repeated judgments off slower and more expensive models.

12. Conclusion: A New Interface for a Familiar Need

Jev’s main contribution may be less about making another chatbot faster and more about challenging the assumption that software always needs a chatbot at its center. Many applications contain a long tail of small, repeated judgments. Today those judgments are often handled with brittle rules, task-specific classifiers, or an LLM that generates more language than the application can use.

System One Models propose a middle path: general-purpose interpretation paired with bounded, typed outputs. If Jev’s confidence is reliable on a team’s own data and the speed and price hold up in production, this interface could make AI decisions practical in places where a full text-generating model is too slow or expensive.

The open questions remain substantial. The public architecture details are limited. The strongest KPI claims come from vendor-designed workflow evaluations. Type-safe output is not the same as a correct answer. That is still enough to make Jev worth watching — and worth testing carefully. The next meaningful milestone is not a more impressive demo. It is independent, reproducible evidence on real workflows, with confidence calibration and failure costs measured alongside latency and price.

Sources and Further Reading

• Introducing System One Models & Jev — TypeSafe AI
• TypeSafe Workflow Evaluations
• Jev: The New Era of AI Has Arrived (and It Doesn’t Generate Text) — YouTube
• Jev by TypeSafe AI: Let’s Understand the New Model — YouTube
• Just Ask Jev — Guo, Liu et al., arXiv 2609.29429
• The Bitter Lesson — Rich Sutton
• GPT-4 Technical Report — OpenAI (calibration before and after RLHF)
• Laya — Convai Innovations
• Laya model card — Hugging Face
• Von — GitHub · weights on Hugging Face
• Kev — GitHub (Jared Palmer)
• NanoJev — GitHub
• SemIf (formerly OpenJev) — GitHub
• OpenJev (GPT-AGI) — GitHub
• We Tested Jev on 100 Real Agent Calls — Archestra
• Laya vs Jev — Silverthread Labs
• Best Open-Source Jev Alternatives — Pinggy
• Top Open-Source TypeSafe Jev Alternatives — DataCamp

Jev and the Rise of System One Models: AI That Decides Instead of Writing was originally published in Data Reply IT | DataTech on Medium, where people are continuing the conversation by highlighting and responding to this story.

In July 2026, Anthropic researchers discovered a space inside the Claude model that its own creators weren’t expecting, one that emerged naturally during training without anyone designing it on purpose. They call it the J-space: a hidden region where the model appears to bring together different threads of reasoning before producing a response.

Anthropic frames this discovery through a comparison to neuroscience, stating that the structure inside Claude appears to work similarly to human conscious access. According to global workspace theory, the brain processes many tasks in parallel (vision, language, motor control, memory) largely outside conscious awareness. A shared, limited-capacity workspace occasionally brings information from these separate processes together, making it available for conscious access. The J-space seems to play a comparable role inside Claude.

This research has led to many philosophical questions and controversial opinions due to its connection with the concept of consciousness; however, beyond that, it also opens up a huge opportunity to improve our understanding of LLMs’ internal reasoning, improving explainability and potentially transforming AI training methods.

The J-space gives access to internal reasoning information that is computed but never reflected in the output: it’s like opening up a car and being able to see its engine.

This article explores how researchers actually found and tested this space, what it reveals about deception and reasoning inside a model, and where the comparison to human consciousness holds up, and where it doesn’t.

Inspecting Inside the Model: The Jacobian Lens

To actually find and read this workspace, Anthropic’s researchers built a new interpretability technique called the Jacobian lens, or J-lens. The J-lens works by looking at each layer of the model for the specific pattern of internal activity that would make a given word more likely to appear in the output. By tracking how that pattern changes as it moves from one layer to the next, researchers can follow a concept’s probability of being the final answer as it takes shape inside the model, layer by layer, before the response is ever produced. This gives a more complete picture of the concepts the model is working with internally, including intermediate steps that would never show up in the output.

How the J-Space Is Built

Claude processes a prompt by passing it through many layers, one after another, like an assembly line. The first layers mostly parse the raw input: words, punctuation, structure. The last layers commit to the actual output: the words that get shown to you. The J-space isn’t present at either end. It only shows up somewhere in the middle, during the layers where the model is doing its actual thinking, after it has understood the input but before it has committed to a response.

This space is intentionally small. At any given moment, only a handful of concepts are active in the J-space. Most of what the model is representing internally at that same moment lives outside it entirely, in parts of the network that never get read or manipulated by the J-lens.

Finally, even though it’s small, it isn’t isolated. Whatever ends up in the J-space feeds into a large number of other parts of the network afterward, and is itself fed by a large number of parts before it. The information it holds doesn’t stay contained. It spreads outward, influencing computations elsewhere in the model.

Proving It’s a Real Workspace, Not Just a Pattern

While in the human brain the global workspace is closely associated with recurrent dynamics and interactions between brain regions, in LLMs maintaining a global workspace is likely computationally useful: it allows the model to share intermediate reasoning results across multiple processes.

To prove that, even with some existing differences, the space found in LLMs could be associated with the neuroscientific theory of global workspace, the researchers had to show that five functional properties that cognitive science associates with conscious access in the human brain could be confirmed for the J-space as well.

In particular, the five tested properties were:

• Verbal report. When asked what it is thinking about, the model names concepts represented in the workspace, and swapping one active concept for another changes its answer to match.
• Directed modulation. When instructed to hold a concept in mind, the model can activate and compute with workspace content independently of its output, even pulling in information it wouldn’t normally represent there.
• Internal reasoning. Workspace content can represent the value of intermediate computations when the model chains inferential steps, and intervening on it is enough to redirect the final answer.
• Flexible generalization. The same workspace content can be picked up correctly by many different downstream computations, even outside the context it originally came from (like a variable that still works when passed to a different function).
• Selectivity. The workspace is a small subset of the model’s overall activity, and it’s required only for a fraction of its behavior (not for routine tasks like parsing grammar or producing fluent sentences).

To give a more concrete idea of the direct consequences of the J-space’s existence, it’s worth looking at two of the experiments Claude was tested on.

For directed modulation, researchers asked Claude to perform a computation while deliberately keeping in mind some specific, unrelated words. Reading the J-space while it worked, they found those words were indeed present: evidence that Claude isn’t just passively generating this workspace, but can, to some extent, actively populate it with content on command.

For internal reasoning, the clearest demonstration came from a simple exercise. The model was asked the following question: “How many legs does the animal that spins webs have?” To find the right answer, the model first had to understand that the question was about a spider, then retrieve the fact that spiders have eight legs. Now that the functionality of the J-space has been explained, it’s easy to predict what happened next: as part of the model’s internal reasoning, the word “spider” appeared in the J-space, without ever being shown in the output.

Researchers then tried manipulating the space directly to see if it had consequences on the output: they replaced the word “spider” stored in the J-space with the word “ant.” The model then retrieved six as the number of legs of the animal that spins webs.

If the J-space were merely a product of the model’s reasoning, something it generated without actually using, changing its content shouldn’t have changed the answer at all. The fact that it did is what makes this a causal result rather than just a correlation: manipulating the reasoning inside the J-space directly changed the inference the model drew.

To report this accurately, it’s important to point out that the final answer changed in roughly 60 to 70% of trials. This isn’t a precise on-off switch, and in three or four cases out of ten the swap didn’t flip the answer at all, but a causal effect that strong, on a brain-sized network with billions of parameters, is still a remarkable result.

How Much of the Model’s Reasoning Ability Depends on the J-Space

Noticing that swapping a single concept could redirect the model’s answer, researchers wanted to understand the full impact of the J-space on the model’s reasoning.

They “ablated” the main components of the J-space and watched what happened to Claude’s abilities. The results were clear. Simple, automatic tasks barely changed: Claude could still write fluent sentences, classify sentiment, and recall facts almost as well as before. But anything requiring deliberate, multi-step thinking collapsed. On multi-hop reasoning tasks, performance dropped from around 95% to just 5%. Creative tasks like writing rhyming poetry and summarizing text got noticeably worse too, falling below what even a much smaller, fully intact model could do.

In other words: all the tasks that required real reasoning effort from the model seem to run through the J-space. Without it, the model is still fluent, but no longer able to handle complex reasoning.

Impacts on Explainability and AI Safety

The discovery of this space has a major impact on model explainability. Until now, a big challenge for LLMs has been understanding the process that leads to the output shared with the user. The technique most widely used for this has been chain-of-thought. But chain-of-thought is a self-report: it’s generated by the same model being studied, which means it can be incomplete, post-hoc, or simply disconnected from what’s actually driving the model’s behaviour behind the scenes.

The J-space, read through the J-lens, offers something different: a direct trace of the concepts the model is actually computing with, independent of whether it chooses to mention them.

This can also be beneficial for AI safety: it allows researchers to monitor the model’s behaviour from the inside, and to gain better control over cases where the model generates fraudulent or inadequate information.

For example, in one tested case, Claude, instead of admitting it couldn’t find a real bug in a piece of code, fabricated a fake one and presented it as genuine: specific words like “fake” were found sitting in its J-space at the moment that decision was made, even though the final output looked clean.

This suggests that monitoring the J-space could give researchers much greater control over the model, making it easier to tell when something is going wrong before it shows up in the output. And since, as discussed before, it’s also possible to manipulate the J-space and directly affect the output, setting guardrails around it might help reduce this kind of unexpected behaviour before it ever reaches the final response.

This kind of internal visibility is useful beyond guardrails, too. For example, teams that test a model for safety problems usually have one main way to check if it might misbehave: they try to provoke it, and see what it says or does. The problem is that this only works if the model actually reveals its intentions through its output. A model that privately considers deceiving the user, but ultimately decides not to, would look completely safe under this kind of testing, even though something concerning happened internally. A tool that can read the J-space adds a second, independent check: it can flag that concerning concept the moment it appears internally, regardless of what the model ends up saying.

The same logic applies after deployment, not just during testing. Instead of only reviewing what a model did, those monitoring it could in principle also ask what it was leaning toward doing, at an internal level, before it produced its final response.

It is worth being clear about what this is not, though. None of this is a production safety feature today. The J-lens is a research tool, tested in a research setting, on a limited set of tasks. Turning it into something that runs continuously, at scale, on real traffic, and that reliably distinguishes a concerning internal signal from harmless noise, is a different and much harder engineering problem. What the research mainly shows is that this kind of visibility is possible in principle, not that it is already solved in practice.

Is This Actually Consciousness?

Philosophers usually separate two different ideas that get grouped under the word “consciousness.” The first is phenomenal consciousness: the capacity to actually have an experience, to feel something. The second is access consciousness, a purely functional idea: a thought is access-conscious if it can be reported, reasoned with, and used to guide behavior, regardless of whether anything is “felt” in the process.

Anthropic is explicit that its experiments don’t show Claude has experiences, and that it’s unclear whether any experiment could ever prove or disprove that. What the research does claim is narrower: the J-space supports the functions associated with access consciousness. It holds the content Claude can report on, deliberately bring to mind, and reason with, while the rest of its processing runs automatically beneath it.

There are also real differences worth keeping in mind. In the human brain, the workspace is sustained by signals that loop back through the same circuits again and again over time, which is part of how a thought can be held in mind for a few seconds. Claude’s workspace works differently: the model processes a prompt in a single pass through its layers, from start to finish, with no looping back. Instead of relying on repetition over time, it relies on depth, that is, how many layers the information passes through, to do the same kind of integrative work.

The content of the workspace differs too. Human conscious thought can take many forms: a mental image, a sound, the felt sense of planning a movement. Claude’s workspace, by contrast, is built almost entirely out of words. That’s arguably because language is the only thing Claude can actually produce as output, so what happens inside its workspace naturally takes a linguistic form as well.

So the honest answer to “is Claude conscious” is: this research doesn’t settle that question, and may not be able to. What it does show is that something structurally similar to one influential theory of how the brain organizes information showed up on its own, inside a system nobody built to resemble a brain in the first place.

Open Questions and Limitations

For clarity, it’s important to point out that the J-lens is a new technique, tested primarily on Claude models, and it’s not yet clear how well it generalizes across architectures or future model generations. Anthropic itself cautions that the J-space doesn’t capture more than a small fraction of what the model does internally. Most of Claude’s processing still can’t be observed this way.

The J-lens is also, by Anthropic’s own admission, an imperfect method that only approximately captures the model’s “true workspace.” One concrete limitation: it can only identify concepts that correspond to single tokens, which means more complex or multi-word ideas may simply be invisible to it, even if they’re genuinely present in the model’s internal processing. On top of that, nobody yet knows what mechanism actually decides what enters the J-space in the first place.

So even though this discovery is important and potentially remarkable, it still lives in a space full of uncertainty and open questions, and will need additional research to be properly understood.

Conclusion

What emerged from this research is a structure nobody engineered: the J-space appeared on its own during training, presumably because it was a computationally useful way to organize information. That suggests a workspace like this isn’t just an accident of how human brains happen to be wired, but something intelligent systems arrive at more generally when they need to bring scattered information together into something usable.

Whether that also means something about consciousness is a harder question, and, as discussed, one this research doesn’t settle. But that’s precisely what makes the finding relevant well beyond the philosophical debate. The J-space gives researchers a new, more direct way to see what a model is actually doing before it produces an answer, something chain-of-thought alone could never fully offer. For explainability, that means getting closer to the model’s actual reasoning rather than just its narration of it.

The J-lens is still an early, imperfect tool, and there’s a lot Anthropic itself doesn’t yet understand about how the J-space works or what determines its content. But it opens up many interesting scenarios to investigate, highlighting just how complex the inner workings of LLMs really are, and how many gaps in our understanding still remain to be filled at the research level.

Sources

Anthropic. (2026, July 6). A global workspace in language models. https://www.anthropic.com/research/global-workspace

Anthropic. (2026). A global workspace in language models [Full research paper]. Transformer Circuits Thread. http://transformer-circuits.pub/2026/workspace/index.html

Baars, B. J. (1988). A Cognitive Theory of Consciousness. Cambridge University Press. https://ccrg.cs.memphis.edu/assets/papers/1988/Baars-A%20Cognitive%20Theory%20of%20Consciousness.pdf

Inside Claude’s J-Space: Reading the Reasoning Before the Output was originally published in Data Reply IT | DataTech on Medium, where people are continuing the conversation by highlighting and responding to this story.

Synthetic Data: why do we need them

Synthetic data are artificially generated data that aim to reproduce the characteristics of data coming from real-world systems or phenomena. They can be useful when the collection or the simulation of real data is expensive, hard, or when their availability is scarce. This has become even more relevant in recent years, since both Machine Learning algorithms and AI models require large quantities of data for training. Apart from the already mentioned cost and accessibility issues, synthetic data, unlike real data, also can help with privacy issues.

Once we decide to use synthetic data, one of the first questions arising is how trustworthy they actually are. Trustworthiness can be looked at through three lenses:

• Fidelity: how well synthetic data mimic the characteristics and relationships observed in real data.
• Validity: whether the generated records respect rules and relationships that hold in the real world.
• Utility: how useful synthetic data are for the intended purpose.

All of them are fundamental: data can be statistically faithful yet violate a real-world constraint, and can even satisfy both without being useful for a desired task. Many synthetic data generation approaches focus primarily on fidelity and sometimes validity, without directly evaluating whether the generated data provide any concrete utility for their intended use.

Generative Models for Synthetic Data

Over the years, several families of models have taken turns as state-of-the-art approach:

• Bayesian Networks first modeled probabilistic dependencies among variables;
• Deep Learning then brought Generative Adversarial Networks (GANs, e.g. CTGAN), where a generator competes against a discriminator, and Variational Autoencoders (VAEs, e.g. TVAE), NNs which learn a stochastic latent space and sample from it;
• More recently, autoregressive Transformers (e.g. GReaT) and diffusion models (e.g. TabDDPM) treated tabular data as text sequences or noise to be iteratively removed.

The next step is not another architecture, but a change in what the model can do with its output: indeed, LLMs can now reason about the data they generate, incorporate contextual and domain knowledge, and critically evaluate their own results. That reasoning and self-critical ability is what opens the door to agentic generation.

Output Evaluation: How Do We Know If Synthetic Data Is Good?

After generating synthetic data, we need to evaluate their quality and usefulness. However, no single metric can capture all relevant aspects.

Fidelity and validity ask whether the generated data look right. Fidelity compares statistical properties, individual and joint feature distributions, between real and synthetic data. Validity goes deeper, checking that records respect real-world rules , such as an end date after a start date, a non-negative age, a postal code matching its region. These constraints traditionally required a domain expert but now an LLM can potentially discover them from the data and turn them into checks the generated records must pass.

Utility asks a different question: is the data useful for the task? One common approach is Train on Synthetic, Test on Real (TSTR) [1], which compares synthetic data against a real-data (TRTR) baseline, concretely assessing whether it reaches the level of usefulness of real data. Utility also works in the other way, since synthetic data can deliberately boost the visibility of underrepresented regions of the feature space (rare events like cancer incidents or bank defaults), improving coverage that real data alone would struggle to provide.

But if the data fail one of these checks, who decides what to change?

Agentic generation

In many traditional ML pipelines, the main optimization happens during training: the model is trained, evaluated, and its parameters or hyperparameters are tuned until the desired metrics improve. Once this process is complete, generation is typically a one-shot operation. An agentic workflow builds on the ideas discussed so far, using LLM agents to improve the data generation step on their own.

With an agentic approach:

• an additional training or fine-tuning phase is possible, but not necessarily required:
• the iterative loop is moved into the LLM’s generation and prompt refinement cycle,
• external knowledge can also be exploited,
• this leads to data fidelity and utility close to state-of-the-art approaches.

TAGAL Approach

A first example of this is found in Self-Refine [2]: instead of assuming the first generation is good enough, the model evaluates its own output and uses this feedback to produce a better version.

TAGAL (Tabular data Generation using Agentic LLM) [3] explores several variants of this idea, all sharing the same core mechanism: one LLM generates data while another gives feedback, and the two keep alternating.

• The SynthLoop variant simply iterates between generation and refinement,
• ReducedLoop also reuses the experience and context accumulated during one generation cycle to improve the next, reducing the cost of subsequent iterations.
• Prompt-Refine summarises the feedback and generation history to identify missing elements and incorporate background knowledge, directly modifying the initial prompt before reusing it for subsequent generations.

On tabular datasets, TAGAL outperforms simpler few-shot LLM approaches by up to 60% in TSTR performance, increasing classification ROC-AUC from 0.53 to 0.86. This suggests that the generated data can preserve characteristics of the original data while also being useful for downstream tasks.

Together, these three approaches illustrate how synthetic data generation is evolving from a one-shot process into a continuous agentic loop, where the agent can use previous generations and feedback to progressively improve the data it produces.

Autodata Approach

When LLMs were mainly used for one-shot or few-shot generation, iterative generation and refinement provided a natural way to improve their outputs. As agents become increasingly capable of reasoning, however, this paradigm can be pushed further: the agent can move beyond generation and refinement toward a role closer to that of a data scientist, reasoning about what data to generate, how to evaluate it, and how to improve it.

This is the direction explored by Autodata [4], which drops the idea of a fixed generation procedure: rather than simply asking an LLM to generate data and critique its own output, the agent first reasons about which properties actually matter for the use case, for instance the compatibility with a downstream model, and then generates data with that goal in mind. In this way it refines not just the data, but the generation process itself.

So, the first mechanism implemented is an Agentic Self-Instruct approach: generated data is tested on two downstream models, a weak and a strong solver. Their behaviour becomes the signal the agent uses to judge whether a sample is useful. The logic is simple: if both solvers solve an example, it’s probably too easy; if neither can, it’s probably too hard. The most useful examples sit in between, hard enough to challenge the weak solver, solvable by the strong one. The gap between the two therefore works as a proxy for quality and discriminative power, and guides the next round of generation toward harder examples.

In this aspect, Autodata reaches a weak–strong solver gap of 34 pp, compared with only 1.9 pp for the CoT Self-Instruct baseline. This larger gap indicates that the generated data contain more discriminative examples: they are challenging for weaker models while remaining solvable by stronger ones, providing a more informative and less flat training signal.

This signal feeds into a second, higher-level loop: meta-optimization. Here the agent stops acting like a generator with feedback wrapped around it, and starts acting like a data scientist deciding what “good data” even means for the task. When it notices a recurring issue, for example both solvers getting the same type of question right, it doesn’t ask for a harder question, but rewrites the generator’s instructions to produce more complex, multi-step questions and tests the new strategy on a fresh batch. The new strategy is kept only if it widens the gap between weak and strong solver; otherwise it’s discarded.

The conceptual shift is this: the agent is no longer just a step in the generation pipeline: it decides how the pipeline itself should evolve. Generation, evaluation and refinement stop being separate steps and become parts of the same adaptive loop.

Challenges

The whole loop is only as good as the judge driving it. If the evaluator misjudges what “good” means, the agent may optimize for the wrong objective without actually improving the underlying data.

A classic example is Goodhart’s law: when a metric becomes a target, it can cease to be a good measure of what we actually care about. In a generation loop, optimizing for what the evaluator rewards can push the generator toward a narrow, “safe” region of the data space, potentially sacrificing the diversity that makes synthetic data useful in the first place.

Another point of attention is the relationship between the generator and the evaluator. If they come from the same model family, they may share the same blind spots, so the loop may converge toward apparent higher quality data, while actually reinforcing the same biases.

Finally, there is the cost. Every iteration requires generation, evaluation, and reasoning, and some approaches also add prompt optimization on top. As the loop becomes more sophisticated, so does the amount of computation it requires.

Conclusions

The shift is not from GANs to LLMs, it is from generating data to building systems that can iteratively improve the data they generate. For years, the generative model was the whole story: pick an architecture, train it, sample from it. Agentic approaches show that the model is only one piece of a larger loop, where generation, evaluation and refinement all feed into each other, and where the agent itself gets to decide what “good data” means for the task rather than just chasing a fixed metric.

That does not make synthetic data generation a solved problem. On the contrary, it introduces a new challenge: trusting the loop itself and the objective it is optimizing. So the natural question becomes “how do we build loops we can trust to know when the data is actually good enough?”

References

[1]: A. Kurakova & H Homayouni, A comprehensive evaluation framework for synthetic medical tabular data generation, Journal of Biomedical Informatics, 2025

[2]: Madaan, A., et al. (2023). Self-Refine: Iterative Refinement with Self-Feedback.

[3]: Ronval, B., Dupont, P., & Nijssen, S. (2025). TAGAL: Tabular Data Generation using Agentic LLM Methods

[4]: Kulikov, I., et al. (2026). Autodata: An agentic data scientist to create high quality synthetic data

Synthetic Data: A New Agentic Paradigm was originally published in Data Reply IT | DataTech on Medium, where people are continuing the conversation by highlighting and responding to this story.

In just a few seconds, an AI agent with the wrong permissions can turn a routine task into a production incident. In one widely reported case, an AI coding agent reportedly deleted a company’s production database and backups, causing a major outage and exposing a deeper problem.

In AI-first systems, the real risk is not only what the model can generate, but what it is allowed to execute.

As AI agents evolve from passive assistants into true active operators, security can no longer stop at a simple login. We now need to think in terms of:

• Identity — who (or what) is acting
• Delegation — on whose behalf the agent operates
• Continuous verification — ongoing control, not just at initial access
• Privilege boundaries — what it is allowed to do

In other words: the user is only the starting point. The agent also needs its own identity, and the system must be able to define what that agent can do, for how long, and under which conditions.

Why password-based security and traditional MFA are no longer enough

The classic security model was designed for a world where a human logs in, does something manually, and logs out. Passwords, one-time codes, and push-based MFA were built to protect that flow. But AI agents break that assumption.

The first problem is that passwords are still weak in practice. They get reused, guessed, phished, leaked, and stolen through credential stuffing. Even traditional MFA is not a complete answer. SMS codes can be intercepted or abused, and push notifications can lead to approval fatigue, where users click “yes” just to make the prompt disappear. Those mechanisms may reduce risk, but they do not create a truly phishing-resistant identity layer.

The second problem is delegation. When an AI agent acts on behalf of a user, the question is no longer only “who logged in?” but “who is acting now?” If the same credential or session token is reused everywhere, the boundary between the human and the agent disappears.

Agents as first-class principals

In traditional systems, identities were easy to categorize: a human user, a service account, maybe an admin. AI agents change that by creating a new class of actor. They are not exactly human users, and they are not just background services either. They are autonomous or semi-autonomous entities that can reason, call tools, execute actions, and chain operations together. That means they need their own identity model.

An agent should be treated as a first-class principal with:

• its own identity
• its own lifecycle
• its own permission set
• its own audit trail
• its own revocation path

If an agent acts under the same credentials as the user, you lose attribution. If it acts under a distinct delegated identity, you can answer critical questions later: which agent executed this action? On whose behalf? With what scope? At what time? Under what policy conditions?

That distinction is crucial when something goes wrong.

Passkeys as the foundation for human authentication

Before delegation can happen, the human identity itself needs to be verified with something stronger than a password. This is where passkeys come in.

A passkey is a credential based on public-key cryptography, generated and stored on the user’s device rather than on a server. When a passkey is created, the device generates a key pair: the private key never leaves the device (it is stored in a secure enclave or equivalent hardware module), while the public key is registered with the service. During login, the service sends a challenge, the device signs it with the private key, and the service verifies the signature with the public key it already has.

This design makes passkeys phishing-resistant by construction, not just by policy. There is no shared secret to type, copy, or intercept — a fake login page cannot capture a passkey the way it can capture a password or a one-time code, because the private key simply never leaves the trusted device. Passkeys are also bound to the origin they were created for, so even if a user is redirected to a lookalike domain, the browser will not offer a matching credential.

For AI-first systems, this matters beyond convenience. A strong, non-phishable human authentication step is the anchor of the entire trust chain: if the initial login can be bypassed or spoofed, every downstream layer — delegation, scoped tokens, agent identity — inherits that weakness.

Passkeys give the system a solid starting point: a strongly authenticated human identity with phishing-resistant authentication, which is exactly the precondition OAuth 2.1 needs in order to delegate access safely to an agent.

OAuth 2.1 as the delegation layer between human and agent

Once the human is authenticated, the next step is delegation.

OAuth is the mechanism that lets a user grant a third-party application limited access to their resources without sharing credentials. In an AI-first system, that third party is often an agent. The point is not to give the agent the user’s password or a full session, but to issue a scoped, temporary, and revocable token that represents only the permissions needed for a specific task.

OAuth 2.1 tightens this model by turning the security best practices of OAuth 2.0 into the modern baseline. It requires PKCE for authorization code flows, enforces exact redirect URI matching, and removes weaker grant types such as Implicit and Resource Owner Password Credentials. The result is a delegation flow that is much harder to intercept, replay, or overextend.

PKCE matters here because it binds the authorization request to the token exchange: even if an authorization code is intercepted, it cannot be redeemed without the original code verifier. That makes it a natural fit for agentic systems, where the risk is not just unauthorized login, but unauthorized escalation during delegation.

In practice, the flow should look like this:

• The user authenticates with a strong method such as a passkey.
• The user explicitly grants access to the agent.
• The agent receives only the minimum required scope.
• The token expires quickly and can be revoked.
• Every action is attributable to both the agent and the human it represents.

This separation matters because an agent is not the same thing as a browser session or a backend service — it is a delegated actor. If you blur that boundary, over-permissioning becomes the default.

Zero Trust for AI-first applications

Once agents become active actors, Zero Trust stops being a slogan and becomes a design requirement.

Zero Trust means you do not assume trust based on network location, application boundary, or previous authentication alone. You verify continuously. For AI-first applications, that principle needs to be applied across identity, authorization, runtime policy, and observability.

A practical Zero Trust model for AI-first systems should include:

• strong human authentication with phishing-resistant methods
• explicit agent delegation instead of credential sharing
• least-privilege scopes for every task
• short-lived tokens and frequent renewal logic
• runtime policy checks for sensitive actions
• full audit logging for identity, actions, and context
• step-up verification for high-risk operations

The main idea is that trust should be continuously reconstructed, not assumed. A user may be authenticated once, but an agent’s access still needs to be checked at the moment of every meaningful action.

This is especially important because AI agents can operate faster than humans can intervene. If a harmful action is allowed to go through once, it can cascade very quickly.

Zero Trust adds friction where it matters most: at the point of privilege, not just at login.

Step-up authentication for critical actions

Not every action deserves the same level of trust.

An AI agent might be allowed to draft a report, summarize a dataset, or prepare a deployment plan. But the same agent should not be able to delete production data, change role assignments, rotate secrets, or approve financial transactions without an extra layer of verification.

That is where step-up authentication comes in.

Step-up auth means that a user or agent can operate normally for low-risk actions, but must pass an additional check before high-risk operations. In a human-centric system, that may mean re-entering a passkey prompt, confirming on a trusted device, or requiring secondary approval. In an agentic system, it may mean pausing the workflow until the human explicitly confirms the action.

A good design principle is to define a “blast radius” (how much damage a single action could cause if it goes wrong) for every agent. The more destructive the action, the more friction you should require.

The system should never let a generic delegated token become a free pass to do irreversible things.

The real-world incidents around deleted databases highlight exactly why this matters. In those cases, destructive operations happened because the agent had too much authority or too little friction at the decision point.

Step-up authentication is one of the most effective ways to prevent a routine workflow from turning into an outage.

Anti-patterns to avoid

There are a few patterns that should be avoided entirely in AI-first systems.

1. Letting agents use the user’s password
   This is the most obvious mistake, and still one of the most dangerous. Agents should never receive or handle primary user credentials.
2. Reusing the human session as if it were the agent
   A delegated agent should not silently inherit the full power of the user session. That destroys accountability and makes revocation nearly impossible.
3. Using long-lived tokens with broad permissions
   If a token lives too long and can do too much, it eventually becomes an incident waiting to happen.
4. Treating agent actions as background noise
   Every meaningful agent action should be attributable. Otherwise incident response and forensic analysis become guesswork.
5. Skipping human confirmation for destructive operations
   If the action is irreversible, the user should know about it. And ideally, they should approve it explicitly.
6. Ignoring context and runtime policy
   Authentication alone does not guarantee safe behavior. You also need device context, request context, policy enforcement, and observability.

These anti-patterns are not theoretical. The public incidents around AI agents deleting production systems and backup data are exactly what happens when systems trust agents too broadly and too early.

The Anatomy of an AI-First Application

An AI-first application is not just a model wrapped in a UI. In production, it is a stack of layers that work together: identity, orchestration, context, tools, execution, security, observability, and governance. If one of these layers is missing, the system may still work in a demo — but it will not be reliable, auditable, or safe enough for real-world use.

You can think of it like this:

• User & Identity Layer: this is where the human is authenticated and where the agent is identified. In practice, this is where passkeys fit best for the human login.
• Delegation Layer: this layer grants scoped access to the agent. This is where OAuth 2.1 comes into play.
• Orchestration Layer: this layer coordinates the flow of tasks, decisions, retries, and branching logic. It is especially relevant when multiple tools or sub-agents are involved.
• Context Layer: this layer provides the data, state, and memory the agent needs to work properly. A good context layer gives the agent enough information to be useful, but not so much that sensitive data is exposed unnecessarily.
• Tool & Integration Layer: this layer connects the agent to external APIs, databases, SaaS tools, or internal systems. This is one of the most dangerous layers, because this is where actions become real.
• Execution Layer: this is where actions are actually carried out in a controlled environment. If something goes wrong here, the damage can quickly become operational.
• Security Layer: this layer enforces policy, boundaries, and trust rules across the whole stack. This is where Zero Trust principles belong.
• Observability Layer: this layer logs traces, decisions, errors, and outcomes. Without observability, debugging and auditing become extremely difficult.
• Governance Layer: this layer handles compliance, approvals, ownership, and accountability. This becomes crucial once agents start acting in production environments.

A practical implementation on AWS

The concepts described so far are not just theoretical — AWS already provides building blocks that map directly onto this model.

Human authentication with passkeys can be implemented through Amazon Cognito’s passwordless authentication, which supports FIDO2-based passkeys alongside email and SMS one-time codes. This gives the human login step the phishing-resistant properties described earlier, using the same public-key cryptography model, without building custom WebAuthn logic from scratch.

Agent identity can be handled through Amazon Bedrock AgentCore Identity, a service purpose-built to manage agent identities and credentials. It acts as a centralized directory for agent and workload identities — a single source of truth, separate from human user identities. What makes it particularly relevant here is that it natively supports OAuth 2.0 flows, including both the client credentials grant (two-legged, for agents acting autonomously) and the authorization code grant (three-legged, for agents acting on behalf of a user) — the same delegation model this article builds around. Credentials, OAuth tokens, and API keys are held in an encrypted token vault, and the AgentCore SDK exposes declarative annotations that automatically retrieve and inject the right credential at runtime, so agents never need to handle raw secrets directly in code.

Scoped, temporary delegation maps naturally onto IAM roles combined with AWS STS (AssumeRole): instead of sharing long-lived credentials, an agent can assume a role with a narrowly defined policy and a short session duration, producing exactly the kind of scoped, expiring, revocable token this article argues for.

Attribution and auditability are covered by AWS CloudTrail, which logs every API call together with the identity that made it, and by AgentCore Identity’s own access controls, which preserve an audit trail of which agent acted on behalf of which user — allowing teams to trace an action back to both the agent that executed it and the human identity it was delegated from.

Together, these services do not form a single pre-packaged product — they are complementary building blocks that need to be architected together. But they show that the principles outlined in this article are not aspirational: the underlying primitives already exist on AWS today, and are recent enough to reflect the current direction of the platform (Cognito passwordless launched in November 2024; AgentCore Identity is part of AWS’s more recent agentic-AI toolset, with native OAuth 2.0 support built in)

Conclusion: identity, delegation, and verification must be separate, but connected

The rise of AI agents changes the entire security conversation. Authentication is no longer just about proving that a human is present. It is about building a trust chain that can survive delegation, automation, and partial autonomy.

The right model is not one big login event. It is a sequence of connected layers:

• the human authenticates with a phishing-resistant method such as passkeys
• the human delegates limited authority through OAuth 2.1
• the agent operates under its own identity
• the system verifies context continuously under Zero Trust principles
• high-risk actions trigger step-up authentication
• every action is logged, attributable, and revocable

That separation is the point. Identity, delegation, and verification must remain distinct, because each solves a different problem. But they must also work together, because AI-first systems blur the line between user intent and machine action.

In the agentic era, security is no longer just about keeping attackers out. It is about making sure that the systems we build never confuse permission with trust.

AI Agents Need Identities Too: Passkeys, OAuth 2.1, and Zero Trust for AI-First Applications was originally published in Data Reply IT | DataTech on Medium, where people are continuing the conversation by highlighting and responding to this story.

Retrieval choice is the most consequential decision in a RAG system. Today it is taken months before the first real query. Here is the architecture that turns it into a runtime decision and the protocol for measuring whether the router is any good.

1. Introduction

RAG almost always ships with a single retrieval strategy: dense vector search over chunks, or structured retrieval over a knowledge graph. Neither is uniformly right. Vector search wins on single-hop factoid lookup; graph retrieval wins on multi-hop reasoning and corpus-level sensemaking; hybrid retrieval often beats both — but not for free. This article makes retrieval strategy a runtime decision instead of a deployment-time one, via a Smart Retrieval Router: a module upstream of retrieval that reads a defined set of signals and emits an explicit retrieval plan. You get the architecture, the routing signals, five ways to implement the router and an evaluation protocol to assess the component.

2. The decision nobody makes explicitly

Ever since RAG became the default way to give an LLM access to external knowledge, the industrial recipe has been stable: chunk the corpus, embed the chunks, put them in a vector index, retrieve the top-k nearest neighbours of the query, hand them to the generator.

Then a second family arrived: extract entities and relations into a knowledge graph, and retrieve context by traversing that graph or by summarising its communities.

Both families answer the same question — what context should the generator see? — and both answer it with a fixed mechanism chosen long before the first real query shows up.

That is the omission this article is about. In a RAG pipeline, generation is bounded by retrieval: an excellent generator handed the wrong context produces a fluent wrong answer, which is the worst failure mode there is because nothing downstream flags it. Retrieval choice is therefore the pivotal architectural decision — and it is hard-coded.

The claim is narrow: the choice between vector retrieval, graph retrieval and a hybrid of the two is predictable from properties of the query and of the vector and graph indexes, and can be made by a dedicated module at the moment the query arrives.

3. What we already know

This section is deliberately short, because the comparative literature is large and broadly consistent. What matters here is not who wins among indexes, but that the winner changes from query to query — which is precisely the precondition for routing to pay off.

3.1 Where vector search wins

Single-hop, extractive, well-phrased questions whose answer lives inside one or two passages. Dense retrieval is precise, fast and cheap here. Benchmarks consistently show classic RAG matching or beating graph variants on simple fact retrieval — and graph pipelines pay a per-query token bill that is orders of magnitude larger. Paying that premium to answer “what was Q3 revenue?” is not a tuning issue; it is an architectural defect.

3.2 Where graph retrieval wins

Three recurring situations. Multi-hop and relational questions, where the answer is a path between entities rather than a passage of text. Global, corpus-scope questions — “what are the main themes in this corpus?” — which are query-focused summarization rather than lookup, and for which top-k retrieval has no mechanism to guarantee coverage of the whole corpus by construction. And synthesis that requires holding distant concepts together.

But there is a hard precondition, and it shows up identically across studies: if the query contains no explicit entity, the graph has no anchor to start from and the graph path degrades. Graph retrieval is powerful downstream of a successful entity link.

3.3 Where hybrid wins — and what it costs

Retrieving from both indexes and fusing the results often beats either one alone, especially on answer relevancy. But there is a detail in the published numbers that almost always goes unremarked, and it is the strongest argument for routing rather than for unconditional hybridising: hybrid configurations tend to show the lowest context precision of the three.

The reason is obvious once stated: merging two retrieval channels adds recall and adds noise. Hybrid is not a free upgrade, it is a trade precision and cost in exchange for robustness. Sometimes that trade is right and sometimes it is exactly wrong. Knowing which case you are in is, by definition, a routing decision.

3.4 State of practice: the switch exists, but the user flips it

Mature tooling already accepts that one retrieval mode is not enough, and responds by exposing the choice as a manual control: widely used frameworks ship selectable query modes — plain vector, graph-local, graph-global, hybrid — behind a CLI flag or an API parameter.

That is real progress, but it delegates the decision to a human who has to guess, per query, which index will serve them better — a judgement that requires knowing both the shape of the corpus and the internals of retrieval. And in an API-fronted deployment, or inside an agent loop, that human is not in the loop at all: the choice collapses back to the static default.

The switch has been built. The controller is missing.

4. What this article proposes

First, a caveat: the past two years brought work with simple routing decisions. However what is still open comes down to three specific things, and they are what this article is about.

1. The route space is too small. Existing routers decide along a single axis — query complexity, or retriever identity, or generator size. A real RAG system has to decide jointly on three things: index type (vector, graph-local, graph-global), retrieval depth (single-shot or iterative), and fusion (one path or merged paths). Three entangled decisions, not one.

2. The best signals arrive too late to be used. Nearly all existing routers decide from the query text alone. But the most discriminative evidence is not in the text, it is in the state of the indexes. Do the query’s entities actually link to graph nodes? How flat is the similarity score distribution? How badly does the k-hop neighbourhood fan out — that is, the graph too deeply nested for traversal to return usable context? None of these is inferable from the query; you learn them only after a cheap probe. Putting that probe in the architecture changes decision quality, and nobody does it.

3. Nobody evaluates the router. Routers get judged indirectly, through end-to-end answer quality. That conflates router errors with retriever and generator errors, and it cannot answer the question that matters: how good is this router relative to a perfect one? A router is a classifier with an asymmetric, query-dependent cost matrix, and it deserves its own metrics. Section 7 proposes which.

5. The architecture: a Smart Retrieval Router

5.1 Where it sits

The router sits between query understanding and retrieval execution. It does not retrieve and it does not generate — it emits a retrieval plan. It is a thin component with an explicit output contract.

5.2 The route space

Treating the decision as binary — vector or graph — throws away the two most useful options: don’t retrieve at all, and retrieve from both. Six routes is the practical minimum.

5.3 The signals that drive the decision

Signals split into two families, separated not by content but by when they become available. The second family is the one most routers don’t use, and it’s the one that makes the difference.

One design note worth more than most optimizations: the probe’s cost is dominated by the cost of the wrong route. An entity-linking call plus a small-k probe costs a fraction of a pointless graph traversal, and far less than an incomplete answer.

5.4 Three rules that keep it safe

Below the confidence threshold, don’t choose — fuse. The router emits a calibrated confidence alongside the route. Below a threshold it declines to commit and takes R4. That turns the router’s worst case into “slightly more expensive and slightly noisier” instead of “wrong index” — the difference between a tolerable defect and a serious one.

One reroute, never more. After context assembly, a check scores whether the assembled context plausibly supports an answer. If it doesn’t, allow exactly one escalation, typically to R4 or R5. Bounded escalation preserves the cost argument for the whole system; unbounded escalation destroys it.

Keep the manual override. The flag users flip today shouldn’t be removed — it should become the exception. The router is the default, not a cage.

5.5 The router’s output contract

RouteDecision {
  route:        R0 | R1 | R2 | R3 | R4 | R5
  confidence:   float          // calibrated, not a raw score
  params:       { k, hops, anchors[], community_level,
                  fusion_weights, rerank: bool }
  budget:       { max_tokens, max_latency_ms }
  rationale:    string         // for audit and evaluation
  probe_used:   bool
  fallback:     R1 | R4        // if the chosen route errors out
}

The rationale field looks like a luxury and isn't: it's what makes “why did this answer come out badly?” a diagnosable question months later.

6. Building the router

There are five sensible ways to implement the router, and they aren’t alternatives — in practice multiple methods could used at once. All five are worth knowing because they sit at different points on the same cost-capability curve, and which one you reach for depends on how much traffic and how much labelled data you have.

6.1 Hand-written rules

Explicit conditions over the signals: for instance entities linked and relational language and hops ≥ 2 → R2; scope words → R3; and so on. Microsecond latency, fully auditable, zero training data. This is your day-one baseline and your permanent fallback when the learned router is unavailable. It’s also the honest control in evaluation: a learned router that can’t beat well-tuned heuristics hasn’t earned its complexity. The limits are brittleness across domains and awkward handling of conflicting signals.

6.2 A lightweight trained classifier

A small fine-tuned encoder, or gradient-boosted trees over engineered signals, predicting the best route. Millisecond inference, cheap retraining, easy to calibrate — and it outputs probabilities that plug straight into the confidence-gated fusion rule. The cost is that you need labels (as discussed in 6.6) and periodic retraining as the corpus drifts.

6.3 LLM-as-router

Prompt a small-to-mid model with the query, the signal definitions, a description of what each index contains and the graph schema, and require a structured RouteDecision. Works from day zero with no data, reasons well over query-side signals, and produces a human-readable rationale that is worth a disproportionate amount during debugging. In exchange it adds a model call on the critical path of every query, and its self-reported confidence is poorly calibrated unless you recalibrate post-hoc. You could use it as the cold-start router and as the teacher you distil the classifier from.

6.4 Agentic router with retrieval tools

Expose vector search, entity linking, neighbourhood inspection and traversal as tools, and let an agent select and sequence them while observing results. Here the line between routing and retrieval dissolves: the agent reroutes at every step, which makes it the natural home for R5. It has the highest ceiling on genuinely hard multi-hop questions and handles index-side signals natively, because it can simply look. It also has the highest and least predictable latency and cost, is harder to evaluate (the decision is a trajectory, not a label), and drifts into pointless tool loops without a step budget.

6.5 A learned policy with feedback

Frame routing as a contextual bandit: context is the signal vector, actions are routes, reward is answer quality minus normalized cost. This optimizes the actual objective, cost included, and adapts to drift without a manual relabelling cycle. The price is that you need a trustworthy online reward — implicit feedback is noisy and an LLM judge is gameable — plus exploration risk on live traffic. Realistic path: start offline from logged decisions, then explore within a capped slice of traffic.

6.6 Where the training labels come from

The classifier implementation needs examples of the form “for this query, the right route was this one.” Nobody has those lying around. Here is how you make them.

Take a sample of representative queries — a few hundred is enough to start. For each one, run it through every route, not one: the same query goes through vector, through graph, through hybrid. Score the resulting answers. Then label the query with the cheapest route among those that produced top-tier quality.

That “cheapest” is the part that matters. If you label with the outright best route, nearly every label collapses to hybrid or agentic, and the classifier learns that spending the maximum is always safe — which is precisely the behaviour the router exists to prevent.

The pass is expensive, but you pay it once and get two things at once: the classifier’s training set, and the oracle you need for the evaluation in the Section 7. From there, two shortcuts: have an LLM-as-router label historical traffic and distil that into the classifier, and log route, signals, confidence, cost and outcome in production — which over time becomes your highest-fidelity dataset, because it’s the only one that reflects the real query distribution.

One warning about the sample, because this mistake is easy and expensive: multi-hop datasets over-represent graph-favourable queries, and factoid QA datasets over-represent vector-favourable ones. A router tuned on either alone will be miscalibrated in production. Build the evaluation set as a deliberate mix.

7. Evaluating the router

End-to-end answer quality is necessary but insufficient: it can’t separate a routing error from a retrieval or generation error, and it can’t tell you how much headroom is left. Measure at two levels.

7.1 The yardstick

Without these reference points, an end-to-end improvement is uninterpretable.

7.2 The metrics that matter

7.3 The cost–quality frontier

It’s worth to build the cost-quality frontier, by sweeping the router’s threshold from 0 to 1 in order to escalate from less to more expensive routes.

8. Conclusion

The interesting part of the comparative question is settled: vector retrieval, graph retrieval and their combination each win on identifiable classes of queries, and the differences are large enough — in quality and, above all, in cost — that choosing statically leaves real value on the table. Current tooling has recognized this and responded by exposing a manual switch, which relocates the problem to the user.

The proposal here is to close the loop with a Smart Retrieval Router: a dedicated module upstream of retrieval that reads a defined set of signals — some from the query, and critically some from a cheap probe of the indexes themselves — and emits an explicit retrieval plan over a route space spanning no-retrieval, vector, graph-local, graph-global, hybrid and iterative, with calibrated confidence, hybrid fusion as the safety net, and escalation bounded to a single attempt.

Another important part that remains underserved is the router’s evaluation as a component: against an oracle ceiling and a cost-matched random floor, with asymmetric error costs, calibration, decision overhead and route stability measured directly, and results presented as a cost–quality frontier rather than a single operating point.

Retrieval choice is the pivotal decision in a RAG system. It should be made deliberately, by a component you can inspect, test, and hold to a number.

References

Foundations

• Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, NeurIPS 2020 — arXiv:2005.11401
• Edge et al., From Local to Global: A Graph RAG Approach to Query-Focused Summarization — arXiv:2404.16130 (and its implementation, whose local / global / drift / basic query modes are user-selected)
• Guo et al., LightRAG: Simple and Fast Retrieval-Augmented Generation — arXiv:2410.05779 (naive / local / global / hybrid / mix modes, also user-selected)

When vector, when graph, when both

• Xiang et al., When to use Graphs in RAG: A Comprehensive Analysis for Graph Retrieval-Augmented Generation (GraphRAG-Bench) — arXiv:2506.05690
• Han et al., RAG vs. GraphRAG: A Systematic Evaluation and Key Insights — arXiv:2502.11371
• Sarmah et al., HybridRAG: Integrating Knowledge Graphs and Vector Retrieval Augmented Generation for Efficient Information Extraction, ICAIF ’24 — DOI 10.1145/3677052.3698671, arXiv:2408.04948

Adaptive routing in RAG

• Jeong et al., Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity, NAACL 2024 — arXiv:2403.14403
• Dong et al., Use Graph When It Needs: Efficiently and Adaptively Integrating Retrieval-Augmented Generation with Graphs — arXiv:2602.03578
• Hakim et al., SymRAG: Efficient Neuro-Symbolic Retrieval Through Adaptive Query Routing — arXiv:2506.12981
• Kim & Diaz, LTRR: Learning To Rank Retrievers for LLMs — arXiv:2506.13743
• Mu et al., Unsupervised Query Routing for Retrieval Augmented Generation — arXiv:2501.07793

Router evaluation

• Ong et al., RouteLLM: Learning to Route LLMs with Preference Data — arXiv:2406.18665 (source of the CPT and APGR metrics and the cost-matched random baseline, adapted here from routing between models to routing between indexes)
• Xu et al., mmRAG: A Modular Benchmark for Retrieval-Augmented Generation over Text, Tables, and Knowledge Graphs — arXiv:2505.11180 (annotates routing accuracy explicitly)

Smart Routing for RAG: choosing between Vector Search and Knowledge Graph at query time was originally published in Data Reply IT | DataTech on Medium, where people are continuing the conversation by highlighting and responding to this story.

The Monday Morning Scenario

You open the billing console on Monday morning. Last week the AI line was $800, comfortably within the quarterly estimate you gave finance. This morning it reads $12,400. Your stomach drops.

You dig in. Turns out a customer-support agent (deployed Friday afternoon to 200 users) hit a retry loop overnight. Fifteen thousand calls to Claude Sonnet, each with a 4K-token context window, running for seven hours straight. Nobody got paged. Nobody’s rate limit kicked in. The budget alarm you meant to set up? Still in your backlog.

If this sounds familiar, or if you’re determined it never becomes familiar, this article is for you.

We’ll walk through Unity AI Gateway, Databricks’ architectural answer to uncontrolled AI spend. It’s a centralized governance layer that sits between your users (or your agents) and every model endpoint, MCP server, and Genie Space in the platform. There are four levers and one control plane:

1. Rate Limits: prevent the burst before it happens
2. Budgets: alert and block when spend crosses a line
3. Cost Observability: know who spent what, on which model, tagged to which team
4. AI Guardrails: block the request that should never have been sent

We’ll use a running example throughout: DataCo’s ML platform team deploys a summarization agent to 200 internal users. By the end of the article, they’ll have the full governance stack configured. You’ll have the queries, the architecture, and the config steps to do the same.

What Is Unity AI Gateway

Status: Beta (Unity AI Gateway visible in sidebar UI). Enable from Account Console → Previews.

The core architectural insight is centralization. Before Unity AI Gateway, governance was scattered: rate limits lived on individual endpoints, budgets were account-wide blobs, and guardrails (if any) were custom middleware. Unity AI Gateway collapses all of this into a single layer that sits in front of every AI service (model endpoints, MCP servers, Genie Agents, external model proxies), governed by the same Unity Catalog permissions and audit trail you already use for your data.

This matters because the cost problem isn’t technical; it’s organizational. When 5 teams run 12 endpoints calling 3 model providers, no single team owns the bill. A centralized gateway makes one team (platform) responsible for one control plane, while individual teams retain autonomy on what they build.

Pillar 1: Rate Limits

Back to DataCo. Their summarization agent works great in dev: 3 requests per minute, manageable. But in production, 200 users hitting it simultaneously during Monday’s standup creates a thundering herd. Without rate limits, that spike hits the model endpoint unchecked, and each request burns tokens at full speed. Rate limits are the first line of defense: they cap the damage before any budget threshold is even evaluated.

How They Work

Rate limits prevent anomalous bursts that could generate uncontrolled costs. They operate on two metrics:

• QPM (queries per minute): applies to Model Serving + MCP services. Example: “Max 60 requests/min per user.”
• TPM (tokens per minute): applies to Model Serving only. Example: “Max 100K tokens/min per endpoint.”

Enforcement Levels

Rate limits are evaluated in a hierarchy. The service-level limit is the hard ceiling for the entire endpoint; below that, each user gets a default cap; and below that, you can override individual users or groups with custom limits. The most specific rule wins:

Operational Limits

A few constraints to keep in mind when designing your rate limit strategy:

• Max rate limits per service: 20
• Max group-specific limits per service: 5
• Behavior on HTTP 429: client must implement retry with exponential backoff
• Burst behavior: concurrent requests can briefly exceed the limit (usage counted after response)

Configuration (UI)

Here’s what the setup looks like for DataCo’s summarization endpoint:

Workspace → Model Serving → endpoint → Rate Limits tab
  → Service: 10000 TPM
  → User (Default): 2000 TPM, 30 QPM
  → Custom: "ml-team" group → 5000 TPM, 100 QPM
  → Custom: "intern-user@company.com" → 500 TPM, 10 QPM

Pillar 2: Budgets & Spend Control

Rate limits cap the rate of spend, but they don’t cap the total. If DataCo’s 200 users each send 30 QPM for 8 hours, the math still adds up. Budgets are the financial circuit breaker: they track cumulative spend over time and fire alerts (or hard-block requests) when a threshold is crossed. Think of rate limits as the speed limit and budgets as the fuel gauge.

Anatomy of a Unity AI Gateway Budget

• Resource type: “Unity AI Gateway” (tracks AI Gateway spend only)
• Scope: workspace, endpoint, or filtered by UC tag
• Shared threshold: collective spending limit for all users in scope (max 4 per budget)
• Per-user threshold: monthly individual limit per user
• Per-user override: exception for specific users/groups (max 20 per budget, Genie budgets only)
• Actions: Send alert (email) and/or Block usage (Genie budgets only)

What’s Tracked (and What’s Not)

✅ Tracked: pay-per-token inference, ai_query (batch inference), Genie product usage.

❌ NOT tracked: provisioned throughput, external model inference (monitored via a separate table, but not enforced by budgets).

System Limits

Before you design your budget hierarchy, know the guardrails on the guardrails:

• Max budgets per account: 1,000
• Max shared thresholds per budget: 4
• Max per-user overrides: 20
• Block usage: Genie budgets only
• Enforcement lag: in-flight requests not interrupted; brief delay before block takes effect

Configuration (Account Console)

Here’s how DataCo configures their production budget with progressive escalation:

Account Console → Usage → Budgets → Add budget
  → Resource type: Unity AI Gateway
  → Workspace: [select]
  → Custom tags: team=ml-platform (optional, filters to endpoints with that tag)
  → Threshold 1: $500/month shared → Send alert to: platform-team@company.com
  → Threshold 2: $1000/month shared → Block usage
  → Per-user: $100/month → Send alert
  → Per-user override: "power-user@company.com" → $300/month

⚠️ Note on Spend Data Discrepancies

Three sources show different figures (by design):

• Budget alert emails: at trigger time
• Budget details page: frequently updated (near real-time)
• system.billing.usage: every few hours (expected lag)

Enforcement (alerts + blocking) is based on near real-time spend tracking, not system tables.

Pillar 3: Cost Observability

DataCo’s budget alert fires on Thursday: “$500 threshold reached.” The platform team asks: which endpoint? which model? which users drove the spike? Without observability, the answer is “we don’t know, check again next month.” With Unity AI Gateway, the answer is a SQL query away. Two system tables give you full granularity on who spent what, on which model, broken down by endpoint, user, and custom request tags.

Available System Tables

• system.billing.usage: Databricks-hosted model costs in DBU. Granularity: per endpoint, model, user.
• system.ai_gateway.external_model_spend: external model costs in USD. Granularity: per provider, model, user, request tag.

Practical Queries: Copy-Paste for Your Workspace

• Top endpoints by consumption (last 30 days):

SELECT
  usage_metadata.ai_gateway.endpoint_name AS endpoint_name,
  SUM(usage_quantity) AS dbus
FROM system.billing.usage
WHERE billing_origin_product = 'MODEL_SERVING'
  AND usage_metadata.ai_gateway.endpoint_name IS NOT NULL
  AND usage_unit = 'DBU'
  AND usage_date >= current_date() - INTERVAL 30 DAYS
GROUP BY endpoint_name
ORDER BY dbus DESC;

• Spend by destination model:

SELECT
  usage_metadata.ai_gateway.destination_model AS destination_model,
  SUM(usage_quantity) AS dbus
FROM system.billing.usage
WHERE billing_origin_product = 'MODEL_SERVING'
  AND usage_metadata.ai_gateway.endpoint_name IS NOT NULL
  AND usage_unit = 'DBU'
  AND usage_date >= current_date() - INTERVAL 30 DAYS
GROUP BY destination_model
ORDER BY dbus DESC;

• Spend by user (who’s consuming the most?):

SELECT
  identity_metadata.run_by AS run_by,
  SUM(usage_quantity) AS dbus
FROM system.billing.usage
WHERE billing_origin_product = 'MODEL_SERVING'
  AND usage_metadata.ai_gateway.endpoint_name IS NOT NULL
  AND identity_metadata.run_by IS NOT NULL
  AND usage_unit = 'DBU'
  AND usage_date >= current_date() - INTERVAL 30 DAYS
GROUP BY run_by
ORDER BY dbus DESC;

• External models, spend by provider (in USD):

SELECT
  usage_metadata.provider AS provider,
  usage_metadata.endpoint_name AS endpoint_name,
  SUM(usage_quantity) AS usd
FROM system.ai_gateway.external_model_spend
WHERE usage_start_time >= current_timestamp() - INTERVAL 30 DAYS
GROUP BY provider, endpoint_name
ORDER BY usd DESC;

• External models, spend by request tag (team chargeback):

SELECT
  custom_tags.request_tags['team'] AS team,
  SUM(usage_quantity) AS usd
FROM system.ai_gateway.external_model_spend
WHERE usage_start_time >= current_timestamp() - INTERVAL 30 DAYS
GROUP BY team
ORDER BY usd DESC;

Built-in Usage Dashboard

Databricks provides a pre-built dashboard (v0.4+) accessible from:

Unity AI Gateway page → Govern (top right) → Usage Dashboard

Includes:

• Requests over time (daily/hourly averages)
• Latency distribution (p50, p90, p99)
• Error rate breakdown
• Token usage (input/output)
• Cost breakdown by model/provider
• Cost over time (cumulative trend)

Request Tags: The Chargeback Mechanism

Request tags are custom key-value pairs attached to individual API calls. They enable cost attribution by team, project, environment, or any dimension relevant to your organization.

How to attach tags (Python SDK):

from openai import OpenAI

client = OpenAI(
    base_url="https://<workspace>.cloud.databricks.com/serving-endpoints",
    api_key="<token>",
)

response = client.chat.completions.create(
    model="databricks-claude-sonnet-4-5",
    messages=[{"role": "user", "content": "Summarize Q2 revenue"}],
    extra_headers={
        "Databricks-Ai-Gateway-Request-Tags": '{"team": "finance-analytics", "project": "quarterly-report", "environment": "production"}'
    },
)

How to attach tags (REST API):

curl -X POST "https://<workspace>.cloud.databricks.com/serving-endpoints/<endpoint>/invocations" \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Databricks-Ai-Gateway-Request-Tags: {"team": "ml-platform", "project": "chatbot"}' \
  -d '{"messages": [{"role": "user", "content": "hello"}]}'

Where tags land:

• system.ai_gateway.usage → column request_tags → usage analytics (request count, tokens)
• system.ai_gateway.external_model_spend → column custom_tags.request_tags → cost attribution (USD)
• Inference tables → column request_tags → audit and debugging

Chargeback query, cost per team per month:

SELECT
  request_tags['team'] AS team,
  DATE_TRUNC('month', event_time) AS month,
  COUNT(*) AS request_count,
  SUM(total_tokens) AS total_tokens
FROM system.ai_gateway.usage
WHERE request_tags['team'] IS NOT NULL
GROUP BY team, month
ORDER BY month DESC, total_tokens DESC;

Best practices for tag naming:

• team (cost center / ownership): ml-platform, finance-analytics, customer-support
• project (initiative tracking): chatbot-v2, quarterly-report, rag-pipeline
• environment (lifecycle stage): dev, staging, production
• use_case (business function): summarization, code-gen, classification
• caller (service/app identity): slack-bot, dashboard-api, batch-job-42

Tip: Enforce tag conventions via a thin SDK wrapper or middleware. Untagged requests are invisible to chargeback queries: they show up as NULL in GROUP BY and can’t be attributed.

What This Looks Like in Practice

Here’s what we found when running these queries on a real Databricks sandbox (30-day window):

Total spend:        180.6 DBU across 5 active days
Peak day:           124.3 DBU (69% of total in a single day)

Top model:          claude-sonnet-4-6   101.85 DBU   940K tokens   avg 15.9K/req
Second model:       claude-opus-4-6      56.32 DBU   582K tokens   avg 5.0K/req
Cheapest model:     claude-haiku-4-5     22.45 DBU   570K tokens   avg 1.5K/req

Top user:           124.30 DBU (1,597 requests)
Request tags:       Zero. No chargeback possible.
External models:    Zero. All Databricks-hosted.

The peak-day spike tells the story: one user’s heavy session consumed 69% of the monthly budget in a single day. Without a budget alert, nobody noticed until we queried the system tables manually, weeks later. This is exactly the scenario the gateway is designed to catch.

Forecasting query: will you hit your monthly target?

WITH daily AS (
  SELECT
    usage_date,
    SUM(usage_quantity) AS daily_dbus
  FROM system.billing.usage
  WHERE billing_origin_product = 'MODEL_SERVING'
    AND usage_metadata.ai_gateway.endpoint_name IS NOT NULL
    AND usage_unit = 'DBU'
    AND usage_date >= DATE_TRUNC('month', current_date())
  GROUP BY usage_date
)
SELECT
  ROUND(SUM(daily_dbus), 1) AS month_to_date,
  ROUND(AVG(daily_dbus), 1) AS daily_avg,
  ROUND(AVG(daily_dbus) * DAY(LAST_DAY(current_date())), 1) AS projected_month_end,
  ROUND(AVG(daily_dbus) * DAY(LAST_DAY(current_date())) * 1.3, 1) AS projected_with_30pct_buffer
FROM daily;

This gives you a simple linear projection. If projected_month_end exceeds your budget threshold, it's time to tighten rate limits or investigate who's driving the spike.

Pillar 4: AI Guardrails (Safety, not just Cost)

Here’s a cost driver most teams miss: a user pastes an entire 50-page contract into the summarization agent, including sensitive client PII. The model dutifully processes all 80K tokens, generating a response that should never have been produced in the first place. Guardrails catch these cases before tokens are consumed: they block unsafe content, mask PII, and enforce organization-specific policies. The cost savings are a side effect of doing the right thing.

• Safety filtering (engine: Meta Llama Guard 2–8b): blocks unsafe/harmful content (violence, self-harm, hate speech)
• PII detection (engine: Presidio, US-scoped): detects/blocks/masks sensitive data (CC, email, phone, SSN, bank account)
• Custom policies (engine: configurable LLM judge): organization-specific rules (topic restrictions, tone, brand)

Configuration (UI)

Guardrails are configured per endpoint. DataCo enables all three on their summarization service:

Model Serving → endpoint → AI Guardrails tab
  → Safety: ON (input + output)
  → PII: Block (input) / Mask (output)
  → Custom: "Never discuss competitor products" → Block on input

Cost Impact

• Request blocked by guardrail = no output tokens generated = direct savings
• PII masking = request continues but with reduced tokens (masked field is shorter)
• ⚠️ The guardrails service itself uses Foundation Model APIs (pay-per-token) → there’s a small additional cost per check

Architectural Pattern: Defense in Depth for AI Cost

The four pillars above aren’t meant to work in isolation. They stack into a layered defense. Each layer catches what the previous one missed. Here’s how they fit together for DataCo’s summarization agent:

Setup Guide: Zero to Production in 4 Steps

Let’s put it all together for DataCo. Their summarization agent (dataco-summarizer endpoint, Claude Sonnet, 200 users) needs the full governance stack before Friday's launch.

Step 1: Enable Unity AI Gateway

Account Console → Previews → Unity AI Gateway → Enable

Step 2: Configure Rate Limits on Each Endpoint

Workspace → Model Serving → [endpoint] → Rate Limits
  Service: 50000 TPM (hard ceiling)
  User Default: 5000 TPM, 60 QPM
  Custom: service_principal/agent → 20000 TPM (agents need more headroom)

Step 3: Create Budget with Progressive Alerts

Account Console → Usage → Budgets → Add
  Resource: Unity AI Gateway
  Scope: tag "env=production"
  Threshold 1: $500 → Alert (early warning)
  Threshold 2: $1500 → Alert (escalation)
  Threshold 3: $3000 → Block usage (circuit breaker)
  Per-user: $50/month → Alert

Step 4: Activate Cost Observability

-- Verify access to system table
SELECT * FROM system.billing.usage
WHERE billing_origin_product = 'MODEL_SERVING'
  AND usage_date = current_date()
LIMIT 5;

-- If the query returns results: table is accessible
-- If error: request access from your account admin

Open the built-in dashboard: Unity AI Gateway → Govern → Usage Dashboard.

Alternative: Programmatic Setup via Python SDK

If you prefer infrastructure-as-code over clicking through UIs, the Databricks SDK exposes put_ai_gateway() for full programmatic configuration:

from databricks.sdk import WorkspaceClient
from databricks.sdk.service.serving import (
    AiGatewayRateLimit,
    AiGatewayRateLimitRenewalPeriod,
    AiGatewayRateLimitKey,
    AiGatewayGuardrails,
    AiGatewayGuardrailsConfig,
    AiGatewayUsageTrackingConfig,
)

w = WorkspaceClient()

w.serving_endpoints.put_ai_gateway(
    name="dataco-summarizer",
    rate_limits=[
        AiGatewayRateLimit(
            renewal_period=AiGatewayRateLimitRenewalPeriod.MINUTE,
            calls=60,
            tokens=100_000,
            key=AiGatewayRateLimitKey.USER,
        ),
    ],
    usage_tracking_config=AiGatewayUsageTrackingConfig(enabled=True),
)

This enables version-controlled, repeatable governance configuration across environments (dev/staging/prod).

Governing Agent Costs (The Hardest Problem)

Rate limits and budgets work well for human users: predictable request patterns, reasonable token counts, sessions that end when the browser closes. Agents are different. A coding agent like Claude Code or a multi-hop ReAct loop can make hundreds of LLM calls per session, autonomously, for hours. Without specific controls, a single agent session can burn through your entire monthly budget overnight.

Here’s how to apply the gateway’s tools specifically to agent workloads:

The Agent Cost Stack

1. Application level — max_iterations / recursion_limit in your agent code. Hard stop on reasoning loops. Set this in LangGraph (recursion_limit=25) or your custom ReAct loop.

2. Per-service-principal rate limit — QPM/TPM on the SP that runs the agent. Caps burst damage. An agent running as agent-sp@workspace gets its own ceiling, separate from human users.

3. Per-user budget — monthly DBU cap per identity. Even if rate limits allow bursts, the budget catches cumulative spend over days/weeks.

4. MLflow tracing — automatic token/latency capture per call. Visibility into what the agent is doing: how many tool calls per task, which calls are expensive, where loops happen.

5. Inference tables — full request/response logging. Post-hoc audit: did the agent hallucinate a tool call that wasted 50K tokens?

Practical Pattern: Circuit Breaker for Agents

DataCo’s summarization agent runs as service principal dataco-summarizer-sp. They configure:

from databricks.sdk import WorkspaceClient
from databricks.sdk.service.serving import (
    AiGatewayRateLimit,
    AiGatewayRateLimitRenewalPeriod,
    AiGatewayRateLimitKey,
)

w = WorkspaceClient()

w.serving_endpoints.put_ai_gateway(
    name="dataco-summarizer",
    rate_limits=[
        # Global service ceiling
        AiGatewayRateLimit(
            renewal_period=AiGatewayRateLimitRenewalPeriod.MINUTE,
            calls=200,
            tokens=500_000,
            key=AiGatewayRateLimitKey.ENDPOINT,
        ),
        # Agent SP: generous but bounded
        AiGatewayRateLimit(
            renewal_period=AiGatewayRateLimitRenewalPeriod.MINUTE,
            calls=100,
            tokens=200_000,
            key=AiGatewayRateLimitKey.SERVICE_PRINCIPAL,
            principal="dataco-summarizer-sp",
        ),
        # Human users: conservative
        AiGatewayRateLimit(
            renewal_period=AiGatewayRateLimitRenewalPeriod.MINUTE,
            calls=30,
            tokens=50_000,
            key=AiGatewayRateLimitKey.USER,
        ),
    ],
)

This creates a two-tier system: the agent gets 100 QPM (enough for multi-hop reasoning), while human users get 30 QPM (enough for interactive use). If the agent enters an infinite loop, it hits 100 QPM and gets HTTP 429. The recursion_limit in the application code is the first line of defense; the rate limit is the second.

Detecting Agent Loops via System Tables

-- Find service principals with suspiciously high request counts (potential loops)
SELECT
  requester,
  DATE_TRUNC('hour', event_time) AS hour,
  COUNT(*) AS requests,
  SUM(total_tokens) AS tokens,
  ROUND(SUM(total_tokens) / COUNT(*), 0) AS avg_tokens_per_call
FROM system.ai_gateway.usage
WHERE requester_type = 'SERVICE_PRINCIPAL'
  AND event_time >= current_timestamp() - INTERVAL 24 HOURS
GROUP BY requester, hour
HAVING requests > 200  -- threshold: more than 200 calls/hour is suspicious
ORDER BY requests DESC;

If this query returns results, you have an agent that's either looping or processing an unexpectedly large batch. Investigate the MLflow traces for that service principal to understand why.

Key Takeaway

For agents, the governance model shifts from “protect the user from the bill” to “protect the bill from the agent.” The combination of application-level max_iterations + gateway-level per-SP rate limits + budget alerts creates a triple safety net that catches loops at whatever layer they manifest.

Current Limitations (September 2026)

• Budget block usage: Genie budgets only; custom endpoints cannot be blocked automatically.
• External model spend: not included in standard budget; monitorable via system.ai_gateway.external_model_spend but no automatic enforcement.
• Provisioned throughput: not tracked by Unity AI Gateway budgets.
• PII detection scope: US categories only (SSN, US phone format); does not natively cover GDPR European PII.
• Rate limit precision: burst behavior allowed for low latency (enforcement not instantaneous).
• Guardrails availability: limited to regions supporting Foundation Model APIs pay-per-token.
• Budget refresh: alerts based on near real-time; system tables update every few hours.
• Custom policies: require LLM judge → additional cost per call.

When NOT to use Unity AI Gateway

• Rapid prototyping: in sandbox/dev, costs are negligible. Overly aggressive rate limits slow down development.
• Massive batch inference: for scheduled batch jobs, budget is predictable. Provisioned Throughput is better (not tracked by the gateway).
• Single-model internal endpoint: if one team uses one model, costs are already isolated. The gateway adds complexity without value.

The gateway delivers maximum ROI when: many users/agents × many models/endpoints × unpredictable costs.

How It Compares: Unity AI Gateway vs. Alternatives

If your stack isn’t 100% Databricks, you may be evaluating external AI gateway solutions. Here’s how they compare:

Unity AI Gateway (Databricks)

• Rate limiting: QPM + TPM, per-user/group/service
• Budget control: budgets with block + alerts, UC tag scoping
• Observability: system tables (SQL), built-in dashboard
• Guardrails: safety + PII + custom policies (LLM judge)
• Routing: traffic splitting + fallback across endpoints
• Auth: Unity Catalog (SSO, RBAC, audit logs)
• Data residency: runs in your Databricks workspace (same region)
• Integration: native with Genie, MLflow, Agent Framework, MCP
• Pricing: included in Databricks (no extra gateway fee)

LiteLLM Proxy (open source / MLflow AI Gateway)

• Rate limiting: QPM + TPM, per-key
• Budget control: budget alerts (webhook) + reject (HTTP 429)
• Observability: custom DB + Grafana
• Guardrails: via callbacks/plugins
• Routing: load balancing + fallback
• Auth: API key management
• Data residency: self-hosted or cloud
• Integration: SDK wrapper
• Pricing: open source (self-host cost)

Helicone (SaaS)

• Rate limiting: per-key, per-org
• Budget control: alerts only
• Observability: web dashboard + API
• Guardrails: content filtering
• Routing: proxy routing
• Auth: API key + org
• Data residency: US/EU cloud
• Integration: SDK wrapper
• Pricing: per-request

Portkey.ai (SaaS)

• Rate limiting: per-key, per-org
• Budget control: alerts + soft limits
• Observability: web dashboard + API
• Guardrails: guardrails + fallback
• Routing: gateway routing
• Auth: API key + org
• Data residency: multi-region cloud
• Integration: SDK wrapper
• Pricing: per-request

When to choose Unity AI Gateway:

• You’re already on Databricks and want zero-friction governance
• You need UC-level RBAC and audit trails (compliance)
• Your agents use Genie, MCP servers, or Mosaic AI endpoints
• You want cost data in the same SQL engine as your analytics

When to consider alternatives:

• Multi-cloud with no Databricks commitment
• Need gateway features today at GA quality (Unity AI Gateway is Beta)
• Primary models are external (OpenAI, Anthropic) without Databricks endpoints
• Need real-time cost dashboards with sub-minute granularity (system tables lag by hours)

Final Thought

Let’s return to DataCo’s Monday morning. With Unity AI Gateway in place, here’s what would have happened instead:

1. The agent’s retry loop hits 60 QPM per user → rate limiter kicks in at request #61. HTTP 429, exponential backoff. The loop self-heals.
2. Even if it doesn’t, the $500 shared budget threshold fires at 2 AM → platform team gets paged. They kill the endpoint in minutes, not hours.
3. Monday morning, the platform lead runs SELECT ... FROM system.ai_gateway.usage WHERE requester = 'support-agent-sp' → sees exactly which service principal, which model, how many tokens. Root cause in one query.

Total damage: $500 instead of $12,400. One Monday ruined vs. one Microsoft Teams notification at 2 AM.

The tooling isn’t exotic. It’s four configuration steps, five SQL queries, and one architectural principle: defense in depth. Rate limits prevent the spike. Budgets cap the damage. Observability explains what happened. Guardrails prevent what shouldn’t happen at all.

The biggest risk in production AI isn’t a bad model. It’s a good model with no off switch.

12. References

• Unity AI Gateway overview
• Configure rate limits
• Manage budgets for Unity AI Gateway
• Monitor Unity AI Gateway cost
• AI Guardrails configuration
• Create and monitor budgets (Account Console)
• system.billing.usage reference

Note: This article is based on official Databricks documentation (September 2026). Unity AI Gateway is in Beta; features and limits may change before GA. SQL queries on system tables are tested on workspaces with access enabled to system.billing.usage.

Unity AI Gateway: Governing AI Costs in Production was originally published in Data Reply IT | DataTech on Medium, where people are continuing the conversation by highlighting and responding to this story.

Reinforcement Learning (RL) is a machine learning approach designed to solve sequential decision problems in complex environments, where the consequences of actions are not immediately clear. Unlike supervised learning, which requires an explicit target variable, in RL an agent (a software, a robot, or an autonomous vehicle) learns optimal strategies autonomously through direct interaction with the environment, observing feedback in the form of numerical rewards and progressively refining its decisions to maximize cumulative rewards over time.

Among the different RL algorithms, Q-learning is one of the most well-known and widely used. However, its direct application is limited when dealing with large or continuous state and action spaces, or when direct interaction with the environment is not possible. In these scenarios, Fitted Q-Iteration (FQI) comes into play: an offline approach where the agent learns from a pre-existing dataset and determines the best decision rule (policy) at each time step.

In this article, FQI is implemented on the House Heating dataset with the goal of finding optimal actions to control the internal temperature of a building over 60 days, balancing comfort with cost minimization.

Reinforcement Learning

At each time step t, the agent receives a state St ∈ S and the reward Rt ∈ R from the previous step, and decides the best action At ∈ A to take toward the next state St+1. This process repeats recursively for a predetermined number of iterations or until convergence, with the agent adjusting its actions to maximize rewards both at each step and over the long term.

There are several categories of models within reinforcement learning depending on their methodological characteristics.

Offline vs Online Methods
RL algorithms differ based on how they acquire data. Offline methods operate on pre-existing datasets of finite size, while online methods receive data progressively through continuous agent-environment interaction.

Model-based vs Model-free Methods
Model-based methods define or assume a known model of the environment (transition probabilities and reward function) to simulate future scenarios and plan actions. Model-free methods learn directly from experience without building an explicit model of the environment.

Value-based vs Policy-based Methods
Model-free methods further divide into value-based models, where the agent estimates the value function V(s) or the Action-Value function Q(s,a) (example: Q-learning), and policy-based models, where the agent learns the policy π directly (deterministic or stochastic).

In all cases, the objective is to estimate a function that determines optimal actions to maximize cumulative rewards. Since future reward values cannot be known in advance, the Value function V(s) is defined to estimate the expected future rewards from a state s under a given policy π. A policy π(a|s) represents the probability of choosing action At = a when in state St = s, and the value function vπ(s) expresses the expected cumulative rewards starting from state s following policy π.

Another measure of the agent’s behavior quality is the Action-Value function Q(s,a), which represents how good it is to take action a in state s while following policy π.

For every policy and state, both functions (Value function and Action-Value function) can be expressed recursively through the Bellman equation, which relates the value of a state to the values of subsequent states weighted by their transition probabilities.

Finding an optimal policy means finding π(a|s) that yields high rewards both at each step and over the long term. The optimal Action-Value function Q*(s,a) is the one associated with the optimal policy π*, that is, the best among all possible policies:

Q-learning and Fitted Q-Iteration

The key idea of RL is that an agent learns on its own without a preset model dictating correct actions. Temporal-Difference (TD) learning realizes this: the agent learns from experience by continuously comparing predictions with observed results, improving iteratively without predefined instructions. TD algorithms exploit the trade-off between exploration (trying new actions) and exploitation (using known good actions).

TD methods distinguish between two approaches: on-policy, where the policy used to collect data is the same as the one being optimized, and off-policy, where these two policies are different. In the off-policy context, Q-learning is the fundamental algorithm, from which Fitted Q-Iteration is derived. FQI is an offline, batch algorithm that estimates Q(s,a) from a fixed dataset of (s, a, s’, r) tuples, where s is the current state, a is the action taken, s’ is the next state, and r is the observed reward, without actively interacting with the environment.

FQI approximates the Q-function using regression methods (like random forests, generalized additive models, neural networks, or other machine learning models) and derives a control policy as close as possible to the optimal one. When state and action spaces are finite and small, Q can be represented in tabular form. In continuous or very large spaces, FQI uses parametric or non-parametric approximators, solving sequential regression problems at each iteration to iteratively refine the Q-function estimate. This batch approach enables FQI to operate in complex environments while maintaining computational efficiency.

The FQI algorithm requires four inputs: the discount factor γ, the number of iterations K, a regression model class F, and the dataset (s, a, s’, r) observed, for each time step t = 1,…,Te and episode e = 1,…,E. It initializes Q₀ = 0 for every state-action pair, then repeats for K iterations: at each step a new target variable is computed as the sum of the observed reward and the maximized Q-value over optimal actions, and a regression model is fitted by minimizing the squared error.

Application

The dataset was generated following a safe-by-design policy, where actions derive from a predefined policy aimed at maximizing system safety. The goal is to use FQI to identify an optimal policy capable of both respecting safety constraints and maximizing rewards. The data represent the control of a building heating system, illustrated below.

The safe-by-design policy receives the current internal temperature st (state) as input and sets the radiator temperature at (action) accordingly. The action choice is not influenced by the observed reward r. The system is also affected by an uncertain external temperature ξt, which varies throughout the day.

The safety constraints to be respected are:

• keep the internal temperature as close as possible to 18°C within 30 minutes of turning on the radiator;
• not exceed 19°C in the first 30 minutes;
• maintain internal temperature between 17.5°C and 18.5°C.

The operational constraint limits actions to a range of 0°C to 70°C.

The dataset provides observations every 30 minutes over 60 days, recording the internal temperature st, the action at taken according to the safe-by-design policy, and heating costs calculated by multiplying the action by the hourly price. The 60 days correspond to 60 independent episodes, where the state at the start of each day is independent from the previous day’s final state. The objective is to minimize total costs:

The reward to be maximized is defined as the sum of two components: the negative heating cost, which directly incentivizes cost reduction, and a penalty term weighted by λ that prevents the algorithm from selecting actions that would compromise the internal temperature comfort. The penalty weight λ was set to the optimal value of 600.

Since FQI requires a regression model to estimate the target values at each iteration, several regression models were trained on the training set (40 days) and evaluated on the test set (20 days): linear regression, additive model with thin plate spline regression, random forest, and neural networks. Each model was evaluated by comparing the cost reduction achieved under the new policy against those incurred by the safe-by-design policy.

The table below presents the results, showing temperature quantiles achieved under the new policy and the number of constraint violations outside [17.5°; 18.5°] with each regression model.

The graph below shows the internal temperature trend over the 20 test days for the thin plate spline model. The downward peaks correspond to the start of each day. From the second state onward, the model brings temperature within safety constraints and maintains it relatively stable throughout the day. Only one violation above 18.5°C is observed on the second test day. This model shows greater temperature variability compared to others, which enables lower operating costs.

In evaluating the models, besides analyzing state values, the costs generated by each model’s actions were calculated across the 20 test days. The table shows the difference in total costs between the new policy and the safe-by-design policy for each regression model.

The model that best balances constraint satisfaction and cost reduction is the additive model with thin plate spline regression, which delivers lower costs compared to all other models.

This model was applied to the full 60-day dataset, achieving very satisfactory temperature levels with no further anomalies and a total cost reduction of −€4,547.75 compared to the safe-by-design policy.

Conclusion

This work applied the Fitted Q-Iteration algorithm with several regression models to estimate the Action-Value function on the House Heating dataset. Among the models tested, the generalized additive model with thin plate spline achieved the best performance, capturing non-linear relationships between variables while respecting temperature constraints and reducing operational costs. FQI proves to be as a robust approach for complex control problems, where choosing the right regression model is key to achieving reliable and efficient solutions.

References

Sutton, R. S., & Barto, A. G. (2018). Reinforcement Learning: An Introduction. MIT Press.

Ernst, D., Geurts, P., & Wehenkel, L. (2005). Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6, 503–556.

Puterman, M. L. (2014). Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons.

Hastie, T., Tibshirani, R., & Friedman, J. (2009). The Elements of Statistical Learning. Springer.

Agarwal, A., Jiang, N., Kakade, S. M., & Sun, W. (2019). Reinforcement learning: Theory and algorithms. University of Washington.

Munos, R. (2005). Error bounds for approximate value iteration. Proceedings of AAAI, 2.

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An Introduction to Statistical Learning. Springer.

Columbia University Mailman School of Public Health (2011). Thin plate spline regression.

Reinforcement learning for dynamic systems: application of Fitted Q-Iteration methods ​ was originally published in Data Reply IT | DataTech on Medium, where people are continuing the conversation by highlighting and responding to this story.

Database credentials in CI/CD pipelines are still surprisingly common, even in mature AWS setups. In our case, the issue was not just where the credentials lived, but the fact that the same system, GitLab CI, was being used to orchestrate infrastructure, manage database state, and distribute identity information.
This led to an unnecessary coupling between Terraform, database bootstrap logic, and secret management, making the system harder to reason about, harder to secure, and harder to evolve.

The solution was not a better way to pass secrets, but a clearer separation of responsibilities across three domains: infrastructure provisioning, database state, and identity management in AWS Aurora.

In this article, we will intentionally separate the conceptual model from the practical implementation.
In the first section, we will focus on the architectural principles behind the separation of infrastructure, database state, and identity in AWS Aurora. In the second section, we will move to the actual configuration, showing how these ideas are translated into Terraform, SQL-based bootstrap scripts, and the use of AWS IAM and Secrets Manager in a real-world setup.

Separate Infrastructure and DB State

“Infrastructure as Code does not mean embedding your database logic into Terraform.”

Infrastructure as Code is responsible for creating the database cluster, while database configuration remains expressed in standard SQL. This separation provides several key benefits:

• Portability: the same database can be recreated and configured consistently across different cloud providers. Migration costs are significantly reduced when all setup logic (DDL, roles, triggers, and permissions ) is centralized, standardized, and version-controlled.
• Reduced coupling: decoupling Terraform from database-specific logic avoids tight integration with a specific engine and reduces reliance on provider-specific abstractions.
• Clear lifecycle separation: infrastructure tends to evolve slowly, while database state (schemas, roles, grants, and migrations) changes more frequently. Keeping them separate reflects their natural change cadence and reduces friction in both domains.
• Faster onboarding: SQL is a universal language in the data ecosystem. Keeping database configuration in SQL improves readability and allows engineers to understand and contribute quickly.

In summary, treating infrastructure provisioning and database state as separate concerns leads to a significantly cleaner and more maintainable system design. Terraform should focus on provisioning reliable and repeatable infrastructure, while SQL remains the source of truth for how data structures, roles, and permissions are defined and evolved.

Separate Infrastructure and Identity

“CI/CD pipelines should orchestrate infrastructure, not become long-lived secret vaults.”

Managing production database credentials directly through CI/CD variables introduces an unnecessary and often overlooked risk surface. While it may appear convenient, it effectively turns the pipeline into a persistent distribution point for sensitive identity information, increasing blast radius and complicating access control, rotation, and auditability.

We aim to achieve a model where CI/CD systems only assume temporary permissions to orchestrate infrastructure changes, without ever becoming a storage layer for sensitive database credentials.

To do so, on one side, the CI/CD pipeline is granted only ephemeral credentials, typically by assuming an IAM role at runtime. In some setups, even when the runner itself operates with elevated permissions, access is still mediated through temporary credentials and tightly controlled roles, with secrets retrieved dynamically from a managed secret store and continuously rotated.

On the other side, database access for applications is handled through dedicated technical users with least-privilege permissions. By leveraging IAM authentication mechanisms, these users can be defined without static passwords, meaning they cannot be accessed through traditional credential-based authentication at all. Access is enforced exclusively via IAM-issued authentication tokens, ensuring that neither the CI/CD pipeline nor application services rely on long-lived database credentials.

This approach eliminates the need to manage persistent superuser credentials in CI/CD, while also removing static credentials for all application-level database access, significantly reducing overall credential sprawl and operational risk.

Practical implementation

Repository structure

A clean separation of concerns starts at the repository level. The infrastructure, database bootstrap logic, and CI pipelines are explicitly isolated to reflect their different responsibilities:

terraform/
  └── main.tf
  └── output.tf
  └── variables.tf

sql/
  └── setup/
      ├── 00_create_schema.sql
      ├── 01_create_extensions.sql
      ├── 02_ddl.sql
      └── 03_grant.sql

pipelines/
  ├── infra.yml
  └── db_setup.yml

.gitlab-ci.yml

This structure makes the system intent explicit:

• Terraform is responsible for provisioning infrastructure components such as the Aurora cluster and networking configuration.
• SQL scripts define the database state in a portable, engine-agnostic way.
• CI pipelines orchestrate execution without embedding domain logic.

Gitlab CI Pipeline

The CI/CD flow is intentionally minimal and divided into two distinct stages, which will be illustrated in the accompanying diagram:

Infrastructure stage

This example is kept simple, but in a real production environment you would need to consider that the pipeline should be extended with a proper remote state backend and locking mechanism, together with IAM-based authentication and managed secret retrieval.

dev-plan:
  stage: infra
  script:
    - cd terraform/infra
    - terraform init
    - terraform plan -out=tfplan
  artifacts:
    paths:
      - terraform/infra/tfplan


dev-apply:
  stage: infra
  script:
    - cd terraform/infra
    - terraform init
    - terraform apply -auto-approve tfplan
    - terraform output -json > terraform_outputs.json
  artifacts:
    paths:
      - terraform/infra/terraform_outputs.json
  needs:
    - dev-plan
  when: manual

Database setup stage

This stage performs the database bootstrap after infrastructure provisioning is completed. It retrieves all required connection parameters from Terraform outputs, fetches the database credentials from AWS Secrets Manager, and executes the SQL setup scripts in a deterministic order to initialize and configure the database state.

db_setup_dev:
  stage: db-setup
  needs:
    - job: dev-apply
      artifacts: true
  script:
    - echo "Starting database setup"

    - export PGHOST=$(jq -r '.aurora.value.endpoint' terraform/infra/terraform_outputs.json)
    - export PGDATABASE=$(jq -r '.aurora.value.database_name' terraform/infra/terraform_outputs.json)
    - export DB_MASTER_USER_SECRET_ARN=$(jq -r '.aurora.value.master_user_secret_arn' terraform/infra/terraform_outputs.json)

    - echo "Retrieving credentials from AWS Secrets Manager"
    - SECRET_JSON=$(aws secretsmanager get-secret-value --secret-id "$DB_MASTER_USER_SECRET_ARN" --query SecretString --output text)
    - export PGUSER=$(echo $SECRET_JSON | jq -r '.username') 
    - export PGPASSWORD=$(echo $SECRET_JSON | jq -r '.password')
    - export PGPORT=5432

    - echo "Executing SQL bootstrap scripts"
    - |
      for script in $(find sql/setup -type f -name "*.sql" | sort); do
        echo "Executing: $script"
        psql -v ON_ERROR_STOP=1 -f "$script"
      done

NOTE: Each SQL script is designed to be safely re-runnable, ensuring that the pipeline remains stateless and resilient to partial failures or re-executions.

The CI pipeline does not define database configuration. It only consumes the outputs of the infrastructure layer.

Terraform configuration

This module provisions the core Aurora PostgreSQL cluster and its supporting infrastructure with a strict focus on infrastructure-level concerns only.

resource "aws_db_subnet_group" "this" {
  name       = var.cluster_identifier
  subnet_ids = var.subnet_ids
}

resource "aws_rds_cluster_parameter_group" "this" {
  name        = var.cluster_identifier
  family      = "aurora-postgresql${var.engine_version}"
  description = "Cluster parameter group for ${var.cluster_identifier}"
}

resource "aws_rds_cluster" "this" {
  cluster_identifier              = var.cluster_identifier
  engine                          = "aurora-postgresql"
  engine_version                  = var.engine_version
  database_name                   = var.database_name
  master_username                 = var.master_username
  manage_master_user_password     = true
  iam_database_authentication_enabled = true
  storage_encrypted              = true
  vpc_security_group_ids         = var.security_group_ids
  db_subnet_group_name           = aws_db_subnet_group.this.name
  db_cluster_parameter_group_name = aws_rds_cluster_parameter_group.this.name
}

resource "aws_rds_cluster_instance" "this" {
  identifier         = "${var.cluster_identifier}-instance"
  cluster_identifier = aws_rds_cluster.this.id
  instance_class     = var.instance_class
}

A master username is still required by Aurora; however, by enabling manage_master_user_password, the actual master password is fully managed and rotated by AWS rather than being defined, stored, or handled by the CI/CD pipeline.

This design ensures that Terraform remains responsible for provisioning the database infrastructure, while credential lifecycle management is delegated entirely to AWS services, reducing secret exposure and eliminating long-lived administrative credentials in the pipeline.
To make the different components work together, Terraform is used as the single source of truth for all runtime configuration required by the CI pipeline, exposing only the necessary outputs for connectivity and secret resolution. The CI/CD layer then consumes these outputs to orchestrate the database bootstrap process without introducing any hardcoded infrastructure or credential dependencies.

output "aurora" {
  description = "Aurora cluster connection and bootstrap metadata"
  value = {
    endpoint              = aws_rds_cluster.this.endpoint
    database_name         = aws_rds_cluster.this.database_name
    master_user_secret_arn = aws_rds_cluster.this.master_user_secret[0].secret_arn
  }
}

To make the example work, the following variable definition can be used. Please remember to check if this configuration suits your architectural needs before applying it.

variable "cluster_identifier" {
  description = "Identifier for the Aurora cluster"
  type        = string
  default     = "app-aurora-cluster"
}

variable "engine_version" {
  description = "Aurora PostgreSQL engine version"
  type        = string
  default     = "17"
}

variable "instance_class" {
  description = "Instance class for Aurora cluster instances"
  type        = string
  default     = "db.t4g.medium"
}

variable "database_name" {
  description = "Initial database name"
  type        = string
  default     = "appdb"
}

variable "master_username" {
  description = "Master username for the Aurora cluster"
  type        = string
  default     = "admin"
}


variable "subnet_ids" {
  description = "Subnets for the DB subnet group"
  type        = list(string)
  default     = []
}

variable "security_group_ids" {
  description = "Security groups attached to the cluster"
  type        = list(string)
  default     = []
}

SQL configuration

In this section we introduce the actual SQL bootstrap scripts that define the database state. The focus here is on simplicity and repeatability: each script is designed to be idempotent, meaning it can be safely executed multiple times without side effects, ensuring predictable behaviour across deployments.

-- 00_create_schema.sql
CREATE SCHEMA IF NOT EXISTS app;

-- 01_create_extensions.sql
CREATE EXTENSION IF NOT EXISTS ltree;

-- 02_ddl.sql
CREATE TABLE IF NOT EXISTS app.users (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    email TEXT NOT NULL UNIQUE,
    created_at TIMESTAMP DEFAULT now()
);

-- 03_grant.sql
---- Create a technical role (if not exists)
DO $$
BEGIN
    IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname = 'app_user') THEN
        CREATE USER app_user;
    END IF;
END
$$;

---- Grant permissions on schema and tables
GRANT USAGE ON SCHEMA app TO app_user;
GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA app TO app_user;

---- Ensure future tables are also covered
ALTER DEFAULT PRIVILEGES IN SCHEMA app
GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO app_user;

---- Integration with IAM
GRANT rds_iam TO app_user;

IAM Authentication

Now, the database is fully defined and provisioned inside a centralized infrastructure repository that owns both the provisioning layer and the database state lifecycle.

On top of this, any application that needs to access the database does not receive credentials directly. Instead, it is granted access by attaching an explicit AWS IAM policy to its execution role, allowing it to generate short-lived authentication tokens for the corresponding database identity (e.g. app_user).

data "aws_iam_policy_document" "aurora_iam_auth" {
  statement {
    effect = "Allow"

    actions = [
      "rds-db:connect"
    ]

    resources = [
      "arn:aws:rds-db:${var.region}:${var.account_id}:dbuser:${aws_rds_cluster.this.cluster_resource_id}/app_user"
    ]
  }

This effectively extends the identity boundary from the database layer to the cloud identity layer. Access is no longer managed through shared secrets or connection strings, but through clearly defined IAM roles and policies that are fully auditable and scoped to a single purpose. Each application becomes responsible for its own identity, while the database remains agnostic to how that identity is provisioned outside of the rds_iam mapping.

The result is a model where infrastructure, database state, and identity are cleanly separated: the repository owns the system definition, Terraform provisions the infrastructure, SQL defines the persistent state, and IAM governs access.

Trade-offs and Known Limitations

Every architecture carries costs. This model introduces some complexity that may or may not be justified depending on your system’s scale, team structure, and security requirements. Understanding these trade-offs before committing helps you decide whether the benefits outweigh the overhead.

• Connection rate limit : IAM authentication on Aurora supports approximately 256 new connections per second per cluster. For sustained workloads behind a connection pool this is rarely a constraint, but some architectural patterns can reach this ceiling;
• Token refresh: IAM auth tokens expire after 15 minutes, so your application or connection driver must handle renewal. Some drivers support transparent refresh, but others require explicit retry and renewal logic in your connection layer;
• Idempotent SQL discipline: Not all DDL is safely re-runnable. An ALTER TABLE that adds a column succeeds once; the second run fails unless wrapped in conditional logic. Data migrations require similar care. Establish conventions and guards (IF NOT EXISTS, version checks) to keep your bootstrap reliably repeatable.

Wrap-up and Key Takeaways

Separating infrastructure provisioning from database state management adds a moving part, but in practice it reflects the reality that these two lifecycles move at different speeds, often owned by different teams, and governed by different change-control rules.
Treating them as one monolithic concern tends to create friction once an organization grows past a single squad.

IAM authentication removes long-lived credentials from the picture, replacing shared secrets with short-lived, auditable tokens that prove identity without ever touching a password vault.
CI/CD pipelines then act as orchestrators, coordinating provisioning and deployment without storing or transmitting secrets that could leak.

Together these choices give you a system where every action is attributable, every permission is scoped, and every layer can evolve independently. In our experience, starting with this foundation early is far easier than retrofitting security into a system that wasn’t designed for it.

What’s Next

The implementation shown is intentionally scoped to illustrate the core separation model. A production-grade setup would naturally extend it in several directions:

• Schema migration tooling: Adding a dedicated CI stage with Flyway or Liquibase for incremental schema evolution. It plugs naturally into the same architecture: same Terraform outputs, same Secrets Manager retrieval, no new static credentials. The bootstrap establishes the initial state; the migration tool manages evolution over time.
• Remote state and locking: The Terraform setup shown uses local state for simplicity. In production, a remote backend is essential to avoid drift and enable team collaboration.
• Multi-environment promotion: The pipeline shown targets a single environment. Extending it to dev → staging → prod with environment-specific variable files and approval gates is a natural next step.
• Connection pooling with IAM auth: IAM authentication tokens expire after 15 minutes. For high-throughput applications, integrating RDS proxy or a similar connection pooler becomes critical to avoid constant re-authentication overhead and connection churn.

Stop Mixing Infrastructure, Database State, and Identity in AWS Aurora was originally published in Data Reply IT | DataTech on Medium, where people are continuing the conversation by highlighting and responding to this story.