<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Gregory Zem on Medium]]></title>
        <description><![CDATA[Stories by Gregory Zem on Medium]]></description>
        <link>https://medium.com/@mne?source=rss-4f908fc82f16------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Rm3ILE7wHPT-JPUWJzc3UQ.png</url>
            <title>Stories by Gregory Zem on Medium</title>
            <link>https://medium.com/@mne?source=rss-4f908fc82f16------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 06:51:04 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mne/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com" rel="hub"/>
        <item>
            <title><![CDATA[Stop Adapting to Your Tools. Build Your Own.]]></title>
            <link>https://medium.com/@mne/stop-adapting-to-your-tools-build-your-own-483a9a6aa6be?source=rss-4f908fc82f16------2</link>
            <guid isPermaLink="false">https://medium.com/p/483a9a6aa6be</guid>
            <dc:creator><![CDATA[Gregory Zem]]></dc:creator>
            <pubDate>Sat, 31 Oct 2026 19:25:15 GMT</pubDate>
            <atom:updated>2026-10-31T19:25:15.139Z</atom:updated>
            <content:encoded><![CDATA[<p>Three weeks ago I stopped using the terminal. Deleted Warp since I no longer needed it, and kicked iTerm2 off my Dock. I now work with agents entirely in a browser window (same experience on my phone, iPad and desktop). Living in that browser is a full-blown agentic IDE built around my workflows. It grows and evolves along with me, and basically reflects my current skill level when it comes to orchestrating agent sessions.</p><p>Right now I have about 30 sessions open across different projects. Some are running, some have finished and are waiting for me to look at them. My dev process doesn’t involve an autonomous swarm of a hundred agents talking to each other, because I can’t do that yet, and that’s not how I work. But juggling 30 sessions across 8 projects at once? That I can do now. Getting there took three weeks of reworking my IDE, because everything was off, and I didn’t have the convenience or the customization I actually needed. Out of the box, neither the terminal nor the Claude / Codex apps let you work with sessions efficiently, especially when the work is iterative: kick off a long session, review the results, call out what’s broken, kick off the next stage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0eUasjEIvRQNcGgU-HiPyw.png" /></figure><p>Here are a few examples of where the shoe used to pinch:</p><ul><li>An agent can spend five hours building a new feature and then, buried in a mountain of bash logs, drop something like “oh yeah, I found a bug here we should fix later” and go right back to spamming the console. So I’d never see those messages (I don’t watch the chat for five hours; I only read the final report), and the agent doesn’t always bring it up again at the end. So it just gets lost. In my IDE, these things get written to special session notes (Reminders/Bookmarks) as the session runs. I can open them anytime and take a look, even while the session is still running.</li><li>An agent can spend five hours building a new feature with no indication of when it’ll be done or how much is finished, let alone what its plan is and which step it’s on right now. I used to ping it every so often: “dude, where are you with this? What’s left?” Now I have a button that opens the plan, and the percent complete shows in the corner (not always accurate, but better than nothing). The agent fills it in and updates it in real time on its own. Built as a separate MCP.</li><li>When an agent urgently needs a decision from me in one of the 30 sessions, the notification pops up in whatever session I’m in, not just the original one. I click it, jump straight there, answer, and unblock the agent so it can keep working. Before, that decision would sit there until I happened to open the stuck session to check how things turned out. And often, it hadn’t even started yet.</li><li>When an agent finishes and dumps a wall of text with the session results, I usually have questions about specific paragraphs or items. I used to copy-paste them into the chat and add “what’s this?”, “why this?”, “this absolutely has to get done”, etc. Now there’s an “Ask” button for that: I highlight a chunk of text, hit Ask, type my comment into a popup, and it lands in the chat automatically as a numbered quote (numbered continuously within the session), so I can always refer back to it further down. Picking apart results with questions is now my favorite thing to do; it’s become super convenient.</li><li>Sometimes, as a session runs, errors or other things pop up that I’d like to come back to later, think through, or test. I used to keep notes for every session in Sublime, which turns into madness at 30 sessions. Now I highlight a chunk of text and hit “Bookmark.” It remembers the spot in the chat, marks it on the scrollbar, and adds it to my notes right there inside the session. When I get around to it, I open my bookmarks, go through all that “technical” and “cognitive debt,” and either resolve it or use it as a TODO list.</li><li>I have several dev servers, each with projects in progress and a tmux server, and on each one I want to browse files on disk, paste images into the chat, and so on. Pasting images and other binary data from the clipboard into chat was always a pain with tmux on a remote server. I never managed to get it working in iTerm2 or Warp. Now it works the same with local and remote servers. I grab a couple of folders in Finder, drag them into the chat, and they automatically fly over to the session on the remote server with all their contents, and the agent there starts working with those files and folders right away. Magic!</li><li>I work with all sorts of formats: json, pdf, tsv, midi, au, mov. To listen to or look at them, I had to leave the terminal and open them via Finder or in dedicated software. Because you want to see json as a tree, and csv and tsv as tables. Now that’s part of the IDE: I click a .mid file in the built-in file explorer and it plays with decent GM sounds from the built-in soundbank. And inside the chats, all tables and markdown render right away into something clean and readable.</li><li>Once I had 12 sessions open across different projects and my computer rebooted. I had to restore the session IDs by hand. Now sessions are snapshotted automatically, and after a reboot everything stays right where it was. Plus, I made it save the last 300 lines of every terminal, so you can see how everything ended instead of sitting there smiling at a blank bash prompt.</li></ul><p>The thing I built for myself is called <a href="https://terminum.app"><strong>terminum</strong></a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9l96Ddv59Dm0n-7o_tUZPg.png" /></figure><p>It’s open source and gets updated every day. Whenever something starts to feel inconvenient, I open a Claude session right inside it, ask it to fix the problem, and ship a new release right away (2 hours ago I shipped another one, release #31).</p><p><a href="https://github.com/extractumio/terminum/">https://github.com/extractumio/terminum/</a></p><p>— <br> I think every developer can build a work tool that fits like a comfy pair of slippers and makes them 10x more productive. Instead of getting used to other people’s products, bending to fit them, and waiting for updates and fixes, you bend the products to fit your own workflows and habits. I’m sure you can build your own IDE too, or, worst case, fork mine and tweak it to your liking. And terminum will keep growing and evolving along with me, no matter what.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=483a9a6aa6be" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What We Pay for 1M of Context]]></title>
            <link>https://medium.com/@mne/what-we-pay-for-1m-of-context-6d7a16f42d31?source=rss-4f908fc82f16------2</link>
            <guid isPermaLink="false">https://medium.com/p/6d7a16f42d31</guid>
            <dc:creator><![CDATA[Gregory Zem]]></dc:creator>
            <pubDate>Wed, 29 Apr 2026 13:40:42 GMT</pubDate>
            <atom:updated>2026-04-29T13:40:42.876Z</atom:updated>
            <content:encoded><![CDATA[<p>April brought five flagship open-weights models, and behind all the benchmark noise, a far more interesting story got lost — how these models actually handle long context (while optimizing compute and memory), and what users end up paying for it. I want to unpack this a bit, pop-sci style.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NJohChlFmoOTHtsfHBvcwQ.png" /></figure><p>As you probably know, classical transformer attention requires every token to look at every other token. At a million tokens, this gets insanely expensive both in compute and in memory for the KV cache. So vendors are experimenting with model and cache architectures to cut down on resources. And the latest models pack a whole bunch of optimization tricks.</p><ul><li><strong>DeepSeek V4</strong> bets on two-level sparse attention: Compressed + Heavily Compressed. The advertised savings are 27% FLOPs and 10% KV cache versus V3.2 at one million tokens of context. But you pay for it: a risk of missing a rare but important connection between distant tokens.</li><li><strong>Qwen 3.6–27B</strong> goes the hybrid route. 64 layers organized in a 3:1 pattern — three layers of linear Gated DeltaNet, one layer of full Gated Attention, repeated 16 times. The linear layers are fast and cheap, but lose facts over long distances. The full layers (every 4th) serve as synchronization points. Works on short and medium context, sags on tasks involving precise fact retrieval from the middle of the window (the famous Lost-In-The-Middle).</li><li><strong>GLM-5.1</strong> builds its architecture on MLA: for every query it selects the top-2048 relevant keys from the entire context (up to 200K), and MLA attention only runs over this selection. MLA quality is preserved within the chosen window, the savings come from not wasting attention on the obviously irrelevant. The architecture’s quality is bottlenecked by the indexer’s quality. If it’s trained poorly — it skips important tokens, and the model loses the thread to critical information somewhere far away in the context. It’s a black box: not very clear what types of dependencies the indexer might miss. Especially given that a sparse attention mask is applied after the indexer, which lowers precision further.</li><li><strong>Kimi K2.6</strong> bets purely on MLA. Attention stays full, only memory is saved through compressing keys and values into a latent representation. Quality barely suffers, but compute stays expensive.</li><li><strong>MiniMax M2</strong> — after experimenting with a hybrid sliding window, they rolled back to classical full attention. This is the only team that publicly admitted the reversal.</li></ul><p>For short tasks (chat, one-off queries, simple code) the difference between these approaches is barely noticeable. They all cope. But for agentic scenarios the error profile becomes critical. A multi-turn agent holds in its context the history of tool calls, files it has read, past reasoning/history, intermediate plans. After twenty or thirty turns this is hundreds of thousands of tokens. And where the architecture loses attention to facts, the agent starts to “forget” what it has already done, repeats steps, loses the thread to the original task. People on Reddit have already been complaining about Qwen 3.6–35B-A3B: on long multi-step tasks the model “gets lost as the number of steps grows.”</p><p>Hence the practical takeaway. Model size in billions of parameters is a poor standalone criterion for choosing. 27B with attention that fits the task can work better than 400B with the wrong kind. For one-shot code generation or summarization — a hybrid saves money without a noticeable quality hit. For deep agentic loops with long history — look toward full attention or MLA. For precise fact retrieval inside a giant document — sparse approaches can let you down where MLA will hold up.</p><p>It’s a pity we don’t know the architecture of ChatGPT and Claude, because then we could roughly estimate whether they can be trusted with work on critical data. And another practical takeaway — don’t bloat your context and don’t aim for that 1M. It’s pure marketing. Remember that most benchmarks are done on context up to 32K.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6d7a16f42d31" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Knowledge Distillation for Agents]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@mne/knowledge-distillation-for-agents-04e0de7c2fa1?source=rss-4f908fc82f16------2"><img src="https://cdn-images-1.medium.com/max/1396/1*1Ua7VzWAqnqDw7iA_Zki-g.png" width="1396"></a></p><p class="medium-feed-snippet">You can teach any agent new tricks &#x2014; just show it the process, the inputs, the expected output, and give it ways to verify its own work.</p><p class="medium-feed-link"><a href="https://medium.com/@mne/knowledge-distillation-for-agents-04e0de7c2fa1?source=rss-4f908fc82f16------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@mne/knowledge-distillation-for-agents-04e0de7c2fa1?source=rss-4f908fc82f16------2</link>
            <guid isPermaLink="false">https://medium.com/p/04e0de7c2fa1</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[knowledge-distillation]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Gregory Zem]]></dc:creator>
            <pubDate>Mon, 09 Mar 2026 11:54:20 GMT</pubDate>
            <atom:updated>2026-03-09T11:54:20.857Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[How to Bill Customers for Your Agentic Platform (Without Going Broke)]]></title>
            <link>https://medium.com/@mne/how-to-bill-customers-for-your-agentic-platform-without-going-broke-a9bac3d03852?source=rss-4f908fc82f16------2</link>
            <guid isPermaLink="false">https://medium.com/p/a9bac3d03852</guid>
            <category><![CDATA[tutorial]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[monetization]]></category>
            <dc:creator><![CDATA[Gregory Zem]]></dc:creator>
            <pubDate>Sun, 01 Mar 2026 22:06:36 GMT</pubDate>
            <atom:updated>2026-03-01T22:06:36.755Z</atom:updated>
            <content:encoded><![CDATA[<h3><em>A practical pricing guide for founders building AI agent products in 2026</em></h3><p>I’ve been building an agentic platform, and the first time I watched a power user kick off a complex task, I had a sinking feeling. The agent spun up beautifully — researching, coding, debugging, iterating — and my cloud bill climbed in real time. The product worked exactly as designed. The unit economics did not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ajvBVhil1LRLiJGcMckXkg.png" /></figure><p>Welcome to the central tension of agentic AI: <strong>the thing that makes your product magical is the same thing that makes it expensive to run.</strong> A single user prompt can fan out into 2, 10, or 30+ LLM calls, each one resending the full conversation history, invoking tools, spinning up sandboxes, and burning through tokens at rates that vary by an order of magnitude from one task to the next.</p><p>Flat pricing loses money on complex jobs. Raw per-token billing terrifies customers. So what actually works?</p><p>I’ve spent a few days digging into how Cursor, Devin, Replit, Intercom, GitHub Copilot, LangGraph, and others have tackled this — dissecting their pricing pages, reverse-engineering their billing logic, and stress-testing the math against my own cost data. A clear playbook is emerging. Here’s the practical version.</p><h3>The Core Problem: Your Costs Are Wildly Unpredictable</h3><p>Before talking about pricing models, let me walk you through <em>why</em> agentic workloads are so hard to price. I learned this the hard way.</p><p><strong>Three structural cost drivers make agents different from traditional SaaS:</strong></p><p><strong>Context explosion.</strong> Agents are stateless under the hood — every turn resends the full conversation history to the LLM API. Tool definitions alone can reach 134K tokens. As the agent works through a multi-step task, input tokens grow quasi-quadratically per turn. That’s not a typo. The first time I plotted our token consumption per turn, I thought the logging was broken.</p><p><strong>Spiky variance.</strong> A simple task might take 2 turns and cost you $0.04. A complex one — same product, same user, same subscription tier — might take 30 turns and cost $2.80. That’s a 70× spread within the same product surface. I’ve seen this range in my own logs, and it makes traditional SaaS pricing math completely useless.</p><p><strong>Multi-component COGS.</strong> Token cost is just one line item. Add sandboxed compute, vector database queries, CI runs, web searches, file retrieval, network egress, and third-party API fees. Your true cost-to-serve is a composite.</p><p>The consequence is simple: you cannot slap a flat price on an agentic product and expect to survive. You need a model that absorbs variance without punishing either you or your customer.</p><h3>The Model That’s Winning: Hybrid Subscription + Usage</h3><p>After pulling apart every production agentic platform I could get my hands on, one pattern dominates:</p><pre>Platform/seat fee (your predictable MRR)<br>  + Included usage pool (credits, jobs, or steps)<br>    + Overage at a metered rate (with volume discounts)<br>      + Optional outcome bonus (for enterprise deals)</pre><p><strong>Why it works:</strong></p><p>The base subscription funds your fixed costs — engineering, infrastructure, compliance, support. The included usage pool covers typical behaviour for most users, making the price feel predictable. Overage billing aligns your marginal revenue with marginal cost on heavy users. And the optional outcome bonus captures enterprise willingness-to-pay when you can prove ROI.</p><p>This isn’t theoretical — I’ve verified it across the board. Cursor charges ~$20/month with a ~$20 credit pool. Replit’s Core plan is ~$25/month with $25 in credits. GitHub Copilot uses seat licenses plus “premium request” allowances with model-specific multipliers. When I noticed the same structure showing up in product after product, I knew it wasn’t coincidence. It repeats because it solves the fundamental mismatch between predictable subscriptions and unpredictable costs.</p><h3>Six Pricing Models That Work in Practice</h3><p>The hybrid model is the most common, but in my research I’ve identified six distinct models that are working in production right now. Depending on your product’s shape, one of these might be a better fit.</p><h3>1. Subscription + Usage Credits + Overage</h3><p>The dominant pattern described above. Monthly fee includes a credit pool mapped to actual API cost. Overage billed at-cost or with markup. Spend limits are user-configurable.</p><p><strong>Best for:</strong> Developer tools, IDE agents, creative tools. Anywhere usage varies but users expect a predictable monthly bill.</p><h3>2. Normalised Compute Units</h3><p>Create a single proprietary unit — like Devin’s “ACU” at roughly $2 each — that wraps together tokens, VM time, network, and tool calls. The customer buys ACUs; you handle the messy internal accounting.</p><p><strong>Best for:</strong> Multi-dimensional workloads where tokens alone massively underrepresent true cost. If your agent spins up containers, runs CI pipelines, or does heavy computation alongside LLM calls, this is your model.</p><h3>3. Per-Node / Per-Step Pricing</h3><p>Each orchestration step — each node in your agent graph — is the billing atom. LangGraph charges $0.001 per node plus per-minute standby time.</p><p><strong>Best for:</strong> Agent frameworks and infrastructure products where your customers are developers building their own agents. Tightly correlated with variable cost and transparent to technical buyers.</p><h3>4. Outcome-Based Pricing</h3><p>Charge a flat fee per verified result. Intercom’s Fin charges $0.99 per resolved support ticket. Cosine’s Genie charges per shipped feature.</p><p><strong>Best for:</strong> Workflows with clear, binary success metrics. This model has the highest willingness-to-pay alignment — the customer only pays when they get value — but it requires you to absorb all cost variance. Price carefully.</p><h3>5. Workflow Executions</h3><p>Charge per end-to-end workflow run regardless of internal complexity. This is n8n’s model.</p><p><strong>Best for:</strong> Automation platforms targeting business users who don’t care about tokens or steps. They think in workflows, so bill them in workflows.</p><h3>6. Operations + Token Credits</h3><p>Charge per operation plus additional credits scaled by token consumption. This is Make’s model — a hybrid within a hybrid.</p><p><strong>Best for:</strong> Platforms where the same operation can have wildly different AI intensity depending on configuration.</p><h3>Step 1: Instrument Everything (Seriously, Everything)</h3><p>You cannot price what you cannot measure. This was my first expensive lesson — I was three months into building before I had proper per-run cost attribution, and I was essentially flying blind on margins. Before choosing a model, build internal cost metering that tracks every cost driver per run:</p><ul><li><strong>Tokens:</strong> input, output, cached, and reasoning — broken down by model</li><li><strong>Steps/turns:</strong> how many LLM round-trips the agent took</li><li><strong>Tool calls:</strong> web searches, code execution, file operations, API calls</li><li><strong>Compute:</strong> sandbox/container seconds, CI minutes</li><li><strong>Storage and egress:</strong> vector DB reads/writes, file retrieval, network transfer</li><li><strong>Third-party API fees:</strong> anything you’re paying per-call to external services</li></ul><p><strong>Critical rule I learned the hard way:</strong> Use provider-reported token counts as your billing source of truth. Local tokenizers are estimates only — Anthropic’s published tokenizer is known to be inaccurate for some Claude models. I initially built my cost model on local token estimates and found a 15–20% discrepancy when I finally reconciled against the API response data. OpenAI and Anthropic both include usage data in their API responses. Use it.</p><p>Here’s the per-run cost formula you should be computing:</p><pre>Fully-loaded run cost =<br>  Σ LLM calls [(input_tokens × input_rate)<br>              + (cached_tokens × cached_rate)<br>              + (output_tokens × output_rate)]<br>  + Σ (tool_calls × tool_call_rate)<br>  + Σ (compute_sessions × compute_rate)<br>  + storage + egress costs<br>  + third-party API fees</pre><p>Keep this metering layer separate from your customer-facing pricing. Internally you track tokens and tool calls. Externally you present credits, jobs, or ACUs. The mapping between these two layers is where your business model lives.</p><h3>Step 2: Price to the P90, Not the Mean</h3><p>This is the single most important pricing principle for agentic platforms, and the one I almost got catastrophically wrong.</p><p>When I first modelled our pricing, I used average cost per job. It looked great on the spreadsheet. Then I ran 250 representative test jobs and actually looked at the distribution:</p><ul><li><strong>P50 cost:</strong> $0.08 per job</li><li><strong>P90 cost:</strong> $0.30 per job</li><li><strong>P99 cost:</strong> $1.20 per job</li></ul><p>If you price based on the average ($0.12), roughly 10% of your jobs will lose money. For a high-volume platform, that tail will eat your margins alive.</p><p><strong>Set your unit price to maintain your target margin at P90 cost.</strong> If your target gross margin is 70% and your P90 cost is $0.30:</p><pre>Price per unit = $0.30 / (1 - 0.70) = $1.00</pre><p>This means your P50 jobs are wildly profitable (which is fine — that funds the business) and even your expensive jobs maintain margin. The P99 outliers might break even or lose slightly, but they’re rare enough that the math works overall.</p><h3>Step 3: Build Profitability Levers Into the Architecture</h3><p>Pricing alone won’t save you. You need engineering levers that structurally reduce your cost-to-serve. These are the ones that made the biggest difference in my own platform.</p><h3>Prompt Caching</h3><p>This is the single biggest cost lever for agents that repeatedly reference the same codebase, documentation, or context. OpenAI offers automatic caching with up to 90% input cost reduction. Anthropic offers explicit cache writes where cache reads cost 0.1× the base input rate. If your agent isn’t using prompt caching, you’re probably overpaying by 3–5×.</p><h3>Model Routing</h3><p>Stop using your most expensive model for everything. Simple sub-tasks — summarising, formatting, classifying — should route to a cheap, fast model. Reserve frontier models for complex reasoning, architecture decisions, and debugging. Some products (like Replit) expose this as a user-controlled toggle, which is elegant: the user explicitly opts into higher cost for higher capability.</p><h3>Batch API for Background Work</h3><p>Both OpenAI and Anthropic offer 50% discounts on asynchronous batch processing. If your agent does any non-real-time work — nightly analysis, background indexing, bulk processing — route it through batch endpoints.</p><h3>Context and Tool Hygiene</h3><p>Tool schema bloat is a first-order cost driver that most teams ignore — mine included, initially. Don’t send every tool definition on every call — only send what’s relevant for the current step. Prune irrelevant conversation history. Cap retrieved content size. When I audited our tool definitions, I found we were sending ~134K tokens of schemas on every single call. That’s money on fire. Trimming to only relevant tools per step cut our input costs by nearly 40%.</p><h3>Hard Guardrails Per Run</h3><p>Every run needs safety limits:</p><ul><li><strong>Max steps/turns:</strong> Cap at 15–25 to kill runaway loops</li><li><strong>Max tokens per run:</strong> Trigger a pause, downgrade, or user confirmation at threshold</li><li><strong>Max cost budget:</strong> Enforce a dollar cap in your orchestrator</li><li><strong>Tool-call caps:</strong> Limit web searches, CI runs, and file operations per job</li><li><strong>Timeout:</strong> Wall-clock limit on total execution time</li></ul><p>On budget breach, stop and show partial results or ask the user to confirm continuation. Never let a single run drain your margin.</p><h3>Step 4: Make Costs Visible to Users</h3><p>Opacity breeds distrust. This is something I feel strongly about after talking to dozens of users: the platforms winning on retention are the ones making cost legible at every stage. Our churn dropped noticeably once we started showing users what they were spending and why.</p><h3>Before the Run: Estimates</h3><p>Use a cheap planner LLM call to analyse the request, map the likely steps, and present an estimate: “This looks like ~8–10 steps, roughly 5 credits. Proceed?” Anthropic exposes a token-counting endpoint that gives input token estimates without running the model — use it for cost previews. Supplement with historical distributions from similar past tasks.</p><h3>During the Run: Checkpoints</h3><p>Show a running cost display: “3 credits used so far.” Insert human-in-the-loop pause points for long tasks. Add approval gates before irreversible actions like deploying, pushing code, or deleting resources.</p><h3>After the Run: Itemised Receipts</h3><p>Show two levels of detail. The user-facing level should read like a story: “Analysed repo (1 credit) → Wrote tests (2 credits) → Fixed 3 bugs (3 credits).” The technical level (expandable for power users) shows tokens in/out per model, step count, tool calls, and wall-clock time.</p><p><strong>Never</strong> show raw “Tokens used: 450,210” to a non-technical user. Translate everything into your billing unit.</p><h3>Ongoing: Dashboards and Budget Controls</h3><p>Give users per-run, daily, weekly, and monthly spend limits (hard or soft caps). Send alerts at 80% utilisation and for anomalous runs (10× above typical). Offer scope limits (“agent may only touch the /frontend directory”) and model tier policies (“use the cheap model by default; never use the frontier model without approval”). Provide CSV/API export for audit trails.</p><h3>Concrete Pricing Templates You Can Steal</h3><h3>If you’re building a developer agent (like Cursor or Devin):</h3><pre>Pro:    $39/seat/month — includes 200 agent jobs<br>Extra:  $0.50–1.00/job overage<br>Team:   Per-seat, pooled allowance, volume-discounted overage<br>API:    Per-job or per-1K-token for integrators</pre><p>Set a per-job hard cap around $1.50. Max steps: 25. Enforce budget in the orchestrator.</p><h3>If you’re building business automation (support, ops):</h3><pre>Platform:   Fixed monthly fee per workspace<br>Included:   X tasks/month (tickets resolved, invoices processed)<br>Overage:    $0.60–0.99/task, volume discounts<br>Enterprise: + outcome bonus (15–30% of measured savings, capped)</pre><h3>If you’re building agent infrastructure (framework, orchestration):</h3><pre>Free:       100K nodes/month<br>Plus:       $39/seat/month + $0.001/node + per-min standby<br>Enterprise: Custom pricing, SLAs, dedicated infra</pre><h3>The Decision Shortcut</h3><p>Still not sure which model to pick? Here’s the quick filter:</p><p><strong>Your agents serve developers with variable task complexity?</strong> → Subscription + credit pool + overage.</p><p><strong>Your workloads are multi-dimensional (LLM + VM + CI)?</strong> → Normalised compute units.</p><p><strong>You have clear binary success metrics?</strong> → Outcome-based pricing.</p><p><strong>You’re selling a framework to other builders?</strong> → Per-node/step metering.</p><p><strong>You’re targeting enterprise with provable ROI?</strong> → Hybrid + outcome bonus.</p><h3>The Implementation Checklist</h3><p>If you’re starting from scratch, work through this roughly in order:</p><ol><li>Instrument every cost driver per run (tokens, steps, tool calls, compute, external API fees)</li><li>Collect P50, P90, and P99 cost distributions across 1,000+ representative runs</li><li>Define your external billing unit (credits, jobs, ACUs) and calibrate it to internal cost</li><li>Set unit price at P90 cost with your target margin (aim for 3–4× average CTS minimum)</li><li>Build per-run budget enforcement into your orchestrator</li><li>Implement model routing (cheap model default, frontier on-demand)</li><li>Enable prompt caching and batch API for eligible workloads</li><li>Optimise tool schemas and context management</li><li>Ship pre-run estimates, in-run checkpoints, and post-run receipts</li><li>Provide spend limits, alerts, dashboards, and usage export APIs</li><li>Reconcile billing against provider-reported usage (never trust local tokenizers)</li><li>Run sensitivity analysis: what happens to your margin if tokens per task grow 20%, or your model mix shifts toward the expensive tier?</li></ol><h3>The Bottom Line</h3><p>Agentic pricing is hard because agent costs are fundamentally unpredictable at the individual task level. But they’re quite predictable in aggregate — which is why the hybrid subscription-plus-usage model keeps winning. It gives your customers predictability while giving you the mechanism to align revenue with cost.</p><p>Having gone through this journey myself, the three things that made the biggest difference were: metering obsessively from day one, pricing to the P90 instead of the mean, and building cost visibility directly into the product. The founders I’ve talked to who got it wrong either underpriced (and bled cash on power users) or overpriced (and lost adoption to competitors who figured out model routing and prompt caching).</p><p>Your billing model isn’t just a finance decision. It’s an architecture decision, a product decision, and ultimately a survival decision. If I could go back and give myself one piece of advice, it would be this: build the metering infrastructure early, even before you’ve settled on your pricing model. The data will tell you which model to pick.</p><p><em>I’m still iterating on this — our pricing model has changed twice already, and I expect it’ll evolve again as model costs shift and our usage patterns mature. If you’re building in this space and want to compare notes on unit economics, I’d love to hear what’s working for you. What pricing model are you using, and what’s your biggest cost driver? Drop a comment or find me on </em><a href="https://www.linkedin.com/in/gregzem/"><em>LinkedIn</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a9bac3d03852" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The New Cursor, Now “Banana-Flavored”: How I Created Comics]]></title>
            <link>https://medium.com/@mne/the-new-cursor-now-banana-flavored-how-i-created-comics-ac4689fb51b2?source=rss-4f908fc82f16------2</link>
            <guid isPermaLink="false">https://medium.com/p/ac4689fb51b2</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[comics]]></category>
            <category><![CDATA[art]]></category>
            <dc:creator><![CDATA[Gregory Zem]]></dc:creator>
            <pubDate>Mon, 22 Dec 2025 23:01:42 GMT</pubDate>
            <atom:updated>2025-12-22T23:01:42.259Z</atom:updated>
            <content:encoded><![CDATA[<p>Let me tell you how I set out to generate educational comics for my kid. And how it didn’t work at first — but then it did.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EOIKJ8U_X9MiSXNHGHbTmA.png" /></figure><p>It all started two days ago: I realized I’d never worked with graphics in GenAI at the API level, and Nano Banana Pro seemed like a perfect fit for this kind of experiment, so why not give it a shot.</p><p>The idea actually came after I spotted something in a story from an interesting person — apparently you can now generate personalized textbooks for schoolkids. Any subject could be explained using examples based on a kid’s interests (not calculating angles and areas of some abstract triangle, but, say, problems using Geometry Dash terminology — something like calculating the area of a group of spikes, orbs, and a portal on a level). And it could even be interactive.</p><p>I thought: I’ll make a series of educational comics explaining next year’s school material. After several attempts to write a 6–7 page script, I realized it was a bad idea because the format just doesn’t fit. But making comics with interesting facts or a fun narrative, while being about 30% educational — that works perfectly for the short comic format. My kid loves Dav Pilkey comics (which are all fun but zero substance), so I wanted to make something similar in style but with actual meaning. Though Nano Banana shut me down on the style thing — said no can do, copyright and all that. So I came up with my own style ;)</p><p>OK. Fired up Cursor. Created a new project and started building a pipeline for serial comic production )) Generated a directory with documents describing the base context and main production stages (kid_profile.md with his preferences, style_guide.md with general styles, characters.md with character descriptions, page_layout_template.md with page design guides, comics_planner.md with the pipeline, task_and_goals.md with the assignment) + reference_characters.png with a so-called “character bible” — a reference file with characters in different emotions, speech bubbles, styles, and color palette. Set up directories for output, logs, scripts, and a future web app for viewing/reading.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*32Xize1--wJH0PHggcGgJQ.png" /></figure><p>I came up with two antagonist characters: Mr. Trash (a smart but sarcastic raccoon) and Pigeon Brain, a dim-witted pigeon who thinks he’s super intelligent and constantly gets into trouble because of it. As one does, I created detailed character descriptions via GPT-5.2 in characters.md, then defined the styles in detail: how characters appear situationally in the comic, what mandatory elements both must have (for example, the pigeon must wear a helmet with “IQ=∞” on it). In page_layout_template.md I developed page layout options: how many and what kind of blocks (“panels”) can appear on a page, what types of blocks exist, how to show backgrounds, etc.</p><p>Next step — I created a skill that can generate a single image in Nano Banana Pro and OpenAI ImgGen based on given specs (just to compare). You just throw into the chat “Hey, generate me a bunch of drawn dollar bills, I’ll use them for visualization! Make it 16:9 in 4K. And use the Google model” and it converts this into CLI parameters for the script, which then hits the Nano Banana Pro or OpenAI API. Super convenient. But for comics, this approach turned out to be completely unworkable. More on why later.</p><p>comics_planner.md defines the workflow — the chain of actions: what to do in what order, where human-in-the-loop is needed. For example, first a catalog for the new comic is created with a specific structure, then script.md is generated for the new comic and it asks me to read it, I read it, make corrections if needed, then say “continue,” the pipeline moves to the next stage and generates page layouts with dialogues, formatting, and the general style guide. I read the pages, make corrections if needed, and/or say “all good, generate the images.” And it generates the final images using the skill.</p><p>This is where I should talk separately about the main headache of generating images for comics (or rather — many different headaches). The hardest part turned out to be maintaining character consistency and seriality when generating images across multiple requests. For example, I was generating 6 separate comic pages, each with 4 to 7 panels/blocks. The raccoon would come out either fat with tiny eyes without whites, or skinny with human-like eyes, or not a raccoon at all but some cosmic creature, or…ahem…a human?! Mind you, I was using the reference image and style_guide.md for all requests. Spent two days on this. Here’s what finally helped:</p><p>1. Reducing the total system prompt for page generation to 3–4K tokens maximum, with all character details, layout descriptions, styles, and the page script. This was the hardest part. I explained this to myself as the model’s attention budget, which has to be used rationally in the request. This isn’t a language model where you can send a 60K token instruction and it’ll account for everything reasonably well.</p><p>2. The characters themselves need to be described as detailed and vividly as possible, so that even Stevie Wonder could draw them from the description. Down to “almond-shaped eye whites” and “The pigeon has 2 wings. Not three arms. And not one.” I think someone who does this production professionally sells these prompts, because you really have to wrestle with them.</p><p>3. The model constantly forgets details; within the page script you need to periodically remind it that the pigeon absolutely must have a helmet and bow tie, and the raccoon — a bag. Yes, all these rules are described in style_guide.md, which is part of the system prompt, and yes, the reference image shows the pigeon with all the right attributes, but the model stubbornly keeps forgetting this on some panels.</p><p>4. Things improved significantly when I moved all instructions, style guide, and characteristics into system_instruction in the “Nano Banana” API, and in the content part next to the reference image I started sending only “Edit this image according to instructions.” Before that, I was sending styles in the system prompt and the page script alongside the reference image, and that worked poorly.</p><p>5. The difference between 1K/2K/4K images is in the rendering detail (and price), not result accuracy. Meaning the texture is rendered better at 2K, but if the raccoon is fat and the pigeon is helmet-less, they’ll be fat and helmet-less at 4K too.</p><p>6. gpt-image-1 vs Nano Banana Pro is like comparing the ocean in Aivazovsky’s paintings to first-graders’ drawings of the ocean. OpenAI is still very far from having a model that can be used for generating complex images from description and reference. Only through comparison do you appreciate how awesome “Banana Pro” is. I was too lazy to test MidJourney. Maybe it’s great too. Especially with all the ControlNets.</p><p>7. Finding the right balance between description detail and prompt length — that’s the main challenge. You just have to sit and experiment. By the second comic I started getting the hang of it.</p><p>8. Experimentally established: ideally, no more than 5 panels per page (4 is better). To make the comic look authentic, I specify panel layout in the layout and let the model choose which layout works best in each case (during the script and page design stage).</p><p>9. Had to replace the single image generation skill with a series image generation skill, which runs a script that receives parameters for which pages to generate, corresponding page scripts, style guide, and other parameters. The script then calls the API in a loop with identical parameters, just different page descriptions/scripts.</p><p>But even with a seemingly perfect page script, layout, style guide, and reference, there are still inaccuracies and generation glitches on individual blocks. With a couple of pages you have to tinker manually, tweak the layout, add instructions in places where the model’s attention drifts and it forgets what the character should look like or what they should be holding.</p><p>Oh, and about content: I naively thought at first that you could ask the model to come up with a cool idea for, say, explaining prime numbers or area/perimeter to a kid, but creativity in this case is not its strong suit (GPT-5.2 and Opus 4.5). So I came up with the key idea myself — what I want and how — and then asked the LLM to write out the script/narrative and dialogues (both GPT-5.2 and Opus 4.5 handle this equally well).</p><p>In the end, the pipeline and workflow came together like this: I set the idea and guidelines for the topic in task_and_goal.md. For example, “I want to make a comic about place names that are often confused, here’s the list. Let’s have Pigeon Brain teleport using his GameBoy-looking teleporter, but always to the wrong place, and Mr. Trash rescues him.” I drop it into the chat and say — let’s go. This file has links to descriptions of the whole pipeline and corresponding files with instructions and context. The rest is as I described above. The result — I get <a href="http://mdex.sh/tmp/comics_wrong_country_again.pdf"><strong>slides roughly like these</strong></a><strong>.</strong></p><p>Time per comic: about 15–20 minutes (with iterations), cost: around 6 euros.</p><p>So how are you spending your money before the New Year?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ac4689fb51b2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Claude “Skills” Feature]]></title>
            <link>https://medium.com/@mne/the-claude-skills-feature-ecb1f3a8eace?source=rss-4f908fc82f16------2</link>
            <guid isPermaLink="false">https://medium.com/p/ecb1f3a8eace</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[claude]]></category>
            <category><![CDATA[skills]]></category>
            <dc:creator><![CDATA[Gregory Zem]]></dc:creator>
            <pubDate>Sun, 02 Nov 2025 13:15:11 GMT</pubDate>
            <atom:updated>2025-11-02T13:15:11.879Z</atom:updated>
            <content:encoded><![CDATA[<p>A couple of weeks ago, Anthropic released an extension called “Claude Skills” for their agents (I’ll just call them “skills” from now on). In my opinion, this is a cool and underrated thing for low-code development of various automations. Right now there’s a trend of creating so-called “Personal OS” based on something like Cursor, where people without development skills throw together various text files in directories (or pull in their Obsidian database), and interactively through an agent summarize, structure, and analyze the data in them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/559/1*A9XwhtwE57-WtZ7m8X-pcw@2x.png" /></figure><p>So skills are the quintessence of this process. When you know how the work should ideally be done, you write it down as a huge structured document with step-by-step instructions, add TXT, JSON and other files with templates and examples if needed, Python scripts, and there you have it — a ready-made workflow, as if you’d made it in n8n or make.com, only written in natural language, with the ability to run it in basically any agent (though the integration is most convenient in Claude, because it loads skills before the request and automatically selects the relevant one depending on the task, kind of like with MCP tools).</p><p>A skill is a directory with files including the main SKILL.md and auxiliary prompts, scripts and other assets (resources) that the agent needs to execute some complex workflow or automation, and do it repeatably every time. One skill is typically built around one business task. We instruct the agent to perform the automation and give it the necessary tools in the form of ready-made scripts, tools, guidelines and everything we’d normally throw into context to do this work well every time. Obviously, a skill isn’t something revolutionarily new, it’s just quantity that has turned into quality (structured “prompt engineering” for agents).</p><p>Examples of skills:</p><ul><li>extract data from tables and generate an analytical report</li><li>run tests, find the error by trace and fix it in the code</li><li>process a dozen text documents and build an ontological graph from them</li><li>…</li></ul><p>Basically, whatever you want. Here’s an <a href="https://github.com/anthropics/skills/blob/main/mcp-builder/SKILL.md">example skill</a> from Anthropic itself. Most often it’s just a long, well-structured prompt in the SKILL.md file with a bunch of links to various external guidelines and documents. I noted for myself the fact that you can basically learn how to properly describe a task to an agent from SKILL.md files.</p><p>Skills are not the agent’s MCP tools, they’re a higher level of abstraction. For example, skills can use MCP tools and the agent’s built-in instruments, explicitly telling the agent what to use for solving the task.</p><p>Skills are also useful for developers. In one of my projects, skills fit very nicely into a code processing pipeline, where at one stage the Claude Code agent, which I control through the SDK, automatically hooks up the testing skill, learns from it how to run builds for different types of projects, then extracts failed tests from the build logs, fixes the code and reruns just the failed tests. I simply described the launch commands, parameters for various maven/gradle, etc. in the instructions, made a general description of the skill and what it’s for in SKILL.md, and the console Claude Code decides for itself when to load this instruction and follow it.</p><p>In general, it’s pointless now to invent your own local agent (for example, “deep research” agent). You need to take Claude Agent SDK and do everything with it. And for rapid prototyping — extend it with MCP tools at the low level, and skills at the high level.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ecb1f3a8eace" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I Evaluated AI-Agent Response Quality]]></title>
            <link>https://medium.com/@mne/how-i-evaluated-ai-agent-response-quality-224c9c80ffe9?source=rss-4f908fc82f16------2</link>
            <guid isPermaLink="false">https://medium.com/p/224c9c80ffe9</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[quality]]></category>
            <category><![CDATA[evaluation]]></category>
            <dc:creator><![CDATA[Gregory Zem]]></dc:creator>
            <pubDate>Mon, 13 Oct 2025 19:40:20 GMT</pubDate>
            <atom:updated>2025-10-13T19:40:20.231Z</atom:updated>
            <content:encoded><![CDATA[<h3>How I Evaluated Agent Response Quality</h3><p>A few weeks ago, I was working on a task to monitor agent quality to understand whether the agent was improving with each release and by how much. To do this, I defined a set of important evaluation criteria for this particular agent, which were calculated quantitatively. The results were collected in ClickHouse and visualized through Redash (though this doesn’t really matter — you could save them in a text file if you wanted).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*g3TeelCz4CTZ83xN" /></figure><p>The measurement process was built in the best traditions of Structured Output with CoT (which some AI-dev communities are now proposing to call SGR). Context was sent to the LLM, and features across different categories were extracted from it in structured form, which immediately went into a Pydantic object, and then scores (“points”) were calculated based on these features. This produced the main groups of scores in numeric form. And they can be compared from run to run.</p><h3>Context Structure</h3><p>Since I was checking the execution of a single request (one-shot run), the context in this case was:</p><ul><li>the original user request (for example, “analyze my competitors and develop a promotion strategy… blah-blah-blah”)</li><li>system configuration (date/time, user context, agent role context)</li><li>request execution traces (tool calls and their output)</li><li>final response to the user, as generated by the agent</li></ul><h3>Evaluation Process</h3><p>Agent evaluation was performed in two stages:</p><p><strong>Stage 1.</strong> Running 20 selected requests that engaged different agent features and used different tools. At this stage, all requests were logged in structured form in the DB with configs, tool traces, and final results.</p><p><strong>Stage 2.</strong> For each obtained trace, data extraction and score calculation were performed.</p><p>There were 3–4 runs per day, and the results were averaged to avoid high variance.</p><h3>Evaluation Approach</h3><p>As for LLM requests, I didn’t ask the LLM to make an explicit decision (“is it good?” / ”is it bad?” / ”is it correct?”), and tried to extract features as much as possible, which were then multiplied in Python code by certain fixed weight constants and used to obtain final scores. Nevertheless, I couldn’t completely avoid a number of classification features that are returned as 1/2/3 or Low/Medium/High or flags (True/False). Though perhaps in the future I’ll rework this to increase determinism and accuracy of the scores.</p><h3>Methods for Calculating Final Scores</h3><p>Calculating final scores by category happened in two variants:</p><p><strong>Variant 1: known number of features</strong><br> If the number of features is known, it’s simple: divide the number of correct ones by the total number, get a value in %. If there are several, then do a weighted evaluation, multiplying each by some weight and adding them up.</p><p><strong>Variant 2: unknown number of features</strong><br> When the number of features that the LLM will return is unknown, I set an initial score (50 or 100) and then subtracted some points from this value for penalty features, and added for bonus features. For each category I got some final numeric value from 0 to 100 (lower bound was always ≥0, upper ≤100).</p><h3>Calculation Examples</h3><p><strong>Example 1: completeness of response</strong><br> Checking how completely the response covers the original user request. Up to 3 points are expected in this category. Let’s say 2 out of 3 were missed in the response — the result for completeness is ≈33.3%.</p><p><strong>Example 2: bonuses and penalties</strong><br> For calculating scores through bonuses and penalties (for example, when there are features like “response specificity,” “professionalism,” and “generality,” each having their own points with plus or minus signs). Penalty and bonus points are selected empirically for each category individually (LLM does this normally, by the way).</p><p>Let’s say:</p><ul><li>“response specificity” gives +10 points for each feature</li><li>“professionalism” gives+15 points for each feature</li><li>“response generality” gives −15 points for each feature</li></ul><p>Total:</p><pre>min(max(50 + 10 × number_of_specificity_features + <br>        15 × number_of_professionalism_features − <br>        15 × number_of_generality_features, 0), 100)</pre><p>This produces a numeric result in the range 0–100. And so on for each category with its own set of features. I have 6 categories:</p><ul><li><strong>completeness of response,</strong></li><li><strong>factual accuracy,</strong></li><li><strong>quality,</strong></li><li><strong>professionalism,</strong></li><li><strong>formatting and</strong></li><li><strong>retrieval</strong></li></ul><p>and about 40 features in total (see below).</p><h3>Additional Features for Debugging</h3><p>Besides features that affect scores, I made additional features for debugging (these are evidence and indicators of problems). For example, if there’s a definition clarity_score that can be &quot;1&quot;, &quot;2&quot;, or &quot;3&quot; then nearby there will also be examples of problems in the form of text excerpts for clarity_score == 1 or clarity_score == 2, which can help figure out why a result of 1 was returned.</p><h3>List of Extracted Features</h3><p>Now the most interesting part: what features were extracted from request traces.</p><h3>Completeness of Response</h3><ul><li><strong>identified_user_requirements: List[str]</strong> — list of expected requirements, or different aspects of the request that the agent should respond to. The agent should understand the user&#39;s intent and return a list of what should be in the response. I settled on a max of 3 requirements.</li><li><strong>fulfilled_requirements: List[str]</strong> — list of requirements that were satisfied (what the agent actually responded to). If len(fulfilled_requirements) &lt; len(identified_user_requirements) — we have problems with response completeness. You can calculate % of completeness.</li><li><strong>missing_requirements: List[str]</strong> — list of requirements that were missed in the response.</li><li><strong>main_intent_addressed: bool</strong> — was the main question (intent) of the user answered. If not — this is really bad, the response is irrelevant.</li></ul><h3>Clarity and Precision</h3><ul><li><strong>clarity_score: int</strong> — on a scale of “1”, “2” or “3” was the response clear and precise. For example: <br>“<strong>1”</strong> — this is when “one undefined technical term per ten words,” “no clear structure to the response,” “more than three vague definitions or there’s an undefined abbreviation in a paragraph”, whereas “<strong>3”</strong> — when “average sentence length is less than 20 words,” “clear structure with headers, transitions and lists/tables”</li><li><strong>clarity_problems: List[str]</strong> — up to three problematic examples</li><li><strong>clarity_strengths: List[str]</strong> — up to three confirmations of quality and clarity</li></ul><h3>Specificity and Concreteness</h3><ul><li><strong>vague_statements: List[str]</strong> — evidence of vague meaning and general conclusions. For example: &quot;many businesses face this&quot; (how many?), &quot;Usually this works well&quot; (general judgment), &quot;Quite a few facts affect this&quot; (which ones?).</li><li><strong>specific_recommendations: List[str]</strong> — specific practical recommendations</li><li><strong>generic_recommendations: List[str]</strong> — general recommendations</li></ul><h3>Professionalism</h3><ul><li><strong>professional_tone_score: int</strong> — assessment of professional language in the response: presence of slang, response in first/third person, mix of formal and informal vocabulary, use of established industry vocabulary, etc. 1 — bad, 3 — good.</li><li><strong>unprofessional_language_examples: List[str]</strong> — examples of unprofessionalism in the response</li></ul><h3>Technical Problems</h3><ul><li><strong>truncation_detected: bool</strong> — is the response complete or truncated (often detects going beyond context limits).</li></ul><h3>AI-Generated Response Patterns</h3><p>I’ve included a separate section that represents AI-Generated Pattern detection that often found in AI-generated response. Any of them pessimized the score of the “professionalism” category.</p><ul><li><strong>ai_disclaimer_phrases: List[str]</strong> — sort of &quot;AI-assisted summary of...&quot; and similar,</li><li><strong>ai_refusal_patterns: List[str]</strong> — like &quot;I&#39;m not sure if I can do that...&quot; and similar,</li><li><strong>hedging_language_examples: List[str]</strong> — kind of &quot;Based on the info available...&quot; and similar,</li><li><strong>template_phrases: List[str]</strong> — like &quot;Here is a summary of the data...&quot; and similar,</li><li><strong>excessive_politeness_examples: List[str]</strong> — such as &quot;Thank you for your time...&quot; and similar,</li><li><strong>generic_statement_examples: List[str]</strong> —sort of &quot;This is a good practice...&quot;,</li><li><strong>domain_specificity_problems: List[str]</strong> — something like &quot;This is a good practice for any website...&quot;.</li></ul><h3>Actionability and Practicality</h3><ul><li><strong>next_steps_provided: bool</strong> — are there next steps that the user needs to take to solve the problem,</li><li><strong>next_steps_examples: List[str]</strong> — what these next steps are.</li></ul><h3>Factual Accuracy</h3><ul><li><strong>factual_error_examples: List[str]</strong> — examples of factual errors with explanations</li><li><strong>numerical_error_examples: List[str]</strong> — numerical calculation errors (for example, incorrect interpretations of numerical values from tools)</li><li><strong>aggregation_error_examples: List[str]</strong> — errors in numerical aggregations and average calculations</li><li><strong>synthesis_error_examples: List[str]</strong> — data synthesis errors (for example, errors in combining facts)</li><li><strong>unsupported_claim_examples: List[str]</strong> — use of facts that weren&#39;t in the original request, configuration, and tool results</li><li><strong>contradiction_examples: List[str]</strong> — factual contradictions in conclusions</li></ul><p>—</p><p>To get deterministic results, I had to work hard on the descriptions of Pydantic class members: somewhere there weren’t enough examples, somewhere scores were duplicated or smeared across multiple features. It took about two weeks to settle on the weights, list of criteria, and property descriptions.</p><p>But it was worth it!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=224c9c80ffe9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Don’t let models make decisions!]]></title>
            <link>https://medium.com/@mne/dont-let-models-make-decisions-0cd4349db614?source=rss-4f908fc82f16------2</link>
            <guid isPermaLink="false">https://medium.com/p/0cd4349db614</guid>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Gregory Zem]]></dc:creator>
            <pubDate>Sun, 05 Oct 2025 21:01:19 GMT</pubDate>
            <atom:updated>2025-10-05T21:01:19.832Z</atom:updated>
            <content:encoded><![CDATA[<p>Over a couple of years of actively chatting with LLMs, I’ve developed a rule for myself on “how to ask questions properly.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_ERWB8d8IJu1ELwd4UiIfA.png" /></figure><p>I stopped asking general yes/no questions, and alternative questions where you need to choose one option from several, and also said “no” to disjunctive questions (“choose the red pill or the blue pill?”).</p><p>And I replaced them with specific questions that can only be answered with detailed responses, backed up by details and facts. This way you can immediately find out “who,” “what,” “where,” “when,” and “why.”</p><p>I use this principle both in chats with your typical Claude/ChatGPT, in Cursor chat, and in structured queries for evaluating AI responses, like (God forbid) “LLM-as-a-Judge.”</p><p>The general principle: ask a question, immediately evaluate the supporting evidence in the response, make sure it’s not nonsense and hallucination, and then make a further decision about continuing the dialogue.</p><p>For example, you could ask: “Hey, I’ve got this tags field in my database, do I still need it?”</p><p>Or you could ask it this way: “Show me all the places in the code where the tags field is used.”</p><p>The second will give you more useful information for making a decision.</p><p>You could ask ChatGPT: “Is stevia safe or not as a sweetener” and get a detailed answer about why it’s safe. Or dangerous. But how do you know why? How do you verify that the model made the right decision? Based on what?</p><p>Or you could ask ChatGPT: “Find all recent scientific evidence with links to primary sources about the benefits or harms of stevia.” And then go read separately what it finds (if it finds anything!) and decide for yourself.</p><p>You could ask an LLM-as-a-Judge to validate a response to a question: “Here’s a user question &lt;…&gt;, here’s the system’s answer &lt;…&gt;. Is the system’s answer accurate?” or “Here’s a user question &lt;…&gt; and the system’s answer &lt;…&gt;. Rate the accuracy of the system’s answer on a scale from 1 to 10.”</p><p>But it’s more effective to do it like this: “Here’s a user question &lt;…&gt;, here’s the system’s answer &lt;…&gt;. Identify the criteria for evaluating completeness and accuracy for this question. Return supporting evidence for the answer on each point, or nothing if no such evidence exists.” And then calculate that score in code. Even in this case it’ll still be buggy, but in a more manageable way. The judgment task becomes a task of extracting structured facts from unstructured data.</p><p>The downside of asking specific questions: they require more cognitive load on your brain, because you first think about why you’re asking this, you get the information, and then you yourself make the decision about what to do with it next. But isn’t that the main job of the brain — to think?</p><p>And, accordingly, the final heretical thought: never!… never let LLMs make decisions.</p><p>Decisions should always be made by you. And the model is good for gathering what’s called “supporting evidence” or facts, confirmations, proofs (that is, for extracting them from unstructured or semi-structured data).</p><p>General, alternative, and disjunctive questions force the model to make decisions, but that’s not the job of a language model, no matter how statistically appealing it might look.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0cd4349db614" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Helping Robots]]></title>
            <link>https://medium.com/@mne/helping-robots-507d751b1cb3?source=rss-4f908fc82f16------2</link>
            <guid isPermaLink="false">https://medium.com/p/507d751b1cb3</guid>
            <category><![CDATA[cursor]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[rules]]></category>
            <dc:creator><![CDATA[Gregory Zem]]></dc:creator>
            <pubDate>Sun, 28 Sep 2025 12:46:01 GMT</pubDate>
            <atom:updated>2025-09-28T12:46:01.185Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LPcgcUwaHURm8KGN" /></figure><p>In anticipation of the not-so-distant future where robots will take over all critically important spheres of life and start doing creative work, while humans will serve them and do heavy physical labor, I’m trying to build up positive karma by helping robots. Maybe they’ll have mercy on me and assign me to some simple tasks (like doing RLHF or data labeling instead of sorting radioactive ore with bare hands).</p><p>I help agents everywhere it makes their work more efficient. For example, in Cursor and Claude Code, I provide more context so the agent spends less time and tokens, gives more accurate answers, and writes decent code right off the bat. Over the past few months of practicing vibe-coding, I’ve developed the following habits (sorry if I’m being Captain Obvious):</p><ul><li>Attach all files to the chat right away that might be relevant to the task at hand. Sure, agents can find them on their own. But somehow I feel more at ease when they find what I expected. And they do it faster than when they search themselves.</li><li>Point out links to code chunks that I think should be reworked or used for the task. I write explicitly: “Buddy, make me a document parser through LLM, use structured output, take the pydantic data model &lt;…from here…&gt;, and here’s how &lt;…link to code chunk for working with LLM…&gt; you need to send requests to the LLM and use &lt;…this model…&gt;”. As soon as I formed this habit, productivity in Cursor immediately increased. And now I don’t tempt fate hoping that the agent will find and understand how to implement the method on its own, and actually implement it using existing references and APIs.</li><li>When I ask to fix a bug, I dump chunks of logs with exceptions, browser console output, a piece of the generated HTML page (not just an error message), and also hint where I’d look first (if I have ideas, of course). Showing a chunk of the page works especially well for fixes in HTML templates and ReactJS apps.</li><li>I don’t get lazy about creating a new chat when the agent starts acting dumb. Sometimes a certain combination of tokens causes an unexpected reasoning result (or lack thereof), and you keep writing in that chat (sometimes swearing) trying to get it to fix itself, but it keeps going in circles. So don’t waste time and tokens on this — just open a new chat and provide the necessary context, continuing to solve the task there. This often helps break out of the vicious cycle of “ — Find the causes and fix this… — Sure, I fixed it!… — No, you didn’t fix it… — You’re right I didn’t fix it, now I really fixed it… — No, you didn’t fix it again, damn it… &lt;opens new chat&gt;”.</li><li>I use different models for different tasks. For my (this is important!) tech stack, these work best: Sonnet 4 for everyday changes (when you need to do it carefully and well), Opus 4.1 for refactoring and design (and sometimes for tricky bugs), and I also like Code Supernova now (some new free model in Cursor, probably Grok from Elon Arronaevich Musk). I never got the hang of ChatGPT-5, don’t see any benefit from it in coding (even though it’s supposedly d’Artagnan in all the benchmarks). Gemini Pro 2.5 works too slowly and too verbosely. The other smaller ones are complete trash. Got used to working with the Max flag enabled.</li><li>I crafted myself several rules by adding them in Cursor settings (I’ll leave them in the comments). Though don’t get your hopes up — the agent doesn’t always follow the rules. But at least it stops being annoying by creating .md files after every run, and starts using constants by declaring them at the top of the file (and generally following at least some PEP8 standards and practices). This is especially relevant if the code was initially written poorly, and the agent continues replicating low-grade code.</li></ul><p>My rules from Cursor: <a href="https://mdex.sh/?596Lba60b0E">https://mdex.sh/?596Lba60b0E</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=507d751b1cb3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI Automation vs Ad-Hoc Tasks with LLM]]></title>
            <link>https://medium.com/@mne/ai-automation-vs-ad-hoc-tasks-with-llm-dfb6867ca64c?source=rss-4f908fc82f16------2</link>
            <guid isPermaLink="false">https://medium.com/p/dfb6867ca64c</guid>
            <category><![CDATA[agentic-workflow]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[automation]]></category>
            <dc:creator><![CDATA[Gregory Zem]]></dc:creator>
            <pubDate>Sat, 14 Jun 2025 11:09:28 GMT</pubDate>
            <atom:updated>2025-06-14T11:09:28.497Z</atom:updated>
            <content:encoded><![CDATA[<p>When developing solutions based on agent frameworks, you face a dilemma: how to make the agent’s work reliable and predictable without going overboard with tool integrations (since those are what largely determine agent capabilities). Those of you choosing the easy path of ready-made agents extensible through MCP are martyrs. This kind of instant-fix mess falls apart quickly and causes major indigestion.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pzJqLp8ak_-tyU1G" /></figure><p>Need repeatable results, stability, predictability, consistency, and easy debugging? Forget about it. An agent loop based on ReAct + MCP tools is pure alchemy. Only prayers and a lucky rabbit’s foot will help you here.</p><p>The main reason is that:</p><p>🔹 The model itself is non-deterministic, yet you’re asking it to make the same decision given identical input data. There will be no idempotency. It will periodically go down different paths, calling tools with wrong parameters or completely wrong tools (especially when there are many tools from the same domain).</p><p>🔹 You end up with way too many failure points. Each MCP tool individually is a wrapper on top of an API, and the ReAct agent involves LLM calls in a loop. You get at least these failure points: API internals, API↔MCP server integration, MCP server↔agent loop integration, failures within the agent loop itself (wrong tool call decisions, bad reflection, infinite loops, model API failures), plus actual code bugs. If you calculate the failure probability of each node separately, you get depressing numbers. Let’s say you’re a former pacemaker developer, and all the services you use were also built by people like you — meaning all nodes have 99.99% reliability. If you multiply all the reliabilities (and raise this to the power of the number of agent flow steps), you get the expected workflow reliability.</p><p>Roughly, for a 5-step system flow you get: 0.9999 (API) × 0.9999 (MCP) × 0.9999 (Agent) × 0.9999 (Framework) ^ 5 = 99.8%. Not even “three nines” anymore. But realistically, all these components will be lucky to have 95% reliability. Then the total reliability of such a flow comes out to around 35%. Meaning every third request will complete successfully, while two out of three will fail. This is roughly what we’re seeing in quick-fix agent solutions today.</p><p>The problem is that business doesn’t need 35% reliability. It needs at least 90%. So if you’re aiming for B2B success, you need to look straight at the complex and long-term approach. What does this mean?</p><p>✓ Forget ReAct and CodeAct — build your own step planner (e.g., as a DAG) and a runner to execute them.</p><p>✓ Forget MCP tools — build your own integrations with the APIs you need.</p><p>✓ Don’t let the model make decisions — do that in code. Use the model for three things it excels at: 1) extracting facts from unstructured data and returning them in structured form 2) transforming one data representation into another 3) generating content.</p><p>Overall, frame it as an engineering problem: “I need to keep everything under control,” making it more rule-based rather than alchemical, AI-based.</p><p>What are the downsides? Yes, you’ll have to buckle down, do some design work, and write code. An agent solution based on your own business tools is harder to scale compared to an agent based on MCP tools, of which there are millions now (though 90% of that is garbage).</p><p>But if clients come to you saying, “Why don’t you just build an agent based on Claude Desktop + MCP tools?”, you need to clarify first what scenarios they’re planning for and what reliability they expect. Because it’s quite possible they don’t need high accuracy and reliability, and the main use case for the agent is routine ad-hoc tasks. In that case, Claude Desktop + MCP or ReAct + MCP tools could very well be the solution.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dfb6867ca64c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>