Developing Experience-Focused KPIs

Over the last year, I’ve seen many people fall into the same trap: They launch an AI-powered agent (chatbot, assistant, support tool, etc.)… But only track surface-level KPIs — like response time or number of users. That’s not enough. To create AI systems that actually deliver value, we need 𝗵𝗼𝗹𝗶𝘀𝘁𝗶𝗰, 𝗵𝘂𝗺𝗮𝗻-𝗰𝗲𝗻𝘁𝗿𝗶𝗰 𝗺𝗲𝘁𝗿𝗶𝗰𝘀 that reflect: • User trust • Task success • Business impact • Experience quality    This infographic highlights 15 𝘦𝘴𝘴𝘦𝘯𝘵𝘪𝘢𝘭 dimensions to consider: ↳ 𝗥𝗲𝘀𝗽𝗼𝗻𝘀𝗲 𝗔𝗰𝗰𝘂𝗿𝗮𝗰𝘆 — Are your AI answers actually useful and correct? ↳ 𝗧𝗮𝘀𝗸 𝗖𝗼𝗺𝗽𝗹𝗲𝘁𝗶𝗼𝗻 𝗥𝗮𝘁𝗲 — Can the agent complete full workflows, not just answer trivia? ↳ 𝗟𝗮𝘁𝗲𝗻𝗰𝘆 — Response speed still matters, especially in production. ↳ 𝗨𝘀𝗲𝗿 𝗘𝗻𝗴𝗮𝗴𝗲𝗺𝗲𝗻𝘁 — How often are users returning or interacting meaningfully? ↳ 𝗦𝘂𝗰𝗰𝗲𝘀𝘀 𝗥𝗮𝘁𝗲 — Did the user achieve their goal? This is your north star. ↳ 𝗘𝗿𝗿𝗼𝗿 𝗥𝗮𝘁𝗲 — Irrelevant or wrong responses? That’s friction. ↳ 𝗦𝗲𝘀𝘀𝗶𝗼𝗻 𝗗𝘂𝗿𝗮𝘁𝗶𝗼𝗻 — Longer isn’t always better — it depends on the goal. ↳ 𝗨𝘀𝗲𝗿 𝗥𝗲𝘁𝗲𝗻𝘁𝗶𝗼𝗻 — Are users coming back 𝘢𝘧𝘵𝘦𝘳 the first experience? ↳ 𝗖𝗼𝘀𝘁 𝗽𝗲𝗿 𝗜𝗻𝘁𝗲𝗿𝗮𝗰𝘁𝗶𝗼𝗻 — Especially critical at scale. Budget-wise agents win. ↳ 𝗖𝗼𝗻𝘃𝗲𝗿𝘀𝗮𝘁𝗶𝗼𝗻 𝗗𝗲𝗽𝘁𝗵 — Can the agent handle follow-ups and multi-turn dialogue? ↳ 𝗨𝘀𝗲𝗿 𝗦𝗮𝘁𝗶𝘀𝗳𝗮𝗰𝘁𝗶𝗼𝗻 𝗦𝗰𝗼𝗿𝗲 — Feedback from actual users is gold. ↳ 𝗖𝗼𝗻𝘁𝗲𝘅𝘁𝘂𝗮𝗹 𝗨𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱𝗶𝗻𝗴 — Can your AI 𝘳𝘦𝘮𝘦𝘮𝘣𝘦𝘳 𝘢𝘯𝘥 𝘳𝘦𝘧𝘦𝘳 to earlier inputs? ↳ 𝗦𝗰𝗮𝗹𝗮𝗯𝗶𝗹𝗶𝘁𝘆 — Can it handle volume 𝘸𝘪𝘵𝘩𝘰𝘶𝘵 degrading performance? ↳ 𝗞𝗻𝗼𝘄𝗹𝗲𝗱𝗴𝗲 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 𝗘𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝗰𝘆 — This is key for RAG-based agents. ↳ 𝗔𝗱𝗮𝗽𝘁𝗮𝗯𝗶𝗹𝗶𝘁𝘆 𝗦𝗰𝗼𝗿𝗲 — Is your AI learning and improving over time? If you're building or managing AI agents — bookmark this. Whether it's a support bot, GenAI assistant, or a multi-agent system — these are the metrics that will shape real-world success. 𝗗𝗶𝗱 𝗜 𝗺𝗶𝘀𝘀 𝗮𝗻𝘆 𝗰𝗿𝗶𝘁𝗶𝗰𝗮𝗹 𝗼𝗻𝗲𝘀 𝘆𝗼𝘂 𝘂𝘀𝗲 𝗶𝗻 𝘆𝗼𝘂𝗿 𝗽𝗿𝗼𝗷𝗲𝗰𝘁𝘀? Let’s make this list even stronger — drop your thoughts 👇

Most teams pick metrics that sound smart… But under the hood, they’re just noisy, slow, misleading, or biased. But today, I'm giving you a framework to avoid that trap. It’s called STEDII and it’s how to choose metrics you can actually trust: — ONE: S — Sensitivity Your metric should be able to detect small but meaningful changes Most good features don’t move numbers by 50%. They move them by 2–5%. If your metric can’t pick up those subtle shifts , you’ll miss real wins. Rule of thumb: - Basic metrics detect 10% changes - Good ones detect 5% - Great ones? 2% The better your metric, the smaller the lift it can detect. But that also means needing more users and better experimental design. — TWO: T — Trustworthiness Ever launch a clearly better feature… but the metric goes down? Happens all the time. Users find what they need faster → Time on site drops Checkout becomes smoother → Session length declines A good metric should reflect actual product value, not just surface-level activity. If metrics move in the opposite direction of user experience, they’re not trustworthy. — THREE: E — Efficiency In experimentation, speed of learning = speed of shipping. Some metrics take months to show signal (LTV, retention curves). Others like Day 2 retention or funnel completion give you insight within days. If your team is waiting weeks to know whether something worked, you're already behind. Use CUPED or proxy metrics to speed up testing windows without sacrificing signal. — FOUR: D — Debuggability A number that moves is nice. A number you can explain why something worked? That’s gold. Break down conversion into funnel steps. Segment by user type, device, geography. A 5% drop means nothing if you don’t know whether it’s: → A mobile bug → A pricing issue → Or just one country behaving differently Debuggability turns your metrics into actual insight. — FIVE: I — Interpretability Your whole team should know what your metric means... And what to do when it changes. If your metric looks like this: Engagement Score = (0.3×PageViews + 0.2×Clicks - 0.1×Bounces + 0.25×ReturnRate)^0.5 You’re not driving action. You’re driving confusion. Keep it simple: Conversion drops → Check checkout flow Bounce rate spikes → Review messaging or speed Retention dips → Fix the week-one experience — SIX: I — Inclusivity Averages lie. Segments tell the truth. A metric that’s “up 5%” could still be hiding this: → Power users: +30% → New users (60% of base): -5% → Mobile users: -10% Look for Simpson’s Paradox. Make sure your “win” isn’t actually a loss for the majority. — To learn all the details, check out my deep dive with Ronny Kohavi, the legend himself: https://lnkd.in/eDWT5bDN

Three unglamorous website fixes lifted engagement by 145%. A manufacturer came to us with credibility, demand and traffic. But the post-click experience was making buyers work too hard. The product journey reflected how the business organised its ranges. Not how buyers chose. Proof was buried. The enquiry path sat too far from the moments of highest intent. So we focused on three things: On category pages, we clarified what each range was for, who used it and where to go next. On product pages, we moved useful proof closer to the decision. Then we added contextual calls to action where buyers had enough information to enquire. Not one lonely contact button at the end. We also cleaned up the templates, reduced page bloat and gave marketing more control without turning every change into a dev ticket. After the changes, traffic increased by 49%, new users grew by 20.5%, and user engagement rose by 145%. The campaign earns the visit. The website has to earn the outcome. Where is your site making good traffic work too hard?

📊 Average vs. Percentiles: A Product Manager's Guide to Feature Adoption Analysis - Ever wondered why averages can be misleading? Let's dive into a real-world scenario that showcases the power of percentiles in product analytics. 🎯 Scenario: Analyzing Adoption of a New Collaboration Feature. Imagine tracking user engagement with a new feature in the first month. Here's the engagement count data for 15 users: [2, 5, 8, 10, 12, 15, 18, 20, 25, 30, 35, 40, 45, 50, 60] 📈 Key Percentiles: - 50th (Median): 20 engagements - 75th: 35 engagements - 90th: 47.5 engagements - 100th (Max): 60 engagements 🤔🤔🤔 Why Not Use the Average? The average (25 engagements) seems simple but can be misleading: - Sensitive to Outliers: Poor or Power users skew the number. - Misrepresents Typical Behavior: Doesn't show where most users are. - Lacks Distribution Insight: Misses the bigger picture. 🚀 The Power of Percentiles for Product Managers: - Median (50th): Half of users engage ≤20 times (typical behavior) - 75th: 75% of users engage ≤35 times (great for realistic goals!) - 90th: Only 10% engage >47.5 times (your power users) 💡 Actionable Insights: - Aim to increase 75th percentile to 40 engagements/month. - Learn from 90th percentile users - what drives their high engagement? - Improve experience for below-median users to boost overall adoption/ 🎉 Key Takeaways: Percentiles offer a clearer picture of user behavior, helping you: - Identify user segments (casual vs. power users). - Prioritize improvements and plan A/B tests. - Set realistic, segmented goals. - Communicate feature performance effectively to stakeholders. Thanks Shikha Pandey for sharing this input.

How I helped a growth org 2x their impact... With one metric change! The context: - My team and I were supporting a head of growth - B2C subscription business - Trying to increase retention The problem: - Growth teams were working on different areas - Each trying to optimise slightly different metrics - Local metrics were moving but global metrics weren't - Turns out local metrics rewarded cannibalisation effects The solution: - Build a single proxy metric - Make it broad enough to connect to the global topline - But also made sensitive enough to pick up leading signals - Align all teams to use this as their primary decision-making metric - Maintain local metrics for secondary/supporting purposes The results: - Radical alignment around shared goal - Near 2x increase on the cumulative uplift generated on this metric over 6 months (backtested with past experiments) - Correlational evidence of improved movement in the global topline metric The takeaway: 𝗜𝗳 𝘆𝗼𝘂'𝗿𝗲 𝘁𝗮𝗰𝗸𝗹𝗶𝗻𝗴 𝘁𝗵𝗲 𝘀𝗮𝗺𝗲 𝗴𝗼𝗮𝗹, 𝘀𝗵𝗼𝗼𝘁 𝗳𝗼𝗿 𝘁𝗵𝗲 𝘀𝗮𝗺𝗲 𝗴𝗼𝗮𝗹 𝗽𝗼𝘀𝘁𝘀. 𝗬𝗼𝘂'𝗹𝗹 𝘀𝗰𝗼𝗿𝗲 𝗮 𝗹𝗼𝘁 𝗺𝗼𝗿𝗲. Data, used properly, is a great aligner for large teams.

Certainly, while wishlists have emerged as a valuable tool for gauging consumer interest, there are several other methods and metrics that e-commerce platforms can use to measure consumer interest: 1. Cart Abandonment Rate: Observing how many customers add products to their carts but don't complete the purchase can provide insights into potential hesitations or barriers. 2. Product Views: The number of times a product is viewed can indicate its popularity or interest level. 3. Time Spent on Page: Monitoring the average time consumers spend on product pages can hint at their level of interest. 4. Product Reviews and Ratings: A high number of reviews or ratings, even if mixed, can signify strong interest or engagement with a product. 5. Search Query Analysis: Observing which products or categories users are searching for on the platform can indicate trending interests. 6. Social Media Engagement: Shares, likes, comments, and mentions related to products can provide insights into consumer preferences. 7. Referral Traffic: Analyzing traffic from external sites or social media can show where the interest is coming from and which products are driving it. 8. Customer Surveys and Feedback: Directly asking customers about their preferences or interests can yield detailed insights. 9. Sales Data: A straightforward metric, but analyzing which products are selling the most can clearly indicate consumer interest. 10. Click-Through Rate (CTR): Observing how often people click on a product after seeing it in a recommendation or advertisement can be a strong indicator. 11. User-Generated Content: If consumers are posting pictures, videos, or blogs about a product, it showcases genuine interest and engagement. 12. Repeat Purchases: Products that are frequently repurchased can indicate high levels of satisfaction and interest. 13. Customer Service Inquiries: The number and nature of questions related to a product can offer insights into areas of curiosity or concern. 14. Heatmaps: Tools that show where users most frequently click, move, or hover on a page can help in understanding which products or sections grab their attention. 15. Newsletter and Email Open Rates: If consumers are frequently opening emails about specific products or categories, it can be an indication of their interest areas. 16. Retargeting Campaign Success: The conversion rate of retargeting campaigns can provide insights into the residual interest of consumers after their initial interaction. By leveraging a combination of these methods, brands can gain a comprehensive understanding of consumer interest, helping them to tailor their offerings and marketing strategies more effectively. #ecommerce #LinkedInNewsIndia

A critical part of journey management in any large organisation is measuring how your journeys perform. 📊 By setting clear goals, monitoring performance, identifying gaps, and measuring improvement impact, you create a continuous cycle of management and enhancement. Measurement surfaces opportunities and kickstarts improvements. 🚀 Yet many organisations struggle: data sits in silos, teams measure inconsistently, and dashboards report numbers without a coherent story. Product, marketing, sales, service, and digital teams collect valuable insights, but without a common language, they never combine into a unified performance view. The result? Plenty of activity, little clarity on what actually improves customer experience and business performance. Measuring performance along specific journeys—rather than isolated KPIs—provides the right context: the journey itself. 🗺️ This approach transforms your journey framework into an engine for improving both customer experience and business performance holistically, creating a shared structure and language where different KPIs unite. 🧭 Inspired by the Balanced Scorecard, this pragmatic 3x3 Matrix structures performance measurement across two dimensions: 👉 First, it distinguishes 3 performance metric categories: - Customer performance (behavior and sentiment) - Commercial performance (conversion, customer base, revenue) - Operational performance (cost, efficiency, reliability) 👉 Second, it distinct three journey hierachy levels: - Overall customer lifecycle - End-to-end product or service journey - Individual customer tasks These intersecting dimensions ensure each metric sits logically within a complete, coherent view. The visual below shows example metrics for all nine sections, helping you build a balanced measurement framework for journeys. This matrix delivers three immediate benefits: ✨ 1. It aligns siloed KPIs and contextualizes them into a shared journey 2. It enables drill-down and aggregation through connected KPIs across journey levels 3. It surfaces trade-offs and synergies between performance metrics A few quick tips to take into account when drafting or structuring your own journey-driven measurement framework 👇👇👇 🐌 Consider both leading and lagging indicators for a robust measurement approach that balances early warning signs with outcome metrics.  🤲 Don’t collect everything. Start with a North Star KPI for each journey, and add a small set of supporting metrics. Less is more. 💬 Always mix performance metrics with more qualitative feedback and insights that will help you determine why performance is down and how to fix it. Happy measuring! 🎉