Product Validation Methods

Rick Rubin went on stage in Helsinki the day after my talk. Someone asked how he resolves creative differences with artists. His answer was simple: change the conversation from "I disagree" to "let's build it." Then he shared a story: An artist played him a song. The transition didn't work. Rubin told him so. The artist said, "We'll just cut that part in half." Rubin thought to himself: What a dumb idea. But he didn't say that. He said, "Let's try it." The artist played it. It worked. Rubin is a legend. He's produced everyone from Johnny Cash to Jay-Z. Instead, he bit his tongue and let the artist prove him wrong. The principle: when you make an idea tangible, it stops being the person's idea. It becomes something you can both look at objectively and improve together. Once you build it, the truth is obvious. Here's what this looks like in practice: Your designer wants to change the entire homepage layout. You think it's too risky. Instead of three meetings debating it, you say: "Let's build a prototype and test it with 50 users this week." Your sales team wants to restructure the pricing page. Instead of blocking it because you're worried about conversions, you say: "Let's run it as an A/B test on 20% of traffic for two weeks." Your engineer wants to rebuild a core feature from scratch. You think it's overengineered. But instead of killing it in the planning phase, you say: "Spike it out in three days and show me if the performance gain is real." You're not saying yes to everything. You're saying, "Let's find out." Rubin also said something that stuck with me: "If there's disagreement, I always side with the artist's vision. Because to them, it's their career. To me, it's just one piece of my portfolio." Most leaders think backing down makes them look weak. Rubin knows that siding with the person who has the most at stake makes better work happen. Your job isn't to be right. It's to create the conditions where the best idea wins. Stop debating. Start building. P.S. This insight is from this week's newsletter where I break down why Yamaha dominates while Steinway got sold to private equity: https://lnkd.in/efSqP_9K P.P.S. Access additional research links, the podcast, and the full archive in the first comment 👇 Thank you to Nordic Business Forum!

One of the biggest mistakes in growth is confusing “people use it” with “people love it.” So at Lovable, we built a composite qualitative metric to measure actual product love. The Lovable Score is a combination of: Net Promoter Score | NPS -> How likely are you to recommend? Sean Ellis PMF -> How would you feel if you could no longer use us? CSAT -> How satisfied are you? CES -> How easy was it to use? Any one of these on its own is usually too flawed, too narrow, or too easy to misread. But together, they start to tell a much more useful story. Our Lovable score has the following formula and weights: 35% NPS + 25% PMF + 20% CSAT + 20% CES Your mix might be different. But I’d really push every team to define what “love” actually means for their product, then measure it. Wrote more about how we built it here: https://lnkd.in/emQr6TFS And big shout out to Passionfroot, my awesome blog sponsor this week, who you should work with to scale your creator-led growth! #growth #lovable

BREAKING! The FDA just released this draft guidance, titled Artificial Intelligence-Enabled Device Software Functions: Lifecycle Management and Marketing Submission Recommendations, that aims to provide industry and FDA staff with a Total Product Life Cycle (TPLC) approach for developing, validating, and maintaining AI-enabled medical devices. The guidance is important even in its draft stage in providing more detailed, AI-specific instructions on what regulators expect in marketing submissions; and how developers can control AI bias. What’s new in it? 1) It requests clear explanations of how and why AI is used within the device. 2) It requires sponsors to provide adequate instructions, warnings, and limitations so that users understand the model’s outputs and scope (e.g., whether further tests or clinical judgment are needed). 3) Encourages sponsors to follow standard risk-management procedures; and stresses that misunderstanding or incorrect interpretation of the AI’s output is a major risk factor. 4) Recommends analyzing performance across subgroups to detect potential AI bias (e.g., different performance in underrepresented demographics). 5) Recommends robust testing (e.g., sensitivity, specificity, AUC, PPV/NPV) on datasets that match the intended clinical conditions. 6) Recognizes that AI performance may drift (e.g., as clinical practice changes), therefore sponsors are advised to maintain ongoing monitoring, identify performance deterioration, and enact timely mitigations. 7) Discusses AI-specific security threats (e.g., data poisoning, model inversion/stealing, adversarial inputs) and encourages sponsors to adopt threat modeling and testing (fuzz testing, penetration testing). 8) And proposed for public-facing FDA summaries (e.g., 510(k) Summaries, De Novo decision summaries) to foster user trust and better understanding of the model’s capabilities and limits.

👩🦰 Persona Spectrum For Inclusive Design (Figma Kit) (https://lnkd.in/eGD38hs4), a wonderful little accessibility tool for designers to include permanent, temporary and situational contexts in design decisions. Open sources, with all illustrations and assets for presentations and print. By 🐝 Mahana Delacour. --- 🔶 1. Accessibility ≠ Compliance We should never rely on automated accessibility testing alone to “ensure” accessibility. Compliance means that a user can use your product, but it doesn’t mean that it’s a great user experience. Manual testing makes sure that your users actually can meet their goals in their own context. It often feels daunting to get started, but small first steps are a great beginning. First, gather people interested in accessibility. Document what research was done, where the gaps are. And then try to include 5–12 users with disabilities in a dedicated accessibility testing. One way to find participants is to reach out to local chapters, local training centers, non-profits and public communities of users with disabilities in your country. You might want to add extra $25–$50 depending on disability transportation. Once you have access to users, run a small accessibility initiative around key flows in your products. Tap into critical touch points and research them. Eventually extend to components, patterns, flows, service design. A good target is to incorporate inclusive sampling into all research projects — at least 15% of usability testers should have a permanent, temporary or situational disability. --- 🔹 2. Building Accessibility Research From Scratch If you’d like to get started, I highly recommend to check “How We’ve Built Accessibility Research at Booking.com” (https://lnkd.in/eq_3zSPJ), a fantastic case study by Maya Alvarado on how to build accessibility practices and inclusive design into UX research from scratch. Maya highlights the idea of extending Microsoft's Inclusive Design Toolkit (https://lnkd.in/eN5J7EkJ) to meet specific user needs of a product. It adds a different dimension to disability considerations which might be less abstract and much easier to relate for the entire organization. And as Maya noted, inclusive design is about building a door that can be opened by anyone and lets everyone in. Accessibility isn’t a checklist — it’s a practice that goes way beyond compliance. A practice that involves actual people with actual disabilities throughout all UX research activities. More resources in the comments ↓

I made Physical AutoResearch sound simple (conceptually), but it took a village to pull off and lots of design thinking into the robot /loopcraft. The hardest part is everything we need to setup *before* pressing Enter. We released ENPIRE yesterday if you missed it: https://lnkd.in/gXeZzMYb Project site: https://lnkd.in/g3t4qS8Y Here's a behind-the-scene tour: 1. Safety harness Letting 8 robots run unattended overnight means safety has to be more than a hint in the system prompt. ENPIRE hardwires it in 2 layers: (1) hard kinematic limit that trips an immediate task failure and auto-resets as soon as a robot leaves its safety envelope, and (2) a torque-limited compliant gripper so a bad contact or misaligned insertion ends in a safe stall, instead of crushing the robot or the object at hand. We make safety more conservative than usual so humans can sleep tight. In reality, we still need a few human operators to watch over the "robots of loving grace". 2. Definition of /done An agent that can edit its own reward will game it for sure. ENPIRE fixes the goalposts before the fleet can move them. Here's the recipe: Collect a few minutes of success & failure demos -> Ask agent to write code using computer vision tools to classify success and measure against groundtruth -> Agent hill-climbs on classifier until reliably good -> This classifier becomes the real-time reward function that directly computes on sensor streams -> *Freeze* the reward function before AutoResearch. It's sacred, enshrined in a Gym env that no one can touch. 3. System telemetry design Robot-seconds is by far the scarcest resource, followed by GPU-seconds, and finally tokens. We instrument all three and surface them to ENPIRE for live resource awareness rather than letting it hill-climb in a vacuum. We define: - Mean Robot Utilization ("MRU"): the fraction of wall-clock time when the robot is actively executing an experiment. Otherwise the hardware is sitting idle and waiting for the next code commit. - Mean Token Utilization ("MTU"): tokens consumed per minute, our proxy for how hard the agent is actually thinking. A low MTU means the agent is stalled, waiting on a robot rollout to finish instead of doing research. - GPU utilization: fraction of wall-clock time when GPU is active. ... and evaluate on two budget-to-outcome metrics: 1. Tokens-to-Success: token budget the fleet burns to complete /goal. 2. Time-to-Success: wall-clock time to /goal

Product-market fit (PMF) isn't a binary. The reality is that there are *shades* of PMF and you need an *action plan* to get there. Maja Voje — better known as the GTM Strategist — has worked with 350+ startups to help them achieve & expand on PMF. She's now sharing her tested frameworks with the rest of us. Here's the TL;DR: 1️⃣ Proof of concept: Get 10 testers - These tend to come from your personal network, advisors or warm outreach (from a founder) with a hook - Show "problem-solution fit" by starting to document "can we even solve this problem?" metrics with case studies 2️⃣ Proof of monetization: Get 5 paying customers - These come from retained PoC testers, cold outreach to adjacent segments, case studies sent as warm outreach with a hook, or via influential people in your network - Pro tip: you need an early customer profile before you can get to an ideal customer profile (ICP) 3️⃣ Proof of 1+ scalable GTM motion: Reach 20+ paying customers - Your GTM options: inbound (content), outbound (cold outreach), paid digital, community, partners, ABX and/or PLG - Pro tip: you need differentiated positioning to unlock this GTM motion; Maja's recommendation is to always position in relation to *something* (a service, DIY process, doing nothing or direct competitors) 4️⃣ Proof of a sustainable business model: Reach 50+ paying customers - If you were to only use this 1+ scalable GTM motion, would you be able to become break-even / profitable? - Look at: retention/churn, acquisition costs, customer referenceability 5️⃣ Proof of market expansion: Reach 100+ paying customers in 2+ markets - There's now clear evidence that you're ready to win on more fronts: opening new markets, launching new products, selling to new personas --- Read the full piece in Growth Unhinged: https://lnkd.in/guUFj-5H My favorite quote: "I like to think of PMF as a cycle... Every time I fail to validate something, I remember that Nokia started with toilet paper, Lamborghini with tractors, and McDonald’s with hot dogs." Can't wait to hear what you think 🙏 #pmf #startup #gtm