Stop Running Experiments, Start Earning Them

Описание: If 80% of A/B tests fail, how do you de-risk decisions that touch pricing, product, and brand? Aleksandra (Aleks) Bass, Chief Product & Technology Officer at Typeform, shares how her team “earns the right to A/B test” with medical-grade rigor—moving from literature reviews and user tests to simulated trials before exposing changes to customers. She details Typeform’s repositioning from “forms” to an AI engagement platform—and the pricing and packaging bet behind it: a 15% drop in new business count offset by a 32% increase in ASP and a 25% lift in annual attach.
Aleks unpacks how Typeform AI acts as a co-pilot that doubled activation and boosted one-day conversion, plus the design shifts (CTA altitude and onboarding) that increased adoption. She also reveals why they moved video features down-tier and what a head-to-head test showed: video interviewers generated 14x more words and 10x fewer skipped questions with comparable completion time.
Finally, Aleks breaks down the cultural side—eliminating “anti-knowledge,” standardizing experiment design, and creating a cross-functional review that prevents false learnings—along with how her data engineering team evaluates LLMs for quality, latency, and trust.


Timestamps
[00:45] – Rethinking experimentation: “earn the right to A/B test” with staged rigor  
[03:31] – Pricing and packaging shift: from forms to flows, ASP up 32%, annual attach up 25%  
[05:34] – Typeform AI as a co-pilot: doubling activation and lifting one-day conversion  
[07:27] – Adoption lessons: elevating AI CTAs and reducing friction to use  
[10:02] – Behind the scenes: model selection, quality bars, and why MVP can backfire in AI  
[12:40] – Moving video down-tier: demand signals, cannibalization checks, and net gains  
[14:52] – Video vs. standard forms: 14x more words, 10x fewer skips, similar completion time  
[20:22] – Building an experimentation culture: process resistance, “anti-knowledge,” and cross-functional review  
[30:16] – Leader playbook: visibility, empathy, and incentives for rigorous testing


Takeaways
Implement a staged experimentation funnel—discovery, simulation, then customer A/B—to reduce risk.  
Use pricing experiments to trade volume for revenue quality; pair higher monthly prices with stronger annual discounts to grow annual attach.  
Treat AI as an activation lever: elevate AI-first CTAs and streamline onboarding to boost adoption.  
Add video interviewer options to increase response richness (14x more words) while keeping completion rates steady.  
Enforce experiment hygiene: change one variable at a time, randomize at the right unit (account vs. user), and run long enough for effect size.  
Purge “anti-knowledge” by standardizing design, instituting cross-functional reviews, and only codifying learnings supported by repeatable data.

Follow Aleks:   / aleksbass  
Check out the company: https://www.typeform.com/