A/B Testing

This episode is fundamentally about systematic testing of pop-ups. Shaan discusses testing mystery discounts versus fixed offers, testing 20% versus 15% discount percentages, testing timing variations between mobile and desktop, testing alternative offers like cashback and sweepstakes, and running holdout tests (pop-up versus no pop-up). The entire conversation centres on using data from 3,000 brands and 100 million monthly views to determine what actually works through split testing rather than assumptions. Brands learn how to track opt-in rates, conversion rates, bounce rates, AOV, and revenue from codes to make informed testing decisions.

Matthew shares extensive A/B testing insights throughout, from button colour psychology to micro-commitment text variations. Discover how testing "we'll never call you" actually decreased conversions, why "Buy Now" underperforms "Add to Cart", and how systematic testing across thousands of sites revealed the optimal button text for each stage of the customer journey.

This episode extensively covers A/B testing methodologies, with Tanner explaining how to test headlines, offers, and landing pages through micro-testing with ads before implementing changes on websites. He provides specific frameworks for testing with 400 impressions and $25 budgets.

Master the psychology behind high-converting pop-ups and offers. Understand why some brands achieve 14% conversion rates whilst others struggle at 5%, and implement Caulen's wheel pop-up strategies and customer addressing techniques for immediate improvements.

Learn how AJ Asgari's pharmacy turnkey solutions achieve 40% conversion rates through strategic testing of pre-populated cart systems and outbound calling approaches. Discover advanced methods for measuring success across multiple touchpoints and channels, from phone calls to online checkout completion.

Learn how Focus Funnels systematically tests different hooks and messaging for hero products, using data-driven experimentation to optimise conversion rates and identify winning acquisition strategies.

Learn the truth about A/B testing from Oliver Palmer, who reveals why even giants like Google and Airbnb only see winning results in 10-20% of tests. Discover the traffic requirements that make A/B testing worthwhile, why most small to medium eCommerce sites lack sufficient volume for meaningful results, and when testing actually makes sense as part of your optimization strategy. Understand why software vendors showcase rare double-digit increases whilst reality delivers far more modest improvements.

This episode is a masterclass in A/B testing methodology, covering when to test, how long to run tests, the three heuristics for determining test completion (two weeks minimum, statistical confidence threshold, and traffic requirements), and how to properly interpret results. AJ explains why A/B testing should be the final validation tool rather than the starting point, and discusses the importance of examining both wins and losses.

Kath Pay delivers comprehensive guidance on hypothesis-driven A/B testing that goes far beyond traditional subject line testing. The episode explains how to test entire customer motivations (time-limited offers versus benefit-driven messaging, gift-giving versus bargain-seeking), why understanding the 'why' behind winning variants enables application across future campaigns, and how to modify multiple elements (subject lines, imagery, copy, calls-to-action) when testing hypotheses whilst maintaining internal consistency for robust results.

Padraig Walsh reveals how understanding behavioural psychology—specifically System One versus System Two thinking—provides the framework for what to test on your website. Learn why testing familiar patterns against creative designs, cognitive load reduction through white space, and question-based engagement versus feature broadcasting can dramatically improve conversion rates. Discover the behavioural science principles that explain why certain variations outperform others.

The episode presents a contrarian 'anti-A/B testing' approach, arguing that for smaller sites, fixing obvious problems is more valuable than testing, and A/B testing can become 'smart procrastination' without sufficient traffic volume.

Understand systematic creative testing using modular components. Learn how to control variables effectively by testing hooks, transitions, and text overlays separately to identify what actually drives ad performance.

Central to Leonardo's philosophy that 'only a few tests really matter,' this covers systematic testing approaches that focus on variables with disproportionate impact. Learn how to prioritise testing for maximum growth rather than getting lost in endless optimisation.

Explore Greg Writer's systematic approach to split testing every element of the customer journey, from Facebook ad headlines to checkout page elements. Learn why even experienced marketers are only right 50% of the time and how small improvements compound to create dramatic profitability differences.

Master the testing laboratory approach for creative optimisation, including systematic naming conventions, two-week sprint methodology, and tools for tracking creative performance across campaigns.

Explore systematic approaches to improving Amazon performance through split testing titles, main images, and A+ content using Amazon Experiments. Learn how to integrate successful keywords from advertising back into listing copy to improve organic rankings and create flywheel effects.

Cody shares his methodical testing approach that transformed results over time, emphasizing testing one variable at a time (headlines, fonts, colors, button placement, popup timing) rather than multiple changes simultaneously. Discusses tracking profit per order per traffic source as the key metric.

The episode provides a practical testing framework for paid advertising, with Kenny's '3-5X product price rule' for determining when to pivot or persist with campaigns. He explains which metrics to monitor (CPM, click-through rate, CPC, conversion rate) and how to systematically test creative, targeting, and budget allocation. This systematic approach to testing prevents both premature optimization and wasteful spending.

A significant portion of the episode focuses on testing methodology, including creating 10-20 different versions of each video by varying hooks, offers, lengths, and aspect ratios. Joseph demonstrates how testing can reveal a 2.77 ROAS version versus a 1.09 ROAS version of the same video, emphasizing the critical importance of split testing in conversion optimization.

The episode reveals how to use paid advertising as an accelerated A/B testing platform. Listeners will discover why waiting months for organic traffic to reach statistical significance wastes time, when paid traffic can identify winning variations within a week. Jared explains what to test—pricing strategies, product descriptions, offers, headlines, and call-to-action variations—then how to roll winning versions out across entire sites. This approach transforms paid ads from pure acquisition channels into strategic testing tools that improve overall website performance and conversion rates.

The episode challenges conventional A/B testing wisdom with Kath's holistic testing framework. Users discover why testing subject line length whilst measuring only open rates leads to poor business decisions—short subject lines achieve higher opens but longer, more specific subject lines drive higher conversions. Learn how starting with hypotheses (testing 'benefits versus savings' as motivations) rather than variables creates robust findings across multiple campaigns, and why measuring conversions rather than engagement metrics reveals what actually drives revenue.

The episode emphasizes continuous testing and optimization as the foundation of successful traffic management. Users will understand why logging in every Monday to review and adjust Google Ads campaigns can improve ROI by 400%, the importance of testing different ad copy and landing page combinations through granular campaign structures, and how to use data to inform which approaches deserve more investment versus which should be turned off. Chloe's examples demonstrate that optimization isn't a one-time setup but an ongoing practice of testing, measuring, and refining based on actual performance data.

Adam shares their validation methodology using the scientific method and lean startup principles, including creating simple websites, spending $500 on Google AdWords to collect emails, and using crowdfunding as a validation mechanism to test if strangers would actually pay for the product.

While not the primary focus, Dan touches on testing different email approaches and timing based on data rather than assumptions. He emphasises using Google's conversion window data and customer feedback to inform your testing strategy, moving beyond simple subject line tests to examine fundamental assumptions about customer behaviour.

Discover Skip's systematic approach to testing and optimisation, where every campaign variable can be measured and improved independently. Learn how to use his conversion equation to identify which elements to test first for maximum impact.

Apply Disney's systematic process improvement approach to your business optimization. Learn the 'what, how, why' framework for testing customer service processes, measuring experience improvements, and iterating on customer touchpoints. Discover how to test service changes systematically to create measurable improvements in customer satisfaction.

Rishi shares specific A/B testing examples including a 50% conversion improvement achieved by personalising the shopping experience with a simple time-preference question, demonstrating rigorous scientific testing methodology applied to buyer psychology principles over five years of experimentation.

The discussion challenges the reduction of growth hacking to merely A/B testing Facebook ads. Users gain perspective on how testing fits within holistic strategy—as one tool among many rather than the entire approach. The episode contextualises when testing delivers value versus when foundational strategy work must come first.