Is It Time We Talked About Turning Off Your Ads? (And Yes, You Read That Right)

---
episode: 302
title: "Is It Time We Talked About Turning Off Your Ads? (And Yes, You Read That Right)"
type: guest
guest: Brenden Delarua
recorded: 2026-07-09
publish-date: 2026-09-04
duration: "52:40"
source: "Riverside recording — AssemblyAI (speaker diarised)"
---

**Matt**: So hello and welcome to the eCommerce Podcast. My name is Matt Edmundson and it is great to be with you on this exceptionally hot day here from sunny England. As we are recording this, we are in the midst of another heatwave. Yes, we are. So I'm calling you from my non-air-conditioned studio, but I'm not bitter in any way. No, I'm not. It's great to be with you. For those of you who've never been to the show before, we talk about ecommerce, I myself run ecommerce businesses, have done since 2002. So I've been around a while and I just love to get guests on the show who are experts in their field, just like Brendan today, and just grill them for an hour on their topic of choice and learn a lot. So I'm here to learn just as much as you. If you're like me, you're gonna have a notebook, you're gonna have a pen, you're gonna make lots of notes as you go through. But if you don't have a notebook and pen, don't panic. We've still got you covered because there'll be an in-depth blog post about today's day's show with all the notes, the links from the guests and all that sort of stuff. We put that on the website at ecommercepodcast.net. You will find that there. It's all free. And you'll also find all kinds of other things at ecommercepodcast.net. So go check out the website and find out all the amazing tools and archives and everything that's just on that website, which will help you run your e-com business. But yes, and of course, if you're welcome, if you're coming back to the show, if you're a regular subscriber, Don't let me, don't let me neglect saying a very warm welcome to you. You are absolutely a legend. Thank you for subscribing. Thank you for keeping listening. Thank you for all your support. It keeps us doing what we're doing. So really appreciate you guys tuning in week in, week out. And hopefully wherever you are in the world, you have air conditioning right now. Now, that said, let's get Brennan on. Brennan, how are we doing?

**Brenden**: I'm doing well. I have AC, so I'm actually pretty comfortable right now.

**Matt**: This is where you get to rub it in, right? That's okay. That's okay. We were talking about air conditioning before we hit the record button about whether we should start a transatlantic air conditioning company.

**Brenden**: I think we should. I think there's a big market for it right now. I bet every single person in the UK would buy it.

**Matt**: Yeah, I think we could do pre-sales, pre-orders. We would clean up.

**Brenden**: I mean, you're the guy to do it, right? Like, we can sell e-com, we can, you know what I mean? You could buy it all online.

**Matt**: Yeah, yeah, yeah, absolutely. Well, I tell you, I'll do the website, you do the marketing, we've got a business.

**Brenden**: If we want the marketing to be really bad, yeah, sure, I'll do it. I'll do it, whatever you want.

**Matt**: Brilliant. Absolutely brilliant. So, Brendan, for those of you who, uh, for those that don't know you, as well as air conditioning, uh, just give us a quick sort of, uh, 30-second elevator pitch about who Brendan is and what you get up to.

**Brenden**: Yeah, so it all started with my mom and my dad. No, I won't go that far. I won't go that far. No, I've been— yeah, I'll give a serious intro. I've been in paid media for the last 11 years working with a lot of different brands from B2B to e-com to lead gen to pretty much everything under the sun. I've worked for small mom-and-pop shops to large Fortune 500 companies at large agencies. But most recently I've left my last agency almost 3 years ago to pursue a causal measurement company. So like measuring what's causing sales for larger e-com marketers in a company called Stella. Yeah, I don't know how in-depth I wanna be, but Stella does incrementality testing, media mix models, and then always-on incrementality to help brands truly measure what's causing growth. invest more in what's actually, you know, improving the P&L.

**Matt**: Yeah, yeah, yeah. Well, we're going to get into all of that. I love this causal measurement phrase. I think that's quite— I'm gonna use it a lot.

**Brenden**: Yeah, I made it up just right now. I just made it up.

**Matt**: Brilliant. Well, let's just capitalize on that. So you've been doing this 4 years, you say? Right? You did, you sort of left the agency and started this 4 years ago.

**Brenden**: A little under 3 years. But yeah, we started Stella. We were doing causal measurement at the agency. It kind of came like, I tell this story a lot because it's like, well, one, it's a true story, but it's like the only story. It's like basically when I was a media buyer, maybe like 2 or 3 years into my journey as like a full-time media buyer, I had a pretty large account and they, you know, The marketing leaders there wanted me to hit a ROAS target. So across Meta, Google, Criteo, and some other platforms at the time, I can't remember what we were all on, maybe like Simplify or something. But I hit all the ROAS targets. I like smashed them. I even exceeded them. And when that QBR came for me to present the results, I presented like record-breaking ROAS. Like I hit the ROAS goal, but I got, it was, I always say on calls I got pressed, but really it's just the CFO was curious as to like, Why is revenue down? Like, if you're showing us that like ROAS is so high, like, why am I not seeing that in our actual bank account? And, you know, that sat with me for a while. I took time to think through it. And what I— the conclusion I came to, which a lot of marketers are starting to come to, is ROAS is not correlated with profit, even though a lot of people think ROAS and accounts are. So that's how I got started. So at the agency, while I was still managing accounts, I was really getting into causal inference or causal impact, geo holdouts, media mix models, because I started being able to lean on that type of measurement for truth. And obviously when you're working on client relationships, it helps when you're speaking their language. Like you don't care what the platforms say, you just really want contribution margin or net profit to go up and to the right, which is exactly what causal measurement does if you do it right.

**Matt**: Okay. Well, I, okay. I mean, I'm, I'm, before we get too headlong into this, because I have a lot of questions already, um, you obviously worked with clients varying sizes over the years. What's the biggest mistake there? I mean, maybe you've covered this already with your You comment about ROAS not being correlated to profit, but what's the biggest mistake we're making with paid media that when someone comes to you, you're like, 9 times out of 10, this is the first problem we solve?

**Brenden**: I think it's, yeah, I, well, I think it starts with the understanding of what you're looking at in an ad platform. Like what is ROAS? ROAS is a metric based on correlation. Someone saw or clicked an ad and then later converted. So that's why it deprioritizes certain channels like CTV or audio because you can't really click, right? Like you try to launch a billboard campaign and you grade it based on ROAS, like you're going to have a hard time doing that. And then as a marketing leader, your goal, the way like you keep your job or your client is by providing that ROAS that the client is asking for. So you start to, you know, maybe invest more in retargeting or branded search because that's a quick win where we can increase ROAS and the client doesn't know any better. But eventually it'll catch up to you where the business is not growing, which ultimately is kind of the path that I took a long time ago. And that's how I fell into that mistake. Like if you want a high ROAS, I can get you a high ROAS, right? But ultimately what I think it is, is business owners or people kind of give, giving the orders downstream. Don't understand how ROAS or CPA or CPL is a flawed metric. Because ultimately what every single business wants, which is what I'm finding whether they say it or not, is net profit. Like if you have a lead gen client, I know this is an e-com podcast, so don't, don't get mad at me, but like you might, you might get a target of like MQLs, marketing qualified leads, or SQLs, sales qualified leads. But ultimately the goalpost will keep moving until you get down to bookings and customers and actual net revenue or net profit. Like that is the goalpost will always move with e-com. It's much closer because someone purchases and you see that sale come in. You have the click-based attribution, you have the ROAS, but you don't know what caused that sale to happen. And that's kind of the difference. So I think it's the biggest mistake is like not fully understanding what multi-touch attribution is actually doing behind the scenes.

**Matt**: Okay, so if this is the biggest mistake, I guess that we're making in not understanding this multi-touch attribution behind the scenes and moving away. I think people go with ROAS for 2 reasons. One is what everyone talks about and 2, it's one simple metric, right? It's something that every platform tells you. It's an easy thing to look at. So where do we start, Brendan? I mean, lead us through the process of where we start to think differently about this.

**Brenden**: Yeah, yeah. I think every e-com brand has launched like before the ads have started, right? And then you launch ads, you launch Meta ads is usually the first place people go and you immediately see lift. You immediately see like sales coming in. So you, that's, those are causal sales. Those sales are caused because you turned on Meta. And for smaller brands, it's not as much of an issue, even though it's still there as like a what's causing sales to happen, right?

**Matt**: Mm-hmm.

**Brenden**: You launch Meta, you start to see immediate impact. Then you launch Google and you see even more impact too. And I mean, there's a lot to be said about like in e-com that maybe Meta is teeing up demand that Google's just taking credit for and capturing.

**Matt**: Mm-hmm.

**Brenden**: But typically when you're small enough, every new channel you launch, you start to see this like Jump in revenue, jump in revenue, jump in revenue. But then where incrementality really comes handy is for brands that are like making over 10 mil annually and they're running on like 2 to 3 or more platforms. And the idea is for e-com businesses, usually the biggest expense are the ads. And imagine if you turned off Meta and nothing happened, or if you spend $2 million a month on Meta. Imagine if you spent, or imagine if you pulled back to 1.5 million instead, and that was actually optimal spend. Like the last 500,000 is just completely wasted. You're past that point of saturation. So every business deals with this issue. It just becomes a larger business or a larger issue at scale, right? And starting small and understanding what's actually causing sales to happen. is a great practice or habit to get into to allow you to scale. The idea is we're trying to find what's causing sales to happen so we can invest more in those regions, those areas, right?

**Matt**: Those channels. Okay. It's an interesting idea, isn't it? I suppose, does it— does this idea of incrementality— you mentioned you should be turning over $10 million a year. What happens if I'm doing $3 million or $2 million a year?

**Brenden**: Yeah, so the concept of incrementality, knowing what's causing sales, it still applies. The methodology of measuring it is a little bit trickier, right? Typically what I recommend, and a lot of people, because I talk about causality, they think I'm against multi-touch attribution. I'm not. It's just, it's good to understand what multi-touch attribution is, but for smaller brands around $3 mil, What I would recommend is making sure you have a proper multi-touch attribution tool and then also setting up post-purchase surveys. So getting the qualitative data and having your users tell you where they, where they came from, also why they converted. I love asking the question on post-purchase surveys of what almost stopped you from converting because that gives you tons of insights as to like in your marketing messaging or targeting. But the post-purchase surveys really tell you the why. And even like a brand new business can start a post-purchase survey and a multi-touch attribution tool. You don't have to be a huge spender to be able to start getting 2 different data points. For example, I have a client I run ads for that is running heavily on YouTube ads on Google.

**Matt**: Yeah.

**Brenden**: And YouTube is typically infamous for having a terrible click-to-convert ratio.

**Matt**: Yeah.

**Brenden**: So the in-platform ROAS is low. So if you're really scaling YouTube, your account ROAS might be very low, right? Like if YouTube is the majority of your spend on Google, your account ROAS is going to suffer. So we were having a conversation around like, why are we spending so much on YouTube if YouTube has like a 0.2 ROAS? But when I pull up the post-purchase survey data from no commerce, Which shout out Nocommerce, I love their tool. You can see like, and I forget the exact numbers, but it was like 7% of all purchasers, or at least that filled out the survey, said they came from YouTube. And with that proportion of revenue, 7% of revenue compared to how much we're spending on YouTube was actually super profitable.

**Matt**: Yeah.

**Brenden**: It was like a 3x return based on what we can take out of the qualitative data. So now we have 2 different touchpoints, right? So we have the click-based multi-touch attribution, and now we have qualitative data. And then once you get to a level where you're getting enough volume of sales, you can start doing proper holdouts.

**Matt**: Okay, that, that makes sense. I love the YouTube story. That's quite fascinating. There's a couple of things. If I can dig into this a little bit, Brendan, multi-touch attribution tools, what would you recommend and why?

**Brenden**: Yeah. There's some I recommend. I feel like the owners of the companies don't like me because I talk about causal measurement so much. Yeah, we've had some exchanges in the DMs. I'll just say it, for e-com tools, I love Triple Whale and I love Northbeam for multi-touch attribution.

**Matt**: Yep.

**Brenden**: Yeah, I will say at Stella, we are building a multi-touch attribution tool as well. However, we're not building it to be a big player in the multi-touch attribution space. We just want to get closer to the full customer journey so we can start assigning incremental values to every touchpoint. But yeah, I'd like to plug Stella, but I honestly, I don't, the product is not there yet for the multi-touch attribution side, but it is Coming, coming along.

**Matt**: As a little side point then, uh, as I'm busy, uh, writing notes here, I'm really curious, uh, in your, so you're doing this multipoint attribution tool, um, or MAT, uh, which might just be an easier way to say it. The, so you're thinking about building one.

**Brenden**: Yeah.

**Matt**: So my question to you is, is, is why? What is it that the current tools don't have that you want to see in your system? What is the scratch that they're— that's not getting— what's the itch that's not getting scratched is the right phrase.

**Brenden**: As to why we're building one?

**Matt**: Yeah. What is it? Yes. Why you're building it. What's it going to have that the current tools don't have? If I can be as bold to ask.

**Brenden**: Sure. I mean, multi-touch attribution is interesting. A lot of people think they can vibe code a multi-touch attribution tool. So what goes into a multi-touch attribution tool is a lot of pixels and working with like the DNS or your domain provider to capture a lot of different signals and store those signals. So for example, if I don't know who you are, I don't know you're Matt and you come to our AC e-com website. to get AC in the UK. I don't know you're Matt until you convert, right? But I can see your IP address, I can see your device ID, I can see your UTMs, I can see your click ID. So I need to be able to anonymize you. So you are now like an anonymous ID. Every time you visit the site, we can ping the same anonymous ID.

**Matt**: Yeah.

**Brenden**: We capture all of your touchpoints. So if you came in from TikTok, we pull in Your TikTok UTMs and your TikTok click ID. You came in from Microsoft, we pull that, Google, whatever, each interaction. And then once you convert and we find out your name's Matt, we can stitch all that back and say, okay, Matt converted, but his first time ever on the site was 3 months ago and he came in from a TikTok ad. Or the last conversion was a branded search ad for Microsoft, right?

**Matt**: Mm-hmm.

**Brenden**: So we can see like first touch or last touch. Or we could look at it like what's called a linear attribution model where we fairly credit every touchpoint, right?

**Matt**: Yeah.

**Brenden**: So that's just the basics of any multi-touch attribution tool. What we're trying to do at Stella is, and you'll find out with me rambling on this podcast, causality is so complex. And I talk to marketers all the time that think they know incrementality or like to trick themselves into like, yeah, yeah, we do holdouts. But they really don't. And even if they do, they don't know how to operationalize it. So it's not just about measuring, is something incremental, yes or no? It's about how do we invest more in that? How do we change? What do we, what levers do we actually pull as operators in Meta, Google, TikTok, wherever to actually invest more in incrementality? Because sometimes, like based on that example I gave with YouTube, sometimes Increasing incremental ROAS is opposite of increasing platform ROAS. Like if I were to invest more in YouTube, platform ROAS is going to drop, but potentially incremental ROAS will increase. So as a media buyer, and then, you know, it's also a lot of like changing how the entire organisation thinks, because if the CFO is not on board and they're still holding you to a ROAS target, you know, You're gonna have some friction there where you're like, ROAS is flawed. I'm trying to track iROAS, but then the CFO doesn't understand it. Anyways, the reason we're building it at Stella, and we've already built it, it's just, and it works. It's just, I would say the dashboards aren't fully there yet, but the functionality works perfectly.

**Matt**: Yeah.

**Brenden**: So the idea is we look at every touchpoint and we're running holdout studies, we're running media mix models, we're running this automated causal analysis on top of all of that. Um, to track how incremental each touchpoint is. For example, if you came in from a Meta ad, a TikTok ad, and then converted from branded search, if branded search is not— and spent, let's say you spent like $100 or 100 quid, as you would say.

**Matt**: Very good.

**Brenden**: Yeah, yeah, yeah. I'm learning. I'm bilingual.

**Matt**: Yeah, absolutely. This is really going to help our marketing for the air conditioning company.

**Brenden**: Yeah, yeah, yeah, yeah. We can say, we can, we can say we're bilingual. We speak American English and original English.

**Matt**: Yeah, yeah, yeah. The OG.

**Brenden**: So, so sorry, let me, let me follow this thought. I'm sorry. Let's say there's 3 touchpoints, right? Meta, TikTok, and then you convert from branded search and spend $100.

**Matt**: Yep.

**Brenden**: First touch attribution would say Meta drove that $100. Last touch would say Google drove that $100. Every platform is going to say they drove that $100. You, if you total up all of them, you'll see 3 conversions. You'll see 1 conversion from Meta, 1 from TikTok, 1 from Google within the ad platforms, but really you only got 1 sale. So that's the difference. But the issue is maybe branded search was just there to capture because someone was familiarized from TikTok and Meta, then searched for the brand and was going to convert anyways. So what our model does is starts assigning, like if we do a branded holdout, we might find branded is not very incremental that if we we turn it off, we don't lose as many sales to where we can pass back a portion of that $100 revenue based on how incremental each touchpoint was from all of our historical analysis that we've done for your brand.

**Matt**: Right.

**Brenden**: So that's the take. So then what happens there is we can then pass back that signal. So instead of optimising your ad campaigns for purchase, which is looking at gross sales, we can pass back incremental purchase That has full context of your business: Shopify, Amazon, retail. It's looking at Google, TikTok, Meta, and we can even go a step further and send back incremental net profit or incremental contribution margin, so you can optimise your accounts towards actually what's causing sales to happen. So that's that's what we built, and I know that was a little complex, but we're trying to solve a complex problem. We're trying to simplify it. enough to where marketers can plug in their platforms, start measuring, start testing, and feeding back the right signal to ad platforms.

**Matt**: Love that. It reminds me of a chap who came on the podcast a couple of years ago, Neil Hoynt, who was a data guy at Google, right? He wrote the book Converted. And he was talking about an example. He was like, how do you do attribution? Because they tracked one lady who bought a single pair of shoes off a website and there were 236 touchpoints with the company before she purchased. Some were paid media, some were organic search, some were social media, everything, right? And so Neil was like, at what point do you attribute that sale? Because like you say, everybody claims it, which is why we have this problem, isn't it, with ROAS? Because everybody's claiming sales to 100%, where it could be, you know, split across all your channels. And so it sounds like what you're doing, if I can put it this way, is you're solving as best you can a complex attribution problem by assigning values to these multi-touchpoints, right? To try and create a system which gives you a number at the end, which makes sense to the net profit in your bank account. Versus what the platforms are telling you versus what you understand about the business. And it sounds a bit like hearing you talk about this, Brennan, that what it's not one size fits all and each business might well be unique in how it attributes across the multi-touchpoints.

**Brenden**: Exactly. There's a lot of, we even released them at Stella, but like we released like a bucketed report where We should be releasing one soon again of just all the holdouts we've done. And then so advertisers can kind of read those reports and see like what's the average IROAS of whatever platform.

**Matt**: Yeah.

**Brenden**: But it really is different. So for example, the obvious, or the obvious example that a lot of people use when they talk about incrementality is branded search, right? Like if I'm typing in Matt's AC e-com brand, which I hope that will be a domain you buy. I'm likely going to end up on that site regardless of if I click a paid ad or if there's a competitor ad showing. If I have high intent to go to your website, I'm probably going to make it to your website somehow.

**Matt**: Yeah.

**Brenden**: So a lot of people point to like branded search is non-incremental. However, I've worked with a few brands, even like at Stella, that they're, they're like the big category leader in the space. So their name brand is the Non-branded, like, I'm trying to think of what I can say. Basically, like, their name of the company was also the name of the product they sell. Yeah. So for their case, branded search was incredibly incremental because if, even if people didn't know about that brand, if they're typing in that product and then there's a brand that is like the product.

**Matt**: Yeah.

**Brenden**: You know, they're going to convert a lot of people, especially the people that are like, they want the quality product of what they're searching for. I know I'm being vague. I'm just trying not to give away client details, but—

**Matt**: No, no, no. I totally get the point. I sell omega-3 and it's like if I called my company Omega-3.

**Brenden**: Exactly. Yeah, yeah, yeah. Like you are the omega-3. Yeah, it's pronounced omega, Matt, by the way.

**Matt**: Sorry, I'll try my best. I'm very sorry.

**Brenden**: That's okay. I'm sorry. But exactly. But then there's also the case of like Maybe you're in an industry where a lot of competitors are being very aggressive towards your brand terms and you having branded search is helping you drive increment. So it really depends. And what I always say is, I met a guy who said, I don't need to test. I have all these benchmarks from everyone putting out data. And I was like, it's really super different for every brand. You should be testing yourself.

**Matt**: Yeah.

**Brenden**: As well as you can.

**Matt**: Well, let's, if we can, let's get into testing. And I want you to first, if I'm, if I can ask you to just define for the sake of clarity, a few terms which you've used a couple times for those that might not know, iROAS and holdouts.

**Brenden**: Sure. iROAS stands for incremental return on ad spend, which is ultimately what most, most brands that are optimising for a ROAS target, what they really want and they don't know is the incremental ROAS target, which means every dollar we put into an ad platform, we get back money that would not have happened without that ad platform, without that investment. That's what we like to think or trust is what ROAS is telling us. But like I said earlier, based on how ROAS is made as a metric, It's correlation-based. It does not actually tell— it tells you who saw or clicked an ad and then later converted, but it doesn't tell you what caused that revenue to come back. So that's what I ROAS is, the incremental return on ad spend.

**Matt**: Fantastic.

**Brenden**: Incrementality or holdout studies are ultimately like the easiest way to explain it is just simply turning off ads in certain regions and monitoring your source of truth revenue. So for a lot of e-com brands, that's Shopify. in those regions compared to the regions that stayed live. And there's a lot of statistical nuance that goes into it, but that's ultimately the core of what holdout studies are. You're holding out a region and you're trying to see, does revenue drop in that region at all? And if so, how much? And can we be confident that that drop is because we turned off the that media channel.

**Matt**: So how do we, how do we do these holdout tests then? Because I can hear however many thousand people listening to the show right now going, hang on a minute, you want me to turn off ads? You want me to do what?

**Brenden**: Yeah, yeah. And we get that a lot. We actually signed a client that for Stella, that was not aware that they were going to have to turn off ads, even though we like explained it to them. They're like, we're gonna have to turn off ads. And I'm like, yes. So yes, so there is a risk, right? When you turn off ads, if the ads are incremental, there's a risk of losing revenue that would've been caused by those ads being live. However, the whole point is to measure, to see what is incremental and how incremental it is.

**Matt**: Mm-hmm.

**Brenden**: Because it's not about, you know, it's not about is this platform incremental, yes or no? It's about how incremental is it right now at our current spend point and how incremental could it be potentially? We might be underinvested in multiple channels and we might be at a point or past a point of saturation for other channels, right? Like we might be with the exact same total budget. We might be able to drive more revenue if we just change the allocation. So that's the part where Like, it is a risk, but the idea is the value that comes from, um, the insight can actually propel the business much further.

**Matt**: So how do—

**Brenden**: where do I—

**Matt**: I guess my question is, let's say I'm running, I don't know, 100 different ads, um, across Meta, um, I'm doing Google Ads, I've got my Amazon paid things going on because I'm trying to sell on Amazon as well. I've got, I mean, I've got multiple different places. Am I doing like one at a time? Am I doing, what's the, I guess, what's the strategy of attack if I'm going to start to do this that I need to think about?

**Brenden**: Yeah. I mean, that's the fun part. That's like the part we get to figure out. Um, so what we typically recommend at Stella is to start at a channel level holdout. So let's turn off all of Meta in certain regions. And there's different types of holdouts.

**Matt**: So regions, you mean geographical regions?

**Brenden**: Geographical regions, yeah. So we want to make sure no one in that area is getting any, getting served any impressions from whatever channel we're testing, right? Because then we want to see like, you know, the drop in revenue that starts to come and then how, how long, how long does it take for us to see that drop so we can start looking at the lagged effects of Meta as well. Usually holdouts run for 20 to 30 days depending on the channel.

**Matt**: Oh, wow. Okay.

**Brenden**: CTV or like more top of funnel channels might run for 45 days. But usually that's what we recommend is between like 20 to 45 days at maximum for a holdout study. You can also, the other side of this is actually turning on ads in certain regions. So like if you're not activated on like Twitter ads or I don't know, like Tatari or CTV, You can activate in certain regions and hold out other regions too to see an incremental lift in those regions, which is a little less scary for brands, right? But yeah, so we're looking at geographical areas to turn off because we don't want to serve any impressions. And then we want to see how does that affect our actual source of truth revenue. Oh, so yeah, yeah, I'm sorry. I was Yeah. So we start with the channel level. So all of Meta, all of, and then once that 20-day increment is up, you go to Google, you go to TikTok, every other channel you're on. The idea here is we're trying to establish incremental base, baselines. We wanna see how incremental is Google as a whole. And we all know Google has tons of different types of campaigns. We have PMax, we have search, we have shopping, we have YouTube, we have demand gen, we have display.

**Matt**: Yeah.

**Brenden**: So we can look at, okay, we turn off Google and we lose X amount of our revenue, which gives us a 3x incremental ROAS of Google. Now we can start testing again with a, with a new process where we go into each tactic. So maybe we hold out all non-branded campaigns and we keep live all branded campaigns. So we can start to see like what's weighing that ROAS from Google down and what's pulling it up, right? So Where should we be investing? And then you can do the same on Meta. Like depending on how your Meta accounts are set up, you could do a holdout on just acquisition campaigns, like just prospecting campaigns compared to retargeting, or depending on your spend level, you could just do a holdout on one specific campaign at a time. So you can start to assign the incremental value of each individual campaign. So it's kind of like this cyclical pattern where you do channel level. tactic level, campaign level.

**Matt**: Yeah.

**Brenden**: And once you finish that sequence, you should actually restart and go back to the channel level. The idea being incrementality and holdout studies are a snapshot in time. It all depends on macroeconomic factors, the ads you're running, your bid strategy, the algorithm that Meta has right now or Google, right? Next month, things might change. How incremental your ads are on Meta, On Meta in February is going to be so different than your ads in November when like Black Friday, Cyber Monday come up. So it shifts over time. So it's, it's not enough to do one holdout and like point to that number. We should be doing constant holdouts. And of course, I'm the guy selling you the holdouts. So like I'm biased, but the idea is you can't just reference one holdout for like the next 2 years. It had—

**Matt**: Yeah.

**Brenden**: It will shift. And that number becomes outdated, something we call causal decay. Like the moment you get that IROAS report back, the next day it's going to start to shift.

**Matt**: This is really fascinating. I guess, how do you deal with, like I say, the 1,000 people listening to the show going, hang on, I'm going to lose sales here. What, what, what have you seen customers gain? Because I think that There's going to be a large amount of inertia to this idea, which I understand. But obviously you do this successfully. And so you see customers gain, obviously, insight, you can reallocate budgets better and more efficiently. But how long is it before, I guess, people start to see the payback? How deep do their pockets need to be to ride this out? I'm, I'm, I'm, you must get asked these questions all the time, right? And I'm really curious.

**Brenden**: Yeah. So I think the funny part is, like I said, like there's a lot of marketing leaders that are diving into incrementality or causal measurement because they're hearing it a lot. They see a lot of blog articles. They're gonna watch this podcast with me talking about incrementality. They see people like they, I think there's a shift in the market right now where people think that's where I need to be. I need to be around the incrementality, but there's like a lack of understanding. Right? Like there's a lack of understanding how to use it. So I even see people at Stella, like we try to work with our clients directly. Stella is a self-serve platform, but we do have some clients that are like on a consultancy type of retainer where we are like making changes for them. But the idea is incrementality studies are just one part of it. It's like a snapshot in time of how incremental something was at that point in time, at that spend level, with that bid strategy, with your ads, with however many competitors are in your market. It's a snapshot in time. What we think is the most powerful way of doing causality is then moving into another form of statistical measurement, which I'm— I know, I know it's hard to follow. I'm hardly— I'm hardly following what I'm saying right now. Also, you booked this at 4 AM Eastern time. So like, I am like, it is very early right now. So it's not though. It's like, it's 9:42. We're good.

**Matt**: Yeah. And you booked it. I just want to be clear. Anyway, carry on.

**Brenden**: Whoa, whoa, whoa.

**Matt**: Don't point fingers, Matt.

**Brenden**: When you Point a finger at me, you point 3 back at you. Yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah. It's the heat in your room. It's getting to your head.

**Matt**: It is. It must be getting to me. Yeah, yeah, yeah, yeah, absolutely.

**Brenden**: Yeah. So, um, so holdout studies are one form of— now I have to be serious again.

**Matt**: Hold on, let me go serious face.

**Brenden**: Okay, so holdout studies are one form of statistical methodology, but it's a snapshot in time. There's another form of statistical methodology that we also support at Stella called Media mix models. Media mix models look at like a 2-year time span of your data and it starts looking at other things like what's something called Fourier contribution, which is seasonality. So it looks like how do ads perform in March compared to April compared to May? It starts to look at things you can add in control variables of like, this is when I have sales throughout the year. These are the emails I send. This is the non-branded Uh, term on Google Trends, like you can see kind of like, okay, the popularity of the industry is increasing or maybe it's decreasing. Maybe a new competitor launched and you see their search queries start to increase because they're taking up more market share. Yeah. Um, media mix models are more holistic and you can calibrate them with holdout studies. And it's actually better if you do calibrate with a holdout study. So the idea then becomes media mix models can start looking at forecasting. So a media mix model can understand the variance in your data, like what causes revenue to go up or down. Again, calibrating with a holdout weights it in a different form of causal analysis too. So now we have 2 different forms of causal analysis working together. But then once it understands like, okay, if we spend an extra $100,000 on Google, we know how that should affect revenue.

**Matt**: Yeah.

**Brenden**: We know how ads should perform in April or May or like whatever the next quarter is. Like we understand that seasonality. Media mix models are great at then using what's called like a budget optimizer or a scenario planner. Most media mix modeling tools have something along those lines. Obviously we have one at Stella. So what we do is we build a media mix model that is very accurate based on metrics like MAPE, R-squared, in-sample, out-of-sample. There's a lot of ways to verify how accurate a media mix model is. And then you go into the budget planning or a scenario planning and you forecast out the next like 3 months. So like, this is our budget for the next quarter. This is, uh, you know, how, what, how can we use this budget to maximize incremental revenue? And that's what media mix models will tell you. It'll tell you, you know, this channel's at saturation, we need to pull back. This other channel, we need to increase revenue. And this is With your given budget, this is how you maximize incremental revenue. So that's the power of causal measurements, this umbrella that has holdouts and media mix models. But our opinion, yeah, our opinion is you should use both. You shouldn't just do holdouts or just media mix models. You should be doing them both together because media mix models are more forward-looking to understand How much can I be spending on these platforms before my next dollar is not returning as much as it could be somewhere else?

**Matt**: Yeah, which is incredibly powerful information.

**Brenden**: Exactly. But like, I— and I understand it because I live and breathe it now. I didn't understand it as much as I would have liked to when I was at the agency, right? Like, because obviously I'm doing this every day, but I'm still seeing that pattern emerge where Maybe some marketing leaders understand that it's powerful, but it gets over— it gets overwhelming looking at all this data science because they're marketers. They're not data scientists, right? They're not statisticians. So that translation into what I'm showing you and then what it's telling you, like what to do next, that's what we're trying to bridge at Stella. So not just measuring causal growth, but knowing what to do next, where to invest.

**Matt**: Yeah.

**Brenden**: increase causal growth.

**Matt**: That's the, the trick, isn't it? Because, um, I can imagine actually a lot of people listening to the show are going to go, well, this sounds all great and, um, I like the idea of it, but how I get started and how I maintain this is a very different thing. Because it's— it doesn't sound like it's something that I can just go to Claude and go, Claude, could you do that for me? Um, I, I—

**Brenden**: it's—

**Matt**: it sounds like it's a lot more involved, which I then puts a high barrier to entry on it for, I would have thought, a lot of small business owners.

**Brenden**: Yeah, exactly. Like I said, like, I think it's just important. You don't, you don't necessarily have to do holdout studies, right? Like, that is the— people call it the gold standard of measurement. I don't know who came up with that, but I love it because it's good for business. But yeah, but you don't have to do a proper holdout study to understand the concept of incrementality and know even if you're making $1 million or million quid just for your audience a year with your e-com business, like you're still dealing with these same issues. They're just at a much smaller scale. And as you scale, it's going to get harder and harder to know what's causing growth to happen.

**Matt**: Yeah.

**Brenden**: So it's important to not like, I guess the main message for like the smaller advertisers, smaller e-com brands is Understand what ROAS or CPA is actually tracking. Make sure you have multiple forms of measurement. What I like to do is multi-touch attribution, post-purchase surveys, and then when you're big enough, causal analysis too. I think that really closes the loop on like what's actually causing growth for your company.

**Matt**: Yeah, yeah.

**Brenden**: But a lot of people are stuck on just multi-touch attribution, or even worse, just looking at in-platform metrics, which eventually will fail you the way they failed me. Yeah. So I think that's the biggest thing is like, you don't have to do holdouts to understand the concept of what's causing sales to happen might be different than what you're seeing in ad platforms.

**Matt**: Yeah. Fantastic. I— one question, I suppose I'm thinking about this. If you use different agencies across the ad platforms, like you've got a Facebook agency over here or Meta agency, you've got a Google guy over here. Um, with this kind of system, are you better off having, I guess, one agency across all platforms that understands what's going across everything, or does it still work effectively with multi-agency?

**Brenden**: So I, I've just always been a thing, regardless of causal measurement, I think it's really powerful to have one agent agency managing your entire ecosystem because there's better collaboration.

**Matt**: Yeah.

**Brenden**: I think my clients, so there's a few consulting clients I work with like directly and like I control the whole strategy of what's happening with their, you know, with their paid media. And I think that is so effective rather than having so many cooks in the kitchen and not understanding like, yeah, you know, we pivoted Google to this strategy because we wanna see this, but then on the other side, your Amazon agency sees an effect happening and they need to spend or else they don't get their percentage of spend. And like all the incentives are like misaligned. So I'm always a big advocate of just having like one agency or one main brain for the strategy. Obviously there's like the VPs of marketing at the company that could kind of control that. But I think without the direct communication, I think also sometimes agencies feel competitive that like maybe you have an agency running your Meta and a different one running Google, but maybe the Google agency also has a service line.

**Matt**: Yeah.

**Brenden**: Where they run Meta, so they might be trying to poke holes in the Meta strategy to get that business.

**Matt**: Mm-hmm.

**Brenden**: I'd rather just have it all to one. When it comes to causal measurement, it is hard to coordinate all this. It just makes it harder, especially when there's more cooks in the kitchen. What I would say is it can come down to the main decision maker, the marketing leader that understands where we need to be investing in the ad platforms. Stella, we have a tool we call the Always-On Incrementality Tool. It does an automated causal analysis. We're very candid with people that like the more you automate, the less accurate it becomes.

**Matt**: Yeah.

**Brenden**: So we always recommend do a proper holdout, do a proper MMM, and then calibrate the Always-On tool. The Always-On tool will show you at a campaign level where to be spending that week to maximize incremental returns on whatever sales channels you've connected. So it can be Shopify or it could be Shopify and Amazon combined if the company wants that. So if all your agencies are plugged into Stella and can see like what's causing growth to happen, the idea is Stella becomes the central brain of the operation that all your ad buyers can start to see if everyone's aligned on the same goal of driving incremental growth rather than just like platform reported ROAS.

**Matt**: Mm-hmm.

**Brenden**: that's a great place to start. I think, yeah, I don't know. I know I was rambling. But basically, like, I think it's always a net positive to have like less people controlling the strategy.

**Matt**: Yeah. Very good. And the— just quickly, well, actually, let's do this. Because I'm aware of time. So first things first, question for Matt. This is where I ask you for a question for me. And I will go away and answer on social media. What is your question for me, Brendan?

**Brenden**: Yeah, I, I want to know your craziest, like, client story or your craziest, like, account story. Like, if you've worked with other marketers, um, I have a bunch in my head that I don't think I can ever say, but I want to hear from you. I want to hear what your story is.

**Matt**: Crazy, crazy account. Okay, I'll answer that on social media. If you want to know my answer to that question, come follow me at Matt Edmondson. But Brendan, listen, I— if people want to reach you, if they want to connect with you, they want to find out more about Stella, what's the best way to do that? Where do we go?

**Brenden**: Yes, if you're a serious marketer and you want me to be serious and give you real advice, you can follow me on LinkedIn at Brendan de la Rúa. That's where I'm a little bit more buttoned up, as buttoned up as I can be. And if you want more jokey, haha content, that's still educational and talking about marketing and causality, you can follow me on my TikTok @brendanbuilds. I try to play a good balance on TikTok and LinkedIn and Twitter where I'm posting different things on all the platforms. On LinkedIn, if you wanna really learn about incrementality, that's where I would follow me. If you want a goof and a gaffe and some sarcastic content, I'd follow me on TikTok.

**Matt**: Always like sarcastic content. Uh, we will of course put those links in the show notes. So if you're listening on a podcast player, just scroll down, uh, you'll see the show notes, the links to Brendan. Uh, if you're watching on YouTube, just click in the description, the links will be there. Of course, if you're subscribed to the newsletter, they'll be in your inbox. Um, and they'll be on the website, like I say, with the blog post, they're going to be everywhere. And if you're a Slingshot AI user, just ask Sam to put you in touch with Brendan and he'll be It will do it. So Brendan, listen, firstly, thank you. Great conversation and eye-opening in many ways. And I think you've answered a lot of questions people have about ROAS and why it's not working well for them. But before we go, we like to do this thing called saving the best till last. And this is where I would love to hear your top tips on the post-purchase survey. So you've mentioned that a couple times as one of the key things to do. And you mentioned one of your favourite questions was what almost stopped you from buying, which I think is a great question, or words to that effect at least. But if I'm going to start to think about these post-purchase surveys, again, people get very nervous because they think it's friction. What's your top advice for anyone running an ecommerce business out there? The microphone is yours for the next few minutes. Over to you, my friend.

**Brenden**: Yeah. One shout out, Jarrell Blades, the VP of growth at Tushy. He's the one who really explained that to me about asking what almost stopped you from converting.

**Matt**: Right.

**Brenden**: I think, you know, there's a book, The 7 Habits of Highly Effective People, that talks about begin with the end in mind. So when you're building out a post-purchase survey flow, you kind of have to wonder like, what are the questions that are gonna help me actually optimise the funnel? Because post-purchase surveys are happening after someone's already made the purchase. You've already converted them, but now the idea is take as much information as you can that's going to help you convert more people better, right? Like, what's the reason they're buying it? I work with a lot of supplement brands that, you know, have people have different ailments or injuries that they might be buying a certain product, and there might be new ones that start popping up, uh, and where we can start making ads specifically on whatever they're talking about, right? Yeah. So appeal to a wider audience people, which allows us to scale meta, right? Post-purchase surveys are just, they also tell you like more word of mouth where on marketing, everything's going to be click-based, like on multi-touch attribution, you're going to see someone clicked an ad or saw an ad, but really what caused them to convert was their friend told them about it, or they went over to their friend's house and used their shower and they had this like shampoo they've never had before. And it, you know, made them go bald or maybe opposite.

**Matt**: Yeah.

**Brenden**: Grew their hair. It was finasteride the whole time. You know what I mean? And then they go and buy it and that's why, and they ended up clicking an ad because they saw it. Right. But I really love post-purchase surveys. Like any company can have them, lead gen, e-com, B2B. Like why did you convert? What ad do you see? There's some post-purchase surveys, like if you use Nocommerce on their premium plan, they have conditional logic. So if you say, I saw an ad from Meta, that might serve you. The next question is, which ad did you see? So now we can start asking like, which ad? Was it a video? Was it a static image? And a lot of times when you're going through the customer journey in general, you're seeing multiple ads. So the ad that they're gonna tell you they saw, if they're being truthful, there's also that caveat with post-purchase surveys. Yeah. Some people just click whatever.

**Matt**: Yeah.

**Brenden**: Is the ad that was most memorable, the ad that they remember the most, right? It might not be the one that they just clicked. Clicked to confirm. 'Cause we can QA that with the UTMs, with the post—

**Matt**: Yeah.

**Brenden**: With the multi-touch attribution data, we can see which ad they clicked on, but maybe they're saying they remember an ad that they did not click on.

**Matt**: Right.

**Brenden**: That we didn't see. And we're like, wow, so this ad must be very impactful here. So that's what I love about post-purchase surveys. We're also building one at Stella. It already works. We're just waiting for approval from Shopify. So we'll have the full brain within Stella, where you'll have your causal measurement, you'll have your multi-touch attribution and your post-purchase survey all within Stella. So we can start tying back the entire customer journey and sending back the right signal to all of your ad channels, which I think is going to be really cool. But it just comes from the love of post-purchase surveys and the value that you get from that qualitative data.

**Matt**: Fantastic. Fantastic. Brandon, listen, uh, thank you so much for coming on the show, man. Really enjoyed the conversation. Love what you guys are doing at Stella and, um, hope the launch goes well of all the different myriad tools you've got coming up, uh, anytime soon. Um, love the sarcasm, uh, and appreciate the, uh, the tip on air conditioning.

**Brenden**: Matt, I will, I will close Stella. I will end Stella. You know, I don't even care. You and me, if you're down, I want to start the AC company. There's so much more opportunity in that than what I'm doing.

**Matt**: There is. There is. Yeah. And the other one I thought about, uh, whenever I come to the States, I'm like, we should do solar power here in America because it just seems crazy expensive to me. And in America, for some reason, I'm not quite sure why. Um, whereas in the UK, it seems to be much more sensible and we have less sun.

**Brenden**: Our constitution says, uh, we don't like solar panels. Solar power, uh, the founding fathers said do not. Get solar panels. We actually get a lot of people door-to-door selling solar all the time, and we're like, you ain't from here.

**Matt**: You ain't from around here.

**Brenden**: Yeah, we like fossil fuels. We like pollution. Yeah, that is American. That is uniquely American.

**Matt**: Yeah, definitely. I get it. I do. I do. Brilliant. Yeah, what a legend. Really enjoyed this conversation, Brendan. Thank you so much, man. Really appreciate it.

**Brenden**: Yeah, thank you for having me.

**Matt**: Well, there you go. Another fantastic conversation lined up on the eCommerce Podcast. Wasn't that great? Loved it. Make sure you do connect with Brendan across his various channels, whether it's LinkedIn, whether it's TikTok, whatever it is. I'm sure he would love to hear from you. But yeah, everything, like I said, about eCommerce Podcast on the website ecommerce podcast.net. You can find everything out about us. Uh, but that's it from me this week. Have a phenomenal week wherever you are in the world. Uh, but from Brendan and from myself, bye for now.