The Case For Conversion Rate Optimisation: Experiment To Learn

welcome to the ecommerce podcast with matt edmondson a show that brings you regular interviews tips and tools for
building your business online
[Music]
hi and welcome to the e-commerce podcast with me your host matt edmondson all of
this week's notes and links can be found on our website for free at ecommercepodcast.net forward slash
and this week we are talking about how you can stop guessing about your website
and the changes that you make and what's going to affect the bottom line we're going to get into the nitty-gritty
of how to do it the right way so don't go anywhere hey there are you a business owner here
at oregon digital we know firsthand that running an ecommerce business can be really hard work
as the online space gets more competitive it is becoming even more challenging to stay ahead of the curve
we totally get it so we want to help you succeed by offering a wide range of services from fulfillment marketing
customer service and even coaching and consulting just so that you can do what matters most save yourself the time and
the money and let us handle the day-to-day tasks this way you can run your business without having to worry
about the boring stuff so what do you say are we a good fit for each other come check us out at oriondigital.com
and let us know what you think
thanks for joining us on the ecommerce podcast it is great that you are here
now whether you are just starting out you may be even just thinking about getting involved in e-commerce or you
may have been around a few years a bit like my good self wherever you are on your ecommerce journey
you are in the right place this is a show for you that's going to help you grow your e-commerce and digital
business and to do that every week we bring you great show sponsors who are
going to help you as well as amazing guests experts in their own field with
stories and insights that can help us start adapt and grow online and this
week's phenomenal guest is the legendary aj davis she is a conversion rate optimization
specialist or cro for those in the know she is the founder
of experiment zone now experiment zone if you haven't heard of it helps your online businesses grow uh by improving
your user experience on your website now prior to starting experiment zone in
aj led optimization strategy for fortune companies during her tenure
at clearhead she was also the lead ux researcher on the google optimize
product so yep she knows her stuff she is the right
person to talk to let me tell you an absolute wonderful lady i really really enjoyed this conversation this interview
with aj so without further ado here we go so aj welcome to the ecommerce
podcast great to be with you all the way from sunny austin texas uh where we were
talking before you came uh before we started the recording that you were been playing pickleball
which was a completely new phrase to me you should absolutely check it out if
listeners or if you haven't played it you've got to do pickleball it is i it's not what i do professionally by any
means but it is so much fun it's a great way to spend your time yeah apparently it doesn't take much to become world champion at pickleball we
were discussing i'm sure that the pros would agree with us
absolutely well listen it's great to have you here uh all the way for like say from sunny austin texas uh beaming
in now uh we are going to be talking about conversion rate optimization uh and i
remember our prequel where we talked about this um and she came up with this phrase we're going to talk about how to do conversion
rate optimization well unlike everybody else and so i thought oh big claim let's
get into this one um so before we jump off that cliff uh why don't you give us a quick intro to all
things aj all right well i'm a conversion rate optimization
specialist and i just have a very big passion for figuring out how to
communicate with users and to measure all the things that they do so we can get them to become customers so what i
my mission in life is to get visitors to become customers on your website your mission in life is to get visitors
to become customers on your website i like that that's a good mission to have um but what kind of i guess a little bit
of the backstory how did you get into cra or conversion rate optimization how did you get into that
i was a user experience researcher at google for a product called google optimize
and i had an opportunity to be the researcher from you know pen and paper white boarding exercises of what the
product could look like and who it was solving problems for up and to launch and so as a result i got to interview
and show prototypes to a bunch of people already doing conversion rate optimization and as a researcher i'm
supposed to remove my emotion i'm supposed to sort of be this blank slate collecting information and i'd walk away from the sessions and
go oh that sounds like so much fun and the reason it's fun for me is because you take the information and the
problems that you learn about with research and then you get to apply it in a real world experiment to make sure it
worked the way you expected so it's fun because we get to make changes fast we get to have big impacts on businesses
and at the end of the day we're also making life a little bit easier for people wow so it started with it started with a
dream at google uh and you decided you know one day i'm just gonna do this for myself i'm gonna take the plunge and and uh and set and
set up an agency doing cro is that is that how it happened just one day you woke up with an epiphany
it was a bit of an epiphany yes i have i actually went to work for an agency here in austin for a year before starting
experiment zone so i got a chance to really test it out make sure it's truly what i wanted to do and then started
experiment zone and have been doing it ever since oh wow wow well it's great that you're here and um
obviously very good credentials uh one might add so let's let's start off the conversation about conversion rate
optimization and um define what it is because i i find
um like with all three letter acronyms used in e-commerce cro being one of them um
there is such a wide breadth of understanding in by what people mean by this term so what
do you mean when you talk about conversion rate optimization i think the first thing that a lot of
people think of with cro is a b testing and that is a big part of it so it's
being able to take an experience of a website and make a second version of that same experience and then see how
people respond to each to see the differences between them i think it's a much bigger experience
than that i think it goes beyond a b testing a b testing is a really powerful tool something we use all the time but
in junction to that you really have to build up an understanding of what is it that these prospects or these visitors
on your site what do they respond to what questions or concerns do they have and then where are those little bumps
along their path to becoming a customer that you can smooth out and so you can measure it with an a b test you can know
the impact to revenue with an a b test but deciding what it is you should test in a b test is really quite an art and
you have to do a lot of research and analytics work before you decide what to test that's a really interesting point
actually that um that when it comes to a b testing which is a phrase actually if you've been
around e-commerce um for a while you would have come across and it's just literally like you say
creating two versions of say the same web page and then sending people some of the people's page a some of the
people to page b and seeing which one converts better or i'm showing everybody the same page and only the headline and
stuff journaling or the same ads but you tweak certain things um it's interesting that you you mentioned
about how actually that's not a straightforward that that actually requires a little bit of thought in
terms of knowing what to test uh with the a b testing so how would you
how would you figure that out how would you know what to test yeah you have to pull from your user
experience research tool belt like i like to say so you've got a batman utility belt right
yeah it's a very practical belt though in our case but you've got to you know a b testing
is kind of the final tool that you use it's the decision of do we do this action or this change or not and then
you learn from it to see if they responded to that change but to decide the change itself usability testing is a
super powerful tool that a lot of people in ecom don't know about or aren't using as much as they should
so a usability test is taking five participants typically through a customer journey on your site and you
ask them to do certain tasks and you get their thoughts as they go through it importantly you're not asking
them what they would change you're asking them what do you expect will happen what do you think about what's happening here what do you expect will
happen on the next page and the collection of that feedback if you analyze it the right way will
surface your big problems that you've got to focus on so usability testing's really good at
capturing about of the problems that you have and helping you focus and prioritize on solving those ones instead
of just guessing as to what things people might respond to
i feel like uh aj this rabbit hole is getting deeper and deeper uh so you talk about which great usability test now you
mentioned five participants i guess in my head i'm thinking is that all
jeremy do i not need participants i mean is there is there a reason why you say five is that the magic number it
seems like a small number to me it is just the right number so the more
people you add to usability study sure you're going to collect more and more problems but the goal of a usability
study isn't to know all of your problems the goal of the usability study is to know your biggest problems
and so there's been a lot of academic research in this field that points to the magic number being somewhere between
four to five people with an assumption that all those people have something in common so
the way we think about it in ecommerce is that they represent one of your user personas so perhaps it's the mobile
visitor who's never seen your site before and is looking to purchase the type of thing you're buying or you're
offering and so you have five people with similar backgrounds on the same device
looking with the same goals in mind and then you can really capture about of the problems that exist
okay so would you do it so um and forgive me for being nitpicky here i'm
just trying to think of i mean there's a million questions in my head so i'm just going to ask at least some of them um if
you've got like that idea if you're going to take a customer persona who is looking on your website on their mobile
um and they're buying something for the first time that's quite a specific set of circumstances do you then take five
people who are um on a desktop who are persona number two and they're not necessarily interested in buying they're
more interested in finding out about something do you mean they may be starting the journey in a different way
would you have five would that be a separate usability test yeah absolutely so anytime you have
different a different learning goal you'd want to run a different usability study or a different user group would be a different usability study and so the
question for where to start with those because you don't have time and budget to do usability studies every day with
all the different possible combinations of desires and needs as you focus on where you believe the biggest pain
points are so kind of stepping back to our tool belt we're going to take out our analytics tool okay we see big drop off
on mobile at this one like maybe from the pvp add to cart to check out so we
want to make sure we construct a study that really examines why we're getting higher drop off there and focus on that
user group first so it's really the combination of signals we're looking for signals from the data that says hey
there may be an opportunity to improve here signal from a usability study that says here's why people are struggling and
then that ultimate signal from an a b test that says yes you found a solution that's going to help address that
problem and change that number you found in analytics wow okay so i mean that's
i like that because it's data driven by your analytics so you're using the data to analyze where your big issues are
where your big problems are and then you're getting people that fit that criteria are you watching how they use
it are you recording it you screen grabbing it are they in a controlled environment how what does that look like
yeah there's a variety of fidelities available and the research options around usability testing so my
background i was trained in a usability lab where i sat next to a participant who was holding the device underneath
the recorder so that the people in behind the scenes that kind of one-way mirror could see exactly what we were
doing that person was taking notes i was running the script and it was very high fidelity research
well as we've matured as an industry and had to you know be open to maybe
compromises or to reaching customers where they are more tools have come out so there are things like moderated versus unmoderated
which means i can ask follow-up questions if i'm moderating but i also become an influencing factor in the
study so then unmoderated gives you a standard script you give them all the same questions and if your script's
really good then they can go through it all on their own and then you have varying levels of fidelity based on where you find your
participants what kind of screen or questions you use um i could just dig and dig into this whole about usability studies because
you know there's a reason there's a reason that it takes so it takes so much skill to be really good at usability
studies anybody could plug in a usability study on usertesting.com and probably take away a few valuable things
but if you really want to get it right with the right question the right participants the right recruitment study
you've got to work with someone who actually has experience in that instead of somebody who's just kind of spread thin across a lot of different expertise
which i is true of most things i think if you want to do something well get an expert or someone who's done it at least
a few times to come help you um or be willing to learn yourself i think
is you know and then be prepared to make those mistakes right and i um i i can see now why you talked about how
to do conversion rate optimization well unlike everybody else it wasn't one of my favorite phrases from when we talked lesson
um just simply because i don't know many people who do the usability studies i
think most people go straight to the a b testing and they go right i'm i want to switch out this image or not i
want to change this headline or should i mean it's not it's it's based on what they think and it's in effect it's guess a and guess b
right and so um at least that's how i see it is that how you see most conversion rate optimization or
i think it's very idea driven so i think a lot of times it's based on ideas that you see a competitor doing or ideas that
you just kind of think of as maybe an interesting thing to try out and those have their own place because creativity
is a big part of this as well is finding the right solution and willing to be a little bit innovative in your testing to
see if you can find something that's kind of that golden nugget but if you want to have a consistent program where you're consistently learning and
consistently getting wins and if you want to be able to learn from losses which is its own kind of
conversation because a lot of times it's easy to say oh it lost let's move on to the next
thing so if you want to have a repeatable experimentation culture you have to have
it be more systematic if you want to look for some quick improvement improvements right now
absolutely go test some ideas i don't want to discourage anyone from doing that but there's different levels of
maturity and kind of in between those two is the ability to prioritize and choose the
right scoring criteria for the ideas so that the best ideas are the ones that you focus on so it's the inputs
understanding the problems prioritization figuring out which of those solutions potentially map to a
problem map to something that's highly visible and then ultimately having the right idea that tackles all the things
that you've just learned if you um i i i don't want to put you on the spot
here aj but have you got an example maybe of something you've seen do this well that can maybe help us understand
the the system the process that you that you've talked about almost like not a case study but you know i mean a sort of
an example um that you can think of yeah i would say we uh we have a client
that we've been working with uh and last summer we were doing some user research with them and in this case they have really
difficult to recruit visitors their target audience is really specific they're expensive to get in
communication with because they're a specific type of professional and so we did research where as a proxy we talk to
their sales team and so as subject matter experts we ask them to walk us through how they talk to
that customer or that prospect what things that person cares about what they need
and with all that input we ended up creating a new landing page and testing against an old landing page and we saw a
lift of over to conversion and all we did was connect the dots we
connected the dots from what their sales team already knew about the customer we translated that into a ux that would
convey what the sales team was conveying and reinforcing that message and of course it was successful
so just one of many examples of if you can connect all the dots in the right way you're going to get to
improve success for your company and is that i mean i the the thing that's ringing loudly in
my ears is the hundred percent increase um is that typical or is that untypical
untypical okay because absolutely nobody listening to this should expect to get a doubling of their conversion
just based on one test it was absolutely a home run grand slam whatever analogy you want to use on that
because it so clearly connected the dots and provided something that was missing from
the current experience most a b testing is going to be incremental it's going to be five percent here three percent here twenty
percent here and the the um collection of those increases will result in a much
higher roi or revenue for the company but because we are so unpredictable as
humans we can't really forecast very accurately what the impact will be of a test and so that's why we reinforce
our scoring with all this other input because if we have lots of signal that it's going to work we feel more confident that it's going to have a
bigger lift and then by actually a b testing and validating it we can be sure that it had an impact that we expected
so it's a mixture then um conversion rate optimization is a mixture then of both art and science isn't it it's not
all data-driven some of it is using your creative intuition i suppose
um but it's not all just creative intuition there is some data and then it's validated by data at the end is am
i hearing that right yeah it's just this kind of cycle of uh brainstorming what the problem can be
confirming that that's the problem brainstorming the solution confirming that's the solution so it's a lot of
creative process and i think what keeps me really energized about doing this work is there's a lot of surprises along
the way and those surprises if you can dig into them are gonna unveil even more opportunity and so being able to take
that time to be really thoughtful and follow up on things that don't go as expected is what kind of keeps keeps me
ticking figuring out how do we keep solving what customers need what do you mean when you say there are
surprises along the way because it sounds like there's a bit of a story behind that single word so i'm curious
what you understand by that yeah you know i think i think there's kind of an assumption and i see this with a lot of um what often happens is
someone will come to us in the sales process and they'll say oh we got a list of fixes to make on our site from this
other company and they call them like conversion rate improvements um and i can look at the list and i can
say well we've run that test it lost we've run that test and it's lost we've run that test and it's lost and so
what's often happening in the industry is people are saying these are heuristics these are best practices you
should do them but they're not suggesting to do them as an a b test and so i think it's an important
distinction that customers are going to behave differently on different sites so an example is for ecom
free shipping banner it's super popular you see it on every site because customers forget the shipping policy and
they need to see it constantly so a little skinny pencil banner at the top of the page helps enforce that message
that hey we've got free shipping we've got free returns like you can buy from us i feel more confident
well we've run that test with customers and seen it not work that's really interesting maybe eight
out of ten times it works but in the the one or two times it doesn't work it should work so we need to understand
why it didn't work and so sometimes it's the specific threshold of the free shipping
sometimes it's the pricing and the products and how it interacts with that sometimes it's if you click through and
the policy isn't clear on the other side or it kind of takes you in a different journey so there could be a thousand
things that could go wrong on your site and if you don't if you're not very intentional in figuring out why they don't work or the fact that they don't
work you miss you have a missed opportunity and you lose out on revenue i now understand what you mean by the
word surprise because i was actually surprised then with the whole free shipping so you've actually seen websites not benefit by putting the free
shipping banner on there i.e it's is it the opposite of benefit actually they've lost sales as a result of
putting that banner on there yes wow well very rare it's very rare but we
still test it because we see that sometimes it doesn't work and if it doesn't work it points to a huge problem
as opposed to some small thing about the message not being present yeah there's got to be a reason because you would
expect it no i get that so this is this is quite interesting because you're right most of the gurus out there
probably me included not that i'd necessarily call myself guru but jeremy it's uh would advocate if you do free
shipping to promote that on the on the front page and i i i'm i'm i'm quite i can see like say why you're surprised
so just because it's best practice doesn't mean it's best practice for you and we have to test that and and realize actually is that working yes or no
um how would you
i i get what you're saying in terms of i have a website over here right and i'm running this website i've sent over a
couple million we've got customers you know thousands of customers we've got a lot of data from all over the world and
we can we can see the analytics so i i can get that i can go well let's start here and let's try that and i have a
budget and we go what about um the guy over on this side
maybe who is three months into his e-commerce business you know
turning over a couple of grand um is doing everything himself you know
managing the website or herself they're managing the website they're picking and packing the boxes jeremy
how where do they start with cro because obviously it's important for both right
it is and i think there's two solutions for the early stage e-commerce businesses um one is to bring in an
expert to do an audit and so saying hey this is this is the information that's
hard to find here are some things that we've seen work in other testing that we should consider and kind of put in place
because if out of times free shipping works it's probably worth putting it on there rather than not on there and testing into it when you have
more traffic for example the other thing that's really powerful is to come back to that deep hole about
usability studies you can do usability studies very inexpensively um you won't
have the fidelity you'll have if you need a very specific user group but you can still get feedback from customers
and so the thing that when i talk to a new store owner i talk to them about is have you talked
to your customers and have you seen them use your site because most of the time people think
that they're seeing their customers using the site because they do click recordings and they can kind of see mouse movement but it's even more
powerful to sit someone down in a cafe and say can you walk through the site pick out a product and go through
purchase you don't have to buy it today i'll just want to see how it goes and you're going to learn things every single time you do
that and it will maybe be less perfect of information than like a full usability study with a great recruit but
it will be a signal that you can act on so for just getting started removing
yourself as a variable and getting inputs from other people is the best place you can do that's fascinating that is fine so you
would actually advocate going to a coffee shop with someone and why would you not if it's a decent
coffee shop but you would actually go and sit down in person with this customer and it's not just about using i mean software like hot jaw for example
which monitors clicks and the the mouse movements um but you would actually say no no i'm
just going to go to a coffee shop i'm going to sit down with a customer buy my coffee and i'm going to see what see what happens and
this person is sort of tentatively in my target uh persona here jeremy and we've
there's some correlation with what's going on um and that actually that yields good
results i mean that's what i mean because for a for a small business owner i mean a coffee shop sounds ideal you
know you can sit and work in a coffee shop i suppose if you're doing an internet business and just have one or two customers come in that day um
sounds remarkably easy when i say it like that but i i have you run into problems doing this
i think the the well kova is one so if you can't if you can't meet in person
so just kind of play out that it could be a virtual coffee as well but if we're let's just play out the full like
in-person scenario um you're often influencing your results so it's like
the moderator effect or the moderator bias and so it's even more so when you are the store owner asking somebody for
feedback about your store and so it's important to try to minimize the impact of yourself in the results
and you also want to make sure you're not just recruiting your friends or family because they're going to be a lot
of yeses this is great i understand it this is what a pretty picture you want to to work with someone who's
going to give you an honest and open feedback about the experience and so from the user experience research world you
would you could set up a script in the script and if you go to usability.gov the us government has templates you can
just start using today and it will set it up with things like the purpose of this is for us to learn
about the software we are testing the software we're not testing you we want your open and honest feedback
about how this experience is and then you construct the scenarios and tasks to be very factual
you'd like to now find a gift for your friend how would you go about doing that and you don't you don't dig in you don't
show a lot of emotion um one of my favorite things to kind of think about because i think a lot of people don't
realize they do this is as humans we're really when we're interested in something someone's saying we lean in we
take notes we do all kinds of things that signal to that person that i love what we're talking about
and so as a moderator you want to try to remove and reduce that impact you don't want the person sitting next to you to
know that you're interested in what they're saying more than the thing they said before okay that's interesting
my go-to yeah you want the blank slate but the other thing that's really great just for people who aren't as experienced with
research is if you bring your own separate computer and you just like take notes the whole entire time then they don't
know that you jotted something down so sometimes i just press the letter a over and over and over until i find something
to actually type up that's brilliant with your deadpan face have you got a good poker face is that
well there you go now there's another phrase that you mentioned um that i want to get into in april and that's a
culture of mistakes which was a phrase we used a couple of times but before we get into that in part two
let's just take a moment to thank this week's show sponsors did you know that nutrition is one of
the keys to maintaining the energy you need to drive your business forward vegetology creates incredible unique
supplements in an eco-friendly ethical and sustainable way that feed your body
with the precise nutrients it needs we're not just making you healthier we're helping to protect our planet too
our products are vegan friendly and approved by the vegan and vegetarian society plus they're gluten free so they
fit perfectly into any lifestyle they also contain no artificial colors or flavors making them good for your taste
buds too you can feel good about your food choices with our healthy natural supplements we have something for
everyone whether you want to boost your immune system or just get more energy every day and we're always working on
new ingredients so that we can provide even better products in the future so what are you waiting for get started now
by heading over to vegetology.com
[Music] hey there are you a business owner here
at oregon digital we know firsthand that running an ecommerce business can be really hard work
as the online space gets more competitive it is becoming even more challenging to stay ahead of the curve
we totally get it so we want to help you succeed by offering a wide range of services from fulfillment marketing
customer service and even coaching and consulting just so that you can do what matters most save yourself the time and
the money and let us handle the day-to-day tasks this way you can run your business without having to worry
about the boring stuff so what do you say are we a good fit for each other come check us out at oreodigital.com and
let us know what you think
okay i am back with aj and a big shout out to our show sponsors thanks to our show sponsors it
enables us to do what we do so go check them out use these guys they're good guys but anyway aj let's crack on with this the culture
of mistakes was an another interesting phrase that sort of popped out uh in our
conversation that i thought this is a really interesting phrase what do you mean by a culture of
mistakes i think that with the culture of mistakes it's the ability to call things
in experiment and know that things can lose and remove the personal kind of effects
of having a loss so i'll flip the flip the coin a little bit and say if you're just a culture
about wins you would remove some transparency because when things don't go right you wouldn't tell
people about it and you remove the ability to learn when things don't go as expected
and so on the flip side culture from losing or from experimentation gives you an opportunity
to say okay let's take time to examine what happened and so this you know this is something
that i've learned from being in the product development like software space but it applies for my entire team now is
that whenever something uh is a new thing or a new initiative we're doing we define it as an experiment we say here's
the problem we're going to solve with this here's our proposed solution and we evaluate it for a certain amount of time
with measurables as much as we can that are going to say yes this worked yes it didn't
and along that path of measurement we understand is it working or not and then at the end of the day it's the ability
to say it lost and the ability to shut something down that really enables your team to like
move forward and propel the business so similarly with experimentation an
experiment a culture of learning from losing ask you to double click into things if
they're a loss so we'll take that i'll give you another example from ecom world we've run a lot of testing to
increase visibility of navigation generally you want people to find the
navigation so that they can find the products that they're looking for we've had a series of times when we've
successfully done that we've gotten a lot more people to interact with the navigation but then they purchase less
and they purchase less like all the steps along the funnel they go to fewer product pages they add to cart less like
the funnel just is showing drop off that it was a bad choice to send them to the nav
so a culture of winning would say great now we're going to just remove the navigation altogether customers don't
need the navigation let's test that next um sure maybe that's a hypothesis
but even more powerful is to say okay well we got people to do this action let's examine that more closely and so
taking that loss and examining it is where the power lays lies that's where the power lies because if you look at
the navigation and you start seeing oh possibly people were using the nav more
but it was more confusing so let's do some research to understand where are they getting stuck in the
hierarchy of our navigation so it kind of comes back to what we were talking about the first part of the show which is just not being afraid to
declare that something lost and creating a plan around that loss so you have like a learning plan or a development plan
for the client or for the team i think um and the reason why i loved this uh
concept is because i think actually it we do live in a society and a culture which is a culture of wins and you you
know um it's i was talking to a friend of mine yesterday who's uh
who's nothing to do with digital he's a he's a doctor he's a very clever man um he is very high up in the field of
breast cancer and so i mean super clever compared to me right
um and it was fascinating talking to him because he was talking about how
the thing that he's noticed in however many years he's been in medicine is this one of the things that has evolved is this culture of perfection
um and so everybody is striving all the time to be perfect and if we're not
perfect then something is wrong do you mean it's um it's it's a really fascinating way in
which we've sort of evolved to think uh so this is not right this is not perfect
therefore it is wrong and his wife was telling me he said you'll be amazed how many people come in and
get diagnosed with cancer and will go this is what have i done to create cancer right
and their whole thing is i don't know if you've done anything do i mean but it's that whole
uh something is wrong therefore it's imperfect whereas um if you can
they talked about how if you can change the way that you think um about this idea of perfection you are
usually much better at dealing with something like cancer which i just thought was a phenomenal conversation right i hope i'm not doing the
conversation an injustice and what you've done here is you've taken that same principle that same uh
policy which says actually it's okay to get things wrong it's okay to make
mistakes in fact it's probably a good thing uh if we make mistakes because we can
learn from them um is it am i reflecting back correctly
what you what you just said yeah and i think that the one thing i'd layer into that is that when we make
mistakes because we were absent-minded or malicious or had bad intent
those are not acceptable mistakes those are things we should point a finger at and address very quickly but also don't
want to support that but on the flip side if we hold back our ability to take risk because we want to be perfect
then we're already failed like we're failing at the thing that we just decide not to try yeah and so something that i hold as a
personal value is when i see something i'm afraid of or i see something that is a bit of a risk to me
if it doesn't fall in that first bucket of being like immoral or you know it's going to kill me today like if it's
something that i can safely do but i'm sensing some sort of reaction holding me back because i maybe want to be perfect
perfect or maybe i'm going to embarrass myself by doing it i usually like take a closer look at
that and say okay well now i'm gonna have to do it because i hesitated
yeah that's awesome that's awesome and i think it's a really interesting life philosophy if i'm honest with you um
this idea of you know what um i remember
i don't know i'm i think a little bit older than you but but by substantial margin um there
was i think we we established in the pre-show you were so we're good yeah that's right
you just get and stay there that's it there was a movie which you may or may not have heard of and i don't want to
make any assumptions called the empire strikes back which is part of the star wars uh range of movies and when i was a
kid this this movie was really interesting to me because i think star wars had a big impact uh on a lot of
people my age to the point where in the uk uh there was a petition drawn up to set the
jedi knight up uh as an official religion here in the uk because if you got enough signatures
you could in theory get something through uh parliament and the government looked at that and went no we're just
not gonna i mean it was it was hysterical but that's such such as the impact of star wars and there's a scene
in this movie where yoda and luke skywalker are in a swamp
uh and luke is trying to use the force to raise this x-wing fighter and he
doesn't succeed and what happens is he says to yoda i'll try and yoda says no do or do not there is no try
and i thought it was a really interesting um phrase when i watched it back with my
kids watching it with them i thought that's interesting how that ideology i think has impacted
a lot of my generation growing up no no we we do or we do not there is no trade it's binary it's on or it's off there's
no analogue i mean it's that whole kind of thinking and actually for the whole purpose
behind experimenting is trying it's having a go it's just i don't know if this is going to work
i'm hopeful it will i'm not stupid to go i'm gonna try something that i know is gonna fail
um but i i'm just gonna have a go and see what works right and i i think it's a really interesting life philosophy all
of that said back to e-commerce how do you take that um ideology that
sort of um culture of mistakes how does that impact cro how does that make me
um a better selling uh that's a wrong phrase how does that help me sell better online is probably the
right phrase yeah i think if you look at customer expectations are
changing over time their experiences intersecting with other parts of their lives or other stores are changing over
time and so if you are not experimenting if you're not trying new things you will
be stale you will become flat and so if you want to be growing and if you want
to stay up on the not just the latest and greatest but the things that actually make it easier for customers
then you've got to be experimenting because it's not static it's not one and done it's not check i did cro xero is
the decision to iteratively approach how your customers experience your site and
so if you're not taking multiple swings at the bat if you're not stepping up and trying new things consistently with an
intent to learn about your customer so you can apply it elsewhere like that you just will not succeed as a
business in the long run that's really good i'm i'm thinking about the guy um or i i
just defaulted guys because that's me right but it's i'm thinking about the person who
um is launching or relaunching their website right so you've um
they've they've got their business it's growing and they've gone like right i need to do i need to update my website which happens to everybody all the time
i need to update it um in this
in this model then the cro this culture of mistakes this testing this innovation
this trying new stuff um i'm going cool i've got my new website do i
do i just launch it or do i do some of the the things that you're talking about beforehand do i get people
to come and use it and try it um and the reason i'm asking this question if i'm honest with you is because i
would say of the people i know in this space just go i've got a new site let's launch it let's see what happens
um without any testing jeremy and i'm i'm curious to know your thoughts on that
well launching a brand new site is a great way to generate business for my company because people launch a site without
having done any of that homework and things go wrong and without a doubt something does whether it's overall
conversion and revenue is down or it could be certain parts of the funnel have more friction that weren't there in
the last experience if you think about it you had a site where you had lots of data going through
it you were able to learn and adjust and it improved and smoothed out over time
and instead of you're comparing like the smooth process or the smoothed out process against like this like rocky
jagged mountain that you've just put in place and so you just don't know where those pitfalls are without getting some
data on it so of course i'd be an advocate of testing that new experience i think that
you know the way i like to think about it is that you know if you asked me five years ago i would say you should test into every
single change and you should instead of rolling out a big site you should be testing every single thing to get to
that end state um i think my thinking on that's kind of come a little bit more centered since then because it is
important to have big brand changes or to address things that are happening outside of the website directly into the
site so a kind of a middle ground like reasonable place to land is to say okay
we have a new homepage design it's part of this bigger redesign effort but instead of changing all the
variables at once and kind of crossing our fingers and hoping it works we could instead say let's build this
new home page and test it against the old one just to make sure that net effect of the changes is neutral to
positive because if it's not then we don't want to roll it out and so often times when our clients are
looking to do redesigns we say look let's do it this way we'll do page by page we'll test out the new templates
possibly we don't do it on all your traffic so we can kind of test on a smaller group get a sense of how they
respond make sure nothing is totally broken in the new switch and then you can roll out you know your
five to eight template pages once you've tested each of those so it becomes much more feasible to make that
change over three to six months with testing as opposed to doing it all at once with a lot of risk or backing into
it over a few years that's really interesting and so you
do you it sounds to me like you're going to need more than five people though for this kind of or maybe it still is five
am i just going to go and get five of my customers and go right this is the uh current home page this is the new
home page tell me what you think uh and i appreciate it's more complicated than that do you mean but is it a case of uh
uh doing that with the home page then doing it with the new product page layout just go and get some great
customers and just sort of see what they say yeah let me clarify when i was talking about page by page i would do an a b
test on it so for brand new big changes i would absolutely want the real world data on the experience so i would split
test a new home page versus an old one the place for usability usability testing would be a variation on
usability testing's called prototype testing so if you had an interactive either design or build out site
experience then you could have you know five customers from each of those important personas go through the new
design site identify pain points the time for that is usually before you've developed it you want it to be in the
early phases so it's not expensive to address that feedback so you can iterate on the design put it in front of more
customers do it again and then do that a b testing once you've built it all out
i think the reason i went straight for a b testing is so often companies kind of get over their skis with a new design
they've gotten a big pitch from a great design agency they're really excited about the new look and feel of their site and all the promise that is made
around improvements to conversion and so in order to de-risk we've already made this commitment we've already
decided to invest in a redesign then you want to break it down into those page or template based a b test
and so you split you had so by a b testing your what your meaning is you're going to send some of your traffic to
that page a portion of the traffic to that page and a portion of traffic to the other page and you're going to monitor the results sort of side by side
and see which one converts better i've seen it done whereby you have um
where people have launched websites and it will say something like hey welcome to our new look and feel website and you
know the homepage sort of talks about some of the changes they're all very excited about it and it's like but if you still prefer our old website you can
click here um i'm i've not seen that as much these days actually but i'm i'm curious have
you ever tried that as an idea does that work yeah uh we actually had a customer do
this maybe two years ago where they introduced like a redirect to the new site with a click back to the old one
just like you're describing um in this in that case they had other issues with usability on the site so
they ended up the new site didn't work but we didn't see a lot of clicks back and the reason is it's not a really
normal journey to be oh i'm frustrated let me go figure out an alternative way to avoid this
frustration most people are going to close their browser or they're going to go to another website and so unless it's
very very prominent probably most people aren't going to benefit from having the ability to switch back it can be a way
to de-risk it can be a way to get a signal as to whether people prefer the old site or are confused but there are
better stronger signals than that click back because of all the other consequences that can happen i guess if
i think about you know before you gave some examples about that one example of a client that got
return on uh return you know change and i thought that's marvellous uh bloody marvelous as we'd say here in the
uk um what about this sort of culture of mistakes this experimentation have you
got any sort of stories um that you can share around that yeah i think one of the one of my
personal favorite testing themes is removing stuff so i'd love to share an
example of a test we ran where we we came to a site they were producing very
highly specialized clothing athletic wear for women and for this company you know they felt
like they had to really explain what was different about their product they were new to the market so they felt like
there was this big component of education that they needed to have and so we we didn't necessarily agree or
disagree with that assumption but how it played out on their site was there were big blocks of text throughout the site
and in particular one place where it felt um well at least the analytics data pointed
to there being a bigger drop off on that page than we expected but on a category listing page
the the first whole scroll so before you scroll the page everything above the fold
was just text and the purpose of the page was a listing page of all the leggings
and so if you go to a page where you're looking to be able to visually compare different styles and colors and
designs and you get to a page and you're hit with a block of text that might be
you know discouraging however there is an assumption that they do need education so what should that
look like how should we provide that and is it important at this stage so
similar to what we were just talking about about collecting a signal through some sort of clicker action
we simply collapsed the information and gave a like read more option
and saw i think somewhere around like a percent lift to orders as a result of
collapsing the information it was still there and the second thing we learned was that people didn't click to open it so they
weren't necessarily looking for more information about the brand and so we see this time and again where businesses
feel as though they need to tell the visitor about them um or they need to like reiterate the
message and for certain messages absolutely we should be doing it free shipping value props
like those things that you've gotta your customer has questions about along the journey but for other things we've gotta kind of
drop we've gotta do this like slow drip of information as opposed to kind of giving them a wall of information every
time so a theme i would have like your listeners take a look at for their own site is if you look at the site and you
look at certain pages can you see a product on the page without scrolling and if you can't you're probably not
giving a good hand hold to help them move through the pages that's really intriguing how
um just one paragraph of text had a change
on orders do what i mean and that it's quite extraordinary that something that seemingly
insignificant had that big of an impact um at least it is in my head you mean you
you i guess if you're settling you're you're thinking about i want to get more sales on my website
there's a checklist in your head right more facebook ads do you mean this and the other nobody sits and goes oh
there's a bit too much text in the first fold of my screen i might want to reduce the number of i don't i do that though
there is one person doing it yeah that is you but i i i appreciate that actually it's important right and and
but people don't think about this sort of thing because um they'll have bought a template of shopify or do you mean a designer
whether i've never been using some text here's what we need yes let's put that there and i think it's um
if you're running a business that has been put together by other people like a web design agency
it's hard to then go back and question those because in your head they're the experts um
whereas what you're doing is you're actually saying well let's let's not assume that shall we and
let's uh let's work this through would that that block of text would that be
something that would have come out of the usability testing had you have done it on that side if you did but had you
have done it would is that where that would have come out of other than your own propensity to go there's too much
text there um would i have found out about that that way
yeah that's a good question i'd say sometimes the design choices like that of like is it the right information or
not you're not going to get the participant in most cases to say what is this big block of text doing here i wish this
weren't here you know most people aren't going to reflect on your site design in that level of detail they instead may
signal in the study um oh i can't find the products or oh i found the products once they've scrolled and so it's the
kind of the it's that art of interpreting the response of what caused
them to have that moment of oh here it is or oh i didn't see this
because then you can extrapolate okay the problem is they couldn't find the products right away
how do i know what caused that problem and that's where you're getting into ideation of okay it's probably this block of text that's in this case it was
pretty obvious because that's the one thing taking up the screen but in other cases it can be hard to pinpoint and you can't get it on the
first test and so that's why we like to think about like a test theme where you have a problem and you have a test theme
of people can't see products on this page and you may have five six ten tests that get you closer and closer to people
being able to see products and therefore ultimately be able to convert that's fascinating fascinating so the
i guess the thing that i'm picking up here um aj is actually
uh there's a culture of mistakes there's a culture of cro it seems like you need to have this
um culture of optimization that actually this is not a one-time gig this is not a one-time event
um and from what i'm hearing you say this is this is uh a way of life for anybody
running an e-commerce business that actually optimization needs to be something you consistently do on your website week in week out uh you
should be continually testing things am i am i picking that up right yeah i mean it comes down to if you're spending
money to get traffic to your site let's say you're running paid ads you're investing in seo you have all these
different inputs to get people to your site if you want to multiply the impact of all those efforts the best way and
maybe the only way to do that is conversion rate optimization so if all this traffic comes in and
we're able to increase their conversion rates up suddenly there's a multiplier and all that spend
and so yes it's a culture in a sense that you will always be learning there's always improvements your customers
expectations and behaviors will always change but on the flip side like businesses are also releasing new lines of products or
potentially making design choices or marketing choices that impact how the site functions and if you're not
iterating if you're not taking a look consistently at what the impact of those things are then you're missing out on
that as well so it could be one and done if the only thing you had on your site was like a
single button and one traffic source that was always the same five people coming to the site to click the one
button but because there's so many variables shifting you have to be able to isolate individual variables of
design and customer flow on the site in order to make sure that's correct for all those shifting variables
how long um i guess when i i'm hearing myself ask this question and
i'm thinking is how long's a piece of string is probably the answer but how long do you run a test for
do i mean if i'm is this something that i do and i get the results tomorrow is this something that i do for three months i mean what
sort of time frames are we talking about um the consulting answer of course is it depends but the we have three heuristics
but we do have three three kind of things we're looking for to determine okay we have enough signal on this
particular change so we want to have two weeks of data at a minimum and the reason is because somebody shopping on a
monday morning has different intent or different needs than somebody you know out shopping on their phone on a
saturday so you get two life cycles of customer behavior monday through sunday
two times it doesn't have to be those days but just like a full two weeks you want to hit a confidence threshold
most of our clients are doing confidence if you have a business that's really
risk adverse you can up that to what do you mean when you say confidence threshold sorry
yeah it's the ability to know that this number is definitely different than this number by this much okay and so
it's basically a statistical modeling of if we have enough data and these two things
are different enough from each other you know confidence out of times we have
accurately assessed the situation and we do know it's better one out of times it could just be
random noise just random data and then the
the third thing that we look at is the amount of traffic because for some businesses they are having fluctuations
in traffic and so we what we do is we set a threshold of how much traffic we need for the test
so if we hit the traffic that we need it's been two weeks and we're over confidence then we'll declare the test
to be a winner or a loser and then do a deeper analysis to understand what to do next with that decision
so the i mean i'm thinking here for the sites that we've had in the in the past and the
site so we've currently got e-commerce sites now one of the things that i can tell you makes a big difference is when payday is and people usually in the uk
get paid at the end of the month um and so you always see a spike in sales towards the end of the month and the team dates are always your lowest sales
dates so you know you get this sort of cycle um would i extend my testing period then
from say two weeks to four weeks to cover at least one of the upturns and one of the
downturns yeah possibly and i think that's why we use these three as kind of guiding
principles because for any individual business there might be some seasonality considerations or within the
month considerations so for example one of the things that comes up is if you're testing in
november and december you have a different visitor type they're often gifting for someone else
and so their needs around the brand messaging and the names the products and
sort of the um specific experience is going to be different than somebody who is there for
themselves and so what we typically will adjust for example would be
yes we can we should do testing around the experience like is the button in the right place is the image in the right
place can they see this is the order of things kind of the right flow for customers but we won't test things that
are going to be audience specific for the rest of the year so it kind of becomes again this kind of moves more into that art of
understanding the business the art of understanding the customer groups and how they might behave differently so you
can adjust your testing strategy so you're testing the right things for the right amount of time at the right time
of year very good and so i like that you've got your three areas think about it they're all movable uh
you know you've got to do something that makes sense for your business listen uh aj i appreciate your time with
us and your insight and you've got the old grey mata uh synapsis firing at a
million miles an hour right now how do if people want to reach out to you if they want to connect with you how do
they do that what's the best way to get in touch our website's the best option so experimentzone.com
we've got a contact form so they can go ahead and send me a message directly through our contact form
i also love doing virtual meet and greets too so sending me a note reaching out on linkedin aj davis
fantastic and we will of course link to all of aj's stuff in the show notes which you can
get at ecommercepodcast.net forward slash um i like when i when you said
then uh your website is the best option i i in my head i thought you were going to say com
so you should just get thebestoption.com and and just register that as your domain name it's probably gone now isn't
it really uh but it was experimentzone.com uh was the website not the best option.com uh but
make sure you connect uh with aj and check out what she's doing get in touch i'm sure she would love to talk to you uh but aj thank you so much uh for being
on the show thanks for coming and just sharing your stories and i i'm absolutely fascinated by the stuff
that you do so um thank you thanks so much for having me on matt it's a lot of fun
well a huge big thanks to my very special guest aj davis wasn't she fantastic what did you think did you
take lots of notes did you write them all down are you like me a copious note taker but
don't actually take notes that are that legible or easy to understand uh if you
are you can head over to our website ecommercepodcast.net forward slash
and you can get all the links notes and transcript from today's show for free so do head on over to our
website and grab those and of course if this is not enough ecommerce goodness
for you and why would it be when there's lots of good stuff out there then stay
tuned because next week we are speaking to cody whittick on why giving stuff away leads to more
sales with influencer seeding does that sound a bit cryptic or
interesting well to wet your whistle even further here is an excerpt from my conversation
with cody back up a step further you're targeting the right influencers it's not just
if i continue with the hat brand and i'm happy to fall and it's a running hat that's athletic wear
i'm not just sending it to the stay-at-home mom or you know just anybody
willy-nilly i guess you could say um but yeah you're targeting the right
influencers based on who your brand is who your customer is and who the influencers that influence those
customers yes i am really looking forward to digging into this one let me tell you uh
so make sure you like and subscribe the podcast so you can get the show delivered to your inbox or wherever you
get your podcast or whether you watch it on youtube or wherever you go get it from it will just appear it will just be there that's the magic of subscription
that's the magic of hitting that notification button so if that's you make sure you do that and if you did
enjoy the show then i would really appreciate it if you could give us a rating on itunes as that just enables us
to reach more people or wherever you get your podcast from it just enables us to reach more people the bigger our audience the more likely we are to get
sponsors the more like we are to get sponsors the easier it is to bring this content to you for free so win win it's
what we call a win-win uh in the ideal world so please do give us a rating it will really really help us
and ultimately help you now as i said at the start of the show all of the notes links and transcripts to today's show
are available for free online you can get them at ecommercepodcast.net forward slash to
do check that out that's it from me for this week thank you so much for listening and make sure you come back
next week to catch our conversation with cody it's going to be fantastic honestly you're not going to want to miss it so
until then bye for now
you've been listening to the e-commerce podcast with matt edmondson join us next time for more interviews
tips and tools for building your business online