Okay, now all looks good.
Thank you for joining us today, Frederick.
The floor is yours.
Thanks.
So thanks, everyone, for being here.
My name is Frederick, and I'm going to talk about Matamu, Converged Generated Simulation
today.
Just as a reminder, you are able to join the chat room on the matamucamp.org website.
Just go into the talk card, and you have the link to join the chat, if you ever have some
questions, I can answer it.
Okay, so here is what we're going to talk about today.
First of all, I'm going to introduce myself, then we're going to talk about the prerequisites
for doing Converged Generated Simulation with Matamu.
Then we will talk about how you can find your first optimization ideas, then how you can
prioritize these ideas using some CRO frameworks, then how you can use Matamu UX analytics features
in the process, how to write your testing hypothesis, and how to make an A-B test using
Matamu testing features.
So I'm a digital analytics and CRO consultant.
I've been working with several companies and several organizations in the public sector.
I am specializing in open source web analytics for about two years now.
And I also was a pedagogical director of a digital marketing master, so I'm quite used
to teaching and building teaching programs.
Here you can find my LinkedIn if you are interested.
Okay, so what do we need to do Conversion Rate Optimization with Matamu?
The first thing we need to do is to define what performance is.
So I have a definition here.
Performance expresses the level of achievement of the objective's pursuit.
So here is really something important to be able to know what we're talking about because
we're talking about something which is performance and it's really important to know what we
are meaning when we say performance.
So now we need to define the objectives, which means the business goal.
First of all, I'm going to give you an example.
You may want to increase the part of people who will purchase one of your products if
you are an e-commerce website.
And then you need to define the level of achievement, which means the indicators, the thresholds
and the segments.
So for instance, for the e-commerce conversion rate, the indicator would be e-commerce conversion
rate.
We could decide that the threshold is two persons and we could decide that this threshold
is for all users.
And to keep going through this process, we will use two main tools.
The first one is the KPI framework.
And the second one is the measurement plan and the tagging plan.
When you want to do optimization on analytics with tools like Matomu, you need to have as
many data as possible to have a granular analysis.
So it's really important to have a proper tagging plan and proper KPI.
So what is a KPI framework?
Basically, it's a document that explains how you're going to measure performance through
different level of engagement from a user point of view.
So you start, you always start with your business goal and in the scope of reach, the first
level of engagement, you want to increase the visibility of your sites.
On the level of engagement, you want to increase the number of qualified users on the platforms.
You may also want to engage the audience and you may want to develop the visitor base within
the digital ecosystem if you have several websites or apps.
On the conversion level, you want to facilitate, develop the realization of actions having
a direct impact on the business.
And on the advocacy level, you may want to maintain and develop the relationship with
the site's audience, increase the number of contacts in the CRM, develop visitors from
social networks, and develop interactions on the community on social networks.
So for each of these business goals, here we are on a KPI level, which means macro indicators.
You have, you define here indicators.
So for instance, for reach and increase the availability of websites, you want to increase
traffic volume.
Then you have the description of the metric, you have the Matomo metric, you have the threshold
and you have the source.
So here we're going to take a look at this part here, but it's really, it's really important
to have all the levels of customer engagement so that you know if your preferences is good
or not, for instance, on a monthly basis.
And here you also, you can see that sometimes you have a Matomo as the main source of data
and you also sometimes have external tools such as SEO tools and sometimes social media.
On this document, there are some things that are missing, but I just wanted to show you
a big picture of it.
What is missing is the segment and what is missing is also the calculation of the indicator.
What you need after that is the measurement plan and the measurement plan is the document
where you explain what are all the actions you're going to track with Matomo, mainly
using event tags.
So for instance, here you have the page template and all the actions that the users can do
on this specific page template.
The type of tag feature you're going to use in Matomo and then the architecture of your
event category, action, and name.
So this is really important also because we are talking about KPIs, which means we are
talking about indicators that we will give to the top management, but sometimes you have
also marketing teams that want to increase the level of engagement and the level of some
events, some realization of events, so it's important that you list everything here and
that you can have that in your Matomo instance.
And once you have your measurement plan, then you can go on and write down your tagging
plan.
So here, this is the technical part where you tell the dev teams what type of codes
to push on the page.
So here we have the data layer for the events, and here you can see the tracking code and
all the different values that are pushed to the data layer.
And then last but not least, you also have to add all the different values you will be
using as custom dimensions.
This is really important also to have a really granular analysis and to be able to segment
all your optimizations.
Okay, now you have your KPI framework, you have your tagging plan.
You can start optimizing.
So we defined what performance is.
Now what we're going to do is define what conversion rate optimization is.
It's the name of what we are doing.
You probably know search engine optimization, all kind of stuff doing optimization, but
it's quite specific here because CRO is, I have a definition also, it's the practice
of increasing the percentage of user who perform the desired action on the website.
But I have a problem with this definition because I give an example below.
If you suddenly drop your prices by, let's say, 90%, you will have a really good positive
impact on your conversion rates.
But that's not what you want because only increasing the conversion rate is not enough.
That's why I think we should talk about conversion volume and value optimization and not only
conversion rate optimization.
Okay, now we have defined what we will be doing.
How can we find our first optimization ideas?
I took the example of an e-commerce website because this is really an example where we're
talking about money.
So performance here is really simple to understand.
So when you have your indicators, you can compare the metrics between the different
segments and audiences.
For instance, we decided that our average conversion rate should be, let's say, 2%.
And when we compare the conversion rates by mobile desktop tablets, we see that there
are huge differences.
So now we should ask ourselves questions, which will be, why is there such differences
between devices?
In the real world, we know that most of the time, people make their first visits with
a mobile device and then finish the order on a laptop or desktop.
But here for the example, we will assume that people start their session, their visits on
the device and finish or don't finish the purchase process.
So here we see a huge difference between desktop and mobile.
What should we do?
We should go and look into the e-commerce sales funnel to see if we can see any difference
in the checkout steps between the different devices.
So what you're going to do is you're going to go in the funnel reports.
And here you have the different steps.
So here is authentication, sign up, payment, and conversion.
And what you want to do here is to use segments within Matomo and see if you see any difference.
And if you see any difference, then you may want to try to optimize.
If you see a huge difference, for instance, at the payment information step, let's say
for mobile users, you may want to investigate further.
But this is not what we are going to do now because we're going to see that at step four,
how to use Matomo UX analytics features.
Now we're going to continue and talk about conversion rate optimization frameworks.
Because we need to structure our approach doing optimization.
We could do what we call heuristic analysis.
Just a few words about what heuristic analysis is.
It's just you go through your websites, through your conversion journey, and you take notes
on what's wrong from your point of view.
You might say on the product page, I find that pictures are too small.
I think that the app to cover is not prominent enough.
It's below the fold, stuff like that.
You can also do some kind of benchmark with competitors saying, yes, on the product page
they do stuff that we don't.
And I think this could be a good idea.
But it's quite risky because, once again, this would be your own opinion.
And we are doing marketing, so our own opinion should not wait that much in our decision.
So that's the reason why we should rely on data and on frameworks.
So I gave you here, I'm not sure, three or four examples of CRO framework.
The first one is PIE, which is a model done by wider funnel.
In fact, you have marks here for every lift optimization zones.
And you have three items, the potential of the change, the importance of the changes
and the ease of the changes.
And then you will have a score here, and the higher the score is, the first the optimization
should be done.
Then you have another example of CRO framework, which is the ICE model, which was created
by Sean Ellis.
Here again, you have three items, the impact, what will be the impact if this works, confidence,
how confident am I that this will work in ease, what is the ease of implementation.
But this here, I have a problem with it, how confident am I, since it's something that
I'm going to propose, I think that I will be confident in it, but confidence is not
something really scientific.
So this exists, so this is good to know that it exists, but it's not something that I would
recommend to use.
You have also another model done by Hotwire, and here you have one point or zero point
for each item.
So let's say this optimization idea will support the companion main metrics, yes or no, one
or zero, with the test as changed on the results of many pages, yes or no, again, one or zero.
Is it a change above the fold or below the fold?
Okay, I'm not going to explain everything, but you understand the principle, one or zero,
and the higher the note, the quicker the test should be run.
And the last one, it's the PXL framework proposed by Conversion XL, and again, it's a mark notation
system with one, zero, or two, and the higher the note, the most important, the test.
So what you have to do when you want to do optimization is to go through the data, go
through the websites, and look for huge differences.
And once you find those huge differences, then you have to investigate further.
And when you want to investigate further, that's when UX analytics features come into
place.
You saw that in the checkout step payment info, it's really lower for mobile users,
but you don't know exactly what the issue is when you only look at statistical data
within Matomo.
So you may have an idea, but you have to confirm that your idea goes in the right direction.
So here is what you can use as features in Matomo.
Here it is examples taken from the Matomo demo website.
I think that you all are aware of what Session Recording is, but I'm going to just re-explain
it just in case.
What it does is that it takes video snapshots of users' journey on the website.
So it allows you to watch video playbacks of users' journeys on the website.
So for instance, on the example I took on the payment info, you see that mobile users
don't go through and don't finalize their purchase.
So you can ask Matomo to record this behavior, and then you can watch the playback just to
confirm your hypothesis that might be, I don't know, it's not mobile friendly or I don't
know what.
You can watch the videos just to confirm the idea you have about why it's not working as
it should.
So here just, again, a screenshot of what it does.
You have here the movement of the mouse on the page.
What you can use also as Matomo features for just to confirm your idea is to use the Itmaps
feature.
So imagine that you have this payment info page and you should click on the accept the
terms of sales, and you see that people don't click on it because it's not that visible.
So here the Itmaps will help you to confirm the fact that people don't click on it because
it's, let's say it's below the fold, it's not visible enough.
So that's the reason why people can go through the purchase process and maybe they don't
see it and they can click on the next button, so that's the explanation you have and it's
confirmed by the Itmaps.
Another interesting feature that you can use in Matomo is the form analytics feature, especially
when you are talking about checkout processes, just to make sure that there is no issue with
some specific fields.
For instance, you may see that some fields are left blank or there are some format issues
with some fields.
Here it's interesting to have also the information on the different forms and what type of issues
can occur on the different forms.
Last but not least, the user feedback plugin.
It's a quite new feature.
It allows you to ask feedbacks to users using the Tag Manager.
So if you have that user or mobile user that's in the payment page, payment info, who is
about to leave, you can use an exit page trigger and then present to this person this model,
saying, okay, what went wrong on this page?
What could have we done to make it work?
This is quite interesting.
There was, I guess it was this morning at 9, there was a conference held by Thomas Pearson.
I encourage you to watch the playback of this session because this is really interesting
because it's all in Matomo.
It's not an external tool.
You have all the feedbacks in Matomo.
You have the completion of the feedbacks that can be sent as events.
So this is really something you could take advantage of to better understand what's going
wrong on our pages.
Okay, so now you have your optimization ideas.
You have sorted them by priority.
You have collected more info with UX analytics features and you can describe the issue more
accurately.
So as I said before, for mobile users in the payment info, the accept terms of stay button
is not visible enough.
You saw that on the session recording because you saw that the mouse moving and not clicking
the button.
You saw that also with the events you prepared on your tagging plan for mobile users to see
the click rate of this button is really low.
You also see with the hit maps, you also saw, sorry, with the maps that people don't click
on it, which makes the next button inefficient.
You can click on next to go through the purchase process.
So these users don't understand what's wrong and they leave the page.
So now you are able to write down your test type of this, and this is really important
when you want to do A-B testing.
You should use this pattern if I, and here you describe what you are going to do on the
page, then users will, and then you describe the change you expect in user's behavior.
So here, if I make the accept terms of sales button more visible in checkout step payment
info for mobile users, then they will click on it and be able to go to next steps and
complete their purchase.
This is what you think is going to happen when you make the change.
Once you decide that it's your hypothesis, you will design a page and you will redirect
a certain amount of people to this page and the other will go through the regular page.
You will want to check if there is any difference in the performance, the performance being
the E-commerce conversion rates, and you can start configuring your Matomo A-B test feature.
But before that, you should use a simple size and duration calculators because if we want
tests to be statistically significant, we need to make sure that we have some requirements.
For instance, here, with 3% conversion rate with 95% confidence level, 80% statistical
power, and minimum 50% conversion rate lift and three variants, the calculator says, okay,
if you want that to be statistically significant, you need to have at least 23,000 sample size
per variant.
And also here it gives you the length that the test should be and here for the minimal
detectable effect you would see in the duration of the test.
So once you have this, you know that you have, you need to have this level of sample size,
this test duration, then you can go on and start configuring your A-B tests.
So I recommend that you follow each step on the Matomo interface for that.
Okay, you give a name to your test.
Here you write down your test hypothesis.
Here just the description of the test and the number of variation that you will have
and then again the target pages.
Then you, sorry, I guess it's the same, no, it's not the same.
Here you go through the success metrics.
Matomo will suggest you some metrics that you could use as a success metrics, but you
can decide to use totally different metrics.
Here you give the success conditions.
So here the minimum detectable effects you want to have.
By default it's 15%, but you can write a different number, but it's, yes, it's a minimum.
The confidence threshold is most of the time for all A-B tests, so it's about around 95%.
So just leave it as it is most of the time.
It's okay.
It's just something that says it's the probability that the result will be the same over the
time.
It says, okay, this is not due to a random result.
Here you have the target pages.
A visitor will enter the A-B test when and a visitor will not enter the test when.
Here you have the traffic allocation.
So here you have 100% of people going through the test, but it's not, it won't be that level
most of the time.
Here you have the, again, the traffic allocation for each variation, each, let's say, template
page template in the test.
And here you have the URLs of the different pages.
Here you have the start and the end of the page, of the test, sorry, and the code you
have to put in your page to make the test active.
And here I put a screenshot of an A-B test example coming from the Matomo demo website.
So when you wrote down the hypothesis in the interface, here it's going to be reproduced.
Here you have the description and all the performance metrics.
Since how long the test has been running.
And here you have the overview of the test.
So you have the conversion rate for each variation, the original and they apply now.
So we have an average conversion rate.
And here we have the difference between the two of them and the detected lift.
So here, since we have the statistical significance of 100%, we have a clear winner, which means
that on this test, when you say if we change the top CTAs on the individual job page to
be an action about the job, more users will apply to the job.
So the hypothesis was good because we see here that there is a real difference in the
way people acted over that call to action.
Okay, so I'm done.
Maybe I was a bit quick, I don't know, it's 1.35.
So if you have any questions, just feel free to ask them on the chat.
I'll be happy to answer.
Okay, so I don't see any questions.
So since there is no question, I propose to stop the recording and to end the session.
Many thank you for this interesting session, Frederick.
I guess people might be feeling a little bit shy now, but of course, the chat room will
be open.
So everyone, you can also continue the conversation there if you want to discuss further with
Frederick about this session and what was discussed here.
So Frederick, once again, many thanks for attending Matomo Camp.
Thank you.
Bye.
Transcribed by https://otter.ai