Hello, everyone.
Nice to meet you all again.
It's currently 3.02.
And we will conduct a conference today.
So continue by who is here with me and myself.
So the idea of this conference is to do something which is kind of unusual.
So we're going to do a use case.
And most of the time, when a use case is presented by an analyst, it's most of the time, like
three-quarter of the time, the analyst who is speaking and then a quarter of the time
with the client.
Here, this is going to be the reverse mode.
So I will leave the floor in a couple of minutes to Cathy, who is a client of mine.
And the idea is that Cathy can introduce you to the project that she's working on.
And I will explain how I decided to deal with the project management of including Matomo
within this given project.
So I leave now the floor to Cathy, and I will interact only dealing with my part of the
slides.
OK.
Thanks, Ronan.
I'll just share my slides.
Can I just check that you can see that, Ronan?
Yes, it looks great.
OK.
Lovely.
Thank you.
Right.
So myself and Ronan are going to, as he said, share the presentation today.
And the presentation is on using Matomo to collect data on intervention engagement within
a research trial, and the case study is a project called WRAPT.
So the overview of the presentation is that I'll firstly talk about why I think it's important
to measure intervention engagement within a research trial.
I will tell you a bit about the WRAPT research project as well.
And then Ronan's going to talk about how we use Matomo within the project to collect individual
level data on engagement.
Then I'll present some early insights from our analytics data, and then there'll be a
chance for some questions at the end.
So first of all, why it's important to measure intervention engagement within a research
trial.
Well, there are two reasons.
One is to minimize something called non-usage attrition.
And I'll go on to explain a bit about what that is.
And then also to examine and control for intervention dose.
So firstly, on to minimizing non-usage attrition.
So when we are testing whether a new intervention works, then we typically do something called
a randomized control trial.
And many of you will be aware of what randomized control trials are already, maybe.
But they are, I'll just share my next slide.
So they are used all the time in a clinical context to test the effectiveness of something.
So that something could be a new drug or a new vaccine, as is the case for the COVID
vaccine.
Or in my case, and I'm a health psychologist, I develop interventions that try to encourage
people to change their health behavior.
So that could be, for example, to persuade people to use the COVID vaccine.
So the intervention itself is not a drug or a vaccine or something like that.
It tends to be a behavior change intervention.
But the principle of testing it is exactly the same.
So we would also run a randomized control trial.
And in randomized control trials, we take a sample population.
So this is a smaller group of the wider population we're interested in testing the intervention
on.
And we'll randomly assign them to two groups.
So like the flip of a coin or a roll of a dice or more commonly using sort of internet
based randomization tools.
But the idea is the same, which is randomly assigning to one group or the other.
And in the one group, which here we have as group one, all the people in that group that
have been randomly assigned to it will get the new drug or treatment.
And then we'll measure afterwards how they fare, what the outcomes were for them.
And then in the other group, they'll get the control.
So if it is a drug, it might be something like a sugar pill that has no active ingredients.
And again, we'll measure the outcome for them.
And the important point with the randomization is that by the end, we should have balance
in the two groups.
So just by chance, we should have approximately even numbers of people of different genders
and ethnicities or with similar levels of health need or health condition.
And that's important to make sure that we control for any sort of confounding factors
that might also have an effect on the outcome.
And they should then be evenly spread between the two groups.
So in a clinical context, when a new drug is being tested, adherence to the drug is
not usually a big problem, although something still to be mindful of.
And this is because the drug is prescribed by a clinician, and the participants are usually
very closely supervised.
So you would be asked to go into a clinical context, for example, and you'd be passed
the pill on a tray with a cup of water and watched by a research nurse taking it, who
then recalled that you'd taken it, and that would likely be for every dose.
And also, as a research participant, you may well, especially if you're in an intervention
condition, experience observable and immediate health benefits of taking that drug, assuming
that it's effective, also potentially in the control condition due to placebo effects.
So there's lots of things that could be meaning that adherence to that drug within an intervention
and trial context is not too much of a problem.
It's usually, it's usually reasonably high.
However, in contract, adherence is a major problem when we're testing e-health interventions.
So I mentioned briefly about the kind of work that I do.
So to test health behavior change interventions, and also almost all of the interventions that
I develop and test are of the e-health type.
So they're interventions delivered via the internet.
So here, the intervention is neither prescribed nor critical to well-being.
And this graph, I think, demonstrates quite nicely typical attrition from e-health interventions.
So on the y-axis going up the left, you can see at the very top, there's a 1, and that
reflects 100% of users of the intervention going down to 90, 80, 70, et cetera, right
down to 0 at the bottom.
And then across the bottom on the x-axis, we've got time and modules.
So if we take the blue line to start with, the study by Farrell Dunn, this was a 12-week
program of internet delivery of an intervention.
So it's supposed to engage for 12 weeks in total to have had the full dose of the intervention.
But you can see after a couple of weeks, we've already got less than 50% of people still
using that website, going down to less than 20% at week 3.
And you can see by the end, in fact, it was less than 1% of people still using or still
completing the intervention right at the very end.
And then for the dashed black line, this was an intervention called Mood Gym.
And it's in a trial context again.
And there were five modules here for this intervention.
You can see that less than 70% did module 2, and then module 3, less than 60%, and so
on, down to just over 20% doing all five modules.
And then the solid black line is also Mood Gym.
But in this context, it wasn't in a trial.
It was just people allowed to have access to it, then see who engaged with it and carried
on engaging with it.
So outside the context of a trial in which people might be incentivized to use the intervention
and also encouraged and reminded by researchers to do so, you can see that engagement is even
less.
So non-use of nutrition, so participants ceasing to use the intervention in a trial context,
is obviously a big problem for us.
And if participants are not using intervention in a trial context, it makes it really difficult
to test whether the intervention actually works or not.
So in trials, regardless of, sorry, in trials, the outcome data for every participant is
randomized and included in the analysis.
So if we have people in that, if you remember that first diagram I showed you of a randomized
controls trial.
So we've got people randomized to the two groups, and in that top arm, we've got people
who received the intervention itself.
If you've only got a small proportion of those who are actually using intervention of four,
you might be thinking to yourself, well, just analyze data from those people who use the
intervention itself and ignore everyone else who didn't.
But that threatens the whole integrity of the trial design, which is the randomization
of people to one group or the other.
I explained earlier about how we randomize so we have balance between the two arms.
But if you're only including a subgroup of those people who have used the intervention
to a minimum extent and ignoring everybody else, then you remove that balance, which
is so important to that trial validity.
So the point of ensuring that everybody, regardless of whether they use the intervention or not
in a trial, it's called intention to treat analysis.
So if you randomize someone, you always analyze their data regardless to ensure you've got
this balance in design.
But the problem, of course, is if you've got a group of people in your intervention condition
that haven't used the intervention at all or not very much, then if your intervention
is indeed effective, then when you do the analysis, it has the effect of underestimating
the effect of the intervention because you're including in the analysis in that intervention
condition people who didn't even use it at all or didn't use it very much at all.
And remember that diagram of it dropping off steeply, that is often the case for us.
So it's really, really important that in a trial context that we do absolutely everything
that we can to minimize people not using that intervention or dropping off.
Now ahead of doing a main trial in which we test the effectiveness of the kind of intervention
that I've been talking about, we tend to do something called a feasibility randomized
control trial, which is exactly the same as a randomized control trial with this randomization
to the two groups.
The only thing that's different is it's with much fewer participants.
And the purpose of that kind of study is to help us to prepare for a full randomized control
trial.
And what we're trying to do is resolve or remove all the unknowns.
So for example, to help us to learn what the best methods for recruiting participants is,
what level of incentive we need to recruit people in the first place, but also to keep
them in the trial until the end.
So all those kinds of things are important for us to establish in this early phase before
we do the main thing.
But we can also use these feasibility randomized control trials, these preparation type trials,
is to examine non-nucid attrition, which we've been talking about.
So if we look at attrition in this early stage trial, it can tell us some important things
such as whether attrition is associated with particular demographic factors.
And of course, we're measuring those in the surveys that we give to people before they
have the intervention and then also afterwards.
And so if we can link that demographic data with the data about attrition, we can learn
some important things.
So if we found out that, for example, females in particular weren't using our intervention
or dropping off, then it would be important to maybe interview them as part of some later
studies to find out why that might be the case and to try and rectify that.
And when we look at the shape of that attrition curve, which we were just looking at before,
that can tell us something also important.
So if we had, should I get my mouse to do that?
Yeah.
So if we had a shape a bit sort of like really steep down like that, that would be a really
bad sign for our intervention, tell us that people were disengaging very quickly because
they just didn't like it.
And then this more kind of sigmoid style curve would tell us that we had some sort of initial
interest in it, which then dropped off, as you might imagine, then a bit of a steep drop
off in the middle, perhaps when there's less motivated people were leaving or possibly
representing some sort of usability style problems.
And then we've got a hardcore sort of set of users at the bottom.
So the style and shape, I suppose, of this curve can tell us some important things as
well if we're measuring attrition from our intervention during that feasibility randomized
control trial.
So it's not uncommon for feasibility randomized control trials of e-health interventions to
measure non-nucid attrition, but measures of engagement are usually quite limited, they're
not so fine grained.
So often it's about when people registering within the website itself, so we know who's
visiting and then we know when they stopped visiting and we might also be able to collect
data on whether they've completed modules within that website itself, but that tends
to be the limitation or the limits of it.
And the use of analytics data rarely happens at all, but I've sort of become increasingly
aware recently that analytics data can offer much greater insight into usability issues.
For example, are the particular points during the different stages in which people are going
through the intervention where they're experiencing some technical issues, it's not very nice,
the look and feel of people are dropping off because they're having problems with it.
And these are the kind of things that can be rectified, which if we don't collect this
kind of data, we won't know anything about.
And typically within feasibility randomized control trials, we tend to have a stage of
qualitative research, so we do interviews with our participants to find out more about
their experience of being a research participant and what we can learn to improve the trial.
And this would be a great stage at which to ask people to, so we can draw hypotheses from
these usability issues and then develop them further through those interviews to find out
more about people's experiences.
So those two things would dovetail really nicely together.
So yeah, we can do these feasibility randomized control trials and examine usability issues
with a view to properly preparing a much better and resolving any of these issues in preparation
for the main trial.
And I just want to make the point that I'm very aware that within the tech sort of web
industry sector, that usability and examining the sort of client or customer's use of websites
and all these friction points and things that are going wrong in order to maximize engagement
and getting people through to the end point, whatever that might be, like purchasing something,
is done really commonly and very well.
But really, in my field, it's rarely done at all, not very much at all.
And I just know there's a huge amount to be learned and gained from collaborating with
people who know much about this, and I'd be really interested in hearing from somebody
who would be interested in working with me in that kind of way.
So yeah, I've hopefully sort of made the case for why it's important to look at non user
attrition and for minimizing that in the trial, we know it's important to do that.
And running these types of feasibility randomized control trials, which are really common, is
a great opportunity to look at attrition and try and remedy it and do something about it.
And I'm really, and everyone else in my field, just at the beginning of trying to understand
how best to measure this type of attrition, to understand what all the measures and things
that we can collect are telling us, and also to make the right sort of changes so that
in the main trial, we can actually have as high an attrition as possible.
I mean, if people don't like the intervention, that's another thing we can't, that's a different
question to be answered.
But if it's to do with how the website itself is operating, I think there's lots we can
do about that.
And then the other issue, and I've only got two slides on this one, this is a much smaller
point really, but equally important is around examining and controlling for dose.
So the amount or dose of an intervention that a participant receives in a trial can affect
the strength of the outcome.
So again, back to that randomization that we talked about, so those in the top arm who
received the intervention, so we look to see what the outcomes are.
So if it's a stop smoking intervention, for example, we'll measure number of cigarettes
per day that I had before the intervention and afterwards, and then we'll compare the
two arms to look at the number of cigarettes smoked.
So we need to know the dose of that intervention.
So if it's a web intervention, we stop smoking, we need to know the amount of consumption
of that website.
Have they gone back a lot every day?
Have they reviewed everything, ordered everything they can, that sort of thing, to know the
dose.
And that way we can control for the analysis.
So we can just isolate the effect of the intervention and not the amount they're having, but then
also we can measure the correlation or the strength of relationship between dose and
outcome as well.
These are really important for us to understand, but you can't examine what you don't measure.
So we really need to understand what is the dose, the amount that people are having of
each of our interventions, and this is to do with the main trial itself.
So if we can measure dose and for this, it's really important that the outcomes are linked
to each individual participant in the study.
So if we can measure dose, then we can understand things such as, is there a linear relationship
between dose and outcome?
So in other words, the more of the intervention that somebody has, the greater the benefit.
Or is it there are non-linear relationships at the point of saturation, which doesn't
really make any difference whether you have any more or not?
Or is there minimum dose that everyone needs to have in order for an effect to be achieved?
And analytics data can also be used for this and can provide us with really precise and
detailed measures of the dose for each individual, such as which pages are accessed, how long
people spend on each page, which videos have been watched, and for how long.
So I'd now like to go on telling a little bit about an intervention that I developed
called RAPT.
And I say I, not me alone, me with a fantastic team of other researchers.
So first, just say a little bit about the study we're doing.
So we're at the stage of doing the feasibility randomized control trial, so the preparatory
study before the main trial.
And this aims to establish if the main randomized control trial is feasible and to inform our
preparations.
And we are using analytics data to better understand any possible usability issues so
that they can be addressed ahead of the trial, so to minimize the non-usage attrition that
I talked about.
But also to learn how best to measure dose in preparation for our main randomized control
trial.
So the RAPT intervention itself, I'm going to show you in a minute what the website looks
like, but just a little bit of information about it.
So it aims to increase condom use by addressing factors such as people's attitudes towards
condoms or whether they feel positively or negatively about them and the beliefs that
underlie that, people's self-efficacy for communicating about condoms and their use.
So this is about, broadly speaking, how confident you feel to raise condom use with a partner
for the first time and also how confident you feel in using them correctly.
And it's also about increasing access to condoms, so about making them more available to people
through a condom distribution scheme.
And there are six components in total which are tailored to individual need.
So at the very start of access to the website, we ask our users to answer some very quick
yes-no questions about their main barriers to condom use, and then they get allocated
between one and six of the components.
So the intervention itself, hopefully, is quite closely aligned to their needs and interests.
And there are three products which can be ordered, a trial pack of condoms, a condom
carrier, and access to a service which supplies condoms on a monthly basis.
And there are also three lots of videos.
So I'm just going to change my screen to show you the website itself.
So I'm going to just shout if you can't see this, otherwise I'll just show you that everybody
can.
So here is the wrapped website.
And after you've answered those few quick questions at the start, this is what you will
see.
And this person here has been assigned all six intervention components.
And these are represented by these blocks on the screen here.
So I won't show you all of it, but I'll just give you a bit of a flavor for it.
So with the sample pack here, if we click on that.
So here, people get the chance to customize the pack.
It's a box with a tray.
And then they can choose the color and an insert that goes inside it.
And then they get to, I just, yeah, choose that.
And then they get to order it.
And it has 12 different types of condoms and three sachets of flub inside there.
And then we've got videos as well.
So this one is about a video, and this is of young people demoing how to put on a condom
correctly without any errors.
And there are various other videos as one of the things to order.
That just gives you a bit of a flavor of the website and what it looks like.
So I'll just go back to the presentation again.
So in terms of data collection, the study requires participants in our feasibility randomized
control trial to complete activities over a 12-month period.
So they consent to the study and complete a survey.
And then they get directed to either, well, they're randomized.
They go into one arm, which is to receive the wrapped intervention website I've just
shown you, or they get randomized to a control website, which has the same branding and sort
of color and logos, but has very basic static information on condom use and sexually transmitted
infections.
And then they get sent another survey three months later, and also a test for an STI called
chlamydia, which gets sent, like a test gets sent in the post to them to complete.
And then at six months, another survey, and then at 12 months, another survey, and also
the same test for an STI.
So they've been asked to do quite a lot from us, and they're incentivized to complete those
different activities.
And we're using some database management software called RedCap to consent participants to prompt
them to complete all those different activities that I've just told you about and to record
their data so it sends out the surveys to them and we record their test results for
the STIs in there as well.
Now, RedCap used to direct participants to the two different websites, so there's an
automated email that goes out to people after they've been randomized, which has a link
into one of the two different websites, and they have to click on that, and then they'll
go to each of the two different websites.
And every activity that any participant in our study does is linked to a unique ID, so
everything they do is logged against that unique ID.
So we were going to be running this feasibility randomized control trial, and we knew that
we wanted to be able to measure attrition both broadly on an aggregate level, but also
individually, so we could look at things like demographic data and how that links to attrition,
but then also to work out how best to measure dose for our main trial so that we can understand
those important things about how dose relates to outcomes.
But we really didn't know how to go about doing this.
We wanted something more fine-grained than just the website itself could tell us, and
I was already beginning to become interested in analytics data, and through a bit of serendipity
came across Ronan, and we began working together on this project.
So over to Ronan to explain a bit about how you went about this, and I'm going to advance
the slides for you, Ronan, so just tell me, I'll go to the first one, then tell me when
to move along.
That's perfect.
Thank you very much, Cathy, for preparing all this work.
So yeah, there are many, many things to say here.
I think the first thing to mention is that this has been, to me, really a project that's
really motivated me a lot because it started by training, so the team of Cathy asked me
to come in the UK and come and train our full team, so I took my backpack, I fly from France
to the UK, and I trained our team for about three days.
I think it was three, I don't remember if it was three or five days, but it's the kind
of project which is really transforming you as an analyst because it's a project which
really doesn't look like any others.
It's not like a public website on which you can land on, and you can easily guess and
find what are the different data collection points that you need to implement.
Here in the case of the RAP project, the first thing I learned about is that it was using
a PHP framework that I didn't know, which was Codignitor.
The developer of the website is a third-party agency based in India, so it wasn't like I
was talking to Cathy straight away, and Cathy could implement the different tracking code
that I wanted.
It was really like a three-party project, so the technology is named Codignitor.
I wasn't really scared about the technology Codignitor that I didn't know, I just went
to the famous search engine and found out that it was a PHP framework.
From this, I knew already that it means a project that I could not put myself, my hands
on, but that I would have to give a recommendation to a third-party company, so to set up the
dev company, which means for me project management in terms of analytics projects.
It means that I really needed to well structure my project in order for them to know what
they need to implement, and I needed to ensure as well that Cathy and her team could clearly
understand what we're going to implement, and that it aligns with everything that Cathy
has mentioned before, which are the needs that they have in terms of data collection
for this research project.
The key aspect of the project, as Cathy showed to you, is a few pages.
I think it's a maximum of between 15 to 20 pages, so it seems like that's an easy project,
but with a lot in terms of data collection.
You have a lot of advanced tracking code, which were necessary, including the measurement
of events, including the use of custom dimensions to say level, visit level information data,
and as well, we quickly saw it user ID, so the possibility to know who is the individual,
let's say, who is making those different choices.
We make the choices to go for a data layer, so to use Matomo Tag Manager, because the
idea was to have something which was really consistent, because the thing is that we have
not many pages, but all those pages are critical, and if we were going without a data layer,
so to say, if I was taking in the project by just asking to add the container on all
the pages, and then decide to use by either scrapping or either use automatically, let's
say, the Tag Manager to collect the different data points, I chances that in the meantime,
the dev company would have made changes to the DOM of the page, and everything would
have exploded, and I would have to redo the full data collection.
So that's the reason why we designed a really consistent data layer, which had been implemented
directly by the dev company in order to ensure that if something breaks in terms of data
collection, I wasn't the person responsible of screwing up the data collection, and of
course, it gave more responsibility and involved more the dev team, which was really the right
solution in this specific project, because I couldn't have the end on the source code.
So those points are really critical, and this is more like a project management analytic
system rather than, let's say, an analyst doing all the work on the platform.
We really need, as well, to have a clear quality assessment process, because as Katie shows,
the RAP project is a place where people are ordering, so they're not making some purchase
because the products are for free, but they are making some orders, and once you make
an order, you cannot order back the different elements, so it means that we didn't want
all the time to go to Katie and say, okay, we make some order in order to test our data
collection, please, could you remove us from the system, and we have to make another order
again in order to test the platform, so that's why we needed to really have a clear project
management system in which we can ensure, okay, here the tracking code has been implemented
over here, we test it, it works, okay, so we tick in the box, it doesn't work, and then
we go back to the dev team in order to ask for a new implementation, and the other point
which was of critical importance was the use of custom dimension, so to say to add additional
data which were not collected by default within Matomo to each individual, so as the use of
the premium feature named custom report, which was one of critical importance because as
we just saw it, the need of data is big, and it has to be crossed with different dimensions,
so this is where the use of custom dimension was of critical importance.
I'm done for this slide, okay, so next one is the work that we decided to work, so this
is the project management part, this work has been reviewed by a colleague of mine that
you know who made a conference as well named Frédéric Forster, and this is the way we
work with the agency, so it consisted of myself drafting this document reviewed by my colleague
Frédéric in order to have a document which is listing what the data layer is about and
then showing this document to the team of Cathy to explain to the team of Cathy what
this is all about because that's clearly not the kind of thing that you see within the
Matomo's documentation, here we are more looking at the project management document made by
an analyst and which as well serve in terms of transparency with the techie team for them
in order to see if, okay, this is the engagement that we took together, you add the responsibility
of deploying the code which is on the column number let's say five for you which is named
code to push when the action is made, here as you can see you can clearly see the data
layer which was implemented, so in our case it was for every step of the order made by
the user we could push an event to Matomo and as I previously said we add for this tracking
code to be consistent so that's why we decided to use a data layer and in the last column
of this document you can clearly see that there is our recommendation which is in our
case that this data layer wasn't implemented properly so please review on this given page
that you have well included this given piece of code on the button when someone is pressing
it so really here to understand how to deal with such a big project it's really like project
management before all that that's the most that's the most important they have to precise
as well one thing that we make some choices with with Katy and her team not to go for
the e-commerce tracking code so just to let you know as well that when you deal with such
a project you have some choices to make because as we saw the user can make some orders but
those order are not paid orders so we had the choice to go either for e-commerce tracking
code or either to go for the event tracking code and we decided to go for the event one
because it was like more justified to go for it rather than having to create some additional
data who were not existing within the system such as let's say the the tax amount such
as the order price such as the transactional order that kind of thing I'm good for this
part next one is kind of the similar thing the big difference is that on the previous
slides that I was that we are showing it was about some variables that we are inserting
within the data layer on this one this is another method that we use with the tag manager
which is the push event method which consists of sending custom event to the data layer
in order for it to react on the interaction of the of the visitor and as well this is
how we succeed to handle it okay I'm good for this one okay and last but not least this
slide show in fact all the different variables so those are all the different needs that
we list with with Katy and ask for validation to see if there was any data which were missing
out of this list and once validated of course this is this is what we implemented but here
on small what we are showing is is really the project management part is like there
is a lot of needs from the client but we need at some point to put a limit and say okay
we don't need extra more data do we agree that this list is the limit and you all need
those and if yes of course this is kind of what is finally defining the final price that's
the analyst will put on on the quote let's say and as well yes I'm good with this slide
I think that was the last oh yeah oh and that's as well a critical part of the of the project
and that I really loved in it is that I during the training I showed to Katy and her team
how to deal with the custom reports feature of matmo and I've been surprised to see that's
when I came some days after or weeks after the implementation with the wrap project that
a lot of custom reports were created and that's a good sign of the training works and now
people have the solution well in end and they are creating as many custom reports as they
can and here you can clearly see how the custom reports really enhance the data which were
in the system of matmo because here we got the user ID with the different components
that it shows and the date at which the individual decided to pick up the product I think I have
one slide left yes and here is another one another report which is really showing for
each individual what are the different components and you can clearly see that on line number
one from line number one to line number four the individual decided to pick up all the
components but line number five for some reason this individual decided to not pick up all
the different components which was one of the needs which was requested from this research
which is about okay what are the different choices that people are making and I think
I'm good for my slides yep exactly okay thanks Ronan so in the last section before the Q&A
I'll just present some early insights from our data collection so from the custom reports
that Ronan was just talking about we've been able to learn a lot already we haven't finished
collecting our data yet we're just a few months into the project so we haven't learned everything
that we will be able to learn by the end but I'll show you what we've got so far so this
is just how I use the data and what's working best for me is to think about the flow of
people through the website in order to sort of see where the things going wrong within
the website and if something needs kind of attending to before our full trial so things
that might be leading people to drop out basically so the non-use usage attrition that I talked
about before so 91 people at the point at which I downloaded the data from the custom
reports had at that point been randomized to wrap so gone to that top of the two different
arms in the randomization and of those 56 people clicked on the link so I told you about
how our software RedCat sent an email invite to everybody inviting them to go to one of
the two different websites so this is 91 people sent the invite to wrapped and 56 of them
clicked on the link and went through to the website so not everybody's doing that despite
us incentivizing them with a voucher so we've lost 35 at that point and then we when I first
land on the website they have to create a profile and sort of register so and then they
also complete some tailoring questions so these are the ones that determine how many
of the components of the six components that they get so between them sort of landing on
the website and creating a profile we've lost just two people in that stage of completing
the tailoring questions so I was a little bit concerned I think there's 10 questions
in total at me people might get a little irritated by that sort of section but we've only lost
two there so that's quite a good sign for me and then we've got registration being complete
and then between that stage and oh sorry visiting the home page we've lost nobody so that's
another good sign that that whole stage is working well so our biggest concern really
is that email that's encouraging people to go visit in the first place perhaps is a bit
boring and doesn't sell the website very well although in a trial context it's kind of difficult
to do that because we can't spell out the things that are good about the intervention
website very easily in that context but we could perhaps try harder and then next slide
so Ronan was talking about people being assigned different components and I mentioned how there's
up to six different ones so we were interested in knowing about the proportion of people
that were going to get assigned the different numbers of them because when we started out
they said we really didn't know that so they answer these 10 questions at the start of
the website and depending on their answers they'll get assigned either all of the six
of them or just the one of them there's always a minimum of one and this is how it's worked
out so no one's had just one even though that's a possibility and then very few people have
had two three or four components most people have either been assigned five or six components
so that's useful in terms of helping us plan for the main trial it helps us to work out
resource need because some of the components at least where they're ordering something
has a cost associated that's really useful and then I explained that there were three
different things that people could order and I showed you the sample box that the website
page relating to that so this is the box with 12 different types of condoms to try out in
it so 54 people were assigned the sample box in total that's everyone who went to the website
because everyone gets the sample box by default so 54 were given that and then so on that
main page where you've got the six different boxes that you can click on or to take you
through to the relevant page 54 people have seen the one for the sample box and 48 of
them have actually clicked on that and gone to visit that page we've just lost six at
that point which is not bad and then between visiting that page and then placing an order
we've lost 11 people so that's 11 people maybe who just didn't like the look of it and just
thought that wasn't really for them but that's an assumption but something perhaps to speak
to people later on in the study when we come to our interviews and then those who then
clicked to go through to the next page they're sort of progressing with making an order they've
chosen what box they want what insert they want we've just lost one person and then we
lost two on the next page between the point to seeing that second page and actually placing
the order so yeah I was quite pleased that it shows to me that process of ordering is
quite smooth and straightforward and perhaps not no one's experiencing technical problems
or issues with usability around the ordering process and really it's the same picture with
condom ordering so 49 people ordered that I'm not going to go through all the numbers
on this one but it's the same pattern that we've lost just a handful at the start who
were never visiting that page and then the actual ordering seems to be fairly straightforward
for everybody and the same with ordering the condom carrier as well and then the three
different types of video that people can go and see I'm just going to do one slide on
one of the videos which is the demo demo video that I showed you it's a completely different
picture here so 47 people allocated this video but only 16 actually clicked on that box to
go and visit the demo page that's telling us that that home page is not selling this
demo video at all goodbye even going to click on it well only 16 of our 47 did and then
only of those 16 over only 7 actually then went ahead to click on the video and play
it and then nobody watched it in full and the meantime is about watching the video was
17 seconds so not attractive able to go and visit it when they do go and visit the page
not playing it very much not many people playing it and then when they do disengaging quite
quickly so we've got quite a bit of work to do on the video and the demo video but the
same picture is true of the other types of videos as well they all need looking at so
what we've learnt well we've learnt that we can use analytics data to gain some important
insights into non usage attrition and we've just started to kind of touch the surface
with the data we've got already and we're going to use this to make some improvements
so my conclusions I've kind of touched on touched upon already are that the home page
as attracting users to visit are the first three components the things that you can order
and once on those component pages there's good conversion to ordering there seems to
be no issues with the ordering process but the home page is not attracting people to
visit the video content and those video component pages are also not working very well to encourage
users to watch the videos when they're already there and then the videos aren't being watched
for very long so the first few seconds are encouraging people to carry on and we still
have lots and lots to learn about this field and what is really possible in terms of analytics
so I'm very much a newbie in this situation and our solution for linking individual level
the user ID that we're collecting in redcap for all our surveys and STI testing and linking
that to our individuals as they go through to the two different websites so we can collect
this analytics data that process that Ronan worked out for us by liaising with us and
also our development companies has worked really well has been successful had no problems
with gathering the unique user ID of everyone in Amatomo and seeing what they're all doing
creating these custom reports and gathering all that data from there that's all worked
really well and that means it's going to be possible for us to measure those in our main
trial and in quite a fine-grained way which is really useful and we're just beginning
to think about how we'll calculate a sort of standardized score for everybody in our
trial at an individual level based on the products that are allocated and ordered and
also which videos to watch and how long for so that's the end of the presentation so I'd
just like to acknowledge our funder the National Institute for Health Research and all of the
team working on this with me as well at the University of Hertfordshire and others and
yeah on to any questions thank you very much Katie thank you very much for this great presentation
I'm looking at the chat right now we have approximately let's say maximum five minutes
because after there's a other talker at 4 p.m. I'm just going to pick the first one which
is which was the most useful metamode tools used during this project?
Rony you might have a view on this but for me I suppose I might not be using the right
terms but it was the custom ID and then also the reports as well I mean the reports are
something I go into all the time now to look at to gather the data to help me understand
the usability of the website and on all the slides I've just presented with those flowcharts
all comes from those custom reports.
I'm looking at the chat there's another question with small linking with logistic which is
about will it be possible to download the presentation I mean will it be possible for
you to send it to I mean do you allow me to share it with the audience or would you like
to keep your presentation for yourself or can it be?
That's fine Rony yeah I'm happy for anyone to have it so I'll send it to you.
Yeah well I think I have it in my mailbox so I will just make it as a PDF and then and
then share it back.
I let the audience write down the last question we still have let's say four minutes left.
Do you have for example any questions Cathy that you would like me to ask you for the
audience or do you have any questions for you?
I mean do you have any questions that you would have expected the audience to ask you
and that you would like me to ask you?
Do I have any questions?
I suppose the thing that I'm still struggling with the most is linking all the data in the
custom report so each custom report I can download the data as an excel file so for
example if we're talking about the components allocated I can download it so I've got one
column which is the user ID and then you know did they have component one yes no component
two yes no I can create it like that then I can download another separate report that
says did they visit the sample pack page for example and that'll be a completely separate
one and so at the moment they're all very separate like that and then I have to combine
them all sort of together so I suppose oh and also every time I want to update the data
it's not easy I can use date I suppose but I can't just filter by everything since I
last looked and then download that and add it in I haven't just kept kind of keep manually
looking and then updating my excel data based on that so I don't know whether it's possible
this is more a question for you than the audience but this is something that I'm I'm struggling
most with at the moment and yeah I'd be keen to know about if there's a way to kind of
make that process a bit more straightforward I suppose but might be getting to the limits
of what Matomo can provide okay great yeah I don't have the yet the answer I think that
we are looking for custom reports with far more dimensions I guess that's the key thing
here the maximum is really three custom dimension and from what I remembered yeah every time
you asked something I did my best in terms of optimizations you can yeah you can just
put up to three custom dimension and then filter by some other custom dimension but
at the moment I'm not sure that there's plans to extend the number of custom dimension which
correspond more to what you would like like you extend if you export the full excel file
and you will get as many custom dimension as you need so in our case I don't remember
if we have up to five or ten but then you can refilter back in excel Alfonso says thank
you because you answered his question about which was the most useful Matomo tools and
we don't have any questions left so I guess that's the time to close the conference talk
once more thank you very much Katy for your time thank you very much for the presentation
and as well I would like to thank all the attendees of this conference and I wish you
all a great end of Matomo camp there's just one conference two conferences left starting
in five minutes and then we have the closing ceremony at in one hour so at five p.m. thank
you very much pleasure good luck with the rest of the conference thanks bye bye