recording-subtitles/2021/Wrapped/output.srt

1
00:00:00,000 --> 00:00:07,200
Hello, everyone.

2
00:00:07,200 --> 00:00:09,280
Nice to meet you all again.

3
00:00:09,280 --> 00:00:11,480
It's currently 3.02.

4
00:00:11,480 --> 00:00:15,160
And we will conduct a conference today.

5
00:00:15,160 --> 00:00:21,920
So continue by who is here with me and myself.

6
00:00:21,920 --> 00:00:28,320
So the idea of this conference is to do something which is kind of unusual.

7
00:00:28,320 --> 00:00:30,000
So we're going to do a use case.

8
00:00:30,000 --> 00:00:35,080
And most of the time, when a use case is presented by an analyst, it's most of the time, like

9
00:00:35,080 --> 00:00:39,440
three-quarter of the time, the analyst who is speaking and then a quarter of the time

10
00:00:39,440 --> 00:00:40,440
with the client.

11
00:00:40,440 --> 00:00:44,520
Here, this is going to be the reverse mode.

12
00:00:44,520 --> 00:00:51,280
So I will leave the floor in a couple of minutes to Cathy, who is a client of mine.

13
00:00:51,280 --> 00:00:58,640
And the idea is that Cathy can introduce you to the project that she's working on.

14
00:00:58,640 --> 00:01:07,640
And I will explain how I decided to deal with the project management of including Matomo

15
00:01:07,640 --> 00:01:11,800
within this given project.

16
00:01:11,800 --> 00:01:17,520
So I leave now the floor to Cathy, and I will interact only dealing with my part of the

17
00:01:17,520 --> 00:01:18,520
slides.

18
00:01:18,520 --> 00:01:19,520
OK.

19
00:01:19,520 --> 00:01:20,520
Thanks, Ronan.

20
00:01:20,520 --> 00:01:21,520
I'll just share my slides.

21
00:01:21,520 --> 00:01:34,040
Can I just check that you can see that, Ronan?

22
00:01:34,040 --> 00:01:39,280
Yes, it looks great.

23
00:01:39,280 --> 00:01:40,280
OK.

24
00:01:40,280 --> 00:01:41,280
Lovely.

25
00:01:41,280 --> 00:01:42,280
Thank you.

26
00:01:42,280 --> 00:01:43,280
Right.

27
00:01:43,280 --> 00:01:47,920
So myself and Ronan are going to, as he said, share the presentation today.

28
00:01:47,920 --> 00:01:53,680
And the presentation is on using Matomo to collect data on intervention engagement within

29
00:01:53,680 --> 00:01:59,880
a research trial, and the case study is a project called WRAPT.

30
00:01:59,880 --> 00:02:05,880
So the overview of the presentation is that I'll firstly talk about why I think it's important

31
00:02:05,880 --> 00:02:10,240
to measure intervention engagement within a research trial.

32
00:02:10,240 --> 00:02:14,040
I will tell you a bit about the WRAPT research project as well.

33
00:02:14,040 --> 00:02:19,120
And then Ronan's going to talk about how we use Matomo within the project to collect individual

34
00:02:19,120 --> 00:02:21,800
level data on engagement.

35
00:02:21,800 --> 00:02:25,320
Then I'll present some early insights from our analytics data, and then there'll be a

36
00:02:25,320 --> 00:02:30,080
chance for some questions at the end.

37
00:02:30,080 --> 00:02:35,240
So first of all, why it's important to measure intervention engagement within a research

38
00:02:35,240 --> 00:02:36,240
trial.

39
00:02:36,240 --> 00:02:38,000
Well, there are two reasons.

40
00:02:38,000 --> 00:02:42,040
One is to minimize something called non-usage attrition.

41
00:02:42,040 --> 00:02:44,760
And I'll go on to explain a bit about what that is.

42
00:02:44,760 --> 00:02:49,600
And then also to examine and control for intervention dose.

43
00:02:49,600 --> 00:02:53,080
So firstly, on to minimizing non-usage attrition.

44
00:02:53,080 --> 00:02:58,920
So when we are testing whether a new intervention works, then we typically do something called

45
00:02:58,920 --> 00:03:00,640
a randomized control trial.

46
00:03:00,640 --> 00:03:06,300
And many of you will be aware of what randomized control trials are already, maybe.

47
00:03:06,300 --> 00:03:08,960
But they are, I'll just share my next slide.

48
00:03:08,960 --> 00:03:14,680
So they are used all the time in a clinical context to test the effectiveness of something.

49
00:03:14,680 --> 00:03:20,120
So that something could be a new drug or a new vaccine, as is the case for the COVID

50
00:03:20,120 --> 00:03:21,120
vaccine.

51
00:03:21,120 --> 00:03:27,400
Or in my case, and I'm a health psychologist, I develop interventions that try to encourage

52
00:03:27,400 --> 00:03:29,160
people to change their health behavior.

53
00:03:29,160 --> 00:03:32,680
So that could be, for example, to persuade people to use the COVID vaccine.

54
00:03:32,680 --> 00:03:37,000
So the intervention itself is not a drug or a vaccine or something like that.

55
00:03:37,000 --> 00:03:40,840
It tends to be a behavior change intervention.

56
00:03:40,840 --> 00:03:43,320
But the principle of testing it is exactly the same.

57
00:03:43,320 --> 00:03:47,040
So we would also run a randomized control trial.

58
00:03:47,040 --> 00:03:51,200
And in randomized control trials, we take a sample population.

59
00:03:51,200 --> 00:03:56,600
So this is a smaller group of the wider population we're interested in testing the intervention

60
00:03:56,600 --> 00:03:57,600
on.

61
00:03:57,600 --> 00:04:00,360
And we'll randomly assign them to two groups.

62
00:04:00,360 --> 00:04:06,040
So like the flip of a coin or a roll of a dice or more commonly using sort of internet

63
00:04:06,040 --> 00:04:07,560
based randomization tools.

64
00:04:07,560 --> 00:04:12,120
But the idea is the same, which is randomly assigning to one group or the other.

65
00:04:12,120 --> 00:04:17,520
And in the one group, which here we have as group one, all the people in that group that

66
00:04:17,520 --> 00:04:22,640
have been randomly assigned to it will get the new drug or treatment.

67
00:04:22,640 --> 00:04:27,480
And then we'll measure afterwards how they fare, what the outcomes were for them.

68
00:04:27,480 --> 00:04:29,640
And then in the other group, they'll get the control.

69
00:04:29,640 --> 00:04:34,360
So if it is a drug, it might be something like a sugar pill that has no active ingredients.

70
00:04:34,360 --> 00:04:36,640
And again, we'll measure the outcome for them.

71
00:04:36,640 --> 00:04:40,520
And the important point with the randomization is that by the end, we should have balance

72
00:04:40,520 --> 00:04:41,520
in the two groups.

73
00:04:41,520 --> 00:04:45,960
So just by chance, we should have approximately even numbers of people of different genders

74
00:04:45,960 --> 00:04:52,440
and ethnicities or with similar levels of health need or health condition.

75
00:04:52,440 --> 00:04:57,080
And that's important to make sure that we control for any sort of confounding factors

76
00:04:57,080 --> 00:04:59,320
that might also have an effect on the outcome.

77
00:04:59,320 --> 00:05:04,880
And they should then be evenly spread between the two groups.

78
00:05:04,880 --> 00:05:09,080
So in a clinical context, when a new drug is being tested, adherence to the drug is

79
00:05:09,080 --> 00:05:14,640
not usually a big problem, although something still to be mindful of.

80
00:05:14,640 --> 00:05:19,840
And this is because the drug is prescribed by a clinician, and the participants are usually

81
00:05:19,840 --> 00:05:21,400
very closely supervised.

82
00:05:21,400 --> 00:05:26,760
So you would be asked to go into a clinical context, for example, and you'd be passed

83
00:05:26,760 --> 00:05:31,880
the pill on a tray with a cup of water and watched by a research nurse taking it, who

84
00:05:31,880 --> 00:05:36,160
then recalled that you'd taken it, and that would likely be for every dose.

85
00:05:36,160 --> 00:05:40,600
And also, as a research participant, you may well, especially if you're in an intervention

86
00:05:40,600 --> 00:05:45,240
condition, experience observable and immediate health benefits of taking that drug, assuming

87
00:05:45,240 --> 00:05:49,960
that it's effective, also potentially in the control condition due to placebo effects.

88
00:05:49,960 --> 00:05:56,400
So there's lots of things that could be meaning that adherence to that drug within an intervention

89
00:05:56,400 --> 00:06:00,440
and trial context is not too much of a problem.

90
00:06:00,440 --> 00:06:04,280
It's usually, it's usually reasonably high.

91
00:06:04,280 --> 00:06:11,720
However, in contract, adherence is a major problem when we're testing e-health interventions.

92
00:06:11,720 --> 00:06:14,360
So I mentioned briefly about the kind of work that I do.

93
00:06:14,360 --> 00:06:19,920
So to test health behavior change interventions, and also almost all of the interventions that

94
00:06:19,920 --> 00:06:22,460
I develop and test are of the e-health type.

95
00:06:22,460 --> 00:06:25,400
So they're interventions delivered via the internet.

96
00:06:25,400 --> 00:06:32,360
So here, the intervention is neither prescribed nor critical to well-being.

97
00:06:32,360 --> 00:06:41,000
And this graph, I think, demonstrates quite nicely typical attrition from e-health interventions.

98
00:06:41,000 --> 00:06:46,320
So on the y-axis going up the left, you can see at the very top, there's a 1, and that

99
00:06:46,320 --> 00:06:53,080
reflects 100% of users of the intervention going down to 90, 80, 70, et cetera, right

100
00:06:53,080 --> 00:06:54,960
down to 0 at the bottom.

101
00:06:54,960 --> 00:06:59,800
And then across the bottom on the x-axis, we've got time and modules.

102
00:06:59,800 --> 00:07:08,200
So if we take the blue line to start with, the study by Farrell Dunn, this was a 12-week

103
00:07:08,200 --> 00:07:12,920
program of internet delivery of an intervention.

104
00:07:12,920 --> 00:07:17,920
So it's supposed to engage for 12 weeks in total to have had the full dose of the intervention.

105
00:07:17,920 --> 00:07:22,740
But you can see after a couple of weeks, we've already got less than 50% of people still

106
00:07:22,740 --> 00:07:29,080
using that website, going down to less than 20% at week 3.

107
00:07:29,080 --> 00:07:33,200
And you can see by the end, in fact, it was less than 1% of people still using or still

108
00:07:33,200 --> 00:07:37,480
completing the intervention right at the very end.

109
00:07:37,480 --> 00:07:43,240
And then for the dashed black line, this was an intervention called Mood Gym.

110
00:07:43,240 --> 00:07:44,880
And it's in a trial context again.

111
00:07:44,880 --> 00:07:47,920
And there were five modules here for this intervention.

112
00:07:47,920 --> 00:07:54,440
You can see that less than 70% did module 2, and then module 3, less than 60%, and so

113
00:07:54,440 --> 00:07:58,520
on, down to just over 20% doing all five modules.

114
00:07:58,520 --> 00:08:01,680
And then the solid black line is also Mood Gym.

115
00:08:01,680 --> 00:08:04,920
But in this context, it wasn't in a trial.

116
00:08:04,920 --> 00:08:09,100
It was just people allowed to have access to it, then see who engaged with it and carried

117
00:08:09,100 --> 00:08:11,640
on engaging with it.

118
00:08:11,640 --> 00:08:16,680
So outside the context of a trial in which people might be incentivized to use the intervention

119
00:08:16,680 --> 00:08:22,720
and also encouraged and reminded by researchers to do so, you can see that engagement is even

120
00:08:22,720 --> 00:08:26,200
less.

121
00:08:26,200 --> 00:08:33,320
So non-use of nutrition, so participants ceasing to use the intervention in a trial context,

122
00:08:33,320 --> 00:08:37,200
is obviously a big problem for us.

123
00:08:37,200 --> 00:08:41,880
And if participants are not using intervention in a trial context, it makes it really difficult

124
00:08:41,880 --> 00:08:46,080
to test whether the intervention actually works or not.

125
00:08:46,080 --> 00:08:55,200
So in trials, regardless of, sorry, in trials, the outcome data for every participant is

126
00:08:55,200 --> 00:08:57,760
randomized and included in the analysis.

127
00:08:57,760 --> 00:09:04,200
So if we have people in that, if you remember that first diagram I showed you of a randomized

128
00:09:04,200 --> 00:09:05,200
controls trial.

129
00:09:05,200 --> 00:09:09,160
So we've got people randomized to the two groups, and in that top arm, we've got people

130
00:09:09,160 --> 00:09:11,480
who received the intervention itself.

131
00:09:11,480 --> 00:09:15,100
If you've only got a small proportion of those who are actually using intervention of four,

132
00:09:15,100 --> 00:09:19,760
you might be thinking to yourself, well, just analyze data from those people who use the

133
00:09:19,760 --> 00:09:23,480
intervention itself and ignore everyone else who didn't.

134
00:09:23,480 --> 00:09:27,720
But that threatens the whole integrity of the trial design, which is the randomization

135
00:09:27,720 --> 00:09:29,760
of people to one group or the other.

136
00:09:29,760 --> 00:09:34,240
I explained earlier about how we randomize so we have balance between the two arms.

137
00:09:34,240 --> 00:09:39,800
But if you're only including a subgroup of those people who have used the intervention

138
00:09:39,800 --> 00:09:44,480
to a minimum extent and ignoring everybody else, then you remove that balance, which

139
00:09:44,480 --> 00:09:51,800
is so important to that trial validity.

140
00:09:51,800 --> 00:09:59,680
So the point of ensuring that everybody, regardless of whether they use the intervention or not

141
00:09:59,680 --> 00:10:02,440
in a trial, it's called intention to treat analysis.

142
00:10:02,440 --> 00:10:07,920
So if you randomize someone, you always analyze their data regardless to ensure you've got

143
00:10:07,920 --> 00:10:09,880
this balance in design.

144
00:10:09,880 --> 00:10:13,760
But the problem, of course, is if you've got a group of people in your intervention condition

145
00:10:13,760 --> 00:10:17,520
that haven't used the intervention at all or not very much, then if your intervention

146
00:10:17,520 --> 00:10:22,240
is indeed effective, then when you do the analysis, it has the effect of underestimating

147
00:10:22,240 --> 00:10:25,960
the effect of the intervention because you're including in the analysis in that intervention

148
00:10:25,960 --> 00:10:30,600
condition people who didn't even use it at all or didn't use it very much at all.

149
00:10:30,600 --> 00:10:36,120
And remember that diagram of it dropping off steeply, that is often the case for us.

150
00:10:36,120 --> 00:10:40,960
So it's really, really important that in a trial context that we do absolutely everything

151
00:10:40,960 --> 00:10:49,120
that we can to minimize people not using that intervention or dropping off.

152
00:10:49,120 --> 00:10:54,760
Now ahead of doing a main trial in which we test the effectiveness of the kind of intervention

153
00:10:54,760 --> 00:10:59,000
that I've been talking about, we tend to do something called a feasibility randomized

154
00:10:59,000 --> 00:11:03,760
control trial, which is exactly the same as a randomized control trial with this randomization

155
00:11:03,760 --> 00:11:05,160
to the two groups.

156
00:11:05,160 --> 00:11:08,680
The only thing that's different is it's with much fewer participants.

157
00:11:08,680 --> 00:11:12,760
And the purpose of that kind of study is to help us to prepare for a full randomized control

158
00:11:12,760 --> 00:11:13,760
trial.

159
00:11:13,760 --> 00:11:18,680
And what we're trying to do is resolve or remove all the unknowns.

160
00:11:18,680 --> 00:11:24,200
So for example, to help us to learn what the best methods for recruiting participants is,

161
00:11:24,200 --> 00:11:27,920
what level of incentive we need to recruit people in the first place, but also to keep

162
00:11:27,920 --> 00:11:29,240
them in the trial until the end.

163
00:11:29,240 --> 00:11:33,320
So all those kinds of things are important for us to establish in this early phase before

164
00:11:33,320 --> 00:11:36,080
we do the main thing.

165
00:11:36,080 --> 00:11:44,160
But we can also use these feasibility randomized control trials, these preparation type trials,

166
00:11:44,160 --> 00:11:48,320
is to examine non-nucid attrition, which we've been talking about.

167
00:11:48,320 --> 00:11:53,600
So if we look at attrition in this early stage trial, it can tell us some important things

168
00:11:53,600 --> 00:11:58,040
such as whether attrition is associated with particular demographic factors.

169
00:11:58,040 --> 00:12:01,560
And of course, we're measuring those in the surveys that we give to people before they

170
00:12:01,560 --> 00:12:03,840
have the intervention and then also afterwards.

171
00:12:03,840 --> 00:12:09,400
And so if we can link that demographic data with the data about attrition, we can learn

172
00:12:09,400 --> 00:12:10,520
some important things.

173
00:12:10,520 --> 00:12:14,720
So if we found out that, for example, females in particular weren't using our intervention

174
00:12:14,720 --> 00:12:19,160
or dropping off, then it would be important to maybe interview them as part of some later

175
00:12:19,160 --> 00:12:24,800
studies to find out why that might be the case and to try and rectify that.

176
00:12:24,800 --> 00:12:28,600
And when we look at the shape of that attrition curve, which we were just looking at before,

177
00:12:28,600 --> 00:12:32,600
that can tell us something also important.

178
00:12:32,600 --> 00:12:38,320
So if we had, should I get my mouse to do that?

179
00:12:38,320 --> 00:12:39,320
Yeah.

180
00:12:39,320 --> 00:12:44,400
So if we had a shape a bit sort of like really steep down like that, that would be a really

181
00:12:44,400 --> 00:12:49,840
bad sign for our intervention, tell us that people were disengaging very quickly because

182
00:12:49,840 --> 00:12:52,080
they just didn't like it.

183
00:12:52,080 --> 00:12:59,200
And then this more kind of sigmoid style curve would tell us that we had some sort of initial

184
00:12:59,200 --> 00:13:03,560
interest in it, which then dropped off, as you might imagine, then a bit of a steep drop

185
00:13:03,560 --> 00:13:08,120
off in the middle, perhaps when there's less motivated people were leaving or possibly

186
00:13:08,120 --> 00:13:10,600
representing some sort of usability style problems.

187
00:13:10,600 --> 00:13:14,520
And then we've got a hardcore sort of set of users at the bottom.

188
00:13:14,520 --> 00:13:19,360
So the style and shape, I suppose, of this curve can tell us some important things as

189
00:13:19,360 --> 00:13:24,680
well if we're measuring attrition from our intervention during that feasibility randomized

190
00:13:24,680 --> 00:13:28,840
control trial.

191
00:13:28,840 --> 00:13:35,480
So it's not uncommon for feasibility randomized control trials of e-health interventions to

192
00:13:35,480 --> 00:13:42,000
measure non-nucid attrition, but measures of engagement are usually quite limited, they're

193
00:13:42,000 --> 00:13:43,860
not so fine grained.

194
00:13:43,860 --> 00:13:50,960
So often it's about when people registering within the website itself, so we know who's

195
00:13:50,960 --> 00:13:54,720
visiting and then we know when they stopped visiting and we might also be able to collect

196
00:13:54,720 --> 00:13:58,520
data on whether they've completed modules within that website itself, but that tends

197
00:13:58,520 --> 00:14:03,320
to be the limitation or the limits of it.

198
00:14:03,320 --> 00:14:07,960
And the use of analytics data rarely happens at all, but I've sort of become increasingly

199
00:14:07,960 --> 00:14:13,320
aware recently that analytics data can offer much greater insight into usability issues.

200
00:14:13,320 --> 00:14:19,960
For example, are the particular points during the different stages in which people are going

201
00:14:19,960 --> 00:14:25,400
through the intervention where they're experiencing some technical issues, it's not very nice,

202
00:14:25,400 --> 00:14:28,600
the look and feel of people are dropping off because they're having problems with it.

203
00:14:28,600 --> 00:14:31,320
And these are the kind of things that can be rectified, which if we don't collect this

204
00:14:31,320 --> 00:14:35,480
kind of data, we won't know anything about.

205
00:14:35,480 --> 00:14:39,560
And typically within feasibility randomized control trials, we tend to have a stage of

206
00:14:39,560 --> 00:14:43,360
qualitative research, so we do interviews with our participants to find out more about

207
00:14:43,360 --> 00:14:47,840
their experience of being a research participant and what we can learn to improve the trial.

208
00:14:47,840 --> 00:14:52,560
And this would be a great stage at which to ask people to, so we can draw hypotheses from

209
00:14:52,560 --> 00:14:56,760
these usability issues and then develop them further through those interviews to find out

210
00:14:56,760 --> 00:14:58,160
more about people's experiences.

211
00:14:58,160 --> 00:15:03,960
So those two things would dovetail really nicely together.

212
00:15:03,960 --> 00:15:10,920
So yeah, we can do these feasibility randomized control trials and examine usability issues

213
00:15:10,920 --> 00:15:16,960
with a view to properly preparing a much better and resolving any of these issues in preparation

214
00:15:16,960 --> 00:15:18,440
for the main trial.

215
00:15:18,440 --> 00:15:24,640
And I just want to make the point that I'm very aware that within the tech sort of web

216
00:15:24,640 --> 00:15:32,680
industry sector, that usability and examining the sort of client or customer's use of websites

217
00:15:32,680 --> 00:15:36,720
and all these friction points and things that are going wrong in order to maximize engagement

218
00:15:36,720 --> 00:15:41,640
and getting people through to the end point, whatever that might be, like purchasing something,

219
00:15:41,640 --> 00:15:44,480
is done really commonly and very well.

220
00:15:44,480 --> 00:15:49,160
But really, in my field, it's rarely done at all, not very much at all.

221
00:15:49,160 --> 00:15:53,400
And I just know there's a huge amount to be learned and gained from collaborating with

222
00:15:53,400 --> 00:15:56,920
people who know much about this, and I'd be really interested in hearing from somebody

223
00:15:56,920 --> 00:16:03,720
who would be interested in working with me in that kind of way.

224
00:16:03,720 --> 00:16:11,000
So yeah, I've hopefully sort of made the case for why it's important to look at non user

225
00:16:11,000 --> 00:16:16,080
attrition and for minimizing that in the trial, we know it's important to do that.

226
00:16:16,080 --> 00:16:19,720
And running these types of feasibility randomized control trials, which are really common, is

227
00:16:19,720 --> 00:16:25,320
a great opportunity to look at attrition and try and remedy it and do something about it.

228
00:16:25,320 --> 00:16:28,960
And I'm really, and everyone else in my field, just at the beginning of trying to understand

229
00:16:28,960 --> 00:16:34,240
how best to measure this type of attrition, to understand what all the measures and things

230
00:16:34,240 --> 00:16:38,840
that we can collect are telling us, and also to make the right sort of changes so that

231
00:16:38,840 --> 00:16:42,480
in the main trial, we can actually have as high an attrition as possible.

232
00:16:42,480 --> 00:16:46,480
I mean, if people don't like the intervention, that's another thing we can't, that's a different

233
00:16:46,480 --> 00:16:47,480
question to be answered.

234
00:16:47,480 --> 00:16:51,200
But if it's to do with how the website itself is operating, I think there's lots we can

235
00:16:51,200 --> 00:16:53,680
do about that.

236
00:16:53,680 --> 00:16:57,320
And then the other issue, and I've only got two slides on this one, this is a much smaller

237
00:16:57,320 --> 00:17:05,100
point really, but equally important is around examining and controlling for dose.

238
00:17:05,100 --> 00:17:11,280
So the amount or dose of an intervention that a participant receives in a trial can affect

239
00:17:11,280 --> 00:17:13,120
the strength of the outcome.

240
00:17:13,120 --> 00:17:18,280
So again, back to that randomization that we talked about, so those in the top arm who

241
00:17:18,280 --> 00:17:21,900
received the intervention, so we look to see what the outcomes are.

242
00:17:21,900 --> 00:17:26,880
So if it's a stop smoking intervention, for example, we'll measure number of cigarettes

243
00:17:26,880 --> 00:17:31,080
per day that I had before the intervention and afterwards, and then we'll compare the

244
00:17:31,080 --> 00:17:34,800
two arms to look at the number of cigarettes smoked.

245
00:17:34,800 --> 00:17:39,440
So we need to know the dose of that intervention.

246
00:17:39,440 --> 00:17:43,440
So if it's a web intervention, we stop smoking, we need to know the amount of consumption

247
00:17:43,440 --> 00:17:44,440
of that website.

248
00:17:44,440 --> 00:17:45,680
Have they gone back a lot every day?

249
00:17:45,680 --> 00:17:49,240
Have they reviewed everything, ordered everything they can, that sort of thing, to know the

250
00:17:49,240 --> 00:17:50,240
dose.

251
00:17:50,240 --> 00:17:52,720
And that way we can control for the analysis.

252
00:17:52,720 --> 00:17:58,000
So we can just isolate the effect of the intervention and not the amount they're having, but then

253
00:17:58,000 --> 00:18:02,680
also we can measure the correlation or the strength of relationship between dose and

254
00:18:02,680 --> 00:18:03,680
outcome as well.

255
00:18:03,680 --> 00:18:09,060
These are really important for us to understand, but you can't examine what you don't measure.

256
00:18:09,060 --> 00:18:12,560
So we really need to understand what is the dose, the amount that people are having of

257
00:18:12,560 --> 00:18:19,000
each of our interventions, and this is to do with the main trial itself.

258
00:18:19,000 --> 00:18:26,320
So if we can measure dose and for this, it's really important that the outcomes are linked

259
00:18:26,320 --> 00:18:29,700
to each individual participant in the study.

260
00:18:29,700 --> 00:18:34,480
So if we can measure dose, then we can understand things such as, is there a linear relationship

261
00:18:34,480 --> 00:18:36,200
between dose and outcome?

262
00:18:36,200 --> 00:18:41,120
So in other words, the more of the intervention that somebody has, the greater the benefit.

263
00:18:41,120 --> 00:18:44,840
Or is it there are non-linear relationships at the point of saturation, which doesn't

264
00:18:44,840 --> 00:18:47,420
really make any difference whether you have any more or not?

265
00:18:47,420 --> 00:18:52,840
Or is there minimum dose that everyone needs to have in order for an effect to be achieved?

266
00:18:52,840 --> 00:18:58,560
And analytics data can also be used for this and can provide us with really precise and

267
00:18:58,560 --> 00:19:04,240
detailed measures of the dose for each individual, such as which pages are accessed, how long

268
00:19:04,240 --> 00:19:10,720
people spend on each page, which videos have been watched, and for how long.

269
00:19:10,720 --> 00:19:18,240
So I'd now like to go on telling a little bit about an intervention that I developed

270
00:19:18,240 --> 00:19:19,240
called RAPT.

271
00:19:19,240 --> 00:19:25,440
And I say I, not me alone, me with a fantastic team of other researchers.

272
00:19:25,440 --> 00:19:29,320
So first, just say a little bit about the study we're doing.

273
00:19:29,320 --> 00:19:33,680
So we're at the stage of doing the feasibility randomized control trial, so the preparatory

274
00:19:33,680 --> 00:19:36,360
study before the main trial.

275
00:19:36,360 --> 00:19:42,560
And this aims to establish if the main randomized control trial is feasible and to inform our

276
00:19:42,560 --> 00:19:43,560
preparations.

277
00:19:43,560 --> 00:19:49,440
And we are using analytics data to better understand any possible usability issues so

278
00:19:49,440 --> 00:19:53,840
that they can be addressed ahead of the trial, so to minimize the non-usage attrition that

279
00:19:53,840 --> 00:19:54,840
I talked about.

280
00:19:54,840 --> 00:19:59,440
But also to learn how best to measure dose in preparation for our main randomized control

281
00:19:59,440 --> 00:20:03,400
trial.

282
00:20:03,400 --> 00:20:08,040
So the RAPT intervention itself, I'm going to show you in a minute what the website looks

283
00:20:08,040 --> 00:20:11,040
like, but just a little bit of information about it.

284
00:20:11,040 --> 00:20:16,900
So it aims to increase condom use by addressing factors such as people's attitudes towards

285
00:20:16,900 --> 00:20:20,560
condoms or whether they feel positively or negatively about them and the beliefs that

286
00:20:20,560 --> 00:20:26,440
underlie that, people's self-efficacy for communicating about condoms and their use.

287
00:20:26,440 --> 00:20:31,600
So this is about, broadly speaking, how confident you feel to raise condom use with a partner

288
00:20:31,600 --> 00:20:37,800
for the first time and also how confident you feel in using them correctly.

289
00:20:37,800 --> 00:20:43,640
And it's also about increasing access to condoms, so about making them more available to people

290
00:20:43,640 --> 00:20:47,680
through a condom distribution scheme.

291
00:20:47,680 --> 00:20:53,000
And there are six components in total which are tailored to individual need.

292
00:20:53,000 --> 00:20:58,480
So at the very start of access to the website, we ask our users to answer some very quick

293
00:20:58,480 --> 00:21:02,640
yes-no questions about their main barriers to condom use, and then they get allocated

294
00:21:02,640 --> 00:21:05,500
between one and six of the components.

295
00:21:05,500 --> 00:21:11,800
So the intervention itself, hopefully, is quite closely aligned to their needs and interests.

296
00:21:11,800 --> 00:21:16,600
And there are three products which can be ordered, a trial pack of condoms, a condom

297
00:21:16,600 --> 00:21:22,400
carrier, and access to a service which supplies condoms on a monthly basis.

298
00:21:22,400 --> 00:21:25,880
And there are also three lots of videos.

299
00:21:25,880 --> 00:21:40,320
So I'm just going to change my screen to show you the website itself.

300
00:21:40,320 --> 00:21:43,280
So I'm going to just shout if you can't see this, otherwise I'll just show you that everybody

301
00:21:43,280 --> 00:21:44,280
can.

302
00:21:44,280 --> 00:21:47,240
So here is the wrapped website.

303
00:21:47,240 --> 00:21:50,400
And after you've answered those few quick questions at the start, this is what you will

304
00:21:50,400 --> 00:21:51,800
see.

305
00:21:51,800 --> 00:21:56,000
And this person here has been assigned all six intervention components.

306
00:21:56,000 --> 00:21:59,920
And these are represented by these blocks on the screen here.

307
00:21:59,920 --> 00:22:04,680
So I won't show you all of it, but I'll just give you a bit of a flavor for it.

308
00:22:04,680 --> 00:22:09,120
So with the sample pack here, if we click on that.

309
00:22:09,120 --> 00:22:12,360
So here, people get the chance to customize the pack.

310
00:22:12,360 --> 00:22:14,600
It's a box with a tray.

311
00:22:14,600 --> 00:22:19,120
And then they can choose the color and an insert that goes inside it.

312
00:22:19,120 --> 00:22:27,120
And then they get to, I just, yeah, choose that.

313
00:22:27,120 --> 00:22:28,640
And then they get to order it.

314
00:22:28,640 --> 00:22:36,280
And it has 12 different types of condoms and three sachets of flub inside there.

315
00:22:36,280 --> 00:22:39,240
And then we've got videos as well.

316
00:22:39,240 --> 00:22:47,920
So this one is about a video, and this is of young people demoing how to put on a condom

317
00:22:47,920 --> 00:22:49,880
correctly without any errors.

318
00:22:49,880 --> 00:22:53,960
And there are various other videos as one of the things to order.

319
00:22:53,960 --> 00:22:59,000
That just gives you a bit of a flavor of the website and what it looks like.

320
00:22:59,000 --> 00:23:03,920
So I'll just go back to the presentation again.

321
00:23:03,920 --> 00:23:13,800
So in terms of data collection, the study requires participants in our feasibility randomized

322
00:23:13,800 --> 00:23:17,520
control trial to complete activities over a 12-month period.

323
00:23:17,520 --> 00:23:21,240
So they consent to the study and complete a survey.

324
00:23:21,240 --> 00:23:25,160
And then they get directed to either, well, they're randomized.

325
00:23:25,160 --> 00:23:29,240
They go into one arm, which is to receive the wrapped intervention website I've just

326
00:23:29,240 --> 00:23:33,560
shown you, or they get randomized to a control website, which has the same branding and sort

327
00:23:33,560 --> 00:23:39,400
of color and logos, but has very basic static information on condom use and sexually transmitted

328
00:23:39,400 --> 00:23:41,200
infections.

329
00:23:41,200 --> 00:23:49,040
And then they get sent another survey three months later, and also a test for an STI called

330
00:23:49,040 --> 00:23:55,320
chlamydia, which gets sent, like a test gets sent in the post to them to complete.

331
00:23:55,320 --> 00:23:59,480
And then at six months, another survey, and then at 12 months, another survey, and also

332
00:23:59,480 --> 00:24:03,680
the same test for an STI.

333
00:24:03,680 --> 00:24:07,920
So they've been asked to do quite a lot from us, and they're incentivized to complete those

334
00:24:07,920 --> 00:24:09,720
different activities.

335
00:24:09,720 --> 00:24:15,720
And we're using some database management software called RedCap to consent participants to prompt

336
00:24:15,720 --> 00:24:20,280
them to complete all those different activities that I've just told you about and to record

337
00:24:20,280 --> 00:24:24,720
their data so it sends out the surveys to them and we record their test results for

338
00:24:24,720 --> 00:24:26,720
the STIs in there as well.

339
00:24:26,720 --> 00:24:33,440
Now, RedCap used to direct participants to the two different websites, so there's an

340
00:24:33,440 --> 00:24:37,840
automated email that goes out to people after they've been randomized, which has a link

341
00:24:37,840 --> 00:24:41,160
into one of the two different websites, and they have to click on that, and then they'll

342
00:24:41,160 --> 00:24:44,720
go to each of the two different websites.

343
00:24:44,720 --> 00:24:51,680
And every activity that any participant in our study does is linked to a unique ID, so

344
00:24:51,680 --> 00:24:55,440
everything they do is logged against that unique ID.

345
00:24:55,440 --> 00:25:02,760
So we were going to be running this feasibility randomized control trial, and we knew that

346
00:25:02,760 --> 00:25:09,000
we wanted to be able to measure attrition both broadly on an aggregate level, but also

347
00:25:09,000 --> 00:25:12,920
individually, so we could look at things like demographic data and how that links to attrition,

348
00:25:12,920 --> 00:25:18,160
but then also to work out how best to measure dose for our main trial so that we can understand

349
00:25:18,160 --> 00:25:21,960
those important things about how dose relates to outcomes.

350
00:25:21,960 --> 00:25:24,160
But we really didn't know how to go about doing this.

351
00:25:24,160 --> 00:25:29,880
We wanted something more fine-grained than just the website itself could tell us, and

352
00:25:29,880 --> 00:25:36,240
I was already beginning to become interested in analytics data, and through a bit of serendipity

353
00:25:36,240 --> 00:25:41,400
came across Ronan, and we began working together on this project.

354
00:25:41,400 --> 00:25:47,520
So over to Ronan to explain a bit about how you went about this, and I'm going to advance

355
00:25:47,520 --> 00:25:50,760
the slides for you, Ronan, so just tell me, I'll go to the first one, then tell me when

356
00:25:50,760 --> 00:25:53,400
to move along.

357
00:25:53,400 --> 00:25:54,400
That's perfect.

358
00:25:54,400 --> 00:26:00,880
Thank you very much, Cathy, for preparing all this work.

359
00:26:00,880 --> 00:26:05,800
So yeah, there are many, many things to say here.

360
00:26:05,800 --> 00:26:12,840
I think the first thing to mention is that this has been, to me, really a project that's

361
00:26:12,840 --> 00:26:19,080
really motivated me a lot because it started by training, so the team of Cathy asked me

362
00:26:19,080 --> 00:26:27,240
to come in the UK and come and train our full team, so I took my backpack, I fly from France

363
00:26:27,240 --> 00:26:31,280
to the UK, and I trained our team for about three days.

364
00:26:31,280 --> 00:26:37,080
I think it was three, I don't remember if it was three or five days, but it's the kind

365
00:26:37,080 --> 00:26:43,000
of project which is really transforming you as an analyst because it's a project which

366
00:26:43,000 --> 00:26:45,440
really doesn't look like any others.

367
00:26:45,440 --> 00:26:50,080
It's not like a public website on which you can land on, and you can easily guess and

368
00:26:50,080 --> 00:26:54,200
find what are the different data collection points that you need to implement.

369
00:26:54,200 --> 00:27:00,080
Here in the case of the RAP project, the first thing I learned about is that it was using

370
00:27:00,080 --> 00:27:05,520
a PHP framework that I didn't know, which was Codignitor.

371
00:27:05,520 --> 00:27:13,960
The developer of the website is a third-party agency based in India, so it wasn't like I

372
00:27:13,960 --> 00:27:18,840
was talking to Cathy straight away, and Cathy could implement the different tracking code

373
00:27:18,840 --> 00:27:19,840
that I wanted.

374
00:27:19,840 --> 00:27:28,360
It was really like a three-party project, so the technology is named Codignitor.

375
00:27:28,360 --> 00:27:32,960
I wasn't really scared about the technology Codignitor that I didn't know, I just went

376
00:27:32,960 --> 00:27:36,840
to the famous search engine and found out that it was a PHP framework.

377
00:27:36,840 --> 00:27:41,960
From this, I knew already that it means a project that I could not put myself, my hands

378
00:27:41,960 --> 00:27:50,640
on, but that I would have to give a recommendation to a third-party company, so to set up the

379
00:27:50,640 --> 00:27:56,320
dev company, which means for me project management in terms of analytics projects.

380
00:27:56,320 --> 00:28:00,920
It means that I really needed to well structure my project in order for them to know what

381
00:28:00,920 --> 00:28:08,840
they need to implement, and I needed to ensure as well that Cathy and her team could clearly

382
00:28:08,840 --> 00:28:14,040
understand what we're going to implement, and that it aligns with everything that Cathy

383
00:28:14,040 --> 00:28:19,120
has mentioned before, which are the needs that they have in terms of data collection

384
00:28:19,120 --> 00:28:22,760
for this research project.

385
00:28:22,760 --> 00:28:27,800
The key aspect of the project, as Cathy showed to you, is a few pages.

386
00:28:27,800 --> 00:28:34,520
I think it's a maximum of between 15 to 20 pages, so it seems like that's an easy project,

387
00:28:34,520 --> 00:28:39,720
but with a lot in terms of data collection.

388
00:28:39,720 --> 00:28:46,200
You have a lot of advanced tracking code, which were necessary, including the measurement

389
00:28:46,200 --> 00:28:56,000
of events, including the use of custom dimensions to say level, visit level information data,

390
00:28:56,000 --> 00:29:04,240
and as well, we quickly saw it user ID, so the possibility to know who is the individual,

391
00:29:04,240 --> 00:29:09,360
let's say, who is making those different choices.

392
00:29:09,360 --> 00:29:15,320
We make the choices to go for a data layer, so to use Matomo Tag Manager, because the

393
00:29:15,320 --> 00:29:22,200
idea was to have something which was really consistent, because the thing is that we have

394
00:29:22,200 --> 00:29:29,520
not many pages, but all those pages are critical, and if we were going without a data layer,

395
00:29:29,520 --> 00:29:36,080
so to say, if I was taking in the project by just asking to add the container on all

396
00:29:36,080 --> 00:29:43,040
the pages, and then decide to use by either scrapping or either use automatically, let's

397
00:29:43,040 --> 00:29:49,480
say, the Tag Manager to collect the different data points, I chances that in the meantime,

398
00:29:49,480 --> 00:29:53,640
the dev company would have made changes to the DOM of the page, and everything would

399
00:29:53,640 --> 00:29:57,320
have exploded, and I would have to redo the full data collection.

400
00:29:57,320 --> 00:30:02,840
So that's the reason why we designed a really consistent data layer, which had been implemented

401
00:30:02,840 --> 00:30:08,000
directly by the dev company in order to ensure that if something breaks in terms of data

402
00:30:08,000 --> 00:30:13,320
collection, I wasn't the person responsible of screwing up the data collection, and of

403
00:30:13,320 --> 00:30:23,260
course, it gave more responsibility and involved more the dev team, which was really the right

404
00:30:23,260 --> 00:30:28,560
solution in this specific project, because I couldn't have the end on the source code.

405
00:30:28,560 --> 00:30:36,640
So those points are really critical, and this is more like a project management analytic

406
00:30:36,640 --> 00:30:46,000
system rather than, let's say, an analyst doing all the work on the platform.

407
00:30:46,000 --> 00:30:55,400
We really need, as well, to have a clear quality assessment process, because as Katie shows,

408
00:30:55,400 --> 00:31:01,160
the RAP project is a place where people are ordering, so they're not making some purchase

409
00:31:01,160 --> 00:31:09,000
because the products are for free, but they are making some orders, and once you make

410
00:31:09,000 --> 00:31:15,880
an order, you cannot order back the different elements, so it means that we didn't want

411
00:31:15,880 --> 00:31:21,840
all the time to go to Katie and say, okay, we make some order in order to test our data

412
00:31:21,840 --> 00:31:27,280
collection, please, could you remove us from the system, and we have to make another order

413
00:31:27,280 --> 00:31:32,920
again in order to test the platform, so that's why we needed to really have a clear project

414
00:31:32,920 --> 00:31:38,440
management system in which we can ensure, okay, here the tracking code has been implemented

415
00:31:38,440 --> 00:31:42,920
over here, we test it, it works, okay, so we tick in the box, it doesn't work, and then

416
00:31:42,920 --> 00:31:49,240
we go back to the dev team in order to ask for a new implementation, and the other point

417
00:31:49,240 --> 00:31:56,240
which was of critical importance was the use of custom dimension, so to say to add additional

418
00:31:56,240 --> 00:32:02,600
data which were not collected by default within Matomo to each individual, so as the use of

419
00:32:02,600 --> 00:32:09,440
the premium feature named custom report, which was one of critical importance because as

420
00:32:09,440 --> 00:32:17,800
we just saw it, the need of data is big, and it has to be crossed with different dimensions,

421
00:32:17,800 --> 00:32:23,280
so this is where the use of custom dimension was of critical importance.

422
00:32:23,280 --> 00:32:31,920
I'm done for this slide, okay, so next one is the work that we decided to work, so this

423
00:32:31,920 --> 00:32:38,280
is the project management part, this work has been reviewed by a colleague of mine that

424
00:32:38,280 --> 00:32:44,000
you know who made a conference as well named Frédéric Forster, and this is the way we

425
00:32:44,000 --> 00:32:53,040
work with the agency, so it consisted of myself drafting this document reviewed by my colleague

426
00:32:53,040 --> 00:33:00,120
Frédéric in order to have a document which is listing what the data layer is about and

427
00:33:00,120 --> 00:33:07,080
then showing this document to the team of Cathy to explain to the team of Cathy what

428
00:33:07,080 --> 00:33:10,680
this is all about because that's clearly not the kind of thing that you see within the

429
00:33:10,680 --> 00:33:15,640
Matomo's documentation, here we are more looking at the project management document made by

430
00:33:15,640 --> 00:33:23,240
an analyst and which as well serve in terms of transparency with the techie team for them

431
00:33:23,240 --> 00:33:31,480
in order to see if, okay, this is the engagement that we took together, you add the responsibility

432
00:33:31,480 --> 00:33:38,640
of deploying the code which is on the column number let's say five for you which is named

433
00:33:38,640 --> 00:33:44,240
code to push when the action is made, here as you can see you can clearly see the data

434
00:33:44,240 --> 00:33:50,520
layer which was implemented, so in our case it was for every step of the order made by

435
00:33:50,520 --> 00:33:58,800
the user we could push an event to Matomo and as I previously said we add for this tracking

436
00:33:58,800 --> 00:34:04,720
code to be consistent so that's why we decided to use a data layer and in the last column

437
00:34:04,720 --> 00:34:09,080
of this document you can clearly see that there is our recommendation which is in our

438
00:34:09,080 --> 00:34:16,960
case that this data layer wasn't implemented properly so please review on this given page

439
00:34:16,960 --> 00:34:22,160
that you have well included this given piece of code on the button when someone is pressing

440
00:34:22,160 --> 00:34:28,360
it so really here to understand how to deal with such a big project it's really like project

441
00:34:28,360 --> 00:34:33,560
management before all that that's the most that's the most important they have to precise

442
00:34:33,560 --> 00:34:39,640
as well one thing that we make some choices with with Katy and her team not to go for

443
00:34:39,640 --> 00:34:44,000
the e-commerce tracking code so just to let you know as well that when you deal with such

444
00:34:44,000 --> 00:34:50,360
a project you have some choices to make because as we saw the user can make some orders but

445
00:34:50,360 --> 00:34:56,160
those order are not paid orders so we had the choice to go either for e-commerce tracking

446
00:34:56,160 --> 00:35:00,840
code or either to go for the event tracking code and we decided to go for the event one

447
00:35:00,840 --> 00:35:07,280
because it was like more justified to go for it rather than having to create some additional

448
00:35:07,280 --> 00:35:14,360
data who were not existing within the system such as let's say the the tax amount such

449
00:35:14,360 --> 00:35:22,920
as the order price such as the transactional order that kind of thing I'm good for this

450
00:35:22,920 --> 00:35:34,920
part next one is kind of the similar thing the big difference is that on the previous

451
00:35:34,920 --> 00:35:41,960
slides that I was that we are showing it was about some variables that we are inserting

452
00:35:41,960 --> 00:35:47,040
within the data layer on this one this is another method that we use with the tag manager

453
00:35:47,040 --> 00:35:52,720
which is the push event method which consists of sending custom event to the data layer

454
00:35:52,720 --> 00:35:59,040
in order for it to react on the interaction of the of the visitor and as well this is

455
00:35:59,040 --> 00:36:10,120
how we succeed to handle it okay I'm good for this one okay and last but not least this

456
00:36:10,120 --> 00:36:17,440
slide show in fact all the different variables so those are all the different needs that

457
00:36:17,440 --> 00:36:26,080
we list with with Katy and ask for validation to see if there was any data which were missing

458
00:36:26,080 --> 00:36:33,160
out of this list and once validated of course this is this is what we implemented but here

459
00:36:33,160 --> 00:36:37,800
on small what we are showing is is really the project management part is like there

460
00:36:37,800 --> 00:36:42,400
is a lot of needs from the client but we need at some point to put a limit and say okay

461
00:36:42,400 --> 00:36:50,760
we don't need extra more data do we agree that this list is the limit and you all need

462
00:36:50,760 --> 00:36:57,720
those and if yes of course this is kind of what is finally defining the final price that's

463
00:36:57,720 --> 00:37:03,920
the analyst will put on on the quote let's say and as well yes I'm good with this slide

464
00:37:03,920 --> 00:37:11,720
I think that was the last oh yeah oh and that's as well a critical part of the of the project

465
00:37:11,720 --> 00:37:19,040
and that I really loved in it is that I during the training I showed to Katy and her team

466
00:37:19,040 --> 00:37:26,480
how to deal with the custom reports feature of matmo and I've been surprised to see that's

467
00:37:26,480 --> 00:37:32,320
when I came some days after or weeks after the implementation with the wrap project that

468
00:37:32,320 --> 00:37:38,000
a lot of custom reports were created and that's a good sign of the training works and now

469
00:37:38,000 --> 00:37:43,240
people have the solution well in end and they are creating as many custom reports as they

470
00:37:43,240 --> 00:37:51,880
can and here you can clearly see how the custom reports really enhance the data which were

471
00:37:51,880 --> 00:37:57,200
in the system of matmo because here we got the user ID with the different components

472
00:37:57,200 --> 00:38:05,880
that it shows and the date at which the individual decided to pick up the product I think I have

473
00:38:05,880 --> 00:38:13,440
one slide left yes and here is another one another report which is really showing for

474
00:38:13,440 --> 00:38:18,440
each individual what are the different components and you can clearly see that on line number

475
00:38:18,440 --> 00:38:25,880
one from line number one to line number four the individual decided to pick up all the

476
00:38:25,880 --> 00:38:31,480
components but line number five for some reason this individual decided to not pick up all

477
00:38:31,480 --> 00:38:36,880
the different components which was one of the needs which was requested from this research

478
00:38:36,880 --> 00:38:43,600
which is about okay what are the different choices that people are making and I think

479
00:38:43,600 --> 00:38:51,800
I'm good for my slides yep exactly okay thanks Ronan so in the last section before the Q&A

480
00:38:51,800 --> 00:38:57,260
I'll just present some early insights from our data collection so from the custom reports

481
00:38:57,260 --> 00:39:01,720
that Ronan was just talking about we've been able to learn a lot already we haven't finished

482
00:39:01,720 --> 00:39:07,160
collecting our data yet we're just a few months into the project so we haven't learned everything

483
00:39:07,160 --> 00:39:13,240
that we will be able to learn by the end but I'll show you what we've got so far so this

484
00:39:13,240 --> 00:39:20,200
is just how I use the data and what's working best for me is to think about the flow of

485
00:39:20,200 --> 00:39:24,640
people through the website in order to sort of see where the things going wrong within

486
00:39:24,640 --> 00:39:30,040
the website and if something needs kind of attending to before our full trial so things

487
00:39:30,040 --> 00:39:34,800
that might be leading people to drop out basically so the non-use usage attrition that I talked

488
00:39:34,800 --> 00:39:43,880
about before so 91 people at the point at which I downloaded the data from the custom

489
00:39:43,880 --> 00:39:48,680
reports had at that point been randomized to wrap so gone to that top of the two different

490
00:39:48,680 --> 00:39:56,680
arms in the randomization and of those 56 people clicked on the link so I told you about

491
00:39:56,680 --> 00:40:03,520
how our software RedCat sent an email invite to everybody inviting them to go to one of

492
00:40:03,520 --> 00:40:09,440
the two different websites so this is 91 people sent the invite to wrapped and 56 of them

493
00:40:09,440 --> 00:40:13,080
clicked on the link and went through to the website so not everybody's doing that despite

494
00:40:13,080 --> 00:40:21,960
us incentivizing them with a voucher so we've lost 35 at that point and then we when I first

495
00:40:21,960 --> 00:40:31,400
land on the website they have to create a profile and sort of register so and then they

496
00:40:31,400 --> 00:40:36,560
also complete some tailoring questions so these are the ones that determine how many

497
00:40:36,560 --> 00:40:41,880
of the components of the six components that they get so between them sort of landing on

498
00:40:41,880 --> 00:40:45,880
the website and creating a profile we've lost just two people in that stage of completing

499
00:40:45,880 --> 00:40:50,120
the tailoring questions so I was a little bit concerned I think there's 10 questions

500
00:40:50,120 --> 00:40:55,640
in total at me people might get a little irritated by that sort of section but we've only lost

501
00:40:55,640 --> 00:41:01,680
two there so that's quite a good sign for me and then we've got registration being complete

502
00:41:01,680 --> 00:41:09,840
and then between that stage and oh sorry visiting the home page we've lost nobody so that's

503
00:41:09,840 --> 00:41:13,800
another good sign that that whole stage is working well so our biggest concern really

504
00:41:13,800 --> 00:41:19,120
is that email that's encouraging people to go visit in the first place perhaps is a bit

505
00:41:19,120 --> 00:41:24,240
boring and doesn't sell the website very well although in a trial context it's kind of difficult

506
00:41:24,240 --> 00:41:27,080
to do that because we can't spell out the things that are good about the intervention

507
00:41:27,080 --> 00:41:35,720
website very easily in that context but we could perhaps try harder and then next slide

508
00:41:35,720 --> 00:41:42,880
so Ronan was talking about people being assigned different components and I mentioned how there's

509
00:41:42,880 --> 00:41:48,120
up to six different ones so we were interested in knowing about the proportion of people

510
00:41:48,120 --> 00:41:51,960
that were going to get assigned the different numbers of them because when we started out

511
00:41:51,960 --> 00:41:56,520
they said we really didn't know that so they answer these 10 questions at the start of

512
00:41:56,520 --> 00:42:01,040
the website and depending on their answers they'll get assigned either all of the six

513
00:42:01,040 --> 00:42:06,160
of them or just the one of them there's always a minimum of one and this is how it's worked

514
00:42:06,160 --> 00:42:10,360
out so no one's had just one even though that's a possibility and then very few people have

515
00:42:10,360 --> 00:42:18,720
had two three or four components most people have either been assigned five or six components

516
00:42:18,720 --> 00:42:22,040
so that's useful in terms of helping us plan for the main trial it helps us to work out

517
00:42:22,040 --> 00:42:25,520
resource need because some of the components at least where they're ordering something

518
00:42:25,520 --> 00:42:34,800
has a cost associated that's really useful and then I explained that there were three

519
00:42:34,800 --> 00:42:38,240
different things that people could order and I showed you the sample box that the website

520
00:42:38,240 --> 00:42:43,160
page relating to that so this is the box with 12 different types of condoms to try out in

521
00:42:43,160 --> 00:42:49,920
it so 54 people were assigned the sample box in total that's everyone who went to the website

522
00:42:49,920 --> 00:42:55,320
because everyone gets the sample box by default so 54 were given that and then so on that

523
00:42:55,320 --> 00:43:00,040
main page where you've got the six different boxes that you can click on or to take you

524
00:43:00,040 --> 00:43:05,200
through to the relevant page 54 people have seen the one for the sample box and 48 of

525
00:43:05,200 --> 00:43:09,080
them have actually clicked on that and gone to visit that page we've just lost six at

526
00:43:09,080 --> 00:43:15,800
that point which is not bad and then between visiting that page and then placing an order

527
00:43:15,800 --> 00:43:20,400
we've lost 11 people so that's 11 people maybe who just didn't like the look of it and just

528
00:43:20,400 --> 00:43:24,440
thought that wasn't really for them but that's an assumption but something perhaps to speak

529
00:43:24,440 --> 00:43:29,800
to people later on in the study when we come to our interviews and then those who then

530
00:43:29,800 --> 00:43:33,160
clicked to go through to the next page they're sort of progressing with making an order they've

531
00:43:33,160 --> 00:43:37,320
chosen what box they want what insert they want we've just lost one person and then we

532
00:43:37,320 --> 00:43:42,360
lost two on the next page between the point to seeing that second page and actually placing

533
00:43:42,360 --> 00:43:47,720
the order so yeah I was quite pleased that it shows to me that process of ordering is

534
00:43:47,720 --> 00:43:51,960
quite smooth and straightforward and perhaps not no one's experiencing technical problems

535
00:43:51,960 --> 00:43:58,000
or issues with usability around the ordering process and really it's the same picture with

536
00:43:58,000 --> 00:44:02,920
condom ordering so 49 people ordered that I'm not going to go through all the numbers

537
00:44:02,920 --> 00:44:07,600
on this one but it's the same pattern that we've lost just a handful at the start who

538
00:44:07,600 --> 00:44:12,760
were never visiting that page and then the actual ordering seems to be fairly straightforward

539
00:44:12,760 --> 00:44:20,960
for everybody and the same with ordering the condom carrier as well and then the three

540
00:44:20,960 --> 00:44:24,240
different types of video that people can go and see I'm just going to do one slide on

541
00:44:24,240 --> 00:44:27,800
one of the videos which is the demo demo video that I showed you it's a completely different

542
00:44:27,800 --> 00:44:35,440
picture here so 47 people allocated this video but only 16 actually clicked on that box to

543
00:44:35,440 --> 00:44:40,400
go and visit the demo page that's telling us that that home page is not selling this

544
00:44:40,400 --> 00:44:47,600
demo video at all goodbye even going to click on it well only 16 of our 47 did and then

545
00:44:47,600 --> 00:44:53,080
only of those 16 over only 7 actually then went ahead to click on the video and play

546
00:44:53,080 --> 00:44:58,440
it and then nobody watched it in full and the meantime is about watching the video was

547
00:44:58,440 --> 00:45:03,920
17 seconds so not attractive able to go and visit it when they do go and visit the page

548
00:45:03,920 --> 00:45:09,040
not playing it very much not many people playing it and then when they do disengaging quite

549
00:45:09,040 --> 00:45:14,080
quickly so we've got quite a bit of work to do on the video and the demo video but the

550
00:45:14,080 --> 00:45:21,320
same picture is true of the other types of videos as well they all need looking at so

551
00:45:21,320 --> 00:45:26,320
what we've learnt well we've learnt that we can use analytics data to gain some important

552
00:45:26,320 --> 00:45:30,680
insights into non usage attrition and we've just started to kind of touch the surface

553
00:45:30,680 --> 00:45:34,520
with the data we've got already and we're going to use this to make some improvements

554
00:45:34,520 --> 00:45:41,400
so my conclusions I've kind of touched on touched upon already are that the home page

555
00:45:41,400 --> 00:45:45,480
as attracting users to visit are the first three components the things that you can order

556
00:45:45,480 --> 00:45:50,680
and once on those component pages there's good conversion to ordering there seems to

557
00:45:50,680 --> 00:45:55,400
be no issues with the ordering process but the home page is not attracting people to

558
00:45:55,400 --> 00:46:03,000
visit the video content and those video component pages are also not working very well to encourage

559
00:46:03,000 --> 00:46:05,920
users to watch the videos when they're already there and then the videos aren't being watched

560
00:46:05,920 --> 00:46:11,400
for very long so the first few seconds are encouraging people to carry on and we still

561
00:46:11,400 --> 00:46:15,960
have lots and lots to learn about this field and what is really possible in terms of analytics

562
00:46:15,960 --> 00:46:28,080
so I'm very much a newbie in this situation and our solution for linking individual level

563
00:46:28,080 --> 00:46:36,960
the user ID that we're collecting in redcap for all our surveys and STI testing and linking

564
00:46:36,960 --> 00:46:41,520
that to our individuals as they go through to the two different websites so we can collect

565
00:46:41,520 --> 00:46:47,240
this analytics data that process that Ronan worked out for us by liaising with us and

566
00:46:47,240 --> 00:46:51,680
also our development companies has worked really well has been successful had no problems

567
00:46:51,680 --> 00:46:57,480
with gathering the unique user ID of everyone in Amatomo and seeing what they're all doing

568
00:46:57,480 --> 00:47:01,240
creating these custom reports and gathering all that data from there that's all worked

569
00:47:01,240 --> 00:47:05,120
really well and that means it's going to be possible for us to measure those in our main

570
00:47:05,120 --> 00:47:10,800
trial and in quite a fine-grained way which is really useful and we're just beginning

571
00:47:10,800 --> 00:47:15,520
to think about how we'll calculate a sort of standardized score for everybody in our

572
00:47:15,520 --> 00:47:20,760
trial at an individual level based on the products that are allocated and ordered and

573
00:47:20,760 --> 00:47:30,280
also which videos to watch and how long for so that's the end of the presentation so I'd

574
00:47:30,280 --> 00:47:34,160
just like to acknowledge our funder the National Institute for Health Research and all of the

575
00:47:34,160 --> 00:47:38,920
team working on this with me as well at the University of Hertfordshire and others and

576
00:47:38,920 --> 00:47:47,840
yeah on to any questions thank you very much Katie thank you very much for this great presentation

577
00:47:47,840 --> 00:47:55,240
I'm looking at the chat right now we have approximately let's say maximum five minutes

578
00:47:55,240 --> 00:48:00,440
because after there's a other talker at 4 p.m. I'm just going to pick the first one which

579
00:48:00,440 --> 00:48:08,200
is which was the most useful metamode tools used during this project?

580
00:48:08,200 --> 00:48:15,040
Rony you might have a view on this but for me I suppose I might not be using the right

581
00:48:15,040 --> 00:48:22,400
terms but it was the custom ID and then also the reports as well I mean the reports are

582
00:48:22,400 --> 00:48:28,160
something I go into all the time now to look at to gather the data to help me understand

583
00:48:28,160 --> 00:48:33,240
the usability of the website and on all the slides I've just presented with those flowcharts

584
00:48:33,240 --> 00:48:38,280
all comes from those custom reports.

585
00:48:38,280 --> 00:48:43,240
I'm looking at the chat there's another question with small linking with logistic which is

586
00:48:43,240 --> 00:48:48,760
about will it be possible to download the presentation I mean will it be possible for

587
00:48:48,760 --> 00:48:54,640
you to send it to I mean do you allow me to share it with the audience or would you like

588
00:48:54,640 --> 00:48:58,680
to keep your presentation for yourself or can it be?

589
00:48:58,680 --> 00:49:05,080
That's fine Rony yeah I'm happy for anyone to have it so I'll send it to you.

590
00:49:05,080 --> 00:49:11,440
Yeah well I think I have it in my mailbox so I will just make it as a PDF and then and

591
00:49:11,440 --> 00:49:15,400
then share it back.

592
00:49:15,400 --> 00:49:23,320
I let the audience write down the last question we still have let's say four minutes left.

593
00:49:23,320 --> 00:49:28,160
Do you have for example any questions Cathy that you would like me to ask you for the

594
00:49:28,160 --> 00:49:35,400
audience or do you have any questions for you?

595
00:49:35,400 --> 00:49:39,720
I mean do you have any questions that you would have expected the audience to ask you

596
00:49:39,720 --> 00:49:43,840
and that you would like me to ask you?

597
00:49:43,840 --> 00:49:49,320
Do I have any questions?

598
00:49:49,320 --> 00:49:58,560
I suppose the thing that I'm still struggling with the most is linking all the data in the

599
00:49:58,560 --> 00:50:06,280
custom report so each custom report I can download the data as an excel file so for

600
00:50:06,280 --> 00:50:09,880
example if we're talking about the components allocated I can download it so I've got one

601
00:50:09,880 --> 00:50:14,920
column which is the user ID and then you know did they have component one yes no component

602
00:50:14,920 --> 00:50:20,320
two yes no I can create it like that then I can download another separate report that

603
00:50:20,320 --> 00:50:27,440
says did they visit the sample pack page for example and that'll be a completely separate

604
00:50:27,440 --> 00:50:33,440
one and so at the moment they're all very separate like that and then I have to combine

605
00:50:33,440 --> 00:50:43,840
them all sort of together so I suppose oh and also every time I want to update the data

606
00:50:43,840 --> 00:50:49,880
it's not easy I can use date I suppose but I can't just filter by everything since I

607
00:50:49,880 --> 00:50:53,880
last looked and then download that and add it in I haven't just kept kind of keep manually

608
00:50:53,880 --> 00:51:00,480
looking and then updating my excel data based on that so I don't know whether it's possible

609
00:51:00,480 --> 00:51:05,080
this is more a question for you than the audience but this is something that I'm I'm struggling

610
00:51:05,080 --> 00:51:12,000
most with at the moment and yeah I'd be keen to know about if there's a way to kind of

611
00:51:12,000 --> 00:51:17,280
make that process a bit more straightforward I suppose but might be getting to the limits

612
00:51:17,280 --> 00:51:25,160
of what Matomo can provide okay great yeah I don't have the yet the answer I think that

613
00:51:25,160 --> 00:51:32,760
we are looking for custom reports with far more dimensions I guess that's the key thing

614
00:51:32,760 --> 00:51:39,880
here the maximum is really three custom dimension and from what I remembered yeah every time

615
00:51:39,880 --> 00:51:46,040
you asked something I did my best in terms of optimizations you can yeah you can just

616
00:51:46,040 --> 00:51:51,480
put up to three custom dimension and then filter by some other custom dimension but

617
00:51:51,480 --> 00:51:56,240
at the moment I'm not sure that there's plans to extend the number of custom dimension which

618
00:51:56,240 --> 00:52:01,320
correspond more to what you would like like you extend if you export the full excel file

619
00:52:01,320 --> 00:52:05,000
and you will get as many custom dimension as you need so in our case I don't remember

620
00:52:05,000 --> 00:52:13,880
if we have up to five or ten but then you can refilter back in excel Alfonso says thank

621
00:52:13,880 --> 00:52:19,800
you because you answered his question about which was the most useful Matomo tools and

622
00:52:19,800 --> 00:52:28,320
we don't have any questions left so I guess that's the time to close the conference talk

623
00:52:28,320 --> 00:52:34,840
once more thank you very much Katy for your time thank you very much for the presentation

624
00:52:34,840 --> 00:52:39,640
and as well I would like to thank all the attendees of this conference and I wish you

625
00:52:39,640 --> 00:52:46,840
all a great end of Matomo camp there's just one conference two conferences left starting

626
00:52:46,840 --> 00:52:54,200
in five minutes and then we have the closing ceremony at in one hour so at five p.m. thank

627
00:52:54,200 --> 00:53:21,200
you very much pleasure good luck with the rest of the conference thanks bye bye