hello everyone
nice to meet you all again it's
currently a three or tooth and we
will have conducted a conference today so
i continue by with here weaver with
me and may sulfur one i shall
dunno and so the idea of this
conference is to do some sandwiches are
kind of unusual
so tommy's oh so we're gonna do
a use case and most of the
time when a use cases presented by
an analyst it's most of the time
like three quarters of the time the
analyst was picking and then a quarter
the timing is the the client here
is going to be the reverse a
mod so i will leave the floating
up in a couple of minutes
tour two kaddish who is a client
of mine and the idea that kathy
can i introduce you to projects that
she is walking on and i will
explain how i decided to deal with
the project management of including matter
all women deserve this given project and
so i leave now the floor to
to get here and i will interact
on leash or dealing with my part
of the site
okay thanks training artist stem
charmless lights
can i just chat you can see
that ronan
yes it looks great
lovely thank you and write so myself
and raining again to attitude share the
presentation today and the presentation is on
using matter mighty collect data on intervention
engagement within a research trial and the
case study isn't a project called wrapped
so the overview of the presentation is
the i'll firstly talk about why i
think it's important to measure intervention engagement
within a research trial i will tell
you a bit about the wrapped research
project to swell and then ronan's going
to talk about how we use matinee
within the project to collect endeavor
at your level data on engagement than
our present somalia insights from our analytics
data and and not be a chance
for some questions at the end
so festival why it's important to measure
intervention engagement with a new research trial
with arab are two reasons one is
to minimize something called non usage attrition
and are going to explain about our
protests and then also to examine and
control for intervention tos
so firstly onto minimizing non-eu such attrition
so when we are testing whether a
new intervention works then we typically decent
could a randomized control trial and many
of you will be aware of what
randomized controlled trials are already maybe but
they are am i to share my
next slide
so they are used all the time
in a clinical context to test the
effectiveness of something so that something could
be a new drug or a vaccine
as is the case with the kv
vaccine or in my case am an
imam house psychologists i develop interventions that
try to encourage people to change their
health behavior
that could be for example to to
persuade people to use the kite that
vaccine so the intervention itself and is
not a drug or a vaccine or
something like that it tends to be
an a behavior change intervention but the
principle of attesting it is exactly the
same so we would also run a
randomized controlled trial
and in randomized controlled trials we take
hair sample population so this is a
smaller group of the the white population
were interested in testing the intervention on
and will randomly assign them to two
groups so am i the flip of
a coin old bolla di saw more
commonly using sort of an internet based
randomizer
ocean tools but the idea is the
same it's just randomly assigning to one
group or the other on n and
the one groups with which have habits
great one at the old people are
not great that reminded me assigned to
it will get thief then the new
drug code or treatments and then we'll
measure afterwards and how they fare to
what the outcomes were for them
and then the other great thou get
to control so if it is a
drag it might be something like a
sugar pill that has no active ingredients
and game will mash at the outcome
for them an important point with randomisation
is that by the end we should
have balanced the groups or just by
chance we should have approximately even numbers
of people of different genders and ethnicities
or
all with similar levels of of healthy
your health condition and that's important to
make sure that we control finest of
confounding factors that made my also have
an effect on the outcome they should
then be evenly spread between the two
groups
so in a clinical context when a
new drug is being tested adherence to
the drug is not usually a big
problem although something do to me to
be mindful of 'em and this spirit
the turkey is prescribed by a clinician
and participants saw usually very closely supervised
so you'd be asked to go
go into a clinical context for example
a newbie pass that the pale on
a tray with a cup of water
i am watched by a research nurse
taking it who then recalled that you
taken it and that would likely be
for every dose and also as a
research participant he may well especially if
you are in the intervention condition experience
observable and immediate health
benefits of taking that drug ashamed that
it's effective and also potentially in the
control condition due to placebo effect so
there's lots of things that could be
and meaning that haven't that that drug
with an intervention and and trial context
am is not too much your problem
it's usually irritates usually reasonably high
however in contrast adherents is a major
problem why may testing ie health interventions
as i mentioned briefly about the kind
of what the itu sought to test
health behavior change interventions and also almost
all of the interventions that i developed
contest of the health type said their
intervention delivered by
via the internet so here the intervention
is neither prescribed nor critical to wellbeing
and this graph i think demonstrates quite
nicely and typical attrition from a health
interventions so on the y axis going
up the left you can see the
very top that a one and that
reflects one hundred per cent of users
of the intervention going down to nine
eighty eighty seventy etc or right down
to the air at the bottom and
then across the bottom on the x
axes we've got time and modules so
if we take the blue line is
at to start with the city by
favell done and this was a i
am a twelve week program of an
internet
and delivery of an intervention so exposed
to engage for twelve weeks in total
to have had the full dose of
the intervention but you can see after
a couple of weeks before he got
less than fifty percents of people i'm
still using that that am website going
down and to less than twenty percents
at week three and you can see
by the end by the law
less than one percent people still using
or still completing the intervention and right
at the very end and then four
and the dashed black line this was
an arm and a bench good mood
gym and it's in a trial context
again and there were five modules half
this intervention you can see that and
lessons have to present day
module two and module three less than
sixty percent and so on down to
just over twenty percent doing all five
modules and then the solid black line
is also moot gym but in this
context it wasn't in a trial and
it was just people allow to have
access to it then see who in
engage with that and carried on engaging
with it
and so outside the context of a
trial in which people might be incentivized
to to use the intervention and also
encouraged and reminded by researchers to do
so you can see the engagement is
even less
so non-use patricians a participant ceasing to
use the intervention in a trial context
and
it's obviously a big is a big
problem from a for us and m
if pockets are not using intervention in
trial contacts it makes it really difficult
task where the intervention actually works or
not
so in in trials and regardless of
i'm sorry to interest outcome data for
every participant is randomized included in the
analysis so if we have people in
not to be remembered at first
siakam i showed you have a randomized
controlled trials we've got people are under
my two creeps in and not top
arm we've got people who received the
intervention itself if i only got a
small portion of those who are actually
using intervention of fall you might be
thinking to yourself well just analyze data
from those people who use the intervention
itself and ignore it when asked who
do
patent but that in threatens the whole
integrity of the trial design which is
the randomisation of people to one group
or the other i explained earlier about
how we randomize so we have pilots
between the two arms but if you're
only including a a a sub group
of those people who have used the
intervention to a minimum extent and ignoring
everybody gauss
then you remove that balance which is
so important to that trial validity
so get the point of and of
ensuring that everybody regardless is what they
use the intervention or not in a
trial it's called intention to treat analysis
so if you randomize someone you will
always analyze that data regardless to ensure
we have got this balance in design
the problem of course is if you've
got a group of people in your
intervention condition that haven't used the intervention
at all or not very much then
if your intervention is indeed effective then
when you do the analysis a hussy
fact underestimated the effect the intervention cajoled
including in the analysis in the intervention
condition people who didn't even use it
at all or didn't use it very
much
such tool and remember that that diagram
of the dropping off steeply that is
often the case for us so it's
really really important the in in a
trial context that we do absolutely everything
that we can to minimize people not
using the intervention or dropping off
now ahead of doing a main trial
in which we test the effectiveness of
the kind of intervention that i've been
talking about we tend to do something
called earth a feasibility randomised controlled trial
which is exactly the same as a
randomized controlled trial with this randomisation to
the two groups the only thing that's
different is it's with much fewer parties
pence and the purpose of that kind
of study is to help us to
prepare for a full randomized controlled trial
and what we trying to do is
our resolve will remove all the unknowns
so for example helps to learn what
the best method for recruiting participants is
and what level of incentive we need
to recruit people in the first place
but water to keep them in the
trial until the end so those kind
of things are important for us to
establish this early phase before we do
the main thing
but what we could also use the
sp feasibility randomised controlled trials these preparation
preparation type trials is to examine non
use at attrition which we been talking
about so if we look at tricia
and in this early stage trial it
can tell us some important things such
as whether atr
ration is associated with particular demographic factors
and of course when we measuring those
in the surveys that we give to
people before they have the intervention and
then also afterwards and so if we
can link that demographic data right with
the day trip out attrition we can
learn some important things so if we
found out that for example females in
particular were using our intervention or
dropping off then it be important too
and maybe interview them as pot some
later studies and to find out why
that may might be the case and
try and rectify that and when we
look at the shape of the attrition
curve which we were we were just
looking at before that can tell us
something also important
so if we hearts i should have
kept my
mouse to do that yeah so if
we had a shape a bit set
alight rarely steep down like that that
would be a really bad sign for
our intervention tell us that people were
disengaging very quickly because i just didn't
like it and then this more can
have sigmoid style curve would tell us
that we have
add some sort of initial interest in
it which then dropped off as you
might imagine then a bit of a
steep drop up in the metal parts
when there's less motivated people were leaving
or possibly representing some of usability style
problems and then we've got hard core
subset of users at the bottom so
that the style of and and shape
space
bows of this curve can tell us
some important things as well if we're
measuring attrition from our intervention jury not
a feasibility randomised controlled trial
so it is not uncommon for a
feasibility randomised controlled trials of a health
interventions to measure non use of nutrition
but measures of engagement are usually quite
limited and not so fine grained so
often it it's abouts
when people registering within the website itself
so we know who's visiting and we
know when they stopped visiting we might
talk about collect data on whether they've
completed modules within that website itself without
tends to be the limitation or the
limits of it and on the use
of analytics data rarely happens at all
but i've
she had become increasingly aware recently the
analytics data can offer much greater insight
into usability issues for example are the
particular point steering the and the different
stages in which people are going through
the intervention why they're experiencing some technical
issues that just is not very nice
the look and feel over time people
are dropping off
having problems with it and these are
the kind of things that can be
rectified which we don't cut this kind
of data we won't know nothing about
and and typically within feet split them
into mice control trials we tend to
have a stage of qualitative research we
do interviews with our participants to find
out more about their experience begin to
be such participants and what we can
learn to improve the trial
and this would be a great stage
at which to ask people to which
we can draw hypotheses from these usability
issues and then envelop the further through
those interviews to find out more about
people's experiences those two things with dr
really nicely together
and so yeah we we can do
these feasibility randomized controlled trials and examine
usability issues with a with a view
to properly preparing a much better armor
and resolving any of these issues in
preparation for the main trial and i
just want to make the point that
i'm very aware that within
that and the tax at web industry
sector that usability and examining the acts
of client or customer's use of of
website and all these friction points and
things are going wrong in order to
maximize engagement and getting people through to
the end point where that might be
like purchasing something is done really
commonly and very well but really in
my field it's not it's rarely done
at all not by much toll and
i just know this a huge amount
to be learned and gained from i'm
collaborating with people who know much about
this not be really interested in hearing
from somebody who would be interested in
working with me and that kind of
way
and so yeah i've hopefully sort of
up it made the case for why
it's important to look at and non
use at attrition and for minimizing not
in a trial we know it's important
to do that and running these types
of feasibility randomised controlled trials which are
really common as a great opportunity to
look
attrition and try and remedy at inducing
about it and i'm rarely and if
one else in my field just at
the beginning of trying to understand how
best to measure this type of nutrition
twins damn what's all the mashers and
things that we can collect telling us
and also to make the right sort
of changes so that in the main
trial we can actually have as high
of attrition
the attrition as possible when people don't
like the intervention that's another thing we
can't that's a different question to be
answered by bit to do with how
the website itself is operating i think
there's lots we can do about that
and on the the other issue an
and you've got to slice on this
one this much so small a point
rarely but equally important is around examining
and controlling for dos and
so the amount or dose of an
intervention that a participant receives in a
trial can affect the strength that the
outcome so again back to that randomisation
that we talked about those in the
the top arm he received an intervention
so we look to see what the
outcomes are so if it's an stop
smoking intervention for example
will will measure number of cigarettes per
day the heart before the intervention then
afterwards nam will compare the two arms
to look to look at the number
of cigarettes smoked so we we need
to know the the dose of that
intervention soap it's a web invent should
be stop smoking we need to know
the amount of consumption of that website
have they gone
backlot every day if they reviewed after
then ordered everything i can that sort
of thing to know the dos and
that way we can control for the
in the analyses so we can have
and can just isolate the effect of
the intervention and not the amount that
happening but then also we can measure
the correlation of the strength of the
relationship between dose and outcome as well
knows kind of things are really important
for us to
understand but you can examine what you
don't measure so we really need to
understand what is the dose the amount
that people are having of our hr
interventions and and this has to do
with the main trial itself
so if we can measure dose and
and for this it's really important that
the outcomes are linked to each individual
participants in the study so if we
can measure dose then we can understand
things such as is there a linear
relationship between dose and outcome so in
other words the moth
the intervention the somebody has the greater
the benefit or is it there are
anonymous linear relationships at the point of
saturation at which that really make the
difference whether you have any more or
not or is the minimum dose everyone
needs to have in order for an
effect to be achieved and an analytics
data can also be used for this
and epiphytes with
the precise and detailed mashes of the
dose rates individual such as which pages
are accessed how long people spend on
each page which videos have been watched
and for how long
so am i now like to go
on selling a little bit about a
intervention the i developed called wrapped when
i say i not may alone me
with a fantastic team of other researchers
so am face a little bit about
the study we're doing so we're at
the stage of doing the feasibility randomized
controlled trials of the preparatory study before
the main trial on this aims to
establish if the main randomized controlled trial
is feasible and to inform our preparations
and we are using analytics data
her to better understand any possible usability
issues so that they can be addressed
ahead of the trial so to minimize
the non usage attrition that i talked
about also to learn how best to
measure dose in preparation for our main
randomized controlled trial
and so the rapt intervention itself i'm
going to show you in a minute
what the website looks like but just
a little bit of information about it
so it aims to increase condom use
by addressing factors such as people's attitudes
towards condoms what they feel positively or
negatively about them and the beliefs that
underlie the
peoples south africa seafo communicating about condoms
and niue so this is about and
body speaking how confident you feel to
raise condom use with a partner for
the first time and also how confident
you feel in and using them correctly
and it's also about increasing access to
condoms so about
making them more available to people through
and a condom distribution scheme and there
are six components in total which are
tailored to individual need so at the
very start of access the website we
ask our users to answer very quick
yesterday questions about their main barriers to
condom use and then
get allocated between one and six of
the components so the infection itself hopefully
is quite closely aligned to that their
needs and interests
and there are three products which can
be ordered a trial pack of condoms
a condom carrier and access to a
service which supplies condoms on a monthly
basis and there are also three lots
of videos so i'm just going to
change my screen
to show you the
the website itself
and ceremony to shower if you can't
see this otherwise i'll let you to
she and everybody can so this is
the route websites and after vance those
few quick questions to start this is
what you will see and this person
here has been assigned all six intervention
components and these represented by these blocks
on the screen here
so am i wait and show you
all a bit but i should give
you a better flavor for it so
with the sample pack here we can
combat
so here people got the chance to
customize the pack it's m a box
with a tray and then they can
choose the color on an insult that
goes inside it and then they get
to buy just yet she thought
and then they get to order it
and it has am twelve different types
of condoms and three sachets of lube
inside though
and then we've got videos as well
so this one is about video video
analysis and of young people demoing how
to put on a condom correctly without
any errors and that there is both
of the videos is one of the
things to order but that just gives
you a bit of a flavor
oh from the website and what what
it looks like
so just go back to the presentation
again
and sometimes data collection the study requires
participants in of a feasibility randomized controlled
trial to complete activities over a twelve
month period so they consent to study
and compete a survey and then they
get directed to either with a randomized
that go into and one on which
is to receive the wrapped intent
action website i've just shown you where
they get randomized to a control website
which has the same branding as of
color and logos but has very basic
static information on condom use and as
and sexual transmitted infections and then they
get sent another survey three three months
later and also a test for and
st eichel chlamydia which get sent my
contest gets get sent in the post
to them to complete and then at
six months another survey and then at
twelve months another survey and also the
same test for an std i said
that they've been asked quite to do
quite a lot from us and they're
incentivized to complete those different activists
it is and we using some database
management software called rep right cap to
consent participants and to prompt them to
complete all those different activities i've just
told you bout and record their data
so it sends out the surveys to
them and we record their test results
for the st eyes in there as
well
now but copies used to direct participants
to the two different websites so its
its and that an automated email that
goes out to people after been randomized
which has a link into one of
the two different websites and they have
to click on that and then they'll
go to each the two different websites
and and every activity the any part
has been in that he does is
linked to a unique id ams evident
they do is is locked against that
unique id
so we knew we were going to
be running this feasibility randomized controlled trial
and we knew that we wanted to
be able to measure attrition and both
so broadly on an aggregate level so
individually typical look at things like demographic
data and how the links to attrition
then also to work out how best
to measure dose for
the main trial so that we can
understand that important things about how dose
relates to outcomes but we really didn't
know how to go about doing this
we wanted to make more fine grained
than just and the website itself could
tell us and i was already beginning
to become interested in analytics data and
am through a bit of serendipity
came across a ronan and we began
working together on this project and so
over to ronan to explain a bit
about sam how he went about this
and i'm going to advance the slides
be ronan such as tommy i'll go
to the first one then tommy went
and move along
that looks perfect the central room mature
guilty for preparing all lisa all this
walk i saw ye are many many
things to to sir
i think the first thing to mention
is that
this is thus been to me where
do your projects that's really motivated me
a lot because it started by your
training saw the team of coach he
has me to come in the uk
and come and train from the gym
so i took my backpack off a
flight from france throughout the uk and
i try not to
for about three days it was sweet
i don't come over to israel or
five days but it's swayed the kind
of project which is where they are
transforming you are as a nationalism because
it's a project which really doesn't look
like any others it's not like a
public website on which you can download
and you can easily guess
and fine with all the different that
i collect from the point that you
need to implement healing the case of
the rub project the first thing i
learn about is that it was using
your appear to be from on but
they didn't know which was a coded
meter the developer of the website or
so british
urgency based in india also it wasn't
er like i was talking to to
courteous tried to work and get equal
input among the different a and cut
that out that they wanted it was
weird ica the three portage project so
the technology is named coordinator and i
wasn't very scared about that
technology coordinator they did annoy just weren't
on the phone with such a design
and find out it was appear to
be framework from this a new era
did it means a project that i
cannot put my certified and on but
though would love to give a recommendation
to a third party companies to setup
a project the litter
the dev company which means for me
project management in terms of analytics projects
it means that they really need it
where the structure my project in order
for them to know what the need
to implement and i needed to ensure
as well that gets she and her
team could clearly see understand what we're
going to
employment and that it will line with
everything that getty as mentioned before which
on the the need that the delve
into of data collection for this research
project so the key aspect of the
project as getty assure to you a
few pages it's i think maximum between
fifteen to twenty
pages so it seemed like oh that's
an easy project but with a lot
in terms of that collection so you
have a lot of edwin striking coverage
for illnesses series including the measurement of
evans including the use of custom dimensions
to save lives
will visit level
information lit up and do as well
we quickly saw it or use irish
so the possibility to know who is
the individual letter was making those different
choices
we make the choices to go forget
that tally also two jews met and
we're talking mature because the irish was
to have something which was really consistent
cause the thing is that we have
luqman not many pages but all those
pages are critical and if we were
going without death earlier
so to say if i was thinking
and then the project by just asking
too
the container on all the page and
then decide to use by itself scrapping
or either use automatically and the lets
her do the tag manager to collect
the different a point i chances that
in the meantime the death company will
make changes to the dome of the
page and everything would have exploded and
they
would love to redo the full data
collection so that was the reason why
we design only consistent that a year
which have been implemented directly by the
deaf company in order to ensure that
if something breaks and download data collection
i was and the person responsible of
a screengrab the data collection and of
course it gave
more responsibility and involve more of the
the
the the deft him a which was
really the the right solution in this
specific project because i kind of have
the m on the source code so
those points are really critical and this
is really where we unit this is
more like project management and analytics system
brazil then led to another
at doing read all the walking in
on the platform and we really need
as well to have a clear quality
assessment process because as katyushas it's the
rap project is a place where people
are
all the rings was entered making some
poor choice because the the product out
for free but they are making us
some orders and it's ones you make
up and although you cannot order back
the different elements so it means that
we didn't want all the time to
go to catchy and silky
we make some order in order to
test or data collection please could you
remove us from the system and we
have to to make another order again
in order to test the platform so
that's why we we need it when
you have a clear project management system
in which we can ensure okay here
the tracking code has been implemented over
here we tested it
walks forecasts with ticking the box ways
he doesn't walk and then we go
back to the deaf team in order
to ask for a new implementation
and the other point which was of
critical importance was the use of custom
dimension so to set two other and
additional data which were not collected by
default within maximo to each individual so
as the use of the premium feature
in custom report which was one of
critical input
buttons because as we just saw it's
the need of data is is big
a task to be crossed with different
dimensions so this is where the use
of custom dimensions was of critical importance
and i'm okay for them then for
the slide okay so
the next one is the way that
we decided to to walk so this
is the project management but this walk
has been reviewed by a colleague of
mine that that you know who made
the conference as well named for the
lifestyle and this is the way we
walk with with the urgency sweet consisted
of me
myself drafting this document reviewed by you
may critique frederick in order to have
a document which is listing what the
data leader is about and then showing
this document to the team of catchy
and to explain to the gym of
katy what what this is all about
because that's clearly not the kind of
thing that you see within the matter
was the recommendation here we are more
looking at the project management document made
by an analyst and which as well
itself in terms of transparency with the
techy tema for them in order to
see if okay this is the engagement
that we took together you add the
responsibilities of deploying the code which is
on the column number of let's say
fifty five for you which his name
cut to bushman direction is made here
as you can see you can clearly
see the the data layer which was
implemented so in our case it was
for every step
btw of the older made by the
user a wicked person ever to matter
more and as i previously said we
add for this tracking cut to be
consistent so that's why we decided to
use up that idea and in the
last coloring of this document you can
clearly see that says our recommendation
which is in our case that did
that other yale wasn't implemented properly so
please review on this given page that
you have when including this given piece
of code on the button when someone
is pressing it so we're here to
understand how to deal with such a
big project it's where they like project
manage
she meant before all that that's the
master that's the most important they have
to precise as well one thing that
we make some choices we've we've got
she and asked him not to go
for the e-commerce tracking could suggest to
let you know as well that when
you deal with such a project you
have some choices to make because as
we saw you
user can make some orders but does
calder and are not paid orders so
we had the choice to go is
up for e-commerce tracking code either to
go for the event tracking code and
we decided to go for the event
one because it was like more justified
too afraid rozelle done a thing to
create some additional data will not exist
anger within the system such as that
said that the tax amount such as
the or the price such as the
transactional although that canvassing
and i'm good for the spot
the next one is i kind of
do similar seeing the big difference is
that under the previous
i slice a tie was that we
were showing it was about sam
valuables that we are inserting within the
day on this one this is another
method that we use we the tag
manager which is the push event method
which consist of sending custom event to
the dollar yearly know they'll fight to
react on the interaction of the of
the visitor and as well as his
hobby
sixty to two hundred
ok i'm good for this one
okay and last but not least up
this a slide show in fact ordered
a different valuable so those are all
the different needs that we at least
with with catfish and ask for a
validation to see if there was any
data which we are missing
out of this list and on some
very dated of course this is this
is what we implemented but he on
small what we are showing is is
read project management parties that says a
lot of needs from the client but
we need at some point to to
put a limit and say okay we
don't need extra more data do we
agree
read that this list is the limit
and you only need dos and if
yes of course this is kind of
what is finally defining the the final
price that's the analysis will put an
undercooked meats and and as well yes
i'm good with disliked i think that
was the last oh yeah oh
and that's as well a critical path
to offer the of the the project
and that i really loved in it
is that said during the training i
showed to kaddish and asked him how
to deal with the custom reports feature
of mathml and i have been surprised
to see it it's
when i came some days after or
weeks after implementation with the rap project
that a lot of custom reports were
created and that's a good sign of
the training walks and now people have
the solution wedding then then they are
creating as many custom reporters as they
can and here you can clearly see
hold the
custom reports really immense the the data
which were in the system of mathml
because here we got to use the
id with the different components that
he chews and the date at which
the individual decided to pick up the
product i think i have one state
left
yes and here is another one in
those a report which is really showing
for each individual what are the different
components and you can clearly see that
underlying number one and from london but
want to line number for the individual
decided to pick up all the components
but line number five false
some reason this individual decided to not
pick up all the different components which
was one of the needs which was
requested from this research which is about
okay what are the different choices that
people are making
and a sink can good for my
slides yep executive okay thanks rain so
and in the last section for that
q and a i just present some
early insights from data collection site from
the customer reports that biden was just
talking about with financial analyst already we
haven't finished cutting out eighty
cats and just a few months into
the project say we haven't let evidently
we will be able to land by
the end pop i'll show you what
we've got so far
say this just how i use the
dayton and what swanky past amazed to
think bouts and the flow of people
through the website in order to sort
of see why the thanksgiving wrong and
then the website and sending these kind
of attending tape before of awful trials
i think that might be at least
eighty people to drop out ice
pretty sad that the non used usage
attrition i talked about before so i'm
ninety one people at the point at
which i downloaded the data from the
customer reports at that point be randomized
to wrap second to top the two
different arms and the randomisation and i'm
of those fifty six people clicked on
the links i told you about how
our software read captain
send an invite to everybody and inviting
him to go to one or two
different websites say this is ninety one
people sent the invite to wrapped and
fifty six them clicked on the link
i went through to the website and
everybody still not despite us incentivizing them
with a voucher
and so it lost thirty five at
that point and then we when they
first land on the website they have
to create a profile and i'm register
so i'm
and then they also complete some tailoring
question se the ones that determine how
many of the components of the six
components that they get saved between and
then landing on the website and creating
a profile we've lost just two people
in that stage of competing the tailoring
question so i was a little bit
concerned that this ten questions in title
it might get a little irritated by
that sort of section but and if
i lost to their side that's quite
good sign for me
then we've got registration and being complete
and in between that stage and oops
sorry and visiting the home page we've
last night but a site that's another
good sign that that how the host
ages working while sabic is concerned rarely
is peep the email that is encouraging
people to go visit in the first
place pups is
a bit boring and and doesn't sell
the website very well and although natal
contents kind of difficult to do that
because we can't spell out the things
awkward about the intervention website and very
easily in that context but we could
perhaps try harder
and the next slide so i'm ronan
was talking about people been assigned different
components and and i mentioned how this
up to six different ones so we
were interested in knowing about the portion
of people that were going to get
assigned def different numbers of them because
when we started out the study we
really
didn't didn't know that so they asked
these questions ten questions at the start
the website and depending on their answers
doug assigned either all the six of
them or just the one of them
is with a minimum of one and
and this is how it's worked out
so no one's had just one even
though that's a possibility and then very
few people have had two three or
four components
most people have either been assigned five
or six components and so that's useful
in terms of helping us plan for
the main trial that helps us to
work out resource need cause some of
the components lately while you're there ordering
something has a cost associated with that
sounds really useful
and and then i explain that there
were three different things that people could
ordered and i showed you the sample
box that the website page relating to
that so this is the box with
twelve different types of condoms to try
out in it so fifty four people
were assigned the sample box in total
that's every
one he went to the website cause
everyone gets sample box by default so
fifty four were given that and then
so on that main page where you've
got the six different boxes that you
can click on or take you through
to the relevant page fifty four people
have seen the one for the sampler
box and forty eight of them have
actually clicked on that and gone to
visit that page we've just lost six
that
point which is not bad and then
earn between visting not paid and then
placing an order with last eleven people
that's eleven people may he just didn't
like the look of it and just
thought that wasn't really for them but
that's an assumption but something pops to
speak to people later in the study
and we've come to our interviews and
then and
those who then click to go through
to the next page thus progressing making
an order chosen what box they want
more insight they want which has lost
one person and then we lost two
on the next page between the point
to and say not second page naturally
placing the order so and yeah was
quite pleased that i chose sway that
process of ordering is quite smooth and
straightforward
perhaps not no one's experiencing technical problems
or issues with usability around the ordering
process
unreal it's the same pitcher with condom
ordering say forty nine people or that
are not going to ensure that the
numbers on this one base the same
pattern that we've lost just as a
handful of star who never festering that
page and then and dutch ordering seems
to be fairly straightforward for everybody
and the same with ordering the condom
carrier as well
and then the three different types of
video that people can go and say
i'm just going to do one slide
on one of the videos which is
the demo demo video that showed you
it's a completely different picture here so
for seven people allocated this video and
but only sixteen actually clicked on that
box to go visit the demo page
that's telling us that that ho
page is not selling this demo video
at all if i'm going to click
on it while only sixteen above or
seventy it and and then only of
those sixteen opens only safa notch then
when i had to click on the
video imply it and then nobody watched
it in full on the meantime it's
by watching the video
i was seventeen seconds so not attractive
able to go and visit it when
i do go visit the page not
playing it very much and not many
people playing it and then when they
do disengaging quite quickly so they got
quite a bit of work to do
on the video and the demo video
but the same picture is true of
the other types of videos as well
they only looking at
and so what we've learned while we've
learned that we can use analytics data
to gain some important insights into non
usage attrition and we've just started to
kind of touch the surface with the
data we've got already and we're going
to use us to make some improvements
so and my conclusions have a kind
of touch support arm touched upon already
are the
the home page as attracting users to
fit it up the first three components
the things that you can order a
month on those component pages that good
conversion to ordering that seems to be
no issues with the ordering process and
but the homepages not attracting people to
visit the video content and and those
speedo
component pages are also not working very
well to encourage users to watch the
videos when they're already there and the
videos aren't be watched very long so
the first few seconds i encourage people
to carry on
am and we still have lots and
lots to learn about this failed and
what is really possible intense analytics so
i am very much a newbie in
this
and situation
i am ona solution for linking an
individual level and up that the user
id that we collecting invite cap for
all our surveys and test and sai
testing and linking that to our individuals
as they go through to the two
different websites who collects analytics data that
process that ronan
worked out for us am by liaising
with with us and also at our
development companies as was what really one
has been successful had no problems with
gathering the the unique user id of
everyone in a matter moab and seeing
what the wrongdoing and crises customer reports
and gathering all that data from that
swill worked really well and that means
can be possible for us to
measure dose in our main trial amd
in quite a fine grained way she's
really useful and and we're just beginning
to think about how we'll calculate a
set of standardized score for everybody in
our trial at an individual level based
on the products that are allocated and
ordered and also which videos to watch
and how long for
so that's the end of the presentation
them so just like to acknowledge off
under the national institute for health research
and all of the team working on
this with me as while at the
university of hertfordshire and others and am
yet onto any questions
thank you very much a catfish six
euros for the sir griffith presentation i'm
looking at the chart right now we
have approximately let's say maximum five minutes
because of those i was a talker
at four pm just pick the first
one which isn't
which was the most useful met him
with tunes use during this project
ah roney my view on this but
for me as both a masterpiece in
the right times but was the em
the custom id and then also the
reports as well when the report says
and then i go into all the
time now to to look at to
gather the data to help me understand
the
usability of the website and unauthorised i've
just presented with those flowcharts all comes
from those custom reports
and i'm looking at the chart those
not a question but small dickey with
logistic which is about though would it
be possible to download the presentation i
mean will it be possible for you
to send it to i mean do
you allow me to share it with
the audience or would you like to
keep the presentation
for yourself or can it be
ronan yea i'm happy for anyone to
have it stolen identity oh yeah well
one sinker of it my mailbox just
to make it as a pdf and
then and then surely to shooting back
hmm i lit the audience write downs
the last question with the level itself
four minutes left
do you have for example any any
questions or get you that you like
me to ask you for an audience
or
penny
do i what sorry i'll do you
have any any questions for you i
mean do you ever any question that
you would have expect the audience to
ask you for and that led me
to us
or do i have any questions and
i suppose the thing that i'm still
struggling with the most is
is linking all the data in the
customer report so each each custom report
i can download the data as an
excel file so for example if we're
talking about the components allocates i can
download it so i've got one column
which is the user id and then
noted to have component one yes no
component to yes now i can
it like that then i can download
another sapper report that says did they
do
did they visit the sample pack page
for example now be a completely separate
one and so at the moment they're
all very separate like that and then
i have to combine them all together
so i suppose oh and also every
time i want to update the data
i i it's not easy as pie
can use date i suppose but i
can't just just spell to buy everything
since i last looked in and download
an artisanal happened to just kept kind
of keep manually looking and then updating
my excel day
islam that so i don't know whether
it's possible is more question for you
than the audience rated but this is
something that i'm i'm struggling most with
at the moment and and yeah i
became to know about youth as a
waiter kind of made that process a
bit more straightforward i suppose but my
biggest the
limits of what matinee can provide nuclear
grades yeah it'll have the job the
answer i think that er we are
looking for custom reports with far more
dimensions
i guess that's the that's the key
thing here the maximum usually sweet custom
dimension and from what i remember old
yeah everytime you have something i did
my best thing then will flip to
musicians who can yeah you can just
put up to three custom dimensions then
filter by some as accustomed i mentioned
but at the moment i'm not sure
that desk plans to extend the number
of custom dimension which correspond more to
what you would like to
editor of the expedition xl fight and
you would get as many custom they
mentioned those units were ok they don't
from both free of up to five
or ten but then you can refer
to them back in excel and alfonso
says thank you because you insult to
his question about which was the most
useful met matua
and we don't have any questions left
so i guess that's the time to
close the the conference talk on small
sanctuary much cats you for your time
thank you very much for the presentation
and those while i would like to
thank all the attendees of this conference
and i
wish you all a great end of
matthew mckenzie was just one conference to
conference is left starting in five minutes
and then we have the closing ceremonies
at
and when i will sweat five pm
sanctuary much gets pleasure good luck with
the rest of the conference thanks goodbye
go back