the cape and with this i would
like to start the next thought it
is once again bottomless person or like
said before purchase recording see the business
developer at visually sweden and he has
rubbed with tracking digital analytics since two
posts
intent and has been a contributor to
open source since two thousand and seven
and in this talk outbreak matter mo
and author input how to fix it
thomas will share his experience with us
refund a federal large matter were instances
as most people know matomo can be
pretty slow when the tablet gross
and thomas will share some of the
biggest mistakes and also talk about how
they managed him and think it does
take away thank you for sharing sharing
paying a fairer you should a good
learning pan but i hope you guys
can
i asked a lot of questions founder
we will hopefully get some some interaction
going on as well so this is
actually based all my learnings through and
or our learned or should fight because
it's not only me up to to
tell is working with a number of
course but something we learned during the
last four
yes when we
progressed pretty intense on on the mouth
maintaining matomo for a lot of clients
so let's start off by doing the
short interrupt her
i'm a generalist but in this case
i've kind of had a background in
the nineties were actually worked with hosting
and they're maintaining the servers that were
really physical heavy and large at that
time so that knowledge i've kind of
brought with me doing it during the
years actually
the air got to use pretty intensively
while learning and managing matomo have for
our large clients
maybe you listened to my colleague concrete
yesterday that talked about how we technically
maintained very large installations of metal and
we're just to get an idea of
the size i think we have databases
that go above a beat than a
terabyte of
data that we collect
up to between fifty one and one
hundred million actions per month are stored
in these databases somewhere they're quite quite
big and we maintain them in a
setup using two burnett is where we
can scale things up and do obviously
not the most standard set up but
i will not talk so
much about cabinets today more about the
practical issues and their configurations that we
can can share with you guys
full
this is fair
the scenario that we're proud from any
of you have seen we get different
types of over error messages or reports
that are slowly matomo and obviously our
users can be pretty sad when that
happens
if you've been working with tomo you
probably and had these experiences over time
so that my pro tips if you
want to have this scenario
if i have three main principle guidelines
that you can follow if you want
to have this province the first pro
tips is to give your using matomo
a lot of permissions from stock without
training them
this i can guarantee you you will
run into a lot of nice issues
and province my second pro tips is
to write bad applications and stop tracking
them e matomo and then of course
blame at all for the problems that
you end up having
we're looking a bit more to this
a wrong and my number three pro
tips is to let your i t
department hosts matter more without the appropriate
training i will also discuss it a
bit in detail what this might lead
to
of course i'm a bit today and
joking mode here but i'll try to
give you some examples of why i
gave you these projects so the first
one about giving matomo uses too many
permissions at bottom is quite complex too
of course said
adding up tracking he said is something
that requires quite a lot of all
their skills in web development and also
you ask it and you also need
to know how the product works to
do this properly otherwise you will have
bad data in your database and the
repo
sweet be hard to to understand also
creating segments of custom reports has to
be pretty
you have to know what you are
doing so to speak
because they can actually kill your performance
i will give you examples of this
later on but also doing this wrong
might lead to really really bad user
experiences because you need to know what
you are doing and what you're showing
to your users so you should be
really careful about giving away permissions
and make sure that uses that can
create segments for instance know what they're
doing the last one or the second
one and write bad applications and blame
at all for the issues so this
is actually something you need to be
aware of that was always just a
stupid system
whatever you send to matomo will be
eaten by the system so you can't
really fix things that is broken in
your application or actually can and i
will give you some examples of that
but you need to have the mentality
of understanding that and not always just
a stupid system
so if you send bad data into
moto this could be personal data or
whatever but all will just eat that
and you will store it and you
will have problems in different ways at
the two most important things to manage
properly are ass and titus because these
two will always be
he just stored in matomo when we
send it send it
and your attitude is actually pretty important
to manage in a large installation in
particular
from the third guideline that laterite the
department health nut and without proper training
at will
this goes for many systems not just
matomo bus
why it face and i actually have
lots of experiences with it and we
help a lot of my former clients
they often approach us because they want
help with implementing tracking we're understanding application
but usually i spend about a week
without the department in the beginning just
to fix things that hasn't been properly
set up or taken care of
because maintaining moto moran application actually requires
you to know that the application is
not enough to just set up a
server and install the software and then
you're ready
not always really really have to let
you need to know to maintain properly
because it needs constant updates and the
testing comes with that when we have
a new release all the time when
you really need to test this and
know and understand what was actually stolen
how will that affect
kt our instance of matomo even though
we do this twenty four seven we
work with moto my learn and we
still have problems when there are new
updates because you don't always know exactly
what will happen in your app
not my environment and this is of
course since it's a complex environment you
have plugins you have different integration and
so on and every setup is unique
we need to have constant monitoring in
place and monitoring in this case is
not only about dust but all will
respond it's it can be monitoring an
individual reports or a monitoring and database
for monitoring for errors really connected to
them at all
this is also something you need to
learn over time constant optimization or your
database will probably grow i will show
you examples of that and that means
he will continuously need to optimise and
again constantly change your configurations depending on
the needs of your
users so again feel free to to
do this internally and and but make
sure that the people that is taking
care of matomo has this as their
focus it's a huge difference to to
just
set up a server and install the
software and maintaining an application properly
so let's look into some of the
more specific problems that are related to
users i talked a little bit about
segments and custom reports because these both
to pass affects the aggregation him at
all and aggregation is the jobs that
happens under the hood and to produce
at the fast reports in the interface
if you go to the page use
the report that is actually an aggregation
of the raw data and to produce
that
there are jobs running under the hood
in matomo and they will require more
solar power so every time you create
a new segment for accustomed report you
actually add a new job or two
method before
seen recommendations in the community and and
we tried to have these recommendations as
well to not set up more than
one hundred segments for a site i'm
one hundred segments to be honest can
be quite a lot for your service
to to maintain as well so keep
that in mind
one thing we learned really really the
hard way is that if you create
segments see my total and you could
use these kind of wildcard questions in
your definitions they have a large impact
on your database performance
because what happens is you see the
sql query that is created this is
not the exact query but it's just
an example the important part is the
searched string at the end when you
send a query like that which happens
if you use it contains
setting is that we're not able to
use indexes imitable and and when you
create segments these queries goes to one
of the largest tables in the matomo
database and indexes these away too
medic credits faster basically in my script
but
these type of christ cannot use in
excess so that means that they have
to do a full table scan which
is kind of technical definition here
and that will really really kill your
performance if you have a lot of
data we're talking about
minutes to run sql queries if you
have a large database instead of maybe
seconds if you have a proper index
in place so the the impact can
be messy and especially if you have
a lot of these segments in place
on a large database
so instead you should try to change
your segments into using starts with for
instance if you do that your query
will be different and my skin will
actually be able to perform these queries
a lot faster though it's quite the
po
portland actually on a launched dataset to
to manage this
yes so with this in mind make
sure that only uses with with this
knowledge can create segments and that this
would also lead to that we avoid
bad configurations because i see a lot
of uses that use actually use segments
but in the wrong way i won't
go into too much about that but
alright again the example is if you
want to filter out the pages for
a specific section of your site i
often see users creating segments with this
because they misunderstand how segments work segments
is
is a way to to create the
bucket of users matching certain criteria if
so even though you will filter out
page views for that particular section you
will also fail to pay pictures for
other sessions that the use is matching
your segment criteria first have matched
so yup
different thing so
cape
yet one thing i often sees also
that i
uses
that creates segments that you can do
it on a personal level and suddenly
you see the segment as an administrator
and particularly on sites that a lot
of users i can see that sometimes
there are up to ten segments that
are almost the same thing because useless
crap
let's create some under the hood
under this is also bad because at
me tangibles are running under the hood
when you might only have needed one
of them
so yes let's keep that in mind
and of course this leads to avoiding
performance issues
also this is a nice one look
into that
global you need a peach pie fight
down your server and make sure it
has actually set
this is the user role needed to
add segments in that one so i
think that the default one is view
here and that means that basically any
user in your metal knees can create
segments so make sure to to
make it admin or super user or
write or whatever you or your organization
matches or so so that's the good
thing to look into alright so what
about bad applications
while that
one thing that is really important for
performance is to keep the number of
urine stone
and the thing here is that a
few for instance send a toma can't
differ between uppercase and lowercase so if
you have a page that will report
sometimes with an uppercase and some titans
with a lowercase metal movie handle that
has two different years
nz so make sure to to always
send lower case uranus
make sure to not send your parameters
if you can avoid that because that
will also create unique instances uranus meaning
and the job seem at all one
needs to process the tub for your
unique
also some cms says
sometimes sand trailing slashes the example is
my page and that will not be
the same as my page with an
ending flesh
another common thing that you might want
to avoid is for instance if you
have an issue number with unique you
or else containing numbers you might want
to look into anonymize these so that
you don't store unnecessarily
page views
so yup what i usually do is
that i create actually will show you
a script that without that can do
this later on but but this sometimes
you have an issue at your rattle
and you can end up with thousands
of unique page views so unique your
ants but no one will never look
at the uni
the page you for that so anonymize
them is actually good for performance and
probably also for
the analytics part to to understand the
site
so one way to do this this
is a custom javascript that i had
to recite that it's a it's a
variable so what you can do here
is that you created your cleanup and
in this example i started off like
i'm reading the euro from the browser
and making it lower case
case and the second one will remove
trailing slashes
then i'd have an example on how
you can anonymize an order id so
if you have and you're always saying
horner one two three four five six
this will become water flesh water page
and the way you use this then
if you go to your page view
and you use this variable in your
custom urine and by doing so you
can actually clean your site and and
this is also a good way if
you have problematic applications at this kind
of javascript can actually help you to
anonymize and remove personal data if you
have usually find those canvases so maybe
you haven't appliqued
nation that shares personal identification numbers or
email addresses from clients or whatever i've
seen tons of bad example there examples
there you can actually use this kind
of man your attitude to
to clean things
and another thing that i often see
on some cms systems aren't really terrible
here
so good and your unstructured look something
like this you have maybe are as
following your menu structure or other types
of categories and you can you have
a clear structuring your uranus and since
matan ways the stupid system it will
just take the euros asked
you send them to matomo a bad
example is if your cms will have
this kind of structure and this is
also bad if you're into su
of course but at that the point
here is like if you go to
your page views report you will see
something like this if you do a
nice your instruction you will have aggregated
numbers are matching your sections
but this is really really a nice
way to
or or a nice feature on having
nice applications
you can actually eat if you have
problems you can actually fix some problems
in this slum and so sometimes i've
seen applications where
where we have multiple neurons or for
the same page which is obviously bad
then you can add those
things in in into this year i'll
clean up your musket as well and
maybe air
make sure that they are reporting under
the same europe and you'll see a
broker or a muslim at least
but then there are a lot lots
of way we can help
help him alright so let's look into
optimized seem to move from a more
more of a server perspective
so some companies just seem to know
is that our transitions report is slow
if you use that one
the accustomed date trenches on slope because
what happens if you use a cap
custom date range is that you're actually
querying there the raw data the visitor
log obviously is something that can be
slow
sometimes you can see that today's data
is not coming in as quickly as
you good luck would it it would
like to maybe you don't see data
for today until in the evening or
whatever or uses get five hundred error
or timeout errors will type things like
that there is
no general solutions here but if you're
if you have a lot of this
and this is quite common if your
data grows you might want to look
into your server configurations
first of all look into the the
general guidelines always thought there to see
if your are you even following the
basics before you go into maybe my
more advanced recommendations later on
that
in general you need to understand that
matomo is a p h p application
and we usually refer to that as
a lamp installation
that means we have an apache web
server we have my a scam and
we have p h p c those
are the basic main components there are
more and you can actually change apache
to varnish or some other web server
and my scam can be maria db
so that don't take it to too
much of a
yeah hitler married but when you do
when you optimize these you need to
understand what kind of needs these systems
have so normally apache is a very
cpu intensive system meaning at what you
can optimize theories usually threads sometimes
something's a bit of a memory but
you can you can optimise hollow you
use the cpu
my style is usually memory and they
skimped and saved so make sure to
have the fastest this and then you
can often tweak my escrow at variables
and this is actually one thing when
i talk to either departments that has
been trying to maintain moto mo
do they usually install the database in
kind of the default setup that comes
with my sql and that is probably
the easiest way to make your matomo
server slow because
if you know my schedule you can
optimise i think over two hundred variables
that you shouldn't but you can
but on my scale application like nato
i'm whether it's very database intensive really
really needs a lot of optimizations
and there are great tools to help
you with that a peach pie is
of course super important and for that
we usually use the memory
and super important to cache the hp
because psp is a language that will
reprocess every page every time you see
it basically or everytime a function is
loaded it is actually processed unless you
use cache for caching is key and
when it comes to psp
one thing that is super nice and
particularly on larger installation is to use
the cute tracking plugin which would offload
your database significant sit significantly and also
give you features such as
you can restart your moto e installation
without losing data or really have to
look into this one if you haven't
already
you can also configure tons of things
in moto move
i linked to this little
file on tumblr dot org i can
assure you that actually
just open that
and just to give you an impression
but in your mental installation you have
a file called glow but the psv
this one is actually from github but
just to give you an idea how
much you can actually configure on your
matomo server or your particular
amazon instance you can you can tweak
all of those things and this is
great because it means we can adopt
motto in tons of ways but it's
also it also means that you can
yeah you can get problems if you
don't know what you're doing so one
thing you usually want
to have control of is the number
of rows that you're alone in different
reports
from a user perspective you normally want
to tweak this upwards there because i
think the default is that you show
only five hundred years and seen reports
and if you have a lot of
you rather see obviously want more than
five hundred but setting them too high
and that means the effective performance of
the scanning of forms if you use
the form analytics plugin is something you
should be really careful with that can
make your database grow also keep an
eye out over how long you store
your raw data this is normally something
you should know when you stop
but sometimes people don't have a clue
what that is
because that means growing the database is
growing a lot and also be a
bit careful with plugins because plugins we
consume a lot of memories
and if you don't need them don't
installer
yes my sql that
but yeah this is certainly something we
learned we are using maria db and
what we learned is that there are
actually features in my in matomo that
use functions in mind scale that are
not available in maria db so if
you're having a large database you might
run into problems
if you want to know the deacons
something her grandma call it linked to
those issues
ima ethanol tomorrow but one thing and
on a launch date the base at
least you need to keep an eye
on the database size and you can
use tools like tuning primary over my
oscar junior for two to get the
incense and one thing we also learned
is that over time the number of
table c moto movie grow
and
you might want to check these configuration
options for actually configuration variables in my
scale
because if you don't adjust these according
to your database you might have performance
issues
there are also optimizations that you can
do to make transition reports faster so
there is an africa there
every muslim installation is quite unique so
i wouldn't recommend you to just apply
every index for the the message said
to find you should really try them
and test how they affect your installations
indexes can grow your database quite a
lot and you should do that this
with care
this is obviously something you can look
into and test
another thing that is quite common is
that you have an if you have
a lot of segments and accustomed reports
you might want to start looking into
how to optimize the archiving that happens
under the hood
one thing we've been doing is that
if a client has a lot of
websites normally there is only one or
two websites that have a lot of
data and a lot of statements and
the other websites in the installation are
you should explore so one good thing
is actually to separate these into to
the different jobs meaning what
one job that might be happy and
take a lot of time will not
affect us more as sites so having
two jobs you can just do this
skip either site number one then force
i decide equals one on the on
the job then that when offload or
actually gave faster reports for the small
sites
another example is to
it's also that you run segments separately
that means that they're
if you're not apply any segment those
jobs might be faster but then you
might have other segments
that might take time if someone configured
for instance segments containing wildcards that takes
a lot of time you can actually
separate these into specific jobs maybe these
reports will take hours or two to
process but the other segments we respond
to
faster so that's something you can look
into and start tweaking
another example is that you want to
start optimizing your archiving process as you
can actually use these flags like concurrent
requests for a website or a concurrent
archiver cine can also stop liberating wit
and the
p h p memory that americans can
consume but this is really something you
should
i know what you're doing before before
trying
or you can try them and see
him learn
the one thing that i've been doing
is that i've actually created a plugin
to to warn you a bit this
is actually a plugin i wrote to
learn to how to write plugins sites
may maybe not the most beautiful code
i ever written but it's been a
good learning but what it actually does
is that it adds some reports to
my
my installation so under the hood i
can go in and get a few
recommendations so i have this performance tips
for instance that we gave me some
insights on how my database fees and
i'm doing some calculations on on home
my minds go
database is performing and in this case
for instance we're not really following their
recommendations
one how to configure motto this is
a database setting that we should changed
and also link to a recommendation former
tommaso yup this is something i've been
improving over time and we actually use
it in our database to to get
some feedback another thing i recently added
is regarding
the segments all
i was logged in i will fix
that
oh i can't actually because this one
requires me to be on a vpn
so we won't do that but i
actually created a list
of problematic segments so segment containing one
course you can you can
go to this repository and the and
download x and the
and please give feedback and may be
want to help to contributed to make
it to the photo there
the coding guidelines on what one but
basically i created some ways to easily
get
status variables and the
and things like that
yup what else we have to sign
for questions
and then thank you we have one
question the moon averages would you be
interested in publishing the db health plugin
on the marketplace definitely but i need
some help maybe it maybe lucas can
help me to to code reviewed
but i would definitely be interested in
doing that
if it's already free to download soon
so i guess
if newcastle bought it would be nice
the medal awaited
so if there any other questions of
if you asked him is it just
going to be the few moments for
that
yeah hang on
alright so long new questions it seems
great since i was have been delicious
you must seconds maybe
okay maybe not alright i mean as
as always the thread stays open so
maybe to see english lady you can
you can answer then i would i
would
but with that fill up the thing
elena thank you get it
i mean you're going to have an
hour to i think next hour but
then integrate the app thank you bye