Okay, and with this, I would like to start the next talk.
It is once again for Thomas Persson, like I said before, but just for the recordings.
He's a business developer at Digitalist Sweden, and he has worked with tracking digital analytics
since 2010, and has been a contributor to open source since 2007.
And in this talk, how to break Matomo and also some inputs on how to fix it, Thomas
will share his experience with us with running several large Matomo instances.
As most people know, Matomo can be pretty slow when the data grows, and Thomas will
share some of the biggest mistakes and also talk about how they manage them.
And thank you.
Please take it away.
Thank you.
So, yeah, sharing pains is usually good learning.
And I hope you guys can ask a lot of questions, and, yeah, we will hopefully get some interaction
going on as well.
So this is actually based on my learnings, or our learnings, I should say, because it's
not only me at Digitalist working with Matomo, of course, but some things we learned during
the last four years when we focused pretty intense on maintaining Matomo for a lot of
clients.
So let's start off by doing the short intro.
I'm a generalist, but in this case, I kind of had a background in the 90s where I actually
worked with hosting and maintaining the servers that were really physical and heavy and large
at that time.
So that knowledge I've kind of brought with me during the years and actually got to use
pretty intensively while learning and managing Matomo for our large clients.
Did you listen to my colleague Holger yesterday that talked about how we technically maintain
very large installations of Matomo just to get an idea of the size?
I think we have databases that go above a bit of a terabyte of data, and we collect
up to between 50 and 100 million actions per month stored in these databases.
So they're quite big, and we maintain them in a setup using Kubernetes where we can scale
things up and down.
Obviously not the most standard setup, but I will not talk so much about Kubernetes today,
more about the practical issues and the configurations that we can share with you guys.
So this is a scenario that perhaps many of you have seen.
We get different types of error messages or reports that are slow in Matomo, and obviously
our users can be pretty sad when that happens.
If you've been working with Matomo, you probably have these experiences over time.
So my pro tips if you want to have this scenario is I have three main principle guidelines
that you can follow if you want to have these problems.
The first pro tips is to give your users in Matomo a lot of permissions from start without
training them.
This I can guarantee you, you will run into a lot of nice issues and problems.
My second pro tips is to write bad applications and start tracking them in Matomo, and then
of course blame Matomo for the problems that you end up having.
We'll look in a bit more to this later on.
And my number three pro tips is to let your IT department host Matomo without proper Matomo
training.
We'll also discuss a little bit in detail what this might lead to.
Of course, I'm a bit in a joking mode here, but I'll try to give you some examples of
why I give you these pro tips.
So the first one about giving Matomo users too much permissions, Matomo is a quite complex
tool.
Of course, setting up tracking is something that requires quite a lot of skills in web
development and also JavaScript, and you also need to know how the product works to do this
properly.
Otherwise, you will have bad data in your database and the reports will be hard to understand.
Also creating segments or custom reports has to be pretty, you have to know what you're
doing, so to speak, because they can actually kill your performance.
I will give you examples of this later on.
But also doing this wrong might lead to really, really bad user experiences because you need
to know what you are doing and what you're showing to your users.
So you should be really careful about giving away permissions and make sure that users
that can create segments, for instance, know what they're doing.
The last one, or the second one, write bad applications and blame Matomo for the issues.
So this is actually something you need to be aware of that Matomo is just a stupid system.
Whatever you send to Matomo will be eaten by this system.
So you can't really fix things that is broken in your application or actually can, and I
will give you some examples of that, but you need to have the mentality of understanding
that Matomo is just a stupid system.
So if you send bad data into Matomo, this could be personal data or whatever, Matomo
will just eat that and you will store it and you will have problems in different ways.
The two most important things to manage properly are URLs and titles because these two will
always be just stored in Matomo when we send it.
And URLs is actually pretty important to manage in a large installation in particular.
So the third guideline, then, let your IT department host Matomo without proper training.
So this goes for many systems, not just Matomo, but why it fails.
And I actually have lots of experiences with it.
We help a lot of Matomo clients and they often approach us because they want help with implementing
tracking or understanding the application.
But usually I spend about a week with the IT department in the beginning just to fix
things that hasn't been properly set up or taken care of.
Because maintaining Matomo or any application actually requires you to know the application.
It's not enough to just set up a server and install the software and then you're ready.
And Matomo is really, really a tool that you need to know to maintain properly because
it needs constant updates and testing comes with that.
When we have a new release of Matomo, you really need to test this and know and understand
what was actually installed and how will that affect our instance of Matomo.
Even though we do this 24-7, we work with Matomo and learn, we still have problems when
there are new updates because you don't always know exactly what will happen in your Matomo
environment.
And this is, of course, since it's a complex environment, you have plugins, you have different
integrations and so on and every setup is unique.
We need to have constant monitoring in place and monitoring in this case is not only about
does Matomo respond, it can be monitoring individual reports or monitoring your database
or monitoring for errors really connected to Matomo.
This is also something you need to learn over time.
Constant optimization, your database will probably grow.
I will show you examples of that and that means you will continuously need to optimize
and again, constantly change your configurations depending on the needs of your users.
So again, feel free to do this internally but make sure that the people that is taking
care of Matomo has this as their focus.
It's a huge difference to just set up a server and install a software and maintaining an
application properly.
So let's look into some of the more specific problems that are related to users.
I talked a little bit about segments and custom reports because these both affect the aggregation
in Matomo.
Aggregation is the jobs that happens under the hood to produce the fast reports in the
interface.
So if you go to the page views report, that is actually an aggregation of the raw data
and to produce that, there are jobs running under the hood in Matomo and they will require
more server power.
So every time you create a new segment or a custom report, you actually add a new job
to Matomo to perform.
I've seen recommendations in the community and we try to have these recommendations
as well to not set up more than a hundred segments for a site and a hundred segments
to be honest can be quite a lot for your service to maintain as well.
So yeah, keep that in mind.
One thing we learned really, really the hard way is that if you create segments in Matomo
and you use these kind of wild card questions in your definitions, they have a large impact
on your database performance because what happens is you see the SQL query that is created.
This is not the exact query, but it's just an example.
The important part is the search string at the end.
When you send a query like that, which happens if you use the contains setting is that we're
not able to use indexes in Matomo and when you create segments, these queries goes to
one of the largest tables in the Matomo database and indexes is a way to make queries faster
basically in MySQL.
But these type of queries cannot use indexes.
So that means that they have to do a full table scan, which is kind of technical definition
here and that will really, really kill your performance if you have a lot of data.
We're talking about minutes to run SQL queries if you have a large database instead of maybe
seconds if you have a proper index in place.
So the impact here can be massive and especially if you have a lot of these segments in place
on a large database.
So instead, you should try to change your segments into using starts with, for instance.
If you do that, your query will be different and MySQL will actually be able to perform
these queries a lot faster.
So it's quite important actually on a large data set to manage this.
Yes, so with this in mind, make sure that only users with this knowledge can create
segments.
And this will also lead to that we avoid bad configurations because I see a lot of users
that actually use segments in the wrong way.
I won't go into too much about that, or I can.
The example is if you want to filter out pages for a specific section of your site, I often
see users creating segments for this because they misunderstand how segments work.
This is a way to create a bucket of users matching certain criteria.
So even though you will filter out page views for that particular section, you will also
filter out page views for other sections that the users match in your segment criteria matched.
So yeah, different thing.
So yeah, one thing I often see is also that users that create segments, you can do it
on a personal level.
So only you see the segment.
As an administrator and particularly on sites with a lot of users, I can see that sometimes
there are up to ten segments that are almost the same thing because users create them under
the hood.
And this is also bad because that means ten jobs are running under the hood when you might
only have needed one of them.
So yeah, keep that in mind.
And of course, this leads to avoiding performance issues.
Also this is a nice one.
Look into the global ini.php file on your server and make sure you can actually set
this is the user role needed to add segments in Matomo.
So I think that the default one is view here and that means that basically any user in
your Matomo instance can create segments.
So make sure to make it admin or super user or write or whatever your organization matches.
So that's the good thing to look into.
All right.
So what about bad applications?
Well, one thing that is really important for performance is to keep the number of URLs
down.
And the thing here is that if you, for instance, send, Matomo can't differ between uppercase
and lowercase.
So if you have a page that will report sometimes with an uppercase and sometimes with a lowercase,
Matomo will handle that as two different URLs.
So make sure to always send lowercase URLs.
Make sure to not send URL parameters if you can avoid that because that will also create
unique instances of URLs, meaning the jobs in Matomo needs to process every URL unique.
Also some CMSs sometimes send trailing slashes.
The example here is MyPage and that will not be the same as MyPage with an ending slash.
Another common thing that you might want to avoid is, for instance, if you have an issue
number with unique URLs containing numbers, you might want to look into anonymize this
so that you don't store unnecessarily page views.
So yeah, what I usually do is that I create, actually we show you a script that can do
this later on, but sometimes you have an issue URL and you can end up with thousands of unique
page views or unique URLs, but no one will never look at the unique page view for that.
So anonymize them is actually good for performance and probably also for the analytics part to
understand the site.
So one way to do this, this is custom JavaScript that I've added to the site.
It's a variable.
So what you can do here is that you create the URL cleanup and in this example, I started
off like I'm reading the URL from the browser and making it lowercase.
The second one will remove trailing slashes, then I have an example of how you can anonymize
an order ID.
So if you have an URL saying order one, two, three, four, five, six, this will become order
slash order page.
And the way you use this then is you go to your page view and you use this variable in
your custom URL and by doing so, you can actually clean your site and this is also a good way.
If you have problematic applications, this kind of JavaScript can actually help you to
anonymize and remove personal data if you find those kind of issues.
So maybe you have an application that shares personal identification numbers or email addresses
from clients or whatever.
I've seen tons of that examples there.
You can actually use this kind of an URL to clean things.
Another thing that I often see and some CMS systems are really terrible here.
So a good URL structure looks something like this.
You have maybe URLs following your menu structure or other types of categories and you have
a clear structure in your URLs.
And since Matomo is a stupid system, it will just take the URLs as you send them to Matomo.
A bad example is if your CMS will have this kind of structure.
This is also bad if you're into SEO, of course.
But the point here is like if you go to your page views report, you will see something
like this if you do a nice URL structure.
You will have aggregated numbers matching your sections.
So this is really a nice way to or a nice feature of having nice applications.
You can actually if you have problems, you can actually fix some problems in this one.
So sometimes I've seen applications where we have multiple URLs for the same page, which
is obviously bad.
Then you can add those things into this URL cleanup JavaScript as well and maybe make
sure that they are reporting under the same URL in your or in Matomo at least.
There are lots of ways we can help here.
All right.
So let's look into optimizing Matomo from a more of a server perspective.
So some common issues in Matomo is that our transitions report is slow if you use that
one.
Custom date ranges are slow because what happens if you use a custom date range is that you're
actually querying the raw data.
The visitor log obviously is something that can be slow.
Sometimes you can see that today's data is not coming in as quickly as you would like
to.
Maybe you don't see data for today until the evening or whatever, or users get 500 errors
or timeout errors or things like that.
There's no general solutions here, but if you have a lot of these and this is quite
common if your data grows, you might want to look into your server configurations.
First of all, look into the general guidelines.
Always start there to see if you're, are you even following the basics before you go into
maybe my more advanced recommendations later on.
In general, you need to understand that Matomo is a PHP application and we usually refer
to that as a LAMP installation.
That means we have an Apache web server.
We have MySQL and we have PHP.
Those are the basic main components.
There are more and you can actually change Apache to Varnish or some other web server
and MySQL can be MariaDB, so don't take it too much of a, yeah, it can vary.
But when you optimize these, you need to understand what kind of needs these systems have.
So normally Apache is a very CPU intensive system, meaning what you can optimize there
is usually threads and sometimes a bit of a memory, but you can optimize how you use
the CPU.
MySQL is usually memory and disk intensive, so make sure to have fast disks and then you
can often tweak MySQL variables and this is actually one thing when I talk to IT departments
that has been trying to maintain Matomo, they usually install the database in kind of the
default setup that comes with MySQL and that is probably the easiest way to make your Matomo
server slow because if you know MySQL, you can optimize I think over 200 variables.
You shouldn't, but you can, but MySQL application like Matomo that is very database intensive
really, really needs a lot of optimizations and there are great tools to help you with
that.
PHP is of course super important and for that we usually use the memory and super important
to cache PHP because PHP is a language that will reprocess every page every time you see
it basically or every time a function is loaded, it's actually processed unless you use caching.
So caching is key when it comes to PHP.
One thing that is super nice and particularly on larger installation is to use the Q tracking
plugin which will offload your database significantly and also give you features such as you can
restart your Matomo installation without losing data.
I really encourage you to look into this one if you haven't already.
You can also configure tons of things in Matomo.
I linked to this little file on matomo.org.
I'm going to show you that actually.
Let's open that file just to give you an impression but in your Matomo installation you have a
file called global.ini.php.
This one is actually from GitHub but just to give you an idea how much you can actually
configure on your Matomo server or your particular Matomo instance, you can tweak a lot of things
and this is great because it means we can adopt Matomo in tons of ways but it also means
that you can get problems if you don't know what you're doing.
So one thing you usually want to have control of is the number of rows that you allow in
different reports.
From a user perspective you normally want to tweak this upwards because I think the
default is that you show only 500 URLs in reports.
If you have a lot of URLs you obviously want more than 500 but setting them too high and
that means affecting performance.
Auto scanning of forms if you use the form analytics plugin is something you should be
really careful with that can make your database grow.
So keep an eye on how long you store your raw data.
This is normally something you should know when you start but sometimes people don't
have a clue what that is because that means the database is growing a lot.
And also be a bit careful with plugins because plugins will consume a lot of memory and if
you don't need them don't install them.
Yes MySQL, yeah this is actually something we learned.
We are using MariaDB and what we learned is that there are actually features in Matomo
that use functions in MySQL that are not available in MariaDB.
So if you're having a large database you might run into problems.
If you want to know the details I think Jorge, my colleague linked to those issues at Matomo
but one thing on a large database at least you need to keep an eye on the database size.
You can use tools like Tuning Primary or MySQL Tuner to get the insights and one thing we
also learned is that over time the number of tables in Matomo will grow.
So you might want to check these configuration options or actually configuration variables
in MySQL because if you don't adjust these according to your database you might have
performance issues.
There are also optimizations that you can do to make transition reports faster.
So there is some FAQ there.
Every Matomo installation is quite unique.
So I wouldn't recommend you to just apply every index for databases that you find.
You should really try them and test how they affect your installations.
Indexes can grow your database quite a lot and you should do this with care.
This is obviously something you can look into and test.
Another thing that is quite common is that if you have a lot of segments and custom reports
you might want to start looking into how to optimize the archiving that happens under
the hood.
One thing we've been doing is that if a client has a lot of websites normally there is only
one or two websites that have a lot of data and a lot of segments and the other websites
in the installation are usually small.
So one good thing is actually to separate these into different jobs meaning one job
that might be heavy and take a lot of time will not affect the smaller sites.
So having two jobs you can just do this, skip ID site number one, then force ID site equals
one on the other job and that will offload or actually give faster reports for the small
sites.
Another example is to separate so that you run segments separately.
That means that if you're not applying a segment those jobs might be faster but then you might
have other segments that might take time if someone configured for instance segments containing
wildcards that takes a lot of time you can actually separate these into specific jobs.
Maybe these reports will take hours to process but the other segments will respond faster.
So that's something you can look into and start tweaking.
Another example is that you want to start optimizing your archiving processes.
You can actually use these flags like concurrent request per website or concurrent archivers
and you can also start leveraging with the PHP memory that an archive can consume.
But this is really something you should know what you're doing before trying or you can
try it and see and learn.
One thing that I've been doing is that I've actually created a plugin.
To warn you a bit this is actually a plugin I wrote to learn how to write plugins.
So it's maybe not the most beautiful code I ever written but it's been a good learning.
But what it actually does is that it adds some reports to my Matomo installation.
So under the hood I can go in and get a few recommendations.
So I have these performance tips for instance that will give me some insights on how my
database feels and I'm doing some calculations on how my MySQL database is performing and
in this case for instance we're not really following the recommendations on how to configure
Matomo.
This is a database setting that we should change and I also linked to a recommendation
for Matomo.
So yeah this is something I've been improving over time and we actually use it in our database
to get some feedback.
Another thing I recently added is regarding these segments.
I wasn't logged in.
I will fix that.
I can't actually because this one requires me to be on a VPN so we won't do that.
But I actually created a list of problematic segments.
So segment containing wildcards.
You can go to this repository and download it and please give feedback.
Maybe you want to help to contribute to make it to follow the coding guidelines on Matomo.
But basically I have created some ways to easily get status variables and things like
that.
What else?
We have time for questions.
Thank you.
We have one question which is would you be interested in publishing the DB health plugin
on the marketplace?
Definitely.
But I need some help.
Maybe Lukas can help me to contribute.
But I would definitely be interested in doing that.
It's already free to download so I guess if Lukas help out it would be nice.
You might look into it.
So if there are any other questions feel free to ask them.
I'm just going to give a few moments for that.
All right.
So no new questions.
It seems great.
Someone's typing.
Yeah.
Just a few more seconds maybe.
I mean as always the chat stays open.
So maybe if you see a question later you can answer it then.
But with that for now I think I want to thank you again.
I mean you're going to have another talk I think next hour but have a great day until
then.
Yeah.