the everyone thank you for attending this
session today we will have her her
powers who has been working for that
ten years with analytics seo development and
devops
he stopped the day is how to
manage a moto mo with millions of
page views per month where he will
tell us more about their work at
digital is sweden a company which matches
manages some of the biggest matomo is
the leaders in sweden so i'm giving
the word to her now
thank you
arrest timer
as you heard that i've been working
with politics for many years
i've been programming since i was a
kid in the nineties
i started professionally about eleven twelve years
ago working there
mostly as a
from thunder from the beginning but yeah
i wouldn't label mass front and back
younger
zero exp
i am a generalist and am a
wherever i needed so i'm a problem
solver he could call me
i have recently been on parental leave
so
i'm just back where cancel this presentation
is sir
put together recently so bear with me
but i'm sure i can
any questions if you have any
yeah that's from above so i worked
for digital lists now i've been here
for six years and i've been working
with the devops team
i've been
project manager
and i have been working with moto
more
for a three or four years now
in the digital lists
so what what am i going to
talk about
when we started with matomo it was
still called piwik
and everyone knows that the gdp are
in two thousand and eighteen was sarah
getting air in effect so the gop
were to recover from the seasons we
number
to attract data and take ownership or
data so the swedish tax agency scott
the rackets and
they had a procurement and reached out
to us and asked us if we
knew matomo and we have had pv
kit installations
running for probably ten years so
but we weren't no experts yet and
always took on this said and
yeah so a little bit
sorry didn't it sir
small country in europe if you look
at the population but still we have
ten million plus population in sweden and
yeah
the fiscal declaration i mean the sweden
is very progressive when it comes to
digital age digitalization and the
in python twenty one four point five
million people the their fiscal declaration on
line and
and all of those people went into
this earth to say what the declaration
and they have an utter or my
services as well on other websites so
lot of traffic to their sites and
if you look at the timeline hair
and you think how big applications were
driven back in the days
i
i have it
fire up or withdrawn on a server
i checked our
the kind of cirrhosis and then in
two thousand and fourteen cabinet this came
in and change how the landscape of
applications are deployed today so
and we signed with the captive etiquette
just before two thousand and eighteen and
so we had a bit time before
may twenty fifth i think it was
the date when gd pure was in
effect
and at this time if you downloaded
the source code to pay vegas like
it was called back in the day
you would you could go to the
online documentation and see that
it's a
a lamp stack basically so you have
a leanness linux a part share and
my sql
and pitch p
and for us this is a yeah
we've been working with open source for
twenty years or so so this is
a very familiar
for us
but the challenge share
guessing that cuban earth as was sir
open source in two thousand and fourteen
and then
yeah it was clear how to a
hostess a big installation where we knew
that we were going to have a
lot of traffic so that we put
together a team of our engineers to
try to figure out the best option
to go for it
yeah until two thousand and nineteen we
could find
more documentation on the motto more
website at around the big insulation four
hundred million page views per month or
less
but this is
similar to the conclusion we came up
with at the time
i
but when it comes to
are running this and monitoring in the
us on this silly challenge as we
can just play some servers and hopefully
they will run for a long time
so
we started to sketch our own
infrastructure with guidance from the matomo documentation
of course
at the time when we installed the
matomo maria db was still supported by
matomo
and i'm gonna to come back to
that later but we were yeah how
how should we build a bigger application
a monolithic application you could say sir
and i where everything and compiled into
source code and you will put it
on a server and it will just
men sit there and
run the whole application by itself
this which yeah that
the services you have
the things that you see and
the communication layer under the database basically
or microservices so these were the options
we were looking at that time and
of christ
we can't deny that we had requirements
from art
and now i'm imagining that there's a
big crowd out there and you'll probably
see a rocky horror picture show or
something where there are just shadows the
boo or whatever so we add for
example backups are backups need to have
redundancy when it's at
security certified hosting will need to have
a
the status of the application
we need our scalability because everyone doesn't
go in to do their fiscal declaration
once a month this is something you
do once a year probably
need to be able to restore the
systems if it goes down
you need to have a certify their
standards need to be able to roll
back and when a high availability so
that people from all over sweden can
go in and do their declaration space
basically
so
we went to her
hosting partner we've been working with for
many years and and them they are
certified kubernetes now only one said if
i could notice in in and in
the nordics at the time
and so there will of course proposing
that we should run the this uncovered
lettuce because it's the next day and
that's it in there
in tech ops we wanted to go
with the cabinet to set up and
for us it was
although a black box you could
it is not the traditional way you
should think of but sir corrupted kiss
so
we have to wade through to some
barriers to work the works in the
digital is
we took some crash courses there in
cuban at this and the
i'm looking back at it now and
i remember i was thinking like it
was it was a big question mark
in my head at the time but
as more we begin to cuba netizen
there
the everything around kubernetes as it everything
became more clearer so
to you you need to throw everything
you know about servers and hosting out
of the window and you need to
rethink how you think about applications and
services
so this is a picture i took
from the cabinet this website and you
can see that the traditional deployment you
have there
something similar to the three tire
architecture
or you can have the the dss
and we have some you could have
docker or some kind of error or
alter scaling
hosting
can't remember the name you call him
but all he could have like running
container like microservices
so if you if you want to
know what cabinet is isn't death to
the cabinet as where patient there is
a lot of bird accommodation you can
read
but basically the things that we get
out of the box here to meet
the requirement as was sir
that
we could have rollbacks autumn fullbacks
we can have a great
rob is
at least we can do
tinder application without taking down the service
so if
scott the for example let's say hundred
data and analysts and they go in
and they work office time you know
nine to five and we don't want
to take down the application yes if
you want to run out if we
wanted advice on configuration so this needs
to be very careful
during upgrades
yeah so at the time
because we were new to kubernetes as
were the partner elastics they come up
with the architecture
so this is what it looked like
from the start we could say we
have changed some some bits
through the years but
and the basic concept still and and
the
so when i user either if you
go into the dashboard as an analyst
do you get to
one lad
container you could say
or if you have installed your tracking
scripts websites to go
the tracking data will be selling into
the us
and so this is how we can
scale up and down depending on where
the lotus in the application
and of course
when when you dissect a big application
like matomo
but then you're going to run into
some challenges
and i would say the biggest challenge
at the time was sad configuring and
the database too
handled all their load that was coming
in
and we could use many of the
community everything was open source by the
way i forgot to mention that but
we are open source company face a
lot so we like opensource
and we keep it
we like to keep our infrastructure as
code and and the best way of
doing that at a time with the
help from our partner elastics when they
come up with hell so helm is
a way to keep your
charts as they call them the templates
for your infrastructure and so this is
how we can easily tweak bits and
pieces of the application and without having
to
the access to a server and in
manually and change something so every change
needs to be documented so we can
do this in our version controlling system
with git
yeah so this is hell
and of course we have a
require mass as like cats and ship
out code deploy code that have vulnerabilities
in the software to figure out some
kind of id see i see the
workflow where we can ultimate stuff because
you will you you can really track
all the vulnerabilities that come up all
the time so
the way automates as much as possible
in the cut
we have a drone
a drone is a cia service
where you can plug in a different
kind of applications to a third party
system that will check your code your
base code so if you're peckish all
your motto more into a container in
docker let's say
and all the underlying system applications
that are in this docker image
drone
will stop service and then we will
use something
that is called claire
just the next line
here and also it's an open source
service that will check for vulnerabilities
so everything is running containers so before
we deploy something out to
production site this will run up a
similar environment
where it will check for vulnerabilities and
when we get a green light then
we can do to deploy to the
production and of course being an open
source company and how much the community
means for us we have also open
sourced our hell base set up
so anyone can take our work and
replicate what we have done and if
you go to our github you can
find the matomo cabinet this
they set up the we are using
of course there there's other dependencies so
we use this as a base and
then we have there a different
a project where we the package
and replant well
the worst
dissecting an application like matomo
you want to have the control you
want to automate as much as possible
so when we are dependent on the
cla tools which are
at the time we couldn't use we
can add any new users or so
every deploy basically we will destroy everything
and we build a home more and
if you know metal man how to
install it
basically it's a clean installer every everytime
so to be able to keep an
our database without wiping it and we
had to come up with
different seo tools to automate some of
the stuff so and were contributed this
back to their community and we have
summer plug plugins in the matomo marketplace
one of them will use is the
use of console
but also more most importantly i would
say the extra tools
where we can ultimate a lot of
things like year db backups or installation
of matomo
and
i took this screenshot yesterday and it's
been downloaded over four thousand nine hundred
times i think that's fantastic
yeah and this is there more
abstract or review of our application this
is more about how pumping
that
so if inconveniences were were you haven't
called a pod and a pulled can
run multiple containers
and you can have a different pause
the asked
so for example if you want to
run the core archive job as sn
the presentation before me and
he mentioned the core archive in there
if you go into the server and
you used to say like to he
can run the archiver command and
in a cronjob
in kubernetes this world a this is
called a
they're a job
just a job there
so
and because of the high load or
the millions of page views every month
and not a single corker care jobs
wasn't enough for us so we tried
to come up with different ways of
running this which has led into
other problems too and we have learned
a lot during the years so
and we have had discussions with the
moto moto him to how to solve
this properly in the correct way and
yet i bet to have run into
similar issues like we have and the
but they have probably solved currently
yeah of course monitoring is important to
know when you need to scale up
or scale scale down your application so
of course with you sir
with communities is a very standard to
use the prometheus that will check the
the paris and we use grafana to
visualize are the metrics
which is another story
they're like kibana if you use to
keep an hour elastic stack
i said yeah
jumping into cabinet this was a lot
of fun but it's a
it's it's it's alot to begin to
because you need to know
about a lot of stuff so
fun
yeah
so just to give you a hint
of how much how many cases we're
dealing with here so i tweaked the
numbers are lopez so i am not
i don't want to give away their
real numbers but this is very like
the real deal
we have also other clients with even
more data than this so we know
what we're doing we have done this
for many years now
and if you look here the fiscal
declaration for the swedish texts
there's it's opens up in mars and
you can see there's a peak because
everyone wants to go inland to see
their tax returns if they're getting anything
i still have to pay more taxes
so this is something we can prepare
for and we can scale up our
services
we can also under the
at under the hood we can also
add more
servers are them as hardware if you
want so that's possible too so when
but during migration and that's something i
would not recommend for him once said
if you want been a busy metonymy
skin and so you have enough hardware
in place because doing a my migration
is a tough project
and so here's a little peek from
one of our town where we override
the values in their home shirts so
this is something where we can prepare
like is the fiscal declaration is coming
up in his opening we know that
we can we can add more replica
us
to the dashboard or to the tracker
pods
and if behind the red lion is
yours of the constant one of the
console commands this is also one of
the challenges we have because we couldn't
at the time we couldn't use a
rewrite many an nfl folder and so
we had to come up with
the idea how to regenerate
tag manager containers
so
a way you can do that in
the cabinet has set up mr run
the command so this is where the
console commands are so important for us
yeah we can also add more cpu
more memory to each pod or group
of pods yeah i wouldn't recommend to
do it individually per pod but you
can learn yoga
the basic template and then he can
tweak that for a different services
we have one that is sad tracking
year
the tracking data is sent into i
read the middle layer where it's stores
because the if the database is too
busy we need to keep the data
so it doesn't disappear or take down
the database so this is how we
can
the cure tracking service
and is this a peek at the
name we would only use satellite tools
but now the bigger we get and
the bigger the team gets we need
to ultimate level stuff this is a
dream or i see i see the
workflows
using the tools there and there's called
std which i think is really awesome
also open source of course
where we can now go in and
see all the parts in a nice
graphic view arrow and can check logs
or we can see the health status
and greenheart means that everything is operational
and running late you see the magnitude
of the pods are the microservices
there is running in the background and
yeah and it is an attitude that
i like that i've been using a
lot through the years he said jay
nice and it's a nice cla visualization
tool to see the health and status
of your cluster
so i can highly recommend to use
canines if you're more into sitting in
a terminal than having a graphical overview
yeah so the challenges we had through
the years was the db configuration and
then jumping into trying to debug a
black box from the beginning to find
as throttles or bottlenecks in our application
yeah it was sir
yeah some of the ways we do
now we have the added more
just shared all of those to the
community yet because they are not
ready for taste i would say but
we get enough of data to to
find if if we went around the
moscow journal to check the health of
the database we will have automated a
lot of stops to do that mothers
and from the beginning maria db being
opened
source company and the fact that the
mice girl wasn't no longer open sorcery
was open-source but kind of
because they were acquired by oracle
we chose maria db and when matomo
added more features
two thousand and nineteen i think to
run big madame installations they wouldn't support
maria db and that was a very
frustrating sometimes i did a patch so
i can do stuff now today though
we couldn't do a maria
he'd be back back then
and then of course the core archives
are having a lot of data and
then
one of my colleagues tomas is going
to talk more about some of the
challenges we had had or how how
you can break metal i think his
session name is the title of his
session
i would recommend to join his sessions
because sir
yeah he he will talk more in
detail how how to break puerto rico's
and there we problems from the beginning
with forms that they would automatically include
in every
now recite that you put up in
there matomo
and that gave us a little bit
of headache mo
we have dealt with that
fixed
and the segments go hand in hand
with core archives i would say because
the motto more features in the u
why
and having a lot of analysts analysts
working there and morne putting up their
own segments with squares that are
they're getting a lot of data from
the database could potentially take down your
application
and of course the transitions
because it's squaring their lives
and add indexes to the database and
yeah also i don't think you think
we still have some problem with the
transitions
heatmaps session recordings i would say it
works very well very well sometimes but
sometimes it's it could spam the logs
or we have one of our clients
is using a
a network monitoring series that will
filled with the traffic that it will
detect as a spammer or a denial
of service attacks or whatever so this
is something we didn't foresee and when
one of the class activated heatmaps and
session recordings
so that was interesting
so a quick recap because i think
my time is running out here
yeah so we use docker toma has
a a docker template if you go
into the official matomo github webpage you
can search for their docker
we have were of course hosting our
own docker template but i would say
that's a good start if you want
to run but only in a docker
set up in a cluster environment
and the yeah cuban at this i
have talked about a lot any questions
feel free letter that says i have
ten minutes have to
yeah drawn to automate stuff when we
push our code into guitar or gitlab
or github is actually running their own
seo service but drawing is something you
can integrate with your github repos and
then clear to check for vulnerabilities in
your in your images
in your docker images
github gitlab i mention
there's probably something i have forgotten
but i think you get the big
picture
and he was an interesting yearning i
would say
and i will leave room for questions
and i will see if i can
open up the chats here
mckay is fifty
hey jorge i've shared some of the
questions here in the chat
yeah yeah
from valentine inertia thanks for contributing to
the source with this ripple and you're
welcome
question have you ever used maternal fake
visitor plugin in order to have enough
data within your maternal death in order
to make some crash tests do you
think he makes some crash test with
empty automall database the idea would be
to educate ourselves about system administration
ah yes i have used a fake
research plugin and yes i wouldn't say
that it's the best option to test
a big dataset because it takes a
very long time to
i get enough of the fake visitors
so some of the annotation tests are
the love the band we have
actually had taken a big
yeah
dump of food of course with them
from production to because we want to
have a slyke production as possible
but i would say there's still we
haven't figured out the
the perfect way to do this thing
if i could give an advice
yeah there's some
there's a application i can remember the
name right now this used a lot
too at low test the varnish for
example
i can if it if your ping
me i talk to you
in linkedin or twitter i will share
to my
links later
okay question from her better with millions
of page views how big is your
database have you ever met issues regarding
the size of your db and the
archived process so as to be the
case i would say
eyes noise
i would say around it steady around
eight hundred gigabytes or one terabyte i
would say it goes up and down
when we purchased sunday data sometimes they're
after and our carvings are down some
man some of our
clients have different requirements for example that
they want they want to
do some are acquiring to the database
one year after of course that database
will grow
very big but the class only store
for one hundred and eighty days so
we can start to if delete old
logs after one hundred days and that
saves more if that's better for the
application of levity
and you can also integrate matomo to
a data lake
a duke hadoop cluster or
kafka or something if you want to
be serious or superset it's something we
have been looking into
if you want to do more pivoting
of the dataset some asking big queries
to the database and so yeah and
the archiving process yeah we have had
a lot of issues i think we
have if you want to follow our
issues we have had
through the years you can go into
the moto mais q and he can
search for a correr kaveri we have
had but
i think we have reached a level
of acceptance now and one of the
challenges i think thomas is going to
talk more about and it's about educating
your analysts how to build segments
what else the forms you have goals
and all those different plugins or applications
or the niece or the aura some
kind of archiving and because if you
do too big query on a big
day
dataset with all type of monolithic good
database system
i already know what is it called
a tabular database like my fellow maria
db and you you would eventually hit
some problems or bottlenecks with us you
can always optimize the conflagration but i
think if you want to do big
queries
on
database you will need to integrate with
some other systems like superset that i
just mentioned
one
what else
perhaps i missed a number of events
and scattered i get the figures showed
were purged page views right yes those
correct
ah
i dunno in from the top of
my head i dunno their lands but
a sense because sir
and they are using the tag manager
as much as others some have utter
of plants that uses a lot of
single page applications
then many it doesn't differentiate that much
of course there's a there's more events
of course i could probably look the
him in a minute
i can jump to the next question
is maria db not supporting maternal or
isn't recommended for large installations at metal
yeah so what didn't
the supporting of my maria db am
was it two years ago and
i would say you can still install
with maria db of corridors for small
sites but for large installations if you
want to use some configuration such as
sir
there are some nice girl specific
configurations though not always using
and i think my time is up
there and but whereas you can reach
out and somehow i will share my
slice if you want to get in
touch with me or ask more questions
and i will be more than happy
to share them with you
so thank you for your time i
thank you for listening and thank you
for the madame kept him awesome