Welcome in this session of MatomoCamp.
Hey everyone, we are welcoming Dinamo today, who is a recently graduated computer scientist
working as a developer and business analyst at open source politics.
Today he will tell us more about his integration of Matomo and Metabase.
Welcome Dinamo and looking forward to your talk.
Thanks Sylvia, sorry.
So good morning everyone.
So I'll briefly talk about Metamo and another software we are using currently, which are
Metabase, which is a business analytics tool.
I will introduce further in my presentation.
And basically the purpose of this presentation is to show you how we can improve visualizations
coming from Matomo.
So let's get started.
I first introduce many topics, among them who we are at my company, so open source politics.
The presentation of also the Metabase software, I'll do it shortly.
And in second time, I go through the work we have done on Metabase and the link between
Metabase and Metamo we have done.
And finally, I'll give some insights about what can be improved, what are the next developments
of what we have done, and answers to your questions if you have them.
So let's go first by introducing who we are.
So I'm currently working at a small company called Open Source Politics.
We are part of an ecosystem called the Civic Tech companies.
So basically, Civic Techs are companies that want to bridge the gap between democracy and
digital tools.
So for that, we are working around software called Decidim.
I will present it shortly after.
And as the name of our company suggested, we're aiming at working fully on open source
software, which brought us to Matomo.
So let's get started with the first demonstration of the software we are using.
So here we are.
So Decidim was a software created in 2015 by the City Council of Barcelona.
And as I said, the purpose of that type of software is to bridge the gap between citizen
asking for more democracy, for more possibility to take part in the decision making.
And the fact that currently we are more and more in a digital era, with everybody having
a smartphone on them.
So there is new ways of participation.
It can give new ways of new possibilities for citizens to take part in the democratic
life, not only for elections, they want to take part as more, just not for elections.
So to give you some examples of implementations of Decidim, there is obviously the City Council
of Barcelona, but I will give some instance we have made at Open Source Politics.
So this first platform you see here is Decidim we installed one and a half year ago at New
York City.
So it's not a platform aimed for the whole city of New York, but more for a small district,
for a school on which they wanted to organize their participatory budget.
So basically, they have an amount of money.
And with that amount of money, they make proposals about what we can do with that money.
So creating new spaces, buying some stuff, so on and so forth.
And the second step is to analyze if these proposals respect the criteria and finally
implement them according to the rules.
Another example, a bigger one is the city of Angers in France, which is about 100,000
inhabitants.
And since four years, they are organizing also their participatory budgets.
So again, a certain amount of money organized every year, so I said it, and people are making
proposals about urbanistic culture, so on and so forth.
So here you see the one made in 2021.
And right now, we are in the final steps of releasing the results of that participatory
budget.
And as I said, so we at Open Source Politics are implementing those platforms, and we wanted
to work from the beginning with open source software.
So if you go on any of our platform, you can see the famous Matomo snippet code here.
And we have currently about 100 instances of Decidim using Matomo.
So that is for Decidim.
And you might say right now, what is the link with Metabase?
We had that problem that we have many different data sources.
We had data from Decidim, so basically, proposals.
What are the main topics which are talked about?
Are people commenting about, is there a lot of proposals?
Is there a lot of people registered on the platform?
It's helping, and therefore, city councils, but also organizations or companies that are
implementing Decidim to know if the participation is going well.
Then we had that other tool from the beginning, which is Matomo, which tracked more roughly
the number of visitors, what are the pages which are visited, who are the visitors coming
from, so on and so forth.
And we had no centralizing tool to brought all those data at the same place.
And to also offer like an overview without having to go to Decidim, then having to go
to Matomo, then having to go for like the mailing platform to see all those, we have
all those data, but are separated.
And here came Metabase.
So Metabase is basically to power BI and tableau what Matomo is for Google Analytics.
So it's one of the main open source alternative to make business analytics.
So centralizing all your database in a single place and offer really talkative visualizations
that are simple, efficient to communicate for city councils or organizations.
So after describing it, I will just show it because it's again easier.
So here you have the main page of Metabase, and I'll show you what the work we have done
for a platform called the Agence Nationale Pour la Coalition des Territoires, or shortly
ANCT.
So here you have the Decidim instance.
You have, again, the Matomo snippet we put it here.
And on this Metabase instance, we had different dashboards, which are summing up data from
that platform, but also from Matomo.
So here you have data from Matomo.
So basically number of visits, average time spent on the platform, referrals, what were
the pages who were viewed.
And on the same page below, you have data coming from Decidim.
So the number of users, number of proposals who were made, number of people who were subscribed,
what are the main topics discussed here.
So you have culture, sports, so on and so forth.
And the main proposals who are the most voted.
So as you see, centralizing all data in the same places, and offer really talkative and
easy visualizations to communicate.
So viewing the results like that seems really easy, but obviously it's not.
And I will right now present the work we have done around those two softwares to bring them
back together.
So first, what must be known is that Metabase is really easy to use to add new databases.
I'll share my screen again.
Here you have the admin dashboard, the admin board, sorry, to add new databases to your
Metabase instance.
And as you see, you have here what we call drivers.
So basically endpoint, which helps you to easily add new database sources.
So you have the one like everybody knows, MySQL, Postgres, SQLite.
And there is also a Google Analytics driver, but as you see, there is not a Matomo driver,
not yet.
And that was the first job to do.
The job was to bring, because there is no drivers for Matomo in Metabase, you had to
bring back those data to Metabase in any sort of database, which is accepted by Metabase.
So SQLite, Postgres, MySQL, so on and so forth.
The second problem, which I met as a newbie in Matomo, is that the MySQL database is not
so user friendly.
Like if you're a data analyst and you go through the Matomo databases, so it's databases which
is summing up all websites which are tracked.
So if you want to isolate one website, isolate those referrals, it's not so easy.
But there is something which is really great, which is the API of Matomo.
If you go through the API for the one who knows, I'll just go on the instance of Matomo
for the government, the one of which we gathered data from ANCT.
So here we are.
So that is talkative.
That is really interesting because without knowing anything of the Matomo database, I
can see the number of visits, the referrals here.
And in one click, I can see what is the API endpoint to get those data.
So that was the path we have chosen at Open Source Politics is that we didn't have the
skills to develop specific Matomo drivers.
But we had to do this API which is really well built.
So we started with this API endpoint, and we finished with that scheme of data.
So we have the Matomo instance, we have our Metabase instance, and we have created a small
script, not so small right now, called Matomo pool, which is, as its name suggests, something
which will pull the data from Matomo and bring them back to an external database.
In our case, it was Postgres, but you can imagine all type of database.
So you use the API endpoint, you pool the data you want, you put them back in the Postgres
databases.
And as we have shown previously, you can, with Metabase, add a Postgres database.
So easy peasy, as we said.
So let's get started with that script.
So here it is, I'll make it bigger.
So here you have the main architecture of this program I won't go through because it's
not the purpose of the presentation, but basically we created it as a container, so it can be
instantiated on whatever web host or you want.
And you just have to put here the necessary parameters you need.
So the base URL of your Matomo instance, the DB name of it, the ID site provided by Matomo,
the starting and ending date of the data you want.
And finally, really important, the authentication token provided by Matomo again to recover
the data.
So here I go.
And if I want to ping and get my data, it will gather following different API endpoints,
all data's coming from that instance.
So here the magic is a bit hidden so I can go through.
Here is the config.yaml file on which you see really the fact that we are basing this,
we were based on the API of Matomo.
So here you see some parameters given in the URL, which will be used in the URL.
If you go back to what I previously showed, it's basically what we have here.
So the parameters you have here are exactly the same as here.
And we are adding again, so it's really plug and play.
You can add a new one if you want or delete a new one following your needs.
And finally, when the script is over, you'll have all your data put in a Postgres databases.
So that's for the Matomobu script.
So if you want to participate on it and see it, I put the link on the presentation and
it's completely available and freely forkable of whatever you want on our main page.
So that's for Matomobu.
And I previously shown that thanks to it, we had those visualizations both of DCDM and
Matomo at the same place, which is the purpose of our work.
It was faster than I imagined, so I hope you have a lot of questions.
Through that work, we had a lot of questions which arose and I will go through them.
The first one was about GDPR compliance.
So as you know, Matomo is really ahead in this topic about respecting GDPR by design,
privacy by design.
But the problem created by this script, so Matomobu, and this dataflux is that you have
to make it safe.
So basically, the use of an external container, this Matomobu container, must be safe.
You have to use, in our case, it was a GSM token used to really restrict the access to
that container to people having a special key pass, passphrase, sorry.
There is also the fact that we have maybe more endpoints, which are great right now
and not available.
So I heard that in the new version of Matomo, there is the new endpoints were created.
So we had to improve this Matomobu script to take these endpoints into the script.
And finally, what would be the best is that without using that script, which needs to
be a bit of, to know a bit about development and having, using bash things and so on and
so forth, handling databases, external databases, it would be, the best would be to know how
to code enclosure, which is the language of Metabase, and to code enclosure.
The Matomo driver as the Google Analytics driver is currently coded for Metabase.
So right now, if I share again my screen, I'll go, so, okay, so I think it's over.
I have my Postgres databases and I can, so it's the database way ANCT.
So and I can dump it and recreate it on whatever Postgres remote database I have.
So that was again for the demonstration.
I was really fast, faster than I thought.
So if there is any questions or details I can show, I will gladly answer them.
Okay, I just saw them.
So question from Ronald, have you analyzed how they developed the connector from Google
Analytics at Metabase?
Yes, no, to be honest, I didn't take a look at it.
The problem is that the Google Analytics way of handling data is completely different from
the Matomo one.
To give an example, to request any database from Postgres to MySQL to SQLite, you have
a standard, which is the structured query language, SQL language.
And for Google Analytics, it's a JSON way of making requests.
So it's completely different and therefore I don't think it's applicable to Google Analytics.
Furthermore, there is, I tried to use that connector to Metabase, so using, sorry, to
use, to bring back, to use that Google Analytics driver and the authentications, the parameters
needed for Google Analytics are completely different from the Matomo.
For Matomo, I think you only need basically the ID site, the authentication token, and
that's over.
So I think it would be easier to develop that driver.
I do not code it, but in the future it would be really awesome.
I think it's easier, and it's something we can do in the, like, middle or long term at
Open Source Politics.
I'll take the next one.
What are the new features that Metabase developed over the last six months and or any new coming
nice features which would make the integration of Matomo easier?
So making the integration of Matomo easier, I said it.
It's coding a Matomo driver.
If you have the authentication token and the ID site, I think it's really 30 seconds to
do it.
And new features that Metabase developed over the last six months, they are really great.
I'll go through.
I went really short on the possibilities of Metabase, but you have the possibility to
send by mail all that, sorry, it's the possibility to send the entire dashboard by mail.
So here I am.
So if I click on subscription, I'll go through, there it is, and you can send to whatever
mail you want.
So if I put my personal mail as instance, and I can also join the results.
So here you have the visual results, but you can also can join the CSV sheets of it.
You can send it by mail.
There is also a good nice features about protecting the data, like you can create groups which
are restrained to that type of data or that dashboard.
There is also visualizations.
If I go to asking a question, if I go to that database, I just go and select all users.
That's to show you.
So here they are just showing them as basically a table, but here you have all different visualizations
as they are adding more and more.
So you have lines, pie charts, cloud points.
If I can show you, we recently have done something for the National French Assembly, and you
also have maps of you can add.
So here it is.
It's taking up to load because there is quite a number of results to analyze, but there
is maps.
There are, yes, new visualizations every release.
It's really an active software.
So here you have, so basically we, that our data coming from the CDEM, but I'm really
sure we can do the same for data coming from Matomo.
If you want to have more information about the rune map, there is like a release every
two months of Metabase.
You can go, I will put the link on the chat.
Next question.
How much work in terms of time would you estimate for the creation of a driver potentially cost
if it was you coding it?
So if it was me coding it, I think it would be a lot of money because I don't know how
to code in Clojure.
So I said it, but maybe some people don't know what it is.
So Clojure is a programming language, which is a derivative of Lisp and which is running
on the Java virtual machine.
I won't go through details, but it's really an interesting language.
I didn't looked well on it and I'm not really a fan of it.
I didn't have the time because my language was Python, so I went with Python.
I do not think it would be too difficult.
As I said previously, the Matomo analytics driver seems really more difficult to code
than the Matomo one.
So to answer your question, no, I don't know how much it will cost, but I think it's really
feasible if you find the right person in the Metabase community, which is a strong community,
as strong as the Matomo one, I think, because there is a new release every two months.
So I think it would be really feasible compared to other drivers who are currently developed.
Hi, how happy is your client is about Matomo within Metabase?
Really good question.
Basically, Metabase is not made for viewing everything you can see on Matomo.
The real first purpose is to have different data sources on the same place.
So if you want all data provided by Matomo, go on Matomo.
But if your client, like ours, is not really interested in viewing every day or every week,
how is going the participation, how is going the frequentation of the website?
I think Metabase is a great deal because you put it, the visualizations are really well
built.
I think they are more efficient, they are more talkative than the one provided by Matomo.
Even if the one of Matomo are great, I'm not here to have enemies.
But it's really efficient if you have other data coming from other data sources.
If you do not have them, I do not think it's worth the risk.
But for us, it was really a great deal.
So I said 100 instances of Decidim on which we have Matomo.
And right now, there is like a dozen of them, which are mainly new customers, which are
really happy having that centralization of data in the same place.
So I'll go through next question.
Have you also imported some visitors log raw data from Matomo into Metabase or are you
just working with aggregated data?
Yes, we are just working with aggregated data.
As I said, I'm quite a newbie in Matomo.
I didn't know how basically the database is built.
And when I tried to import the MySQL database of Matomo inside Metabase, really, I didn't
found anything, whereas with the API, it was really straightforward.
You had, you just had to provide the ID site and you had a table of visits and endpoints
for referrals and endpoints for pages and endpoints for everything, so on and so forth.
And it was really more easy to implement that script, Matomo pool, with the API.
But I'm really open to like new ways of implementing it.
I think, for example, if we had to code the Matomo driver, it would be more like efficient
to start again from the MySQL database and then recreate the aggregated data.
Because for data analysts who do not have the time to discover all the data scheme of
Matomo, it would be really easier to go through, to start from the API.
If I, to illustrate it, if I go here and I start with the Matomo database of ANCT, as
you see, it's really straightforward.
You have a table for every type and you have to imagine that every table you have here
is basically a visualization or an API endpoint available on Matomo.
And another question, have you seen any discrepancies in terms of data between Matomo and Metabase
data, typically the count of visits per day?
Yes, we have, no, sorry, I talked too fast.
The data are basically updated every 24 hours for us.
So yes, basically there is differences between the two data, the one we see on Matomo and
the one we see on Metabase, but it's perfectly normal since we are launching that container
once every 24 hours.
That's one, again, of the subject of having a Matomo driver.
If you have a Matomo driver, you can do the request every instantaneously.
If I go back to the admin board and I take, for example, a PostgreSQL database, you can
choose to execute automatically requests.
So here you have that button which basically say if you want to like having a temp database
on which Metabase work and synchronize that database every, I don't know, 24 hours.
And in that case, it's no, the database are directly the one who are available.
So there is no differences.
In our case, there is, but they are not too strong because we are updating the database
every day.
Have you considered other open source data visualization software?
Why did you finally choose to go for Metabase?
Yes, there is also Redash, which is an open source software for business analytics.
I didn't discover much that were as evolved as Metabase.
To be honest, I'm human, so I may have missed them.
The thing was that we already have, we had implemented Metabase at open source politics.
So it was available right now.
The thing is that also it was really active and that was one of the main argument which
pushed us to use it.
Also, the fact that it is more user friendly, so if I go back to the main dashboard and
I ask a question, even without being a data analyst, I can create visualizations.
So I'll go through it to show you.
So if I take like, for example, a Metamo database and I want to have the number of visitors,
so I choose the number of visits and I want to have the sum of number of visits and here
it is.
So it's really straightforward even for people who are not acquainted with databases.
So that was one, the two criteria was that it was already implemented, the community
was really active.
And second one, it was really appealing for consultants, people who are not data analysts
to work with and to take that tool to discover and to learn this tool.
But it's not over.
If you have any suggestion of any other tool which can be used for centralizing data and
sharing them, I'm completely open.
I think I went through all the questions.
Maybe what I can do is I will share all the links about visualizations.
I've shown you there is some public data.
So I've showed you ANCT because the data of ANCT are open.
But if you want to have more details about how we implemented it and what is our roadmap
about this Metamo pool script, I'll gladly answer them.
Maybe I can give you back the stage, Silva.
So we still have a little bit of time.
So if people have more questions, this is the moment to write them down.
Maybe waiting for the question.
I will go through all the pages and all the links I can provide.
So if you want to know more about the work we have done, so again, it's available here
on our main page, if I can take it back to the public chat.
So if you want to improve it or creating issues or pull requests, it's completely open.
There is also what we have done again.
If you want to know more about DCD, here it is.
And finally, more important, that are the main dashboard we started with.
So basically, the data here are completely open.
As I've shown you, it's available on the Metamo instance, which is completely open, given
by the French government.
So another great feature of Metabase that you can share easily.
So here we are.
And here are the public links.
So if you want people to work around it, it's really straightforward.
Now maybe go in deeper details, is there is any question, Silva?
No, there are no more questions, but we also have the metrics chat.
So you can also continue the conversation there if people want to ask more questions
later, basically.
So thank you very much for this very interesting session.
You're welcome.
Thank you for your time, and special thanks to Ronan and Lucas, and also you, Silva, for
helping us to do so.