Continuous integration. Who here uses continuous integration?
Hopefully I'm going to see lots of hands.
Excellent, lots of hands. I had hoped for a few more, but we'll run with it.
Every repository should always have continuous integration of some form or another.
It's the only way to make sure that your project builds outside of your own
machine, because you could have all sorts of weird things on your machine and
there are all sorts of reasons why it might build on your machine and not build
on other people's machines. So CI helps stop this.
It will compile, test your code on every single change that's ever made to your
code, which is a good thing.
It catches problems before they snowball and later on blow up in your face.
You can catch bugs and stop all sorts of other things that you don't want to happen from happening.
CI, the way we have it done, is always done in a clean and reproducible environment.
Every single Linux build that's run, every single Windows build that's run,
every single free BSD build that's run, every single Android build that's run,
all happens in a clean ephemeral Docker container.
There is no such thing as an environment that is dirty.
And most importantly of all, CI should never be limited to just one kind of code.
You can do it for not just C++, but you can lint your YAML, verify your JSON.
You can verify all your Python to make sure that it's following best practices.
There's lots of tools out there for this.
It's even implemented in some of our own repositories. And in fact,
it's used by the CI notary service service that we have
that we'll cover off later and there are also tools to
validate your licensing to make sure that you're complying with the necessary
kind of standards using proper open source licensing have the necessary comments
and tags on there to verify that this code is licensed in this fashion so that
you always know what particular license a bit of code is using.
Generated documentation is accepted yes setting up continuous integration in a gitlab setting,
it's done through your gitlab ci.yml file and it's held at the top end of your
repository for the given branch that you're currently in different branches
can have different ci setups,
if necessary where possible please use our templates that we've kind of already
got set up for you no need to roll your own we already have extensive tooling in place that lets you,
get started with essentially no cost involved.
You just add a little include setup like we've got here on screen, and you're good to go.
If you do have special requirements, feel free to roll your own.
GitLab has extensive documentation on how to do this. It's outside the scope
of this talk though, because as a general rule, we recommend using the templates
that are currently kind of rolled out and ready for everyone to use.
And those templates cover everything everything from FreeBSD,
Android, Windows, as well as two different kinds of Linux, glibc and Musil.
And there's also static builds available, I believe, for Linux and Windows.
But don't quote me on that.
There's quite a bit there.
You might say okay i've included those templates but how do i get my project
dependencies available.
For this there's two different ways of doing
it depending on the kind of dependency you are dealing with if
it's a non-kde project you will need to make sure it's included within the system
images that we have those system images those docker containers are maintained
in a central repository that we have at sysadmin ci images it's a bunch of docker
files if you need to add something,
you can send a merge request there, we'll get it merged in, the image will get
rebuilt and then that dependency will be available for all projects to use.
Relatively clear, simple and straightforward way of getting something added in for everyone to use.
And then there's internal dependencies. We have that specified in our kde.ci yml files.
Those let you depend on given even
dependencies of other KDE projects in a relatively straightforward way.
And you can also do it on a platform by platform basis, because sometimes there's
varying reasons why you might want to, say, pull in Network Manager on Linux and FreeBSD.
But for your Windows builds or your Android builds, Network Manager is not there. It means nothing.
You don't depend on it or use it there. So you can just not have it available.
Makes for really easy ways of doing things.
There are also ways of doing bulk rebuilds of dependencies.
For those of you who are interested in bringing along a new platform or whenever
we upgrade a platform that's particularly used for cute version updates.
We have something called seed jobs that essentially go rebuild all these projects
for a given branch at once.
The job just methodically moves through them one by one by one by one by one
and you get an up-to-date package registry. Thank you.
So, besides the dependencies, there's a few other things that can be configured in the KDCI YAML file.
The probably most important setting, or the most widely used setting,
is making passing unit tests mandatory.
If you would be starting from scratch nowadays, that would obviously be the default.
But from where we are coming from historically, that wasn't an option.
So whenever you have passing tests, reliably passing tests, you want to enable
that. That's a setting per platform.
Not looking at any platform in particular, but some platforms are more challenging than others.
So you might only be able to do that for, say, Linux or something like that.
Another common setting in there that you might need is passing custom CMake
options to your build. So if you have...
Say some demos or examples included in your repository that aren't usually built
that you need to enable explicitly then it of course makes sense to enable that on the ci as well,
most of the other settings are somewhat more exotic so if you happen to write
a valent compositor rather than a normal app they might be relevant you find
all the the settings and and the defaults for them in the CI Utilities repository,
which includes the global defaults. That's the one shown on the right.
That gives you an idea on what you can already configure in there.
And then the same repository contains the Python scripts driving all of this.
So I mean, this isn't hugely complicated.
So if you need something more, this is all extendable.
And then we get to the second big part of all of this, and that is continuous delivery.
So for us, this means building fully functional application packages.
For the CI, we said every repository should have this.
For CD, every application repository should have this.
We support pretty much the same platforms as well as for the CI,
so we have several flavors of Linux, Windows, macOS, and Android.
We'll see a few more details in the following slides.
On platforms where that's necessary, we also have signing and publishing,
the signing of the packages and publishing to repositories or software stores.
The setup is for this slightly more complicated than for CI,
because we not just need to know how to build something,
we also need to know exactly which dependencies to bundle, we need to know what
it is we are actually building there, because that package ends up being user visible.
But the first step is basically the same as for CI.
We have predefined templates for all the supported platforms.
You include them into your GitLab YAML file,
and then you see a second set of jobs showing up in your build pipeline,
which will run after your CI pipeline pipeline passed, which then builds the packages.
And on platforms where that's necessary, you have a third stage that is then
signing and publishing.
By designing and publishing, this is somewhat security-sensitive,
because this is using KDE's official package signing key.
And the package is then sent to a repository that immediately ends up in updates on users' devices.
So that needs to be enabled in the CI Utilities repository.
In addition, there's YAML files
per platform where you select which branch should get this treatment.
Another thing we need for building packages is application metadata.
As I said, this is now the package is user-visible, right?
So we need to actually describe what it is that we are producing there um carl
touched it on on this in in his talk earlier today already,
this is all built on top of the upstream metadata file um i think most applications
have that already it's a big xml file where you enter a description of your
application screenshots screenshots,
the license information,
where to find the source code, release notes, all of that.
And all of that hooked up to the translation system.
And this is kind of our canonical source of choice for that data.
We generate websites out of that.
Flathub is consuming this directly. And for the Google and Microsoft stores,
we are converting that in whatever format is necessary there.
Okay, and then we get to Craft.
So by now you probably already have heard about Craft. So Craft is basically its own distribution.
It supports Windows, Linux and Mac.
Recently we got additional support for Android and we use it in our CD to provide
the packages for those three platforms, with the exception of Snap and Flatpak,
which are not yet craft compatible.
So craft works by building everything from scratch, but we also have a binary cache,
so that enables us to use it on a system like a CD because we can download the
pre-built binaries and don't have to recompile them every single time.
So instead of days, your job runs just for a couple of minutes.
There is craft-users-blueprints, and blueprints are basically a recipe how to build the application.
So you specify the location where you get your source code from,
the dependencies, and which build system to use.
Usually you don't have to tweak anything in there, so you just provide build
arguments that you need.
You can also modify the install step or things like that but usually you don't
need to do that because we build most of the things around it.
If you still have to change something it's probably fault in your CMake file
so you should try to follow the standard there.
Also,
what you should never do is... Sorry, I totally lost my thread.
Yeah, so another thing is what you should not do is install some files from other projects.
So try to keep your CMakeList clean, don't install third-party tools yourself.
Craft will take care of that.
The blueprints you'll find on Invent. You can also have third-party repositories,
so you can have your own collection of blueprints if you want to.
And on the CI you can configure how how Craft works a bit deeper with the Craft.ini,
where you can specify special versions of third-party libraries or special configurations
of third-party libraries.
You can do it, but you should never depend on Frameworks Master because that
will eliminate the use of the cache and the job will just take hours.
The last thing is the ignore list you can have also in your repository by default
we package everything and with the ignore list you can say I don't need the
documentation of OpenSSL in my deployment so you can strip it.
We made the experience that it's way better to remove files than just include
specific files because then you usually miss some plugins or whatever.
On Linux, Craft uses the app image.
You probably all know what it is. In difference to Flatpak or Sandbox,
you just have everything packaged into one single file.
And it's very easy for the developer because they basically have to change nothing.
And it's also quite simple for the user because it's just one file you run.
So no modifications needed, but the downside, it's just some random binary from the internet.
Our images run on most recent systems. It's all limited by ellipse C.
So if your system is older than our build system, we are sorry,
but we can't support your platform.
On Windows, we support the Windows Store, which is currently opt-in,
so you have to be in a specific config file managed by sysadmins.
But after an initial setup of your application in the Windows Store,
it will also be just automatically pushed to the store, and then just the maintainer
has to click the Publish button and you're ready to go.
So beside the Windows door, we also support the classic Windows installer, in our case, ANSYS.
It's just an executable you run and every Windows user will know how to use it.
And last but not least, we have a simple archive. This is just one file,
everything compressed into one file.
You have to extract it, but you can also run those on systems where you don't
have administrative access. So instead of installing it, you can just run it.
That's me again so on android we support building apks so that's the thing you
install on your phone and aab packages that's the thing you need to submit to the google store,
and we have two different publishing jobs the one you definitely want is the
one publishing to to KDE's own F-Droid repositories.
So those are software repositories similar to what you might know from Linux
distributions. You add those to your F-Droid client on the phone.
And when your build pushes an update to that repository, it appears on your phone.
In the usual configuration, that means that your stable branch ends up in the
stable repository, as one would expect. correct, the master builds end up in
the nightly repository.
That's a bit of a misnomer nowadays because it's actually continuous.
The nightly is a leftover from Jenkins times.
And that actually gives you a very quick turnaround time.
So from a push to the master branch to a new update on your phone is a matter of minutes.
And that's not just useful for yourself in itinerary.
This basically gave rise to a completely new type of contribution
that's people running bleeding edge cd
builds in production and that gave
us a huge boost in qa coverage because you get bug reports that are basically
really the latest state you make a change and you get immediate feedback that
you improved it or you fixed it right so i i certainly don't want to miss that anymore.
The other publishing job we have is for the Google Play Store.
That is at best semi-automatic, I would say.
Let's emphasize on semi, not on automatic, which isn't really our fault,
but that is due to all the paperwork and manual procedures you need to do in the Play Store.
So it supports your publishing to the Play Store, but if you want to do that,
that is still involving quite some manual work.
And then we get to macOS, right? Yeah.
So on macOS, we have two different architectures, which we also separate.
So there are two different binary caches.
One is the older Intel architecture, and the other one is ARM.
What we do not have yet is the universal FET binary.
And I think we don't plan to support it yet.
What we get on macOS is DMG bundles, which you can just download and then by
drag and drop easily install.
To get that, you need to have a plist file in your application repository.
You can either use a custom one or you can use some CMake magic to generate
the built-in CMake plist file.
And this is important because it contains the information about the app identity,
which is then used by the signing service to verify it's the correct application which got signed.
Mac OS is a bit special because it's Apple.
And for Apple, you need actual Apple hardware to build and test things.
So that's kind of a limitation, which also hits us on our official server systems.
Because right now we only have one machine, one Apple machine to run everything for macOS.
So you might experience some longer CI queue times when you're on this.
And we are cross-compiling all Intel binaries on an ARM machine.
Another special thing about macOS is the notarization. You can see this nice message over there.
This message appears even if your binary is signed, but not notarized.
Notarization means that in addition to signing,
your binary or your app bundle is sent via an API to Apple servers and they
run some checks and then you will get a certificate which you can include in your bundle.
And only if you did this, this message will appear.
There is a possibility to work it around, but it's not very user-friendly.
Okay, so then we are leaving Craft now and coming to Flatpak.
Flatpak is probably the most or the best imported format of all formats we presented so far.
And there's also a nice KDE team, very active, which will help you in case of questions.
To get Flatpak working, you have to add a Flatpak manifest.json file.
You can also add a YAML file, but JSON is used in most KDE apps.
And one important thing about this is that this manifest is only for nightly Flatpak builds.
For the stable Flatpak versions, they kind of duplicate in a Git repository
in the Flathub namespace.
So that's not that perfect, but I think in the future we might be able to get
rid of this duplication.
For Flatpaks, as I said, there's this JSON manifest. manifest,
Flatpak has a runtime which includes all Qt and KDE frameworks.
So at least this part is shared between apps and in your Flatpak manifest,
you only have to add specific dependencies for your app.
One nice thing about Flatpak Nightly is that you can just download the Flatpak
ref file, which is published on our CDN,
and then can install it via your software center, like, for example, Discover,
and then you will also get automatic updates on every comment that gets built from your repository.
Yeah, so I guess that's about Flatpaks. We also have Snaps, which is another huge topic.
We won't go into detail there. I just recommend to join Kevin's talk later,
I think after the coffee break in this room.
Thank you. And now finally, a couple of other things.
The cd system is not just capable
of building your application binaries the cd
system is quite generic and it can do anything that
essentially requires privilege on the ci workers or the cd workers among those
also includes publishing packages to the python package registry pi pi and it
is also what drives the deployment of all the static websites that kd has on our infrastructure.
Are you seeing a Hugo site, a Jekyll site, Sphinx site, any of those sites,
they're all published using the same continuous deployment infrastructure.
And on the continuous integration side, we also have a bit of things for energy
consumption measurement that's
been mentioned in some of the talks earlier this weekend. And there's a.
That being said, with all the things we've talked about, we're still not complete.
There are still items that are on our wish list. There are still things we were looking to improve.
Still things that we can do better at.
At the moment, we don't have GUI tests on Windows because Windows Docker images
don't support any of the Windows GUI stack. It's completely missing.
The libraries are there, but that's all that's there. the actual components
in Windows that those libraries leverage are completely broken and don't work.
We'd also like to be able to run unit tests on Android. At the moment Android's
completely cross-compiled and you can't really, not without awful emulation
anyway, run unit tests on Android and we would need to spin up a whole Android
stack as well so there's a bit of work to be done there.
Automated testing of continuous deployment packages would be
nice as well so we can validate that we actually have binaries that
can be run rather than just build successfully someone
downloads it and finds oh you're missing this library your
builders are some other all the way corrupt we'd
like to get some kind of tidy up on that at the moment those
of you who are familiar and trying to look at things on a kind of broader
kd level view or a release unit view
will know that it's essentially impossible not without
some awful scripting anyway to track where we're
at in terms of the status so we'd like to get a bit of a status overview
a bit of a dashboard in terms of where we're at and we'd
also like to look at publishing to more places in particular
as julius kind of mentioned just earlier flathub kind
of stands out massively there because at the moment we build flatpaks ourselves
but then have to go update it in a separate place to run through flathub's infrastructure
to get published on flathub.org which doesn't make much sense in a world where
for the Windows Store, we can publish ourselves.
For the Play Store, we can publish ourselves. But on Linux, we can't publish
ourselves. So we should probably look at solving that.
So kind of in conclusion, since we've moved to GitLab away from Jenkins,
things have got a lot more accessible for everyone. You can now have custom jobs.
It's a lot easier to register jobs yourself.
And life in general has gotten a lot better, easier and nicer and more integrated
rather than being on a separate system on a separate site that you only have so much control over.
Things for the most part can be done by a merge
request or directly yourselves in your own repository including in your own
forks you won't get any of the signing automatic uploads or anything like that
whilst you're in your own forks that only works on the mainline repositories
it's been enabled for but for the most part you're free to do whatever you'd like in your own forks,
and for those of you who have additional questions would like to speak to us
more in depth on something we have a bof session on tuesday at three o'clock
in the afternoon known in room three, I believe.
And with that, I'll throw the room open to questions.
Thanks for the presentation. Question about how you are managing changes to the Docker images.
You said that you have a special place for the declarations of what's in those images.
When the Docker image changes, do you test everything that it actually didn't
break stuff when you upgraded, let's say, to the latest Qt version?
So with an upgrade to a cute version we
normally version that one a lot more strongly that's not
something that we just jump into the linux docker
images are based on openSUSE tumbleweed and
we use a specific pinned repository that targets a
specific cute major minor version so
there can't be just a silent upgrade underneath our feet and suddenly
you've moved to 6.7 from 6.6
we keep that one more tightly controlled but for
other changes there's only so much testing we can do
generally someone's adding a library here adding a
library there whatever the case may be adding this
tool that they need for their tool chain there we generally just merge those
build them and then if it's been a while since there's been a build we'll trigger
a seed job the nature of tumbleweed and some of the other distributions that
are in play being rolling distributions means there's likely to be binary compatibility
breaks so yeah to answer the question there's not really too much in the way
of checks we do but we do our best to minimize breakage where possible.
Yeah but i think there is um we we have the mechanisms to do uh compile time
testing where the seat drops indirectly right what we are lacking uh and that is indeed in a problem.
Um running unit tests on all the frameworks against an upgraded qt for example
right so this is indirectly done via the albert job so albert triggers once
a week a rebuild of everything thing and then we see the unit test errors.
So, but we don't have a way to actually see iGate changes to things lower in the stack.
And not just for the CI images, right? We have a similar problem with craft here.
I noticed there was an enable load-sensitive option in the CI config.
How do you deal with tests that are flaky because of various CI issues such as load?
Flaky tests are a bit of a tricky one. Generally, we put that in the hands of
the developer to deal with.
I'm hoping that it's not a case of the ci system itself being flaky there is
a parameter in one of the those that you would have seen the configuration file
there called load sensitive,
that was introduced for certain projects whose tests were very latency sensitive
and when the ci nodes were busy those tests would end up failing nine times
out of ten because the round trips they were trying to do between multiple processes
just weren't completing fast enough because the The processor was busy compiling
someone else's project.
Yeah. I mean, unlike Qt, I don't think we have a special support for flaky tests.
Uh, and I would say most of the tests that are flaky, and I mean,
we definitely have that problem.
Um, those are flaky everywhere, right? So it's, it's not really specific to the CR.
Uh, do you have the ability to test with, uh, devices connected like cameras
or microphones and things?
Not at this time. At the moment, everything's in Docker images and Docker makes
making devices available somewhat not straightforward.
Also, all the servers doing these builds are sitting in a Hetzner data center in Finland somewhere.
We can't just go attach random hardware to these things.
We'll be moving towards VM-based builds at some point, at which point we can
probably tell the VM, please attach this random piece of hardware.
And so at least you might have a chance of doing something but you won't be
able to probably say I want this particular kind of data feed to come through
this particular hardware device.
I have a question about examples of successful configured projects with unit
tests, functional tests, Selenium tests, and so on.
So you have a list of favorite projects that we can take as an example to build
our own configurations or just get the random project.
I think we don't have a specific list of best practice examples.
In general, I would say the new IDE application is, so basically everything
that Carl started is basically following the current best practice.
So they have passing tests, they have Appium tests and have all the CI set up in a way it should be.
The older it gets, the more or it deviates from best practice, I would say.
But maybe it might make sense to have a few showcases that actually use the
full thing, like all platforms, all types of tests that we support. Yeah.
So up here, KDE Connect, we have native Android and iOS apps.
I know we are weird in this. but how much
could we use the the cd that
we already have to maybe sign and
notarize the ios app which i mean shouldn't be so different from the mac bills
right um i think the signing and notarization is not specific to how we built
right so feeding your apk however it is built into to that system should be the same, right?
We indeed, you are special in that sense, and we have no infrastructure for that yet.
But I think at least the later stages should be applicable in the same way.
It should be relatively straightforward to tie that in. It will be you have
to do a bit of extra custom stuff to say, please invoke the notary service to sign my application.
But that shouldn't be a problem. The notary service doesn't care how it was
built. It just cares that you're giving it an APK.
And the same with publishing to Google Play. Maybe we can sit together and solve it. Thanks.
Hi, I would like to know if there's any observability taking into account into
the CICD as in statistics statistics or metrics on which pipelines are failing, how long they take,
how often they fail, all this kind of thing.
Data is probably recorded within gitlab itself but at the moment we don't really
have a way of surfacing that information you could probably pull it out using
graphql but that may or may not be too helpful because we've had some experience
working with graphql in the past with gitlab and the results haven't always
been too perfect it's needed some cleaning up,
the visualizations you get out aren't the best but that's something we could
look at the data will be there because gitlab keeps everything for eternity
unless the project is deleted.
Running tests and other CI jobs require a lot of resources.
So now it uses the infrastructure, right?
So maybe you can provide the ability to contribute our local resources to run GitLab on my server,
for example, and give the ability to decrease the usage of the cloud services
by my resources that I just have but don't use.
So you're talking about running your own GitLab instance or? Or GitLab runner.
So your GitLab instance will use my GitLab runner on my server.
So I contribute my resources to setup.
This is something that you can fully do within your own forks,
within your own personal repositories. You can just go in there.
It has an option to register a runner. It will tell you what you need to do to set that up and do it.
But within the mainline repositories, at the moment, we haven't really made
a decision on that, but chances are we'll probably want to keep that to,
for just reliability purposes alone, keep that to just the general ones that
a sysadmin community maintained, those particular ones.
But for your own personal repositories, you can definitely feel free to use
your own ones as just a case of attaching the runner and making sure that it
has the necessary environment that matches what our jobs expect,
which is a rootless podmin setup.
So possibly a very stupid question, but we have Craft, we have KDE Builder,
we have KDE Source Build.
Can Craft do all these things or maybe KDE Builder or something else?
Do we really need three ways of building the whole thing automatically?
So why Craft? So why we have KDE SRC?
Craft tries to be something different
in a way so it concentrates more
on creating a deployment
in the end so you have an installer that can be distributed and it also has
a focus on reusing this cache so it's not for everyday development or not necessarily for that,
but for creating those installers on all the platforms.
There is a valid point in there, though. It might be worth investigating if
we can generate the Flatpak manifest and the Snap manifest out of the Craft
manifest because that is duplication for the same purpose, essentially.
But I agree, the KDE source build has a different use case.
I understand we only have two minutes left, any final questions, last lucky last?
Seems we've answered all the questions. Thank you everyone.