Software from the Source

In this article, I am outlining an idea for an improved process of deploying software to Linux systems. It combined advantages of traditional, package mangement based systems with containerized software through systems such as Flatpak, Snap, or AppImage. An improved process allows us to make software deployment more efficient across the whole Free software community, have better supported software on users systems and allow for better quality at the same time.

Where we are going
Where we are going
In today’s Linux and Free software ecosystems, users usually receive all their software from one source. It usually means that software is well integrated with the system, can be tested in combination with each other and support comes from a single vendor. Compared to systems, in which single software packages are downloaded from their individual vendors and then installed manually, this has huge advantages, as it makes it easy to get updates for everything installed on your system with a single command. The base system and the software comes from the same hands and can be tested as a whole. This ease of upgrading is almost mind-boggling to people who are used to a Windows world, where you’d download 20 .exe installer files post-OS-install and have to update them individually, a hugely time-consuming process and at time outright dangerous as software easily gets out of date.

Traditional model of software deployment
Traditional model of software deployment
There are also downsides to how we handle software deployment and installation currently, most of them revolve around update cycles. There is always a middle man who decides when and what to upgrade. This results in applications getting out of date, which is bad in reality and leads to a number of problems, as security and bug fixes are not making it to users in a timely fashion,

  • It’s not unusual that software installed on a “supported” Linux system is outdated and not at all supported upstream anymore on the day it reaches the user. Worse, policies employed by distributions (or more generally, operating system vendors) will prevent some software packages from ever getting an update other than the most critical security fix within the whole support cycle.
  • Software out in the wild with its problems isn’t supported upstream, bug reports reaching the upstream developers are often invalid and have been fixed in newer versions, or users are asked to test the latest version, which most of the time isn’t available for their OS — this makes it harder to address problems with the software and it’s frustrating for both, users and developers.
  • Even if bugs are fixed in a timely fashion, the likelihood of users of traditional distributions actually receiving these updates without manually installing them is small, especially if users are not aware of it.
  • Packaging software for a variety of different distributions is a huge waste of time. While this can be automated to some extent, it’s still less than ideal as work is duplicated, packaging bugs do happen simply because distribution packagers do not fully understand how a specific piece of software is built and best deployed (there’s a wide variety of software after all) and software stacks aren’t often well-aligned. (More on that later!)
  • Support cycles differ, leading to two problems:
  • Distros need to guarantee support for software they didn’t produce
  • Developers are not sure how much value there is in shipping a release and subsequent bugfix releases, since it takes usually at least months until many users upgrade their OS and receive the new version.
  • Related to that, it can take a long time until a user confirms a bug fix.
  • There is only a small number of distributions who can package every single piece of useful software available. This essentially limits the user’s choice because his niche distro of choice may simply not have all needed software available.

The value of downstreams

One argument that has been made is that downstreams do important work, too. An example for that legal or licensing problems are often found during reviews at SUSE, one of KDE’s downstream partners. These are often fed back to KDE’s developers where the problems can be fixed and be made part of upstream. This doesn’t have to change at all, in fact, with a quicker deployment process, we’re actually able to ship these fixes quicker to users. Likewise, QA that currently happens downstream should actually shift more to upstream so fixes get integrated and deployed quicker.

One big problem that we are currently facing is the variety of software stacks our downstreams use. An example that often bites us is that Linux distributions are combining applications with different versions of Qt. This is not only problematic on desktop form-factors, but has been a significant problem on mobile as well. Running an application against the same version of Qt that developers developed or tested it against means fewer bugs due to a smaller matrix of software stacks, resulting in less user-visible bugs.

In short: We’d be better off if work happening downstream happens more upstream, anyway.

Upstream as software distributor

Software directly from its source
Software directly from its source
So, what’s the idea? Let me explain what I have in mind. This is a bit of a radical idea, but given my above train of thoughts, it may well solve a whole range of problems that I’ve explained.

Linux distributors supply a base system, but most of the UI layers, so the user-visible parts come from downstream KDE (or other vendors, but let’s assume KDE for now). The user gets to run a stable base that boots a system that supports all his hardware and gets updated according to the user’s flavor, but the apps and relevant libraries come from upstream KDE, are maintained, tested and deployed from there. For many of the applications, the middle-man is cut out.

This leads to

  • vastly reduced packaging efforts of distros as apps are only packaged once, not once per distro.
  • much, much shorter delays until a bug fix reaches the user
  • stacks that are carefully put together by those that know the apps’ requirements best

Granted, for a large part of the user’s system that stays relatively static, the current way of using packaged software works just fine. What I’m talking about are the bits and pieces that the users relies on for her productivity, the apps that are fast moving, where fixes are more critical to the user’s productivity, or simply where the user wants to stay more up to date.

Containerization allows systemic improvements

In practice, this can be done by making containerized applications more easily available to the users. Discover, Plasma’s software center, can allow the user to install software directly supplied by KDE and allow to keep it up to date. Users can pick where to get software from, but distros can make smart choices for users as well. Leaner distros could even entirely rely on KDE (or other upstreams) shipping applications and fully concentrate on the base system and advancing that part.

Luckily, containerization technologies now allow us to rethink how we supply users with our software and provide opportunities to let native apps on Linux systems catch up with much shorter deployment cycles and less variety in the stack, resulting in higher quality software on our users’ systems.

Plasma at Akademy


As every year, also this year, I will be going to KDE’s yearly world summit, Akademy. This year, it will take place in Almería, Spain. In our presentation “Plasma: State of the Union“, Marco and I will talk about what’s going on in your favorite workspace, what we’ve been working on and what cool features are coming to you, and what our plans for the future are. Topics we will cover range Wayland, web browser integration, UI design, mobile and release and support planning. Our presentation will take place on Saturday at 11:05, right after the key note held by Robert Kaye. If you can’t make it to Spain next week, there will likely be video recordings, which I will post here as soon as they’re widely available.

Haste luego!

Parrotfish

I’ve been trying macro photography and using the depth of field to make the subject of my photos stand out more from the background. This photo of a parrotfish shows promising results beyond “blurry fish butt” quality. I’ll definitely use this technique more often in the future, especially for colorful fish with colorful coral in the background.

#photographyfirstworldproblems

White Tip

White Tip Reef Shark
White Tip Reef Shark

This morning, on a dive at Sha’ab Iris, off the coast of Hurghada in the Red Sea, a white tip reef shark visited us for a swim-by.

Surfing progress!

As a founding member of our surf club, I’ve decided to do what what was long overdue and took my second surfing lesson.

Surfspot, Scheveningen's harbour pier
Surfspot, Scheveningen’s harbour pier

I went to a surfschool in Scheveningen at the North Sea, got in touch with an instructor, and after going over the water situation, currents, swell and technique, we went into the water for a good one and a half hours. There was a good swell, and next to the harbour’s pier, we were mostly out of the wind. After building up some skills, like catching waves and paddling into them, I managed to ride out a few waves, onto the beach. Not over a long distance, but at least I didn’t fall for a few seconds, a few times. Pretty good progress. Next try planned on sunday, weather allowing. It’s still the North Sea, and it’s still winter, so things can get nasty…

The water temperature was 9°C, which seems cold. I wore my 7mm full-length suit, 3mm gloves and a 5mm hood. It didn’t feel cold even in my fingertops after getting out of the water, so even in March, the North Sea is already very manageable.

Surfing was great fun, it’s an interesting break from diving in that it’s much more physically active. In diving, you tend to spend as little energy on anything as possible. That means that if you’re a good diver (and in the right conditions), you actually burn very little energy. That means you’re getting cold much quicker. Bodysurfing, on the other hand means that you’re constantly moving through the swell, swimming, paddling, getting up, falling, so you end up burning a lot of energy. The cold splash of water is really welcome then.

As opposed to diving, there is no buddy system in surfing, so you can go surfing on your own (under the right conditions, of course). That makes it a bit more flexible than diving. It also trains different muscle groups, especially arms and shoulders, so it complements diving well.

Waves are awesome. :)

Plasma Meeting: Web, browsers and app bundles

This year’s Plasma Sprint is kindly being hosted by von Affenfels, a software company in Stuttgart, Germany, focusing on mobile apps. Let me try to give you an idea of what we’re working on this week.

Bundled apps

Welcome, KDE hackers!
Welcome, KDE hackers!
One problem we’re facing in KDE is that for Linux, our most important target platform, we depend on Linux distributors to ship our apps and updates for it. This is problematic on the distro side, since the work on packaging has to be duplicated by many different people, but it’s also a problem for application developers, since it may take weeks, months or until forever until an update becomes available for users. This is a serious problem and puts us far, far behind for example deployment cycles for webapps.

Bundled app technologies such as flatpak, appimage and snap solve this problem by allowing us to create one of these packages and deploy them across a wide range of distributions. That means that we could go as far as shipping apps ourselves and cutting out the distros as middle men. This has a bunch of advantages:

  • Releases and fixes can reach the user much quicker as we don’t have to wait for distros with their own cycles, policies and resources to pick up our updates
  • Users can easily get the lastest version of the software they need, without being bound to what the distro ships
  • Packaging and testing effort is vastly reduced as it has to only be done once, and not for every distro out there
  • Distros with less man-power, who may not be able to package and offer a lot of software can make available many more appliations,…
  • …and at the same time concentrate their efforts on the core of their OS

From a Plasma point of view, we want to concentrate on a single technology, and not three of them. My personal favorite is flatpak, as it is technologically the most advanced, it doesn’t rely on a proprietary and centralized server component. Unless Canonical changes the way they control snaps, flatpak should be the technology KDE concentrates on. This hasn’t been formally decided however, and the jury is still out. I think it’s important to realize that KDE isn’t served by adopting a technology for a process as important as software distribution that could be switched off by a single company. This would pose an unacceptable risk, and it would send the wrong signal to the rest of the Free software community.

How would this look like to the user? I can imagine KDE to ship applications directly. We already build our code on pretty much every commit, we are actually the best candidate to know how to build it properly. We’d integrate this seamlessly in Discover through the KDE store, and users should be able to install our applications very easily, perhaps similarly to openSUSE’s one click install, but based on appstream metadata.

Website work

Hackers hacking.
Hackers hacking.

We started off the meeting by going over and categorizing topics and then dove straight into the first topic: Communication and Design. There’s a new website for Plasma (and the whole of KDE) coming, thanks to the tireless work of Ken Vermette. We went over most of his recent work to review and suggest fixes, but also to get a bit excited about this new public face of Plasma. The website is part of a bigger problem: In KDE, we’re doing lots of excellent work, but we fail to communicate it properly, regularly and in ways and media that reach our target audience. In fact, we haven’t even clearly defined the target audience. This is something we want to tackle in the near future as well, so stay tuned.

But also webbrowsers….

KDE Plasma in 2017
KDE Plasma in 2017

Kai Uwe demo’ed his work on better integration of browsers: Native notifications instead of the out-of-place notifications created by the browser, controls for media player integration between Plasma and the browser (so your album artwork gets shown in the panel’s media controller), acccess to tabs, closing all incognito tabs from Plasma, including individual browser and a few more cool features. Plasma already has most of this functionality, so the bigger part of this has to be in the browser. Kai has implemented the browser side of things as an extension for Chromium (that’s what he uses, Firefox support is also planned), and we’re discussing how we can bring this extension to the attention of the users, possibly preinstalling it so you get the improvements in browser integration without having to spend a thought on it.

On and on…

We only just started our sprint, and there are many more things we’re working on and discussing. The above is my account of some things we discussed so far, but I’m planning to keep you posted.

33C3: Works for me

Rocket Science
Rocket Science
The calm days between christmas and new year are best celebrated with your family (of choice), so I went to Hamburg where the 33rd edition of the Chaos Computer Congress opened the door to 12.000 hackers, civil rights activists, makers and people interested in privacy and computer security. The motto of this congress is “works for me” which is meant as a critical nudge towards developers who stop after technology works for them, while it should work for everyone. A demand for a change in attitude.

33C3's ballroom
33C3’s ballroom

The congress is a huge gathering of people to share information, hack, talk and party, and the past days have been a blast. This congress strikes an excellent balance between high quality talks, interesting hacks and electronics and a laid back atmosphere, all almost around the clock. (Well, the official track stops around 2 a.m., but continues around half past eleven in the morning.) The schedule is really relaxed, which makes it possibly to party at night, and interrupt dancing for a quick presentation about colonizing intergalactic space — done by domain experts.

The conference also has a large unconference part, hacking spaces, and lounge areas, meaning that the setup is somewhere in between a technology conference, a large hack-fest and a techno party. Everything is filled to the brim with electronics and decorated nicely, and after a few days, the outside world simply starts to fade and “congress” becomes the new reality.

No Love for the U.S. Gov

I’ve attended a bunch of sessions on civil rights and cyber warfare, as well as more technical things. One presentation that touched me in particular was the story of Lauri Love, who is accused of stealing data from agencies including Federal Reserve, Nasa and FBI. This talk was presented by a civil rights activist from the Courage foundation, and two hackers from Anonymous and Lulzsec. While Love is a UK citizen, the US is demanding extradition from the UK so they can prosecute him under US law (which is much stricter than the UK’s). This would create a precedent making it much easier for the US to essentially be able to prosecute citizens anywhere under US law.

What kind of technoparty^W congres is this?
What kind of technoparty^W congres is this?
This, combined with the US jail system poses a serious threat to Love. He wouldn’t be the first person to commit suicide under the pressure put on him by the US government agencies, who really seem to be playing hardball here. (Chelsea Manning, the whistleblower behind the videos of the baghdad airstrikes, in which US airforce killed innocent citizens carelessly, among others) who suffered from mental health issues, was put into solitary confinement, instead of receiving health care. Against that background, the UK would send one of their own citizens into a jail that doesn’t even respect basic human rights. On particularly touching moment was when the brother of Aaron Swartz took the microphone and appealed to the people who asked how they could prevent another Aaron, that helping Lauri (and Chelsea) is the way to help out, and that’s where the energy should be put. Very moving.

The media team at this event is recording most of the sessions, so if you have some time to spare, head over to media.ccc.de and get your fix. See you at 34C3!