Software from the Source

In this article, I am outlining an idea for an improved process of deploying software to Linux systems. It combined advantages of traditional, package mangement based systems with containerized software through systems such as Flatpak, Snap, or AppImage. An improved process allows us to make software deployment more efficient across the whole Free software community, have better supported software on users systems and allow for better quality at the same time.

Where we are going
Where we are going
In today’s Linux and Free software ecosystems, users usually receive all their software from one source. It usually means that software is well integrated with the system, can be tested in combination with each other and support comes from a single vendor. Compared to systems, in which single software packages are downloaded from their individual vendors and then installed manually, this has huge advantages, as it makes it easy to get updates for everything installed on your system with a single command. The base system and the software comes from the same hands and can be tested as a whole. This ease of upgrading is almost mind-boggling to people who are used to a Windows world, where you’d download 20 .exe installer files post-OS-install and have to update them individually, a hugely time-consuming process and at time outright dangerous as software easily gets out of date.

Traditional model of software deployment
Traditional model of software deployment
There are also downsides to how we handle software deployment and installation currently, most of them revolve around update cycles. There is always a middle man who decides when and what to upgrade. This results in applications getting out of date, which is bad in reality and leads to a number of problems, as security and bug fixes are not making it to users in a timely fashion,

  • It’s not unusual that software installed on a “supported” Linux system is outdated and not at all supported upstream anymore on the day it reaches the user. Worse, policies employed by distributions (or more generally, operating system vendors) will prevent some software packages from ever getting an update other than the most critical security fix within the whole support cycle.
  • Software out in the wild with its problems isn’t supported upstream, bug reports reaching the upstream developers are often invalid and have been fixed in newer versions, or users are asked to test the latest version, which most of the time isn’t available for their OS — this makes it harder to address problems with the software and it’s frustrating for both, users and developers.
  • Even if bugs are fixed in a timely fashion, the likelihood of users of traditional distributions actually receiving these updates without manually installing them is small, especially if users are not aware of it.
  • Packaging software for a variety of different distributions is a huge waste of time. While this can be automated to some extent, it’s still less than ideal as work is duplicated, packaging bugs do happen simply because distribution packagers do not fully understand how a specific piece of software is built and best deployed (there’s a wide variety of software after all) and software stacks aren’t often well-aligned. (More on that later!)
  • Support cycles differ, leading to two problems:
  • Distros need to guarantee support for software they didn’t produce
  • Developers are not sure how much value there is in shipping a release and subsequent bugfix releases, since it takes usually at least months until many users upgrade their OS and receive the new version.
  • Related to that, it can take a long time until a user confirms a bug fix.
  • There is only a small number of distributions who can package every single piece of useful software available. This essentially limits the user’s choice because his niche distro of choice may simply not have all needed software available.

The value of downstreams

One argument that has been made is that downstreams do important work, too. An example for that legal or licensing problems are often found during reviews at SUSE, one of KDE’s downstream partners. These are often fed back to KDE’s developers where the problems can be fixed and be made part of upstream. This doesn’t have to change at all, in fact, with a quicker deployment process, we’re actually able to ship these fixes quicker to users. Likewise, QA that currently happens downstream should actually shift more to upstream so fixes get integrated and deployed quicker.

One big problem that we are currently facing is the variety of software stacks our downstreams use. An example that often bites us is that Linux distributions are combining applications with different versions of Qt. This is not only problematic on desktop form-factors, but has been a significant problem on mobile as well. Running an application against the same version of Qt that developers developed or tested it against means fewer bugs due to a smaller matrix of software stacks, resulting in less user-visible bugs.

In short: We’d be better off if work happening downstream happens more upstream, anyway.

Upstream as software distributor

Software directly from its source
Software directly from its source
So, what’s the idea? Let me explain what I have in mind. This is a bit of a radical idea, but given my above train of thoughts, it may well solve a whole range of problems that I’ve explained.

Linux distributors supply a base system, but most of the UI layers, so the user-visible parts come from downstream KDE (or other vendors, but let’s assume KDE for now). The user gets to run a stable base that boots a system that supports all his hardware and gets updated according to the user’s flavor, but the apps and relevant libraries come from upstream KDE, are maintained, tested and deployed from there. For many of the applications, the middle-man is cut out.

This leads to

  • vastly reduced packaging efforts of distros as apps are only packaged once, not once per distro.
  • much, much shorter delays until a bug fix reaches the user
  • stacks that are carefully put together by those that know the apps’ requirements best

Granted, for a large part of the user’s system that stays relatively static, the current way of using packaged software works just fine. What I’m talking about are the bits and pieces that the users relies on for her productivity, the apps that are fast moving, where fixes are more critical to the user’s productivity, or simply where the user wants to stay more up to date.

Containerization allows systemic improvements

In practice, this can be done by making containerized applications more easily available to the users. Discover, Plasma’s software center, can allow the user to install software directly supplied by KDE and allow to keep it up to date. Users can pick where to get software from, but distros can make smart choices for users as well. Leaner distros could even entirely rely on KDE (or other upstreams) shipping applications and fully concentrate on the base system and advancing that part.

Luckily, containerization technologies now allow us to rethink how we supply users with our software and provide opportunities to let native apps on Linux systems catch up with much shorter deployment cycles and less variety in the stack, resulting in higher quality software on our users’ systems.

7 thoughts on “Software from the Source

  1. I’m not sure, if this works as expected. I mean, traditionally let’s look at Ubuntu and PPAs and let’s forget other distros. Than the simpliest way to package newest KDE is to make a huge PPA and you’re done (in fact this is more less the approach of KDE Neon, if I get it right).

    But, then comes another big player in the software world, let’s say VLC and also makes an own PPA and unfortunately the software in this PPA is build against other library versions, so it has to ship them as well and is incompatible with the KDE PPA. In fact this happens. A long time ago, I used Ubuntu with a lot of software from PPAs with newer versions and my whole system was a mess.

    The alternative of course is containerization, so every programm, that the user uses, ships it’s own set of libraries. But then the upstream has not to maintain only it’s program but also the (security) updates of all used libraries. This won’t work or other said, this is a bunch of responsibility for the one upstream. It has to follow the development of all it’s libraries as well.

    Another downside is the reduced amount of testing. Let’s define two user groups: the normal user, who only want to use the software and is not interested in new features etc. and the power user, who follows upstream development, try new features just for fun and is not frustated if something crashes sometimes.
    I guess this power users give a lot of bug reports with really weird types of configurations, but overall help to improve the software. This won’t happen anymore if there is only one configuration available: The upstream package.

    I understand your concerns, but my kind of “solution” would be: Let’s as a community find a way to unify the packages. As far as I know, all package formats are more less similar, some kind of meta file to specify dependencies, some kind of configuration and how to compile the software and at the end a ready prebuild package. I’m using Gentoo and successfully “repacked” Debian packages to my system. There is also an automatic package generator that converts Pypi-packages to Gentoo packages. I’ve written ebuilds, that are highly oriented on Archs pkgbuilds. It would be fairly cool, to have a build system or generator etc, that takes an upstream software, reads some “meta”-file defined by the upstream out of it and then is able to build a Debian/Arch/Gentoo/Suse/Fedora Package.
    This would enable the upstream developers to take care only of this metafile and the downstream to package new software without a lot of effort, and with this create in a simple way projects like “Debian backports” where you can get newer software on a well tested, but old base system, or let distributions like Gentoo provide packages for all of your versions, including the Git branches.

    1. This is a matter of scale. For smaller downstreams, it wouldn’t work. For an upstream as big as KDE, it would, and it would even mean a lot less duplication of effort since the work to maintain the dependencies would be shared across all distros shipping KDE software, it’s a huge net win for everybody involved.

      It would mean shifting trust to KDE, which I think makes sense because right now the situation is kind of half-baked: Distros trust KDE to ship software that doesn’t harm the users, but they don’t ship the updates right away when KDE releases them, and this half-baked solution leads to the problems I’m outlining above.

      The real solution will be some middle ground, larger upstreams like KDE that satisfy certain requirements (bus number, proven track records, decent review processes, quick response time to security issues, for example should be trusted to directly ship apps to users), smaller applications which can’t satisfy these requirements should be bound to the distro’s maintenance cycles, meaning longer deployment cycles.

  2. Every change produces advantages and disadvantages. You know replacing horse with car, you gain a major advantages, but also disadvantages that appear after like environment pollution.

    I perfectly agree we you, your arguments are strong. I like your ideas very much. But can we also discuss little bit about disadvantages, or new problems that will gonna appear – that does not mean I am against your ideas, I am a fan of it.

    1. Library management is now centralized, because packages are using single library. Using upstream packages, each of the libraries would need to be fixed (packaged for security problems) in upstream – so duplication of work. The smaller the project, the bigger the problem. We would need some tools, to find out which libraries are out of date and make sure upstream package them.
    2. Quality may get worse. When you know the product is going to run on computers for months/years, you need proper testing and bug fixing. When there i a way to package anytime new code, testing is not so important, you just ship new version of software. Proper QA should be moved from distribution to upstream. Small projects do not have resources for QA, so it can pose a problem.
    3. Integration and uniform look and feel. Upstream may package a software in the way downstream may not like it. For example upstream decides to make some radical UI decision, now downstream stops this and may for example make a change to source code to to fill the gap like upstream decides to discard the menus, but distribution is based on uniform look and feel (like user experience = UX) and makes menus back. Downstream is not just repackaging the software, it also takes care of UX.
    4. Power to the upstream. Now software source code is downloaded into downstream and compiled. If code is upstream an evil upstream could easily put some evil code in it and package and distribute it and get major impact to the whole bunch of distributions. Downstream is testing the code and looking into the code and now may spot some evil stuff.

    1. We are talking about open source software. But there is also a proprietary software. Many don’t like proprietary software on open-source platform, but there are. And to get platform more attractive there must a an option to not push out proprietary software. Now it can be a nightmare to use proprietary software on open-source platform. I used to spend a week to install some proprietary software on Ubuntu, I was convinced I am doing something wrong. The product was officially supported on Redhat, Suse and Ubuntu. But obviously (like I found out later) the QA tests where only performed on Redhat. Now proprietary software companies tests only on a few distributions (and explicitly states which one are supported) and test only one the explicit version of distribution (like tested and supported on Ubuntu 16.04 LTS, but not working/not tested on Ubuntu 17.04) and all the rest is not supported. Because of low market share they don’t have an interest and have a resources to support many distros, because of enormous Linux fragmentation it does not make a sense from cost/benefit logic. Uniform package installation like Flatpak, Snap and Appimage (still a problem having three competing solutions, which reduces the interest of proprietary software) would make installation way better. Proprietary vendor would know the software is working on many distributions without a distro specific problem. Investing in Linux platform would increase. End-users willing to install proprietary software would get an option (still may be a problem of compelling UX, but at least an option).

  3. While I understand your points about getting new versions to the users quickly, I’m not sure if containerizing everything is the way to go. As already mentioned in the comments, a side effect of making upstream more of a distributor also comes with responsibilites like keeping all libraries in an up-to-date and secure version. “DLL hell” comes to mind ;-)
    (For the long version, see the slides and/or recording of Richard’s talk )

    IMHO the better answer to the problems you listed are rolling release distributions like openSUSE Tumbleweed. They get new software shipped to users quickly, and you still get libraries etc. managed and updated by the distributions “for free”.

    For the cross-distro meta package file – I like the idea, and it’s probably more realistic than convincing Debian to use RPM or openSUSE to use DEB packages ;-)

    1. I’m not arguing for containerizing everything, I’m arguing that it’s a technology suitable for problems that KDE faces when it comes to getting updates out to the users. KDE has the scale to actually prevent “DLL hell” from happening (we already maintain these stacks for our CI, so it’s not even a lot more work compared to our current situation).

      Rolling distros are not a solution to these problems as that means that the packaging work is still multiplied for every distro, so it’s still a giant waste of resources. (It addresses some other points, but falls short.)

      I’ve actually watched Richard’s talk, and there’s very little I agree with (actually almost everything except for its title, and that was meant in a sarcastic way). My impression was that he argues from inside his own mindset dismissing the advantages upfront. Just check how he doesn’t actually have answers to the questions asked at the end of his talk, that says a lot about the quality of his arguments. Just being dismissive of reality doesn’t make one right…

  4. Sebastian, while I agree on many of your points, it will not happen. Largely because of inertia in the big distributions but also because of choices made in various means that do not necessarily reflect what USERS want to have. You only have to look at “discussions” where choices are made onto the people – systemd, cmake-switch of KDE being more annoying for any single person wanting to compile anything from source, a constantly changing ecosystem in general, not just limited to one distribution alone (take the more recent gnome/gtk devs switching to meson/ninja as build system but also changing the versioning policy for gtk-4 all of a sudden).

    So, in short – I do not think that we will see any LESS chaos. It is just the natural tendency of linux to proliferate more and more and more.

    And I say this after having watched it in the past 15 years. I am not the only one to notice it, by the way – there are some other blog entries, of people noticing that the complexity is constantly increasing, leading to systems that become more and more complex – and complicated.

    You can say the same about KDE by the way, and QT. I am 100% sure that current day KDE and QT are a lot more complex than KDE 3/QT3. We can’t deny this.

    Take the proliferation of C++. Have the new standards made C++ any LESS complex? From what I can tell, with new keywords and options, it has become even MORE complex – and complicated.

    Your idea for containers is a nice one in theory but in practice, it does not work very well. Do containers work only on systemd distributions? What about dependencies – is everything put into that container? Who updates these containers then? What about incompatibilities and version upgrades?

    I see too many conflicting players and as a consequence, absolutely no real improvement will happen.

    One of the best xkcd is this one:

    Because it is so true.

  5. The role of distributions is changing. With the enormous number of free software packages created and distributed upstream it’s not possible to package everything in the same quality as it has been the case in the earlier times of Linux distributions. There are a lot of alternative, orthogonal, or partly overlapping distribution systems. Think for example of the language specific ecosystems such as Node modules, Ruby gems, etc. Or think of Go becoming a popular language for some applications also because distribution is just copying a single binary to a location in your executable search path.

    I think it’s crucial that distributions and upstream work together, even more so than before, because it’s not the simple “upstream releases sources, distributions compile and package them” relationship in many areas. It is about harmonizing these ecosystems, about providing technology such as containers in a way that it serves developers as well as users. And it’s about automated testing. Without that I don’t think any modern software project can work and be integrated for its users.

    KDE is a strong upstream. It has its own ecosystem. But it should carefully look at what else is there and learn from how others are doing it. If KDE works together with distributions to have a direct channel to its users, to automatically test integration with distributions, to build the structures which make it easy and reliable to release software as well as consume software, then I think your vision can work.

Comments are closed.