If you ever have run a Linux-based operating system you are probably aware of the way that software is usually distributed on them: Using a software repository.
Repositories are great for numerous reasons. Want to install an application on Debian? Easy. Just execute
apt-get install ffmpeg and ffmpeg has been installed. Updating? A quick
apt-get update plus
apt-get upgrade and all is done.
As you can clearly see using a repository gives you the great advantage of making it easy for you to update and install applications.
##The responsibilities of packagers
One needs to be aware of the fact that software repositories are not automagically maintained. There are people out there (“packagers”) whose responsibility it is:
- Keep the software updated, especially considering security patches.
- Package new releases.
So you’re basically moving the responsibility of this task to another entity. And this a good thing to do. Time is sparse and precious, so outsourcing tasks like these is often a sensible thing to do.
Furthermore Linux on the server would probably never have received the kind of traction it has nowadays if packages would not be there.
##The hidden dark side of software repositories This certainly sounds all like great stuff. So using packages directly from your distribution just sounds like the right thing to do. Right?
The truth is a little bit more complex though. First of all, you need to completely trust your packagers. This does not only apply to the risk that they could hide malicious code in the packages, it goes much further. These are the people that timely need to react in case a vulnerability in an application is found to publish an update.
Let’s talk a little bit more about packagers. In case of distributions such as Debian these is mainly done by benevolent people in their spare time. This certainly is a nice gesture of them.
But can you really be sure that people that do this stuff as an hobby can deliver this in the quality that you expect and require? Let’s be honest here, probably not. While it is a great thing that they donate their time it is very unlikely that they will always have time to update the packages they maintain in a timely manner. Furthermore, life goes on, people marry, get children or die. If there is no contingency plan for such plans then there is a really big problem just waiting to happen.
Right now, this is exactly what is happening. I have been pointing out this problem already 1.5 years ago and it’s finally time to recall that.
##What is wrong with the packaging model
First of all: This is not intended as a flame. Its sole intent is to make people aware of the massive security problems existent in packaging choices such as done by Debian. Distribution packages are not always the best solution and can sometimes even expose you to critical vulnerabilities. Taking you and your data at risk.
Let’s take a short glimpse at how Debian handles packaging. They have a policy stating that only security relevant changes should be backported and not the absolute version (emphasis mine):
Q: Why are you fiddling with an old version of that package?
The most important guideline when making a new package that fixes a security problem is to make as few changes as possible. Our users and developers are relying on the exact behaviour of a release once it is made, so any change we make can possibly break someone’s system. […] This means that moving to a new upstream version is not a good solution, instead the relevant changes should be backported. […]
What backporting basically means is the packager randomly decides that they take any version of a software and freeze the version to that. Only single security patches are backported.
Of course, you will miss out from many bug fixes but apparently in the eyes of some people stable software is defined by it’s ancient age. So much that for example this critical bug in ownCloud never received any backport, resulting in potential data loss.
So we’re now having multiple home-made problems here:
- Packagers need to go through the source code and find all relevant security patches.
- Packagers need to actually be aware of the fact that a security problem has been fixed.
Just missing one security patch means taking user data at risk. And this is exactly what is happening. Let’s just take a look at one example:
Yikes! You’re seeing it right! The phpMyAdmin version shipped by Debian is totally insecure on any stable release. So there are probably over 20.000 servers out there using an inherently insecure version of phpMyAdmin.
What is even more a shame: Debian has a tool to track open security vulnerabilities in software and nobody is giving a fuck. That should immediately point out to anybody that there is a gigantic problem and all alarm bells should ring.
##Just the tip of the iceberg
And this problem is inherent. If phpMyAdmin sounds like a too obscure example, let’s try Wordpress:
I have spent some minutes on other packages within the “web” category of the packaging system and I’m still more than shocked. Since there are over 1.000 different packages in there I limited myself to some famous software plus the ones that I believed were interesting. And note that this is just the result of some minutes, now think about the time that a real adversary could spend on this…
|Package||Last upstream release date||Last Debian release date||Insecure because of|
|adminer||2016-02-06||2012-02-06||Multiple critical vulnerabilities according to the changelog.|
|ckeditor||2016-02-04||2014-11-10||Missing at least one security patch from 4.4.6|
|collabtive||2015-03-13||2014-10-21||Multiple missing security related changes according to the changelog. No proper vulnerability tracking upstream.|
|dolibarr||2016-01-31||2014-12-05||Changelog lists a ton of issues such as multiple SQL injections.|
|dotclear||2015-10-25||2014-08-20||Multiple security vulnerabilities such as XSS as per the changelog.|
|drupal||2016-02-03||2015-08-20||Missing the security patch for SA-CORE-2015-004.|
|elasticsearch||2016-02-02||2015-04-27||Vulnerable to CVE-2015-4165, CVE-2015-5377 and CVE-2015-5531.|
|glpi||2015-12-01||2014-10-18||Release with security patches has been released in April 2015.|
|gosa||Upstream does not release updates anymore.||2015-07-24||Publicly known code injections.|
|ldap-account-manager||2015-12-15||2014-10-05||Multiple security related issues pointed out in the changelog.|
|mediawiki||-||-||Takes Debian one week to backport security patches.|
|node-* packages||-||-||Consider yourself pwned and reinstall your machine. After this never use Debian packages like this again. Nearly all of them are insecure versions. Just compare https://nodesecurity.io/advisories with what Debian has in the packaging system…|
And that’s where I stopped. Well, not quite true, the actual list is way longer but entering this into the Markdown format was just pain and I guess you understood the problem already. Just try it yourself, spend some minutes going through the Debian packages and you will pretty certainly find some that endanger your whole system.
Note again: This is web software, it’s often exposed to the general public and thus having the latest security updates is critical for your security. A single mistake here can take all your data at risk.
##Call for action Distributions and application authors really need to come together and work on pushing easy solutions to collaborate. openSUSE Build Service is already a nice start for collaboration. But to be honest, probably for some applications packages are not the best solution. Did you know that for example Wordpress comes with automatic updates and updating ownCloud is just replacing single files on the server?
Rolling release distributions like OpenSuse Tumbleweed or Arch Linux are in my opinion really required to gain more traction to make the overall Linux ecosystem more secure. And before somebody brings it up: Containers will probably not save you either.