A deep look at the numbers
It has been over three years now since ownCloud decided in 2012 to issue security advisories for each vulnerability at owncloud.org/security/ following industry best practice. We take this very seriously and create advisories even for very minor issues.
What I have noticed is that people aren’t certain how to take this massive list of advisories or CVE identifiers or they take it as a sign of a bad security track record. If we’d stop putting so much effort into finding and fixing security problems, which would decrease the number of advisories, ownCloud wouldn’t necessarily become more secure. And in fact we are also constantly working on improving and hardening our code base even more.
But advisories do contain information about the security of a project. If there are very few or none, you can guess that the project is not being audited for security issues. Alternatively, perhaps security problems aren’t reported, which takes away the ability for their users to deal with security vulnerabilities. But if there are a lot of security advisories, you have to dig a little deeper to find out what the real state of security is.
And, to illustrate what I mean, let me share the following image by @altonncf:
What this statistic is showing you that statistics are a dangerous thing as it is easy to correlate wrong data to get to wrong conclusions. I recommend to take a look at the Black Hat 2013 talk “Buying into the Bias: Why Vulnerability Statistics Suck” which explains in-depth why analyzing Vulnerability Statistics is difficult.
That does not mean you can’t get a lot of information from a list of vulnerabilities, and in the rest of this blog I will do just that. Comparing the different kind of problems found with the way they were found and plotting that over time will give us some insight in how ownCloud has become what is, in my opinion, the most secure open source file sync and share solution on the market.
Let me first share the definition of some terms we use in the following graphs:
- Vulnerabilities which may allow an adversary to gain complete control over the server or all files on it. This includes for example Remote Code Executions or SQL Injections.
- Vulnerabilities allowing the adversary to gain complete control over a single user session. This includes for example Cross-Site-Scripting vulnerabilities.
- Vulnerabilities that can only be exploited in very rare cases or have marginal impact.
When talking about ownCloud one has to differentiate between the “ownCloud Server” on its own which is the release you can download from https://owncloud.org/install/ as well as “community applications” such as the calendar or contacts application.
Generally speaking, “ownCloud Server” includes everything which is also supported by ownCloud, Inc., the company behind the ownCloud project that employs a lot of the core maintainers to work on “ownCloud Server”. Community applications are however not supported by ownCloud, Inc. and do usually receive not too many commits by employees of ownCloud, Inc. That does not mean we don’t look at security vulnerabilities in them, but they are lower priority and I would discourage using too many community apps on a security critical system.
Of course, you could set up multiple ownCloud instances in separate virtual machines or completely separate systems, some hosting the more sensitive data in a minimal setup, other data being hosted in a more full-featured installation. Our Federated Cloud Sharing technology can help you mesh these systems together so they act, to the end user, as a single ownCloud!
Let’s take a look at the amount of vulnerability reports we have received since 2012, please note that these numbers are not necessarily identical to the amount of advisories as an advisory might have fixed multiple vulnerabilities.
As can be seen over the years the amount of discovered vulnerabilities has gone down significantly which implies that finding security vulnerabilities has gotten much harder over the years. This is, for example, caused by the massive amount of security hardenings that we perform for each release.
What can also be seen is that a significant amount of vulnerabilities are in fact located within the community apps and not the ownCloud Server itself. Meaning that people who do not have these apps installed (including users of the Enterprise Edition) are not affected by these vulnerabilities.
When talking about vulnerabilities it is always important to look at who reported the problem. Was it the vendor who is actively searching for security vulnerabilities or are all vulnerabilities reported by third-parties?
Our statistics clearly prove that most issues are discovered internally. This gets even more visible if we’re going to break this down for each year.
As can be seen in 2012 and 2013 while most bugs are discovered internally there is still a substantial amount of external reported vulnerabilities.
Starting with 2014 and 2015, all critical vulnerabilities have been solely discovered internally and the amount of externally reported vulnerabilities has rapidly decreased as ownCloud heavily increased investment into improving security.
Furthermore, many of these “Critical” issues only occur in very specific environments and sometimes require very unlikely pre-requirements. One of these vulnerabilities, for example, required to be able to control the responses of “dropbox.com” domain (which we connect against using HTTPS and certificate pinning!) as well as having a mounted Dropbox mount in the user profile. One can arguably say that this kind of vulnerability is very hard and unlikely to be exploited in real life. We have anyways fixed this issue and created an advisory for it.
What do these numbers mean? As I wrote previously one should always take statistics with a grain of salt, but we internally have concluded that:
- Finding security bugs has effectively got way harder over the time and we’re working on continuing this trend.
- Critical security issues have got way harder to spot and usually require a deep understanding of the ownCloud code base to be able to discover a critical security issue.
On the other hand, this also implies that everybody who complains about ownCloud being insecure because of having so many advisories is ignoring that most of these issues are in fact discovered internally and fixed in a transparent way. Many vendors tend to not disclose their security problems which makes them look better if one simply looks at factors such as amounts of the advisories.
While this numbers do in no way indicate that ownCloud is a perfectly secure solution, it does show that we really care about security and looking at numbers only is an unsuitable way to compare the security of products.
By the way, we do run a Bug Bounty program at HackerOne and if you believe to have found
a security vulnerability we would like to encourage you to submit it to us. You’ll get a bounty for each valid and qualifying
report – so far, we have received hundreds of reports but only one minor issue in the ownCloud server resulted in a combined
USD 50 payout so I’m feeling quite confident about our security right now. This issue did not affect the integrity or data
security of the instance and only leaked the installation path of the ownCloud instance. (usually