FileZilla fixes show how far we’ve come since Heartbleed

Users of FileZilla, the popular open source FTP client, may have noticed a rather serious looking bug described in the change log for the latest update:

Filenames containing double-quotation marks were not escaped correctly when selected for opening/editing. Depending on the associated program, parts of the filename could be interpreted as commands.

Fixed in version 3.43.0, the flaw is one of seven separate security bugs whose discovery is credited to a bug bounty program run by the European Union, of all things.

The EU’s bureaucratic tentacles reach into many things, but a bit of freeware from an area when cover CDs were a thing still seems an odd place to find them.

Explaining why requires a brief trip down memory lane…

Eric S. Raymond’s seminal work on open source, The Cathedral and the Bazaar, taught us that “given enough eyeballs, all bugs are shallow”.

The idea being that the more people who are actively involved in developing, debugging and testing your code, the easier, faster and cheaper it is to find and fix bugs in it.

It’s an idea that’s central to the success, longevity and robustness of sprawling, noisy, open source projects like the Linux kernel. The development process for Linux, and the many other open source projects propping up our internet ecosystem, is entirely transparent, conducted before a potential audience of billions of eyeballs.

The “many eyes” idea is also important symbolically, as part of the meritocratic culture of open source. However, like a lot of good ideas, we’re prone to over-rely on it and our collective understanding of what “many eyes” meant drifted, over time.

Some time between 1997 and 2014 we developed a bit of a collective blind spot, conflating “many eyes” with “transparency”. For most projects, getting “many eyes” on the codebase requires transparency, but transparency itself doesn’t guarantee any eyes at all.

The scales fell from our eyes in 2014, with the discovery of Heartbleed, a critically serious data leak in the open source cryptographic library, OpenSSL. The bug allowed attackers to quietly plunder the private cryptographic key material required to unpick the encryption keeping them at bay.

OpenSSL was a critical piece of infrastructure whose open source code was relied upon by numerous high profile projects. So how had a flaw that undermined it so wholeheartedly lingered, unnoticed, for years?

Because, relative to its popularity, almost nobody was looking at the code. The notion that transparency alone leads to shallow bugs was thoroughly disabused.

Some projects, like the Linux kernel, are interesting, exciting and well known enough to attract many eyes, but most are not. The discovery of Heartbleed woke us up to the fact that the world’s vast collection of important but unsexy open source projects was going to need a better way to make their bugs shallow.

A few different models emerged.

Looking at OpenSSL specifically, Google and OpenBSD settled on making the project easier to maintain by slimming down the codebase. Although they each succeeded in doing that, the result was fragmentation – two incompatible forks of OpenSSL in the forms of BoringSSL and LibreSSL.

Mozilla, the organisation behind the Firefox browser, established its SOS (Secure Open Source) fund.

The fund makes bugs shallow not with many eyes, but a few very good ones – by paying for security audits. Its focus is projects that are actively maintained and vital to the continued functioning of the internet. Audits are thorough but that thoroughness comes at a cost: SOS has only managed 19 audits in three years.

In the EU, German researcher and MEP Julia Reda established FOSSA (the Free and Open Source Software Audit). The project began in 2014 by establishing an inventory of software used by the EU. In 2016, it funded expert evaluations of the KeePass password manager and the Apache web server, along the lines of the SOS project.

In January 2019, it got new teeth, in the form of about a million dollars worth of bug bounties spread across fifteen separate software projects, including FileZilla.

The idea of bug bounties fits neatly into the rich open source tradition of developers doing what they feel like doing. Beyond some rules about what counts as a security flaw and how flaws should be reported, bug hunters are free to choose where they spend their energy. The bounties act as an inducement to draw them to areas they might otherwise not be attracted to, and as an alternative source of cash to the underground market in vulnerabilities.

Managed via HackerOne, bounties are paid to anyone who finds and reports security flaws in the listed projects, with bonuses available for fixes.

The results, as the latest FileZilla changelog attests, are encouraging. Also on the list of eligible software, alongside FileZilla, was the popular VLC media player. In June it received the biggest security update in its history, thanks to EU-FOSSA bug bounties.

Not content with audits and bug bounties, FOSSA now wants to run hackathons, and Reda wants to see Free Software Security added as a permanent item in the EU budget.