The Public Cloud Security (PCS) group at Salesforce partners very closely with Heroku engineering to review and advise on new product features across the platform, from infrastructure to applications. One of the most rewarding aspects about this partnership and working on this team for me is when we not only identify security concerns, but take an active role in building safe solutions.
Heroku recently announced support for Active Storage in Rails 5.2, which introduces the ability to generate previews of PDFs and videos. As a security engineer, hearing about a new feature in a product that automatically parses media files definitely grabbed my attention. This post takes a look at challenges in supporting Active Storage on Heroku from a security perspective, and how engineering teams and PCS work together to build safe solutions for our customers.
FFmpeg's History
FFmpeg is a tool for handling a wide variety of video and audio files through many libraries and plugins that expand its support for different formats. FFmpeg's power and flexibility make it a popular choice for many different projects, so it's no surprise that the Rails team chose it for handling generation of preview images from video files.
FFmpeg has a long history of publicly disclosed vulnerabilities, with many resulting in code execution. It's a popular target for fuzzers which have been the source of many of these vulnerability discoveries.
Raw CVE count isn't a great metric to determine whether a piece of software is secure or not, but it does show us that this is an attractive target for researchers, and gives us an idea of how frequently we'd need to be concerned about deploying patches for security issues.
Perhaps a more accurate metric is how quickly the FFmpeg team responds to security issues. In a 2015 article about Debian returning to use FFmpeg, security researcher Mateusz “j00ru” Jurczyk had fantastic praise for the FFmpeg team's vulnerability response.
“We have been quite successful working on the above effort with FFmpeg for the last ~3.5 years: every single issue we have found (even the least severe ones) has been fixed in a timely manner.”
The commitment from the upstream project to address security issues big and small gave us confidence to move forward with including support for FFmpeg. Timely triaging and applying application patches in Heroku stack images and buildpacks is a well tested process at Heroku, so we felt comfortable knowing we could quickly get vulnerability patches to customers using Active Storage.
Keeping Up With Upstream
One of the engineering challenges with supporting Active Storage is that the cedar-14 stack is based on Ubuntu Trusty 14.04, which removed support for FFmpeg in favor of libav. That means it isn't as simple as including Ubuntu's FFmpeg package directly in the stack image. While it may have been possible to try and use libav, consider Jurczyk’s description of that project from the same article linked above.
”The situation is entirely different with Libav, which is still affected by hundreds of such bugs, even though we have provided the developers with reproducing testcases a number of times in the past.”
The other problem from a security perspective is that the FFmpeg package for Ubuntu Xenial 16.04 (which heroku-16 is based on) is part of the community maintained Universe repository and was missing backports for several vulnerabilities. We really appreciate the community efforts involved in creating and maintaining backports for a complicated package like FFmpeg, but we were not comfortable in shipping a new feature to customer applications with so many known vulnerabilities.
Without the resources to contribute our own backports, this left us with a new challenge - rolling our own FFmpeg binary.
A Static Solution
While looking at options for the best way to provide customers with the most up to date and secure FFmpeg, I came across an excellent project called sffmpeg that uses CMake to create a static FFmpeg binary. Building a static binary allows us to create a single Debian package that would include any necessary dependencies and work across all of Heroku’s supported stack images.
FFmpeg 4.0 had just been released, and sffmpeg was a couple of versions behind. However, there was already a pull request from Jan Spitalnik to upgrade sffmpeg to 4.0, which we were able to use to start our fork.
The earlier post mentioned some of the licensing concerns with the PDF dependency, and we faced similar issues with FFMpeg. Luckily for the static build, we were able to easily modify the CMakeLists.txt file to change the build configuration options and external project additions to remove non-free software.
This has the added benefit of minimizing the attack surface of FFMpeg by removing unnecessary libraries and dependencies that were included by default in sffmpeg, but not needed for the Active Storage support. Each additional dependency included by FFmpeg to support other media formats is an additional library or tool that could introduce vulnerabilities into customer applications and that we would need to monitor for security patches. By removing things like RTMP stream support or xvid, things that aren’t used by Active Storage, we minimize applications from being vulnerable to issues in unnecessary dependencies.
This reduction in attack surface is apparent when considering that including FFmpeg and the default dependencies would have added around 200MB to an application slug, while the final static build weighed in around 30MB. This approach helps prevent customer applications from hitting the 500MB slug size limits, speeds up customer builds, and reduces our storage requirements for Rails applications. The combination of a clear performance benefit, as well as improving the security story made this an easy choice.
With the static build helper generating the latest, patched version of FFmpeg, and stripped down to the minimum dependencies to create a usable Debian package, all that was left to do was automate the build with CI and get it included in the new Active Storage Preview buildpack.
Automation is essential for ensuring we can react quickly and roll out updated versions when any new security releases are announced. We are using Docker in CircleCI to run our forked sffmpeg build helper. The build script compiles the static FFmpeg binary with all of its dependencies, adds it to a Debian package (.deb), and then uploads the package to AWS S3. Once uploaded, it can immediately be used by anyone using the Active Storage Preview buildpack. The buildpack installs the Debian package into your application dyno by using the -x option to extract it locally, since dyno’s don’t run as root, and adds it to your dyno’s PATH environment variable to make it accessible to Rails. The CI build and push to S3 takes around 20 minutes, allowing us to get patches out to customers very soon after an upstream release.
Conclusion
The result of this effort provided us with a version of FFmpeg that contained the latest security patches, with a reduced attack surface and package size, support for all of Heroku’s stack images, and an automated process for staying up to date with future security releases.
It is a crucial part of Heroku's engineering culture that security is not a blocker, but an enabler and as such our team strives to help find safe solutions to support products and features that make Heroku the best place to run your code.