Celery is by far the most popular library in Python for distributing
asynchronous work using a task queue. If you're building a Python web app,
chances are you already use it to send email, perform API integrations, etc.
Many people choose Redis as their message broker of choice because
it's dead simple to set up: provision a Redis add-on, use its environment
variable as your BROKER_URL, and you're done. But the simplicity of Redis
comes at a cost. Redis does not currently support SSL, and
it doesn't seem like that's going to change any time soon.
Because Heroku add-ons communicate over the public web, that means the contents
of Celery jobs are traveling unencrypted between dynos and Redis.
Luckily, this is a solvable problem. At Heroku, we often use Fernet to symmetrically encrypt information over the wire. Kombu, Celery's messaging library, supports custom serializers, giving us control over how Celery jobs are turned into byte strings to be sent to the message broker and vice versa. We've wrapped all of the default serializers that come with Kombu in Fernet encryption (provided by Cryptography) and published them in an open source library called kombu-fernet-serializers.
To use the library, add kombu-fernet-serializers to requirements.txt, store
your Fernet key in an environment variable called KOMBU_FERNET_KEY, and tell
Celery to use one of the serializers it supplies
via Setuptools entry-points:
import os
from celery import Celery
from kombu_fernet.serializers.json import MIMETYPE
app = Celery('tasks', broker=os.environ['REDIS_URL'])
app.conf.update(
CELERY_TASK_SERIALIZER='fernet_json',
CELERY_ACCEPT_CONTENT=[MIMETYPE],
)
Now you can take advantage of the ease of configuration Redis boasts without having to worry about leaking sensitive information.