External Server Configuration
Warning
It's recommended that you read through the Configuration Overview page before proceeding with this module.
Configuring the Broker
In the broker
section of your app.yaml you will provide all of the necessary settings for Merlin to establish a connection with your broker.
Here, we'll discuss:
- The different Broker Options that are allowed
- How the password field is resolved
- The URL option
Broker Options
Merlin allows for several different broker configurations. This section will detail each of these options and how to configure them. We recommend using RabbitMQ as your broker.
RabbitMQ, AMQPS, and AMQP
See the RabbitMQ Documentation for instructions on how to create a RabbitMQ server.
Once your server is set up, we'll need six keys in the broker
section of the app.yaml file:
name
: The name of the broker (options arerabbitmq
,amqps
, oramqp
)username
: Username for RabbitMQ user that will be used for accessing the servicepassword
: The path to the file that's storing your RabbitMQ passwordserver
: A URL to the server that you're connectingport
: The port that your server is running on. If left undefined and thename
setting is:rabbitmq
oramqps
the default RabbitMQ TLS port (5671) will be usedamqp
the default port will be 5672
vhost
: The vhost for your RabbitMQ service
Using these settings, Merlin will construct a connection string of the form:
Here conn
is amqps
(with ssl) when the name
field is rabbitmq
or amqps
, and amqp
(without ssl) when the name
field is amqp
. If you're using a connection option with ssl, see the Security With RabbitMQ section below.
Example
Let's say we create a file in our ~/.merlin/
directory called rabbit.pass
. In this file we'll store the password for our RabbitMQ server:
~/.merlin/rabbit.pass | |
---|---|
Now we'll update the broker
section of our app.yaml file to be:
broker:
name: rabbitmq
username: rabbit-username
password: ~/.merlin/rabbit.pass
server: server.domain.com
vhost: rabbit-vhost
The connection string that Merlin generates will then become:
Security With RabbitMQ
Merlin can only be configured to communicate with RabbitMQ over an SSL connection and does not permit use of a RabbitMQ server configured without SSL. Therefore, the default value of the broker_use_ssl
celery argument is True
for RabbitMQ.
The keys can be given in the broker config as shown below:
broker:
name: rabbitmq
username: rabbit-username
password: ~/.merlin/rabbit.pass
server: server.domain.com
vhost: rabbit-vhost
# ssl security
keyfile: /var/ssl/private/client-key.pem
certfile: /var/ssl/amqp-server-cert.pem
ca_certs: /var/ssl/myca.pem
# This is optional and can be required, optional or none
# (required is the default)
cert_reqs: required
The ssl config with rabbitmq/amqps in the broker is then placed in the broker_use_ssl
celery argument.
broker_use_ssl = {
'keyfile': '/var/ssl/private/client-key.pem',
'certfile': '/var/ssl/amqp-server-cert.pem',
'ca_certs': '/var/ssl/myca.pem',
'cert_reqs': ssl.CERT_REQUIRED
}
Redis and Rediss
Note
If you're using Redis v6.0.0+ and would like to configure with ssl, you'll need to view the Security With Rediss section below after completing this section.
See the Redis Documentation for instructions on how to create a Redis server.
Once your server is set up, we'll need five keys in the broker
section of the app.yaml file:
name
: The name of the broker (here it will beredis
if running without ssl, orrediss
if running with ssl)password
: The path to the file that's storing your Redis passwordserver
: A URL to the server that you're connectingport
: The port that your server is running on. Default is 6379.db_num
: The database index (this will likely be 0).
Using these settings, Merlin will construct a connection string of the form:
If using ssl, see Security With Rediss for additional setup instructions.
Example
Let's say we create a file in our ~/.merlin/
directory called redis.pass
. In this file we'll store the password for our Redis server:
~/.merlin/redis.pass | |
---|---|
Now we'll update the broker
section of our app.yaml file to be:
broker:
name: redis
password: ~/.merlin/redis.pass
server: server.domain.com
port: 6379
db_num: 0
The connection string that Merlin generates will then become:
Security With Rediss
When using Redis with ssl, aka rediss, (only available with Redis v6.0.0+), there are some additional keys that you'll need to add to your broker
section:
broker:
name: rediss
password: ~/.merlin/redis.pass
server: server.domain.com
port: 6379
db_num: 0
# ssl security
keyfile: /var/ssl/private/client-key.pem
certfile: /var/ssl/amqp-server-cert.pem
ca_certs: /var/ssl/myca.pem
# This is optional and can be required, optional or none
# (required is the default)
cert_reqs: required
The ssl config with redis (rediss) in the broker is then placed in the broker_use_ssl
celery argument.
broker_use_ssl = {
'ssl_keyfile': '/var/ssl/private/client-key.pem',
'ssl_certfile': '/var/ssl/amqp-server-cert.pem',
'ssl_ca_certs': '/var/ssl/myca.pem',
'ssl_cert_reqs': ssl.CERT_REQUIRED
}
Redis+Socket
Celery supports Redis connections using Unix domain sockets. For this setup, three keys are required in the broker
section:
name
: The name of the broker. This will beredis+socket
here.path
: The path to the Unix domain socket file for Redisdb_num
: The database index
Using these settings, Merlin will construct a connection string of the form:
Example
Let's set the broker
configuration to be:
The connection string that Merlin generates will then become:
Resolving The Broker Password Field
The broker/password
is simply the full path to a file containing your password for the user defined by broker/username
.
Example
Say the password to our server is my-super-secret-password
. We'd simply take this password, place it in a file, and then link the path to the file in the broker
section.
~/.merlin/password-file.pass | |
---|---|
Broker URL
A url
option is available to specify the broker connection url, in which case the server name is ignored. The url must include the entire connection url except the ssl if the broker name is recognized by the ssl processing system. Currently the ssl system will only configure the Rabbitmq and Redis servers.
Using the url
setting, Merlin will construct a connection string of the form:
Example
Say we use the default Redis server for our broker:
The connection string that Merlin generates will then become:
Configuring the Results Backend
In the results_backend
section of your app.yaml you will provide all of the necessary settings for Merlin to establish a connection with your results backend.
Here, we'll discuss:
- The different Results Backend Options that are allowed
- How the password field is resolved
- The URL option
Results Backend Options
Merlin allows for several different results backend configurations. This section will detail each of these options and how to configure them. We recommend using Redis as your results backend.
Redis and Rediss
Note
If you're using Redis v6.0.0+ and would like to configure with ssl, you'll need to view the Security With Rediss section below after completing this section.
The recommended option to use for your results backend is a Redis server. See the Redis Documentation for instructions on how to create a Redis server.
Once your server is set up, we'll need six keys in the results_backend
section of the app.yaml file:
name
: The name of the results backend (here it will beredis
if running without ssl, orrediss
if running with ssl)password
: The path to the file that's storing your Redis passwordserver
: A URL to the server that you're connectingport
: The port that your server is running on. Default is 6379.db_num
: The database index (this will likely be 0).encryption_key
: The path to the encryption key (this is automatically generated bymerlin config
)
Using these settings, Merlin will construct a connection string of the form:
To further understand what the encryption_key
is for, see Security With Redis.
If using ssl, see Security With Rediss for additional setup instructions.
Example
Let's say we create a file in our ~/.merlin/
directory called redis.pass
. In this file we'll store the password for our Redis server:
~/.merlin/redis.pass | |
---|---|
Now we'll update the results_backend
section of our app.yaml file to be:
results_backend:
name: redis
password: ~/.merlin/redis.pass
server: server.domain.com
port: 6379
db_num: 0
encryption_key: ~/.merlin/encrypt_data_key
The connection string that Merlin generates will then become:
Security With Redis
Redis versions less than 6 do not natively support multiple users or SSL. We address security concerns here by redefining the core Celery routine that communicates with
redis to encrypt all data it sends to redis and then decrypt anything it receives. Each user should have their own encryption key as defined by
results_backend/encryption_key
in the app.yaml file. Merlin will generate a key if that key does not yet exist.
Security With Rediss
When using Redis with ssl, aka rediss, (only available with Redis v6.0.0+), there are some additional keys that you'll need to add to your results_backend
section:
results_backend:
name: rediss
password: ~/.merlin/redis.pass
server: server.domain.com
port: 6379
db_num: 0
encryption_key: ~/.merlin/encrypt_data_key
# ssl security
keyfile: /var/ssl/private/client-key.pem
certfile: /var/ssl/amqp-server-cert.pem
ca_certs: /var/ssl/myca.pem
# This is optional and can be required, optional or none
# (required is the default)
cert_reqs: required
The ssl config with redis (rediss) in the results backend is then placed in the redis_backend_use_ssl
celery argument.
redis_backend_use_ssl = {
'ssl_keyfile': '/var/ssl/private/client-key.pem',
'ssl_certfile': '/var/ssl/amqp-server-cert.pem',
'ssl_ca_certs': '/var/ssl/myca.pem',
'ssl_cert_reqs': ssl.CERT_REQUIRED
}
MySQL
Coming soon!
Resolving The Results Backend Password Field
The results_backend/password
is interpreted in the following way. First, it is treated as an absolute path to a file containing your backend password. If that path doesn't exist, it then looks for a file of that name under the directory defined under celery/certs
. If that file doesn't exist, it then treats results_backend/password
as the password itself.
Results Backend URL
A url
option is available to specify the results backend connection url, in which case the server name is ignored. The url must include the entire connection url including the ssl configuration.
Using the url
setting, Merlin will construct a connection string of the form:
Example
Say we use the default Redis server for our results backend:
The connection string that Merlin generates will then become:
The url
option can also be used to define a server that is not explicitly handled by the merlin configuration system.