Configuration

This section provides documentation for configuring Merlin’s connections with task servers and results backends.

Merlin server configuration

Merlin works best configuring celery to run with a RabbitMQ broker and a redis backend. Merlin uses celery chords which require a results backend be configured. The Amqp (rpc Rabbitmq) server does not support chords but the Redis, Database, Memcached and more, support chords.

Merlin’s configuration is controlled by an app.yaml file, such as the one below:

celery:
    # directory where Merlin looks for the following:
    # mysql-ca-cert.pem rabbit-client-cert.pem  rabbit-client-key.pem redis.pass
    certs: /path/to/celery/config

broker:
    # can be rabbitmq, redis, rediss, or redis+sock
    name: rabbitmq
    #username: # defaults to your username unless changed here
    password: ~/.merlin/rabbit-password
    # server URL
    server: server.domain.com

    ### for rabbitmq connections ###
    #vhost: # defaults to your username unless changed here

    ### for redis+sock connections ###
    #socketname: the socket name your redis connection can be found on.
    #path: The path to the socket.

    ### for redis/rediss connections ###
    #port: The port number redis is listening on (default 6379)
    #db_num: The data base number to connect to.
     
    # ssl security
    #keyfile: /var/ssl/private/client-key.pem
    #certfile: /var/ssl/amqp-server-cert.pem
    #ca_certs: /var/ssl/myca.pem
    # This is optional and can be required, optional or none
    # (required is the default)
    #cert_reqs: required


results_backend:
    # Can be redis,rediss, mysql, db+ or memcached server
    # Only a few of these are directly configured by merlin
    name: redis

    dbname: dbname
    username: username
    # name of file where redis password is stored.
    password: redis.pass
    server: server.domain.com
    # merlin will generate this key if it does not exist yet,
    # and will use it to encrypt all data over the wire to
    # your redis server.
    encryption_key: ~/.merlin/encrypt_data_key
    port: 6379
    db_num: 0

    # ssl security
    #keyfile: /var/ssl/private/client-key.pem
    #certfile: /var/ssl/amqp-server-cert.pem
    #ca_certs: /var/ssl/myca.pem
    # This is optional and can be required, optional or none
    # (required is the default)
    #cert_reqs: required

The default location for the app.yaml is in the merlin repo under the merlin/config directory. This default can be overridden by files in one of two other locations. The current working directory is first checked for the app.yaml file, then the user’s ~/.merlin directory is checked.

broker/name: can be rabbitmq, redis, or redis+sock. As their names imply, rabbitmq will use RabbitMQ as a task broker (preferred for multi user configurations), redis will use redis as a task broker, and redis+sock will connect to a redis task broker using a socket.

Broker: rabbitmq, amqps, amqp

Merlin constructs the following connection string from the relevant options in the broker section of the app.yaml file. If the port argument is not defined, the default rabbitmq TLS port, 5671, will be used. See the Security with RabbitMQ section for more info about security with this broker. When the broker is amqp, the default port will be 5672.

The prototype url for this configuration is:
{conn}://{username}:{password}@{server}:{port}/{vhost}

Here conn is amqps (with ssl) when name is rabbitmq or amqps and amqp (without ssl) when name is amqp.

broker:
  name: rabbitmq
  #username: # defaults to your username unless changed here
  password: ~/.merlin/rabbit-password
  # server URL
  server: server.domain.com
  #vhost: # defaults to your username unless changed here

Broker: redis

Merlin constructs the following connection string from the relevant options in the broker section of the app.yaml file.

The prototype url for this configuration is:
redis://:{password}@{server}:{port}/{db_num}
broker:
  name: redis
  server: localhost
  port: 6379

Broker: rediss

Newer versions of Redis (version 6 or greater) can be configured with ssl. The rediss name is used to enable this support. See the Security with redis section for more info.

The prototype url for this configuration is:
rediss://:{password}@{server}:{port}/{db_num}
broker:
  name: rediss
  server: localhost
  port: 6379

Broker: redis+socket

Merlin constructs the following connection string from the relevant options in the broker section of the app.yaml file.

The prototype url for this configuration is:
redis+socket://{path}?virtual_host={db_num}
broker:
  name: redis+socket
  path: /tmp/username/redis.sock
  db_num: 0

Broker: url

A url option is available to specify the broker connection url, in this case the server name is ignored. The url must include all the entire connection url except the ssl if the broker name is recognized by the ssl processing system. Currently the ssl system will only configure the Rabbitmq and Redis servers.

The prototype url for this configuration is:
{url}
broker:
  url: redis://localhost:6379/0

Broker: Security

Security with RabbitMQ

Merlin can only be configured to communicate with RabbitMQ over an SSL connection and does not permit use of a RabbitMQ server configured without SSL. The default value of the broker_use_ssl keyword is True. The keys can be given in the broker config as show below.

broker:
  # can be redis, redis+sock, or rabbitmq
  name: rabbitmq
  #username: # defaults to your username unless changed here
  password: ~/.merlin/rabbit-password
  # server URL
  server: server.domain.com

  ### for rabbitmq, redis+sock connections ###
  #vhost: # defaults to your username unless changed here

  # ssl security
  keyfile: /var/ssl/private/client-key.pem
  certfile: /var/ssl/amqp-server-cert.pem
  ca_certs: /var/ssl/myca.pem
  # This is optional and can be required, optional or none
  # (required is the default)
  cert_reqs: required

This results in a value for broker_use_ssl given below:

broker_use_ssl = {
'keyfile': '/var/ssl/private/client-key.pem',
'certfile': '/var/ssl/amqp-server-cert.pem',
'ca_certs': '/var/ssl/myca.pem',
'cert_reqs': ssl.CERT_REQUIRED
}

Security with redis

The same ssl config and resulting ssl_use_broker can be used with the rediss:// url when using a redis server version 6 or greater with ssl.

broker:
  name: rediss
  #username: # defaults to your username unless changed here
  # server URL
  server: server.domain.com
  port: 6379
  db_num: 0


  # ssl security
  keyfile: /var/ssl/private/client-key.pem
  certfile: /var/ssl/amqp-server-cert.pem
  ca_certs: /var/ssl/myca.pem
  # This is optional and can be required, optional or none
  # (required is the default)
  cert_reqs: required

The resulting broker_use_ssl configuration for a rediss server is given below.

broker_use_ssl = {
'ssl_keyfile': '/var/ssl/private/client-key.pem',
'ssl_certfile': '/var/ssl/amqp-server-cert.pem',
'ssl_ca_certs': '/var/ssl/myca.pem',
'ssl_cert_reqs': ssl.CERT_REQUIRED
}

Results backend: redis

Merlin constructs the following connection string from relevant options in the results_backend section of the app.yaml file.

The prototype url for this configuration is:
redis://:{password}{server}:{port}/{db_num}
results_backend:
  name: redis
  server: localhost
  port: 6379

Results backend: rediss

Newer versions of Redis (version 6 or greater) can be configured with ssl. The rediss name is used to enable this support. See the Security with redis section for more info.

The prototype url for this configuration is:
rediss://:{password}{server}:{port}/{db_num}
results_backend:
  name: rediss
  server: localhost
  port: 6379

Results backend: url

A url option is available to specify the results connection url, in this case the server name is ignored. The url must include the entire connection url including the ssl configuration.

The prototype url for this configuration is:
{url}
results_backend:
  url: redis://localhost:6379/0

The url option can also be used to define a server that is not explicitly handled by the merlin configuration system.

results_backend:
  url: db+postgresql://scott:tiger@localhost/mydatabase

Resolving password fields

The results_backend/password is interpreted in the following way. First, it is treated as an absolute path to a file containing your backend password. If that path doesn’t exist, it then looks for a file of that name under the directory defined under celery/certs. If that file doesn’t exist, it then looks treats results_backend/password as the password itself.

The broker/password is simply the full path to a file containing your password for the user defined by broker/username.

Results backend: Security

Security with redis

Redis versions less than 6 do not natively support multiple users or SSL. We address security concerns here by redefining the core celery routine that communicates with redis to encrypt all data it sends to redis and then decrypt anything it receives. Each user should have their own encryption key as defined by results_backend/encryption_key in the app.yaml file. Merlin will generate a key if that key does not yet exist.

Redis versions 6 or greater can use the ssl keys as in the broker section. The ssl config with redis (rediss) in the results backend is the placed in the redis_backend_use_ssl celery argument. The values in this argument are the same as the broker.

redis_backend_use_ssl = {
'ssl_keyfile': '/var/ssl/private/client-key.pem',
'ssl_certfile': '/var/ssl/amqp-server-cert.pem',
'ssl_ca_certs': '/var/ssl/myca.pem',
'ssl_cert_reqs': ssl.CERT_REQUIRED
}