Command Line Interface

The Merlin library defines a number of commands to help configure your server and manage and monitor your workflow.

This module will detail every command available with Merlin.

Merlin

The entrypoint to everything related to executing Merlin commands.

Usage:

merlin [OPTIONS] COMMAND [ARGS] ...

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False
--version	boolean	Show program's version number and exit	False
-lvl, --level	choice(ERROR | WARNING | INFO | DEBUG)	Level of logging messages to be output. The smaller the number in the table below, the more output that's produced: Log Level Choice 4 ERROR 3 WARNING 2 INFO (default) 1 DEBUG	INFO

See the Configuration Commands, Workflow Management Commands, and Monitoring Commands below for more information on every command available with the Merlin library.

Configuration Commands

Since running Merlin in a distributed manner requires the configuration of a centralized server, Merlin comes equipped with three commands to help users get this set up:

• config: Create, update, or select a configuration file
• info: Ensure stable connections to the server(s)
• server: Spin up containerized servers

Config (merlin config)

Create, update, or select a configuration file that Merlin will use to connect to your server(s).

Merlin config has a list of commands for interacting with configuration files. These commands allow the user to create and update configuration files, and select which one should be the active configuration.

See more information on how to set up the configuration file at the Configuration page.

Usage:

merlin config [OPTIONS] COMMAND [ARGS] ...

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False

Commands:

Name	Description
create	Create a template configuration file
update-backend	Update broker settings in the configuration file
update-broker	Update backend settings in the configuration file
use	Use a different configuration setup

Config Create (merlin config create)

The merlin config create command creates a template configuration file that you can customize to connect to your central server. Detailed instructions for completing this template are available in the Configuring the Broker and Results Backend guide.

By default, the generated configuration file is saved at $(HOME)/.merlin/app.yaml. If you prefer to rename the file or save it to a different location, you can use the -o option to specify the desired path. Note that the file must have the .yaml extension.

The default configuration sets the broker to use a RabbitMQ server and the results backend to Redis. While the Redis results backend is mandatory, the broker can be configured to use either RabbitMQ or Redis. To switch to a Redis broker, use the --broker option.

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False
--task_server	string	Select the appropriate configuration for the given task server. Currently only "celery" is implemented.	"celery"
-o, --output-file	path	Optional yaml file name for your configuration. Default: $(HOME)/.merlin/app.yaml.	None
--broker	string	Write the initial app.yaml config file for either a rabbitmq or redis broker. The default is rabbitmq. The backend will be redis in both cases. The redis backend in the rabbitmq config shows the use on encryption for the backend.	"rabbitmq"

Examples:

Create an app.yaml File at ~/.merlin

merlin config create

Create a Configuration File at a Custom Path

merlin config create -o /Documents/configuration/merlin_config.yaml

Create a Configuration File With a Redis Broker

merlin config create --broker redis

Config Update-Backend (merlin config update-backend)

The merlin config update-backend command allows you to modify the results_backend section of your configuration file directly from the command line. See the options table below to see exactly what settings can be set.

Usage:

merlin config update-backend -t {redis} [OPTIONS] ...

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False
-t, --type	choice(redis)	Type of results backend to configure.	None
--cf, --config-file	string	Path to the config file that will be updated.	$(HOME)/.merlin/app.yaml
-u, --username	string	The backend username.	None
--pf, --password-file	string	Path to a password file that contains the password to the backend.	None
-s, --server	string	The URL of the backend server.	None
-p, --port	int	The port number that this backend server is using.	None
-d, --db-num	int	The backend database number.	None
-c, --cert-reqs	string	Backend cert requirements.	None
-e, --encryption-key	string	Path to the encryption key file.	None

Examples:

Update Every Setting Required for Redis

merlin config update-backend -t redis --pf ~/.merlin/redis.pass -s my-redis-server.llnl.gov -p 6379 -d 0 -c none

This will create the following results_backend section in your app.yaml file:

results_backend:
    cert_reqs: none
    db_num: 0
    encryption_key: ~/.merlin/encrypt_data_key
    name: rediss
    password: ~/.merlin/redis.pass
    port: 6379
    server: my-redis-server.llnl.gov
    username: ''

Update Just the Port

merlin config update-backend -t redis -p 6379

Update a Custom Configuration File Path

merlin config update-backend -t redis --cf /path/to/custom_config.yaml -s new-server.gov

Config Update-Broker (merlin config update-broker)

The merlin config update-broker command allows you to modify the broker section of your configuration file directly from the command line. See the options table below to see exactly what settings can be set.

Usage:

merlin config update-broker -t {rabbitmq,redis} [OPTIONS] ...

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False
-t, --type	choice(rabbitmq | redis)	Type of broker to configure.	None
--cf, --config-file	string	Path to the config file that will be updated.	$(HOME)/.merlin/app.yaml
-u, --username	string	The broker username (only for rabbitmq broker).	None
--pf, --password-file	string	Path to a password file that contains the password to the broker.	None
-s, --server	string	The URL of the broker server.	None
-p, --port	int	The port number that this broker server is using.	None
-v, --vhost	string	The vhost for the broker (only for rabbitmq broker).	None
-d, --db-num	int	The backend database number (only for redis broker).	None
-c, --cert-reqs	string	Backend cert requirements.	None

Examples:

Update Every Setting Required for a Redis Broker

merlin config update-broker -t redis --pf ~/.merlin/redis.pass -s my-redis-server.llnl.gov -p 6379 -d 0 -c none

This will create the following broker section in your app.yaml file:

broker:
    cert_reqs: none
    db_num: 0
    name: rediss
    password: ~/.merlin/redis.pass
    port: 6379
    server: my-redis-server.llnl.gov
    username: ''

Update Every Setting Required for a RabbitMQ Broker

merlin config update-broker -t rabbitmq -u my_rabbit_username --pf ~/.merlin/rabbit.pass -s my-rabbitmq-server.llnl.gov -p 5672 -v host4rabbit -c none

This will create the following broker section in your app.yaml file:

broker:
    cert_reqs: none
    name: rabbitmq
    password: ~/.merlin/rabbit.pass
    port: 5672
    server: my-rabbitmq-server.llnl.gov
    username: my_rabbit_username
    vhost: host4rabbit

Update Just the Username

merlin config update-broker -t rabbitmq -u my_new_username

Update a Custom Configuration File Path

merlin config update-broker -t redis --cf /path/to/custom_config.yaml -s new-server.gov

Config Use (merlin config use)

The merlin config use command allows you to switch which configuration file to use as your active configuration.

Usage:

merlin config use [OPTIONS] CONFIG_FILE

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False

Examples:

Use a Custom Configuration

merlin config use /path/to/custom_config.yaml

Info (merlin info)

Information about your Merlin and Python configuration can be printed out by using the info command. This is helpful for debugging. Included in this command is a server check which will check for server connections. The connection check will timeout after 60 seconds.

Usage:

merlin info [OPTIONS]

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False

Server (merlin server)

Create a local containerized server for Merlin to connect to. Merlin server creates and configures a server on the current directory. This allows multiple instances of Merlin server to exist for different studies or uses.

Merlin server has a list of commands for interacting with the broker and results server. These commands allow the user to manage and monitor the exisiting server and create instances of servers if needed.

More information on configuring with Merlin server can be found at the Containerized Server Configuration page.

Usage:

merlin server [OPTIONS] COMMAND [ARGS] ...

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False

Commands:

Name	Description
init	Initialize the files needed for Merlin server
status	Check the status of your Merlin server
start	Start the containerized Merlin server
stop	Stop the Merlin server
restart	Restart an instance of the Merlin server
config	Configure the Merlin server

Server Init (merlin server init)

Note

If there is an exisiting subdirectory containing a merlin server configuration then only missing files will be replaced. However it is recommended that users backup their local configurations prior to running this command.

The init subcommand initalizes a new instance of Merlin server by creating configurations for other subcommands.

A main Merlin sever configuration subdirectory is created at ~/.merlin/server/ which contains configuration for local Merlin configuration, and configurations for different containerized services that Merlin server supports, which includes Singularity (Docker and Podman implemented in the future).

A local Merlin server configuration subdirectory called merlin_server/ will also be created in your current working directory when this command is run. This will include a container for merlin server and associated configuration files that might be used to start the server. For example, for a redis server a "redis.conf" will contain settings which will be dynamically loaded when the redis server is run. This local configuration will also contain information about currently running containers as well.

Usage:

merlin server init [OPTIONS]

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False

Server Status (merlin server status)

The status subcommand checks the status of the Merlin server.

Usage:

merlin server status [OPTIONS]

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False

Server Start (merlin server start)

Warning

Newer versions of Redis have started requiring a global variable LC_ALL to be set in order for this to work. To set this properly, run:

export LC_ALL="C"

If this is not set, the merlin server start command may seem to hang until you manually terminate it.

The start subcommand starts the Merlin server using the container located in the local merlin server configuration.

Usage:

merlin server start [OPTIONS]

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False

Server Stop (merlin server stop)

The stop subcommand stops any exisiting container being managed and monitored by Merlin server.

Usage:

merlin server stop [OPTIONS]

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False

Server Restart (merlin server restart)

The restart subcommand performs a stop command followed by a start command on the Merlin server.

Usage:

merlin server restart [OPTIONS]

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False

Server Config (merlin server config)

The config subcommand edits configurations for the Merlin server. There are multiple options to allow for different configurations.

Usage:

merlin server config [OPTIONS]

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False
-ip, --ipadress	string	Set the binded IP address for Merlin server	None
-p, --port	integer	Set the binded port for Merlin server	None
-pwd, --password	filename	Set the password file for Merlin server	None
--add-user	string string	Add a new user for Merlin server. This requires a space-delimited username and password as input.	None
--remove-user	string	Remove an existing user from Merlin server	None
-d, --directory	path	Set the working directory for Merlin server	None
-ss, --snapshot-seconds	integer	Set the number of seconds before each snapshot	None
-sc, --snapshot-changes	integer	Set the number of database changes before each snapshot	None
-sf, --snapshot-file	filename	Set the name of the snapshot file	None
-am, --append-mode	choice(always | everysec | no)	Set the appendonly mode	None
-af, --append-file	filename	Set the name of the file for the server append/change file	None

Examples:

Configure The Port and Password

merlin server config -p 5879 -pwd /Documents/redis.pass

Add A User and Set Snapshot File

merlin server config --add-user custom_username custom_password -sf /Documents/snapshot

Workflow Management Commands

The Merlin library provides several commands for setting up and managing your Merlin workflow:

• cancel: Cancel a study.
• database: Interact with Merlin's backend database
• example: Download pre-made workflow specifications that can be modified for your own workflow needs
• purge: Clear any tasks that are currently living in the central server
• restart: Restart a workflow
• run: Send tasks to the central server
• run workers: Start up workers that will execute the tasks that exist on the central server
• stop workers: Stop existing workers

Cancel (merlin cancel)

The merlin cancel command allows you to cancel a running study. A study may have multiple runs executing concurrently on multiple workers being watched by a monitor process to ensure every run finishes. This command will help ensure that every piece of this process is cancelled gracefully.

In other words, the merlin cancel command acts as a wrapper around the merlin purge and merlin stop-workers commands. Additionally, any running studies associated with the study being cancelled will be marked as cancelled in the database. This ensures that the merlin monitor command will no longer track these studies.

In short, this command will:

1. Purge the queues from the study (merlin purge -f SPECIFICATION)
2. Stop the workers for that study (merlin stop-workers --spec SPECIFICATON)
3. Mark all runs associated with that study as cancelled

Each of these options can be disabled with their respective options in the table below.

Usage:

merlin cancel [OPTIONS] SPECIFICATION

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False
--no-purge	boolean	Skip purging the queues for the study (skip step 1 above).	False
--no-stop-workers	boolean	Skip stopping the workers for the study (skip step 2 above).	False
--no-mark-cancelled	boolean	Skip marking runs as cancelled in the database (skip step 3 above).	False
--vars	List[string]	A space-delimited list of variables to override in the spec file. This list should be given after the spec file is provided. Ex: --vars LEARN=/path/to/new_learn.py EPOCHS=3	None

Examples:

Basic Cancel Example

merlin cancel my_specification.yaml

Cancel Without Purging Queues Example

merlin cancel my_specification.yaml --no-purge

Cancel Without Stopping Workers Example

merlin cancel my_specification.yaml --no-stop-workers

Cancel Without Marking Runs as Cancelled Example

merlin cancel my_specification.yaml --no-mark-cancelled

Cancel and Substitute Variables Example

merlin cancel my_specification.yaml --vars CUSTOM_QUEUE=new_queue

Database (merlin database)

Alias

The database command can be abbreviated as db.

This command allows you to interact with Merlin's backend database by viewing database info, retrieving and printing entries, and deleting entries. If you ran your study locally, use the --local option here as well when running database commands.

More information on this command can be found below or at The Database Command page. See Merlin's Database for more general information on the database itself.

Usage:

merlin database [OPTIONS] COMMAND ...

or

merlin db [OPTIONS] COMMAND ...

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False
-l, --local	boolean	Use the local SQLite database for this command.	False

Commands:

Name	Description
info	Print general information about the database
get	Retrieve and print entries from the database
delete	Delete entries from the database
gc	Run garbage collection on the database

Database Info (merlin database info)

The info subcommand prints general information about the database, including the database type, version, and brief details about the existing entries.

More information on this subcommand can be found at The Info Subcommand.

Usage:

merlin database info [OPTIONS]

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False

Database Get (merlin database get)

The get subcommand allows users to retrieve entries from the database and print them to the console.

More information on this subcommand can be found at Retrieving Data.

Usage:

merlin database get [OPTIONS] SUBCOMMAND ...

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False

Subcommands:

Name	Description
study	Retrieve and print specific study(ies) from the database
run	Retrieve and print specific run(s) from the database
logical-worker	Retrieve and print specific logical worker(s) from the database
physical-worker	Retrieve and print specific physical worker(s) from the database
all-studies	Retrieve and print all studies from the database (supports filters)
all-runs	Retrieve and print all runs from the database (supports filters)
all-logical-workers	Retrieve and print all logical workers from the database (supports filters)
all-physical-workers	Retrieve and print all physical workers from the database (supports filters)
everything	Retrieve and print every entry from the database

Get Study (merlin database get study)

The get study subcommand allows users to retrieve specific study entries from the database by study ID or name and print them to the console.

Usage:

merlin database get study [OPTIONS] STUDY_ID_OR_NAME [STUDY_ID_OR_NAME ...]

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False

Get Run (merlin database get run)

The get run subcommand allows users to retrieve specific run entries from the database by run ID or workspace and print them to the console.

Usage:

merlin database get run [OPTIONS] RUN_ID_OR_WORKSPACE [RUN_ID_OR_WORKSPACE ...]

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False

Get Logical-Worker (merlin database get logical-worker)

The get logical-worker subcommand allows users to retrieve specific logical-worker entries from the database by ID and print them to the console.

Usage:

merlin database get logical-worker [OPTIONS] LOGICAL_WORKER_ID [LOGICAL_WORKER_ID ...]

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False

Get Physical-Worker (merlin database get physical-worker)

The get physical-worker subcommand allows users to retrieve specific physical-worker entries from the database by ID or name and print them to the console.

Usage:

merlin database get physical-worker [OPTIONS] PHYSICAL_WORKER_ID_OR_NAME [PHYSICAL_WORKER_ID_OR_NAME ...]

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False

Get All-Studies (merlin database get all-studies)

The get all-studies subcommand allows users to retrieve all study entries from the database and print them to the console. This command supports filtering.

Usage:

merlin database get all-studies [OPTIONS]

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False
--name	string	Filter by name	None

Get All-Runs (merlin database get all-runs)

The get all-runs subcommand allows users to retrieve all run entries from the database and print them to the console. This command supports filtering.

Usage:

merlin database get all-runs [OPTIONS]

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False
--study-id	string	Filter by study id	None
--run-complete	choice(true | false)	Filter by run complete	None
--queues	List[string]	Filter by queues	None
--workers	List[string]	Filter by workers	None

Get All-Logical-Workers (merlin database get all-logical-workers)

The get all-logical-workers subcommand allows users to retrieve all logical-worker entries from the database and print them to the console. This command supports filtering.

Usage:

merlin database get all-logical-workers [OPTIONS]

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False
--name	string	Filter by name	None
--queues	List[string]	Filter by queues	None

Get All-Physical-Workers (merlin database get all-physical-workers)

The get all-physical-workers subcommand allows users to retrieve all physical-worker entries from the database and print them to the console. This command supports filtering.

Usage:

merlin database get all-physical-workers [OPTIONS]

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False
--logical-worker-id	string	Filter by logical worker id	None
--name	string	Filter by name	None
--status	string	Filter by status	None
--host	string	Filter by host	None

Get Everything (merlin database get everything)

The get everything subcommand allows users to retrieve every entry from the database and print them all to the console.

Usage:

merlin database get everything [OPTIONS]

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False

Database Delete (merlin database delete)

The delete subcommand allows users to delete entries from the database.

More information on this subcommand can be found at Deleting Data.

Usage:

merlin database delete [OPTIONS] SUBCOMMAND ...

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False

Subcommands:

Name	Description
study	Delete specific study(ies) from the database
run	Delete specific run(s) from the database
logical-worker	Delete specific logical worker(s) from the database
physical-worker	Delete specific physical worker(s) from the database
all-studies	Delete all studies from the database (supports filters)
all-runs	Delete all runs from the database (supports filters)
all-logical-workers	Delete all logical workers from the database (supports filters)
all-physical-workers	Delete all physical workers from the database (supports filters)
everything	Delete everything from the database

Delete Study (merlin database delete study)

Warning

By default, this command will also delete all of the runs associated with a study. To disable this, use the -k option mentioned in the table below.

The delete study subcommand allows users to delete specific study entries from the database by ID or name.

Usage:

merlin database delete study [OPTIONS] STUDY_ID_OR_NAME [STUDY_ID_OR_NAME ...]

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False
-k, --keep-associated-runs	boolean	Keep runs associated with the study that's being deleted	False

Delete Run (merlin database delete run)

The delete run subcommand allows users to delete specific run entries from the database by ID or workspace.

Usage:

merlin database delete run [OPTIONS] RUN_ID_OR_WORKSPACE [RUN_ID_OR_WORKSPACE ...]

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False

Delete Logical-Worker (merlin database delete logical-worker)

The delete logical-worker subcommand allows users to delete specific logical worker entries from the database by ID.

Usage:

merlin database delete logical-worker [OPTIONS] LOGICAL_WORKER_ID [LOGICAL_WORKER_ID ...]

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False

Delete Physical-Worker (merlin database delete physical-worker)

The delete physical-worker subcommand allows users to delete specific physical worker entries from the database by ID or name.

Usage:

merlin database delete physical-worker [OPTIONS] PHYSICAL_WORKER_ID_OR_NAME [PHYSICAL_WORKER_ID_OR_NAME ...]

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False

Delete All-Studies (merlin database delete all-studies)

Warning

This command will also delete all of the runs in the database.

The delete all-studies subcommand allows users to delete all study entries from the database.

Usage:

merlin database delete all-studies [OPTIONS]

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False
-k, --keep-associated-runs	boolean	Keep runs associated with the studies	False
--name	string	Filter by name	None

Delete All-Runs (merlin database delete all-runs)

The delete all-runs subcommand allows users to delete all run entries from the database.

Usage:

merlin database delete all-runs [OPTIONS]

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False
--study-id	string	Filter by study id	None
--run-complete	choice(true | false)	Filter by run complete	None
--queues	List[string]	Filter by queues	None
--workers	List[string]	Filter by workers	None

Delete All-Logical-Workers (merlin database delete all-logical-workers)

The delete all-logical-workers subcommand allows users to delete all logical worker entries from the database.

Usage:

merlin database delete all-logical-workers [OPTIONS]

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False
--name	string	Filter by name	None
--queues	List[string]	Filter by queues	None

Delete All-Physical-Workers (merlin database delete all-physical-workers)

The delete all-physical-workers subcommand allows users to delete all physical worker entries from the database.

Usage:

merlin database delete all-physical-workers [OPTIONS]

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False
--logical-worker-id	string	Filter by logical worker id	None
--name	string	Filter by name	None
--status	string	Filter by status	None
--host	string	Filter by host	None

Delete Everything (merlin database delete everything)

The delete everything subcommand allows users to delete every entry from the database.

Usage:

merlin database delete everything [OPTIONS]

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False
-f, --force	boolean	Delete everything in the database without confirmation	False

Database Garbage Collect (merlin database gc)

Aliases

The gc command can be also be invoked with garbage-collect or cleanup.

The gc subcommand performs garbage collection on the database to identify and remove stale entries. This cleanup process targets:

1. Runs whose workspace directories no longer exist on the filesystem
2. Studies that have no associated runs remaining after the run cleanup
3. Logical workers that only reference runs that no longer exist
4. Physical workers that only reference logical workers that no longer exist

More information on this subcommand can be found at Database Garbage Collection.

Usage:

merlin database gc [OPTIONS]

or

merlin database garbage-collect [OPTIONS]

or

merlin database cleanup [OPTIONS]

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False
--dry-run	boolean	Show what would be deleted without actually deleting anything	False
--skip-runs	boolean	Skip checking for runs with invalid workspaces	False
--skip-workers	boolean	Skip checking for orphaned workers (both logical and physical)	False
--skip-logical-workers	boolean	Skip checking for orphaned logical workers	False
--skip-physical-workers	boolean	Skip checking for orphaned physical workers	False
--skip-studies	boolean	Skip checking for empty studies	False
-f, --force	boolean	Skip confirmation prompt (use with caution)	False

Examples:

Clean Up All Stale Entries

merlin database gc

Preview What Would Be Deleted Without Actually Deleting

merlin database gc --dry-run

Don't Clean Up Stale Runs

merlin database gc --skip-runs

Bypass Confirmation Prompt

merlin database gc --force

Example (merlin example)

If you want to obtain an example workflow, use Merlin's merlin example command. First, view all of the example workflows that are available with:

merlin example list

This will list the available example workflows and a description for each one. To select one:

merlin example <example_name>

This will copy the example workflow to the current working directory. It is possible to specify another path to copy to.

merlin example <example_name> -p path/to/dir

If the specified directory does not exist Merlin will automatically create it.

This will generate the example workflow at the specified location, ready to be run.

For more information on these examples, visit the Examples page.

Usage:

merlin example [OPTIONS] [list | <example_name>]

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False
-p, --path	path	A directory path to download the example to	Current Working Directory

Purge (merlin purge)

Warning

Any tasks reserved by workers will not be purged from the queues. All workers must be first stopped so the tasks can be returned to the task server and then they can be purged.

In short, you probably want to use merlin stop-workers before running merlin purge.

If you've executed the merlin run command and sent tasks to the server, this command can be used to remove those tasks from the server. If there are no tasks currently on the server then this command will not do anything.

Usage:

merlin purge [OPTIONS] SPECIFICATION

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False
-f	boolean	Purge tasks without confirmation	False
--steps	List[string]	A space-delimited list of steps from the specification file to purge	['all']
--vars	List[string]	A space-delimited list of variables to override in the spec file. Ex: --vars MY_QUEUE=hello	None

Examples:

Purge All Queues From Spec File

The following command will purge all queues that exist in my_specification.yaml:

merlin purge my_specification.yaml

Purge Specific Steps From Spec File

The following command will purge any queues associated with step_1 and step_3 in my_specification.yaml:

merlin purge my_specification.yaml --steps step_1 step_3

Purge Queues Without Confirmation

The following command will ignore the confirmation prompt that's provided and purge the queues:

merlin purge -f my_specification.yaml

Restart (merlin restart)

To restart a previously started Merlin workflow, use the restart command and the path to root of the Merlin workspace that was generated during the previously run workflow. This will define the tasks and queue them on the task server also called the broker.

Merlin currently writes file called MERLIN_FINISHED to the directory of each step that was finished successfully. It uses this to determine which steps to skip during execution of a restarted workflow.

Usage:

merlin restart [OPTIONS] WORKSPACE

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False
--local	string	Run tasks sequentially in your current shell	"distributed"

Examples:

Restart an Existing Workflow

merlin restart my_study_20240102-143903/

Restart an Existing Workflow Locally

merlin restart my_study_20240102-143903/ --local

Run (merlin run)

To run a Merlin workflow use the run command and the path to the input yaml file <input.yaml>. This will define the tasks and queue them on the task server also called the broker.

Usage:

merlin run [OPTIONS] SPECIFICATION

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False
--local	string	Run tasks sequentially in your current shell	"distributed"
--vars	List[string]	A space-delimited list of variables to override in the spec file. This list should be given after the spec file is provided. Ex: --vars LEARN=/path/to/new_learn.py EPOCHS=3	None
--samplesfile	choice(<filename>.npy | <filename>.csv | <filename>.tab)	Specify a file containing samples. This file should be given after the spec file is provided.	None
--dry	boolean	Do a Dry Run of your workflow	False
--no-errors	boolean	Silence the errors thrown when flux is not present	False
--pgen	filename	Specify a parameter generator filename to override the global.parameters block of your spec file	None
--pargs	string	A string that represents a single argument to pass a custom parameter generation function. Reuse --parg to pass multiple arguments. [Use with --pgen]	None

Examples:

Basic Run Example

merlin run my_specification.yaml

Pass A Parameter Generator File to Run

merlin run my_specification.yaml --pgen /path/to/pgen.py

Pass A Samples File to Run

merlin run my_specification.yaml --samplesfile /path/to/samplesfile.csv

Do A Dry Run of Your Workflow Locally

merlin run my_specification.yaml --dry --local

Run Workers (merlin run-workers)

The tasks queued on the broker by the merlin run command are run by a collection of workers. These workers can be run local in the current shell or in parallel on a batch allocation. The workers are launched using the run-workers command which reads the configuration for the worker launch from the <input.yaml> file.

Within the <input.yaml> file, the batch and merlin.resources.workers sections are both used to configure the worker launch. The top level batch section can be overridden in the merlin.resources.workers section. Parallel workers should be scheduled using the system's batch scheduler (see the section describing Distributed Runs for more info).

Once the workers are running, tasks from the broker will be processed.

Usage:

merlin run-workers [OPTIONS] SPECIFICATION

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False
--echo	boolean	Echo the Celery workers run command to stdout and don't start any workers	False
--worker-args	string	Pass arguments (all wrapped in quotes) to the Celery workers. Should be given after the input spec.	None
--steps	List[string]	The specific steps in the input spec that you want to run the corresponding workers for. Should be given after the input spec.	['all']
--vars	List[string]	A space-delimited list of variables to override in the spec file. This list should be given after the spec file is provided. Ex: --vars SIMWORKER=new_sim_worker	None
--disable-logs	boolean	Disable logs for Celery workers. Note: Having the -l flag in your workers' args section will overwrite this flag for that worker.	False

Examples:

Basic Worker Launch

merlin run-workers my_specification.yaml

Worker Launch for Just Certain Steps

merlin run-workers my_specification.yaml --steps step_1 step_3

Worker Launch with Worker Args Passed

merlin run-workers my_specification.yaml --worker-args "-l INFO --concurrency 4"

Stop Workers (merlin stop-workers)

Warning

If you've named workers identically across workflows (you shouldn't) only one might get the signal. In this case, you can send it again.

Send out a stop signal to some or all connected workers. By default, a stop will be sent to all connected workers across all workflows, having them shutdown softly. This behavior can be modified with certain options.

Usage:

merlin stop-workers [OPTIONS]

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False
--spec	filename	Target only the workers named in the merlin block of the spec file given here	None
--queues	List[string]	Takes a space-delimited list of specific queues as input and will stop all workers watching these queues	None
--workers	List[regex]	A space-delimited list of regular expressions representing workers to stop	None
--task_server	string	Task server type for which to stop the workers. Currently only "celery" is implemented.	"celery"

Examples:

Stop All Workers Across All Workflows

merlin stop-workers

Stop Workers for a Certain Specification

merlin stop-workers --spec my_specification.yaml

Stop Workers for Certain Queues

merlin stop-workers --queues queue_1 queue_2

Stop Specific Workers Using Regex

merlin stop-workers --workers ".*@my_other_host*"

Monitoring Commands

The Merlin library comes equipped with several commands to help monitor your workflow:

• detailed-status: Display task-by-task status information for a study
• monitor: Keep your allocation alive while tasks are being processed
• query-workers: View information on worker entities
• queue-info: Communicate with Celery to view the status of queues in your workflow(s)
• status: Display a summary of the status of a study

More information on all of these commands can be found below and in the Monitoring documentation.

Detailed Status (merlin detailed-status)

Warning

For the pager opened by this command to work properly the MANPAGER or PAGER environment variable must be set to less -r. This can be set with:

MANPAGERPAGER

export MANPAGER="less -r"

export PAGER="less -r"

Display the task-by-task status of a workflow.

This command will open a pager window with task statuses. Inside this pager window, you can search and scroll through task statuses for every step of your workflow.

For more information, see the Detailed Status documentation.

Usage:

merlin detailed-status [OPTIONS] WORKSPACE_OR_SPECIFICATION

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False
--dump	filename	The name of a csv or json file to dump the status to	None
--task_server	string	Task server type. Currently only "celery" is implemented.	"celery"
-o, --output-path	dirname	Specify a location to look for output workspaces. Only used when a spec file is passed as the argument to status.	None

Filter Options:

The detailed-status command comes equipped with several options to help filter the output of your status query.

Name	Type	Description	Default
--max-tasks	integer	Sets a limit on how many tasks can be displayed.	None
--return-code	List[string]	Filter which tasks to display based on their return code. Multiple return codes can be provided using a space-delimited list. Options: SUCCESS, SOFT_FAIL, HARD_FAIL, STOP_WORKERS, RETRY, DRY_SUCCESS, UNRECOGNIZED.	None
--steps	List[string]	Filter which tasks to display based on the steps that they're associated with. Multiple steps can be provided using a space-delimited list.	['all']
--task-queues	List[string]	Filter which tasks to display based on a the task queues that they were/are in. Multiple task queues can be provided using a space-delimited list.	None
--task-status	List[string]	Filter which tasks to display based on their status. Multiple statuses can be provided using a space-delimited list. Options: INITIALIZED, RUNNING, FINISHED, FAILED, CANCELLED, DRY_RUN, UNKNOWN.	None
--workers	List[string]	Filter which tasks to display based on which workers are processing them. Multiple workers can be provided using a space-delimited list.	None

Display Options:

There are multiple options to modify the way task statuses are displayed.

Name	Type	Description	Default
--disable-pager	boolean	Turn off the pager functionality when viewing the task-by-task status. Caution: This option is not recommended for large workflows as you could freeze your terminal with thousands of task statuses.	False
--disable-theme	boolean	Turn off styling for the status layout.	False
--layout	string	Alternate task-by-task status display layouts. Options: table, default.	default
--no-prompts	boolean	Ignore any prompts provided. This cause the detailed-status command to default to the latest study if you provide a spec file as input.	False

Examples:

Check the Detailed Status Using Workspace as Input

merlin detailed-status study_name_20240129-123452/

Check the Detailed Status Using a Specification as Input

This will look in the OUTPUT_PATH Reserved Variable defined within the spec file to try to find existing workspace directories associated with this spec file. If more than one are found, a prompt will be displayed for you to select a workspace directory.

merlin detailed-status my_specification.yaml

Dump the Status Report to a JSON File

merlin detailed-status study_name_20240129-123452/ --dump status_report.json

Only Display Failed Tasks

merlin detailed-status study_name_20240129-123452/ --task-status FAILED

Display the First 8 Successful Tasks

merlin detailed-status study_name_20240129-123452/ --return-code SUCCESS --max-tasks 8

Disable the Theme

merlin detailed-status study_name_20240129-123452/ --disable-theme

Use the Table Layout

merlin detailed-status study_name_20240129-123452/ --layout table

Monitor (merlin monitor)

Tip When Running Multiple Monitor Instances

When having multiple instances of merlin monitor running for the same study, we recommend turning off the restart functionality with --no-restart for all except one of them, so as to avoid potential race conditions and trying to restart the workflow multiple times at once.

Batch submission scripts may not keep the batch allocation alive if there is not a blocking process in the submission script. The merlin monitor command addresses this by providing a blocking process that checks for tasks in the queues every (sleep) seconds ("sleep" here can be defined with the --sleep option). When the queues are empty, the monitor will query Celery to see if any workers are still processing tasks from the queues. If no workers are processing any tasks from the queues and the queues are empty, the blocking process will exit and allow the allocation to end.

If for some reason your workflow enters a stalled state where the queues are empty, no workers are processing tasks, but your workflow has not yet finished, then the monitor will attempt to automatically restart the workflow for you. This functionality can be disabled with the --no-restart option.

The monitor functionality will check for Celery workers for up to 10*(sleep) seconds before monitoring begins. The loop happens when the queue(s) in the spec contain tasks, but no running workers are detected. This is to protect against a failed worker launch.

When the monitor is first started, it will run automatic garbage collection on the runs and studies in your database. To disable this use the --disable-gc option.

For more information, see the Monitoring Studies for Persistent Allocations documentation.

Usage:

merlin monitor [OPTIONS] SPECIFICATION

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False
--steps	List[string]	A space-delimited list of steps in the input spec that you want to query. Should be given after the input spec.	['all']
--vars	List[string]	A space-delimited list of variables to override in the spec file. This list should be given after the spec file is provided. Ex: --vars SIMWORKER=new_sim_worker	None
--sleep	integer	The duration in seconds between checks for workers/tasks	60
--task_server	string	Task server type for which to monitor the workers. Currently only "celery" is implemented.	"celery"
--no-restart, -n	boolean	Disable the automatic restart functionality for this monitor.	False
--disable-gc	boolean	Disable automatic database garbage collection on monitor startup.	False

Basic Monitor

merlin monitor my_specification.yaml

Monitor Specific Steps

merlin monitor my_specification.yaml --steps step_1 step_3

Monitor With a Shortened Sleep Interval

merlin monitor my_specification.yaml --sleep 30

Query Workers (merlin query-workers)

View information on worker entities in your database.

This will query the Merlin Database for information on logical worker entities and physical worker entities. This can be useful for seeing which workers are running and where.

For more information, see the Query Workers documentation.

Usage:

merlin query-workers [OPTIONS]

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False
--task_server	string	Task server type for which to query workers. Currently only "celery" is implemented.	"celery"
--spec	filename	Query for the workers named in the merlin block of the spec file given here	None
--queues	List[string]	Takes a space-delimited list of queues as input. This will query for workers associated with the names of the queues you provide here.	None
--workers	List[string]	A space-delimited list of logical worker names to query	None
-f, --format	choice(rich | json)	Output format	rich
-l, --local-db	boolean	Use the local Merlin database for querying workers	False

Examples:

Query All Active Workers

merlin query-workers

Query Workers of Specific Queues

merlin query-workers --queues demo merlin

Query Workers From Spec File

merlin query-workers --spec my_specification.yaml

Query Workers Based on Their Name

This will query a worker named step_1_worker:

merlin query-workers --workers step_1_worker

Queue Info (merlin queue-info)

Note

Prior to Merlin v1.12.0 the merlin status command would produce the same output as merlin queue-info --spec <spec_file>

Check the status of queues to see if there are any tasks in them and/or any workers watching them.

If used without the --spec option, this will query any active queues. Active queues are queues that have a worker watching them.

For more information, see the Queue Information documentation.

Usage:

merlin queue-info [OPTIONS]

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False
--dump	filename	The name of a csv or json file to dump the queue information to	None
--specific-queues	List[string]	A space-delimited list of queues to get information on	None
--task_server	string	Task server type. Currently only "celery" is implemented.	"celery"

Specification Options:

These options all must be used with the --spec option if used.

Name	Type	Description	Default
--spec	filename	Query for the queues named in each step of the spec file given here	None
--steps	List[string]	A space-delimited list of steps in the input spec that you want to query. Should be given after the input spec.	['all']
--vars	List[string]	A space-delimited list of variables to override in the spec file. This list should be given after the spec file is provided. Ex: --vars QUEUE_NAME=new_queue_name	None

Examples:

Query All Active Queues

merlin queue-info

Check the Status of Specific Queues

merlin queue-info --specific-queues queue_1 queue_3

Check the Status of Queues in a Spec File

This is the same as running merlin status <spec_file> prior to Merlin v1.12.0

merlin queue-info --spec my_specification.yaml

Check the Status of Queues for Specific Steps

merlin queue-info --spec my_specification.yaml --steps step_1 step_3

Dump the Queue Information to a JSON File

merlin queue-info --dump queue_report.json

Status (merlin status)

Note

To obtain the same functionality as the merlin status command prior to Merlin v1.12.0 use merlin queue-info with the --spec option:

merlin queue-info --spec <spec_file>

Display a high-level status summary of a workflow.

This will display the progress of each step in your workflow using progress bars and brief summaries. In each summary you can find how many tasks there are in total for a step, how many tasks are in each state, the average run time and standard deviation of run times of the tasks in the step, the task queue, and the worker that is watching the step.

For more information, see the Status documentation.

Usage:

merlin status [OPTIONS] WORKSPACE_OR_SPECIFICATION

Options:

Name	Type	Description	Default
-h, --help	boolean	Show this help message and exit	False
--cb-help	boolean	Colorblind help option. This will utilize different symbols for each state of a task.	False
--dump	filename	The name of a csv or json file to dump the status to	None
--no-prompts	boolean	Ignore any prompts provided to the command line. This will default to the latest study if you provide a spec file rather than a study workspace.	False
--task_server	string	Task server type. Currently only "celery" is implemented.	"celery"
-o, --output-path	dirname	Specify a location to look for output workspaces. Only used when a spec file is passed as the argument to status.	None

Examples:

Check the Status Using Workspace as Input

merlin status study_name_20240129-123452/

Check the Status Using a Specification as Input

This will look in the OUTPUT_PATH Reserved Variable defined within the spec file to try to find existing workspace directories associated with this spec file. If more than one are found, a prompt will be displayed for you to select a workspace directory.

merlin status my_specification.yaml

Check the Status Using a Specification as Input & Ignore Any Prompts

If multiple workspace directories associated with the spec file provided are found, the --no-prompts option will ignore the prompt and select the most recent study that was ran based on the timestamps.

merlin status my_specification.yaml --no-prompts

Dump the Status Report to a CSV File

merlin status study_name_20240129-123452/ --dump status_report.csv

Look For Workspaces at a Certain Location

merlin status my_specification.yaml -o new_output_path/

Utilize the Colorblind Functionality

merlin status study_name_20240129-123452/ --cb-help