nf-core/configs: azurebatch Configuration
To be used with the azurebatch profile by specifying the -profile azurebatch when running nf-core pipelines.
Custom queue and storage need to be supplied with params.az_location, params.batch_name, params.batch_key, params.storage_name, params.sas_token.
Required Parameters
--storage_name
Name of Azure blob storage account.
--storage_key
Access key to Azure blob storage account. --storage_key or --storage_sas are required, but not both.
--storage_sas
SAS for access to Azure blob storage account (see relevant permissions on Nextflow documentation). --storage_key or --storage_sas are required, but not both.
—storage_duration
The duration of the shared access signature token created by Nextflow when the --storage_sas option is not specified (default: 48h). Make sure to increase this for large and expensive workflows!
--az_location
The Azure Batch region where the computation is executed in VMs. Default (westus2).
--batch_name
The Azure Batch account name.
--batch_key
The Azure Batch account key.
-w
The Azure Blob container to be used as Nextflow work directory (-w az://work).
--vm_type
VM size to use with Nextflow autopool or when creating a worker pool in Azure Batch. Make sure your Azure account has sufficient quota. Defaults to Standard_D8s_v3. See Azure VM Size documentation for more information.
--autopoolmode
Whether to use Nextflow autopool mode which creates an autoscaling pool for running Nextflow jobs. Defaults to true.
--allowpoolcreation
Allow Nextflow to create a pool for running Nextflow jobs. Defaults to true.
--deletejobs
Allow Nextflow to delete pools after completion. Defaults to true.
--az_worker_pool
Select an existing pool by name to run Nextflow jobs on. Defaults to auto (matching --autopoolmode).
--acr_registry
URL to Azure container registry for private docker images.
--acr_username
Username to access private Azure container registry.
--acr_password
Password to access private Azure container registry.
Azure Batch Setup
Please refer to the Nextflow documentation which describe how to setup the Azure Batch environment.
Config file
//Nextflow config file for running on Azure batch
params {
config_profile_description = 'Azure BATCH Cloud Profile'
config_profile_contact = 'Venkat Malladi (@vsmalladi) & Adam Talbot (@adamrtalbot)'
config_profile_url = 'https://azure.microsoft.com/services/batch/'
// Storage
storage_name = null
storage_key = null
storage_sas = null
storage_duration = '48h'
// Batch
az_location = "westus2"
batch_name = null
batch_key = null
vm_type = "Standard_E*d_v5"
autopoolmode = true
allowpoolcreation = true
deletejobs = true
deletepools = true
az_worker_pool = "auto"
// ACR
acr_registry = null
acr_username = null
acr_password = null
}
process {
executor = "azurebatch"
}
azure {
process {
queue = params.az_worker_pool
machineType = params.vm_type
}
storage {
accountName = params.storage_name
accountKey = params.storage_key
sasToken = params.storage_sas
tokenDuration = params.storage_duration
}
batch {
location = params.az_location
accountName = params.batch_name
accountKey = params.batch_key
tokenDuration = "24h"
autoPoolMode = params.autopoolmode
allowPoolCreation = params.allowpoolcreation
deleteJobsOnCompletion = params.deletejobs
deletePoolsOnCompletion = params.deletepools
pools {
auto {
autoScale = true
vmCount = 1
maxVmCount = 12
}
}
}
registry {
server = params.acr_registry
userName = params.acr_username
password = params.acr_password
}
}