Saving BLAST Results to AWS S3

The pre-installed BLAST workflow template obtains query sequences with efetch from NCBI or curl from any https URL and saves results to a configurable path under the persistent DataVolume. What if you wanted to fetch query sequences from AWS S3 and save results to AWS S3 instead and you did not need the ability to create custom BLAST databases? To achieve that we will first follow the workflow catalog documentation to create a copy of the BLAST workflow and call it Blast S3. Then open the detail view on that Application and make the following changes:

Replace the workflow template

Since we are removing some functionality this template will be simpler and we will make use of the ingress and egress functionality of the volumes instead of using a job to fetch data and saving to a persistent volume. You can use the following template:


{{- $dbpath := list "/data" .BlastDbPath | join "/"  }}
{{- $dbname := .BlastDbName }}

{{- $s3_uri := trim .S3Uri }}
{{- if not (hasPrefix "s3://" $s3_uri) }}
  {{- fail "S3Uri is required and has to start with 's3://'" }}
{{- end }}

{{- $s3_secret := trim .S3Secret }}
{{- if not (hasPrefix "secret://" $s3_secret) }}
  {{- fail "S3Secret is required and has to start with 'secret://'" }}
{{- end }}

{{- $run_name := trim .RunName }}
{{- if not $run_name }}
  {{- fail "RunName is required" }}
{{- end }}

version: v4
volumes:
  data:
    use: {{.DataVolumeProvisioner}}
    name: {{.DataVolumeName}}
  scratch:
    use: {{.ScratchVolumeProvisioner}}
    size: 1GB
    ingress:
      - source:
          uri: "{{$s3_uri}}/{{$run_name}}.fa"
          secret: {{$s3_secret}}
        destination:
          uri: "file://{{$run_name}}.fa"
    egress:
      - source:
          uri: "file://{{$run_name}}.blast.out"
        destination:
          uri: "{{$s3_uri}}/{{$run_name}}.blast.out"
          secret: {{$s3_secret}}
jobs:
  fetch-db:
    image:
      uri: {{.WorkflowContainer}}
    mounts:
      /data: data
      /scratch: scratch
    script: |
      #!/bin/bash
      mkdir -p "{{$dbpath}}" && cd "{{$dbpath}}" || exit 1
      # fix update_blastdb.pl if it's from a conda container
      ubdb="$(type -p update_blastdb.pl)"
      curl="$(type -p curl)"
      [[ -z "$ubdb" || -z "$curl" ]] && exit 1
      if [[ "$ubdb" =~ conda ]]; then
        sed "s:^my \\\$curl.*$:my \$curl = '$curl';:" "${ubdb}" > update_blastdb.pl
      else
        cp "${ubdb}"  update_blastdb.pl
      fi
      chmod 750 update_blastdb.pl

      if  ./update_blastdb.pl --showall | grep -q {{$dbname}} ; then
        echo "{{$dbname}} is a public database available from NCBI"
        now=$(date +%s)
        if [[ -e {{$dbname}}__ ]]; then
          last=$(cat {{$dbname}}__)
          if (( (now - last) < 86400 )) ; then
            echo "  {{$dbname}} is current - update skipped."
            exit
            fi
          fi
          echo $now > {{$dbname}}__
          echo "  updating/downloading {{$dbname}}"
          ./update_blastdb.pl --num_threads=2 --decompress {{$dbname}} && exit 0 || exit 1
        else
          echo "{{$dbname}} is not a public database"
          exit 1
        fi
    resource:
      cpu:
        cores: 1
        threads: true
      memory:
        size: 4GiB
    policy:
      timeout:
        execute: {{.BlastFetchTimeout}}
  run-blast:
    image:
      uri: {{.WorkflowContainer}}
    mounts:
      /data: data
      /scratch: scratch
    cwd: /scratch
    script: |
      #!/bin/bash
      {{.BlastCmd}} -num_threads {{.BlastCores}} -query "{{$run_name}}.fa" -db {{$dbpath}}/{{$dbname}} -out {{$run_name}}.blast.out {{.BlastOpts}} || exit 1
      cat {{$run_name}}.blast.out
    resource:
      cpu:
        cores: {{.BlastCores}}
        affinity: NUMA
      memory:
        size: {{.BlastMemory}}
    policy:
      timeout:
        execute: {{.BlastQueryTimeout}}
    depends-on:
      - name: fetch-db
        status: FINISHED

Adapt the parameters

Remove the following parameters: CustomBlastDbFetchCmd, CustomBlastDbOptions, CustomBlastDbName, CustomBlastDbTimeout, RetrieveQuerySequencesCmd, and BlastOutputPath
Add 2 new parameters:
- Name: S3Uri Type: Text Description: AWS URI prefix for blast inputs and outputs. Default: A URI like s3://<bucket>/<path...>
- Name: S3Secret Type: Text Description: Secret from the secret store with AWS credentials. Default: A URI like secret://<scope>/<name>

Run the workflow

Once you save the modified BLAST workflow template you can run analogously to the stock BLAST workflow template using the appropriate parameters.

Executing a BLAST Workflow Template Viewing BLAST Results