Challenges

Complex, rapidly changing pipelines

Large number of specialized dependencies

https://github.com/StanfordBioinformatics/HugeSeq

Quality differences between calling methods

http://www.bioplanet.com/gcat

Scaling on full ecosystem of clusters

Solution

http://www.amazon.com/Community-Structure-Belonging-Peter-Block/dp/1605092770

Development goals

• Quantifiable
• Analyzable
• Scalable
• Reproducible
• Community developed
• Accessible

Quantify quality

• Reference materials: http://www.genomeinabottle.org/
• Quantification details: http://j.mp/bcbioeval

Known unknowns

• Coverage: summarize what you can't assess
• Structural: large, complex rearrangements

Analysis

Query

https://github.com/arq5x/gemini

Visualize

https://github.com/chapmanb/o8

Parallel scaling

• Infrastructure details: http://j.mp/bcbioscale
• IPython: http://ipython.org/ipython-doc/dev/parallel/index.html

Better parallel blocks

Reproducibility

• Express intentions at a high level
• Revision controlled configuration
• Handle complex distributed logging
• Provenance tracking

Configuration

- files: [NA12878-NGv3-LAB1360-A_1.fastq.gz, NA12878-NGv3-LAB1360-A_2.fastq.gz]
  description: NA12878
  analysis: variant2
  genome_build: GRCh37
  algorithm:
    aligner: bwa
    recalibrate: gatk
    realign: gatk
    variantcaller: [gatk, freebayes, gatk-haplotype]
    coverage_interval: exome
    coverage_depth: high
    platform: illumina
    quality_format: Standard
    validate: NA12878-nist-v2_13-NGv3-pass.vcf

Provenance

• Excellent logging
• Third party version tracking
• Beyond logging:
  • BioLite: https://bitbucket.org/caseywdunn/biolite
  • Arvados: https://arvados.org/

Community developed

• Fully automated installation: CloudBioLinux
• Deployable on multiple clusters (LSF, SGE, Torque…)
• API for new aligners and variant callers
• Open source, hackable and documented

https://github.com/chapmanb/bcbio-nextgen

Automated installation

• Single biggest software problem: running for the first time
• Bootstrap from bare machine to ready-to-go pipeline
• Builds off existing installation work: CloudBioLinux
• Provide example pipelines with real data

http://cloudbiolinux.org

https://bcbio-nextgen.readthedocs.org

Accessible

http://exploringpersonalgenomics.org/

Galaxy

https://bitbucket.org/hbc/galaxy-central-hbc

STORMSeq

http://www.stormseq.org/

Summary

• Community developed pipelines > challenges
• Focus
  • Assessing quality: good science
  • Analysis: enable exploration
  • Scalability: finish in time
  • Reproducibility: show your work
• Widely accessible

https://github.com/chapmanb/bcbio-nextgen