← All projects

Clinical genomics · NGS · Pipelines

Germline variant calling benchmarked against GIAB HG002

A Nextflow pipeline from Illumina reads to germline SNVs and indels, with two callers, GATK HaplotypeCaller and DeepVariant, scored against the Genome in a Bottle truth set for HG002 with hap.py. The result is a validation report of the kind a clinical lab writes before using a pipeline on patient samples: overall accuracy, plus accuracy in the regions where variant calling is hard.

Context
Portfolio project
Data
GIAB HG002, Illumina NovaSeq PCR-free 2×150, ~35×, chr20; GIAB v4.2.1 truth set; GIAB stratifications v3.3
Methods
fastp, BWA-MEM2, samtools markdup, mosdepth, GATK HaplotypeCaller with hard filters, DeepVariant, hap.py
Stack
Nextflow DSL2, conda, Docker
Dot plot of SNV and indel F1 for DeepVariant and GATK across GIAB region types, with both callers near 1.0 outside difficult regions and lower in low-mappability regions
F1 of PASS calls by GIAB region type. Both callers are near-perfect in easy sequence; they separate in low-mappability regions and segmental duplications.
0.997DeepVariant SNV F1 against 71,333 truth SNVs (indel F1 0.995)
4.5×Fewer false-positive SNVs from DeepVariant than from hard-filtered GATK
72 minReads to benchmark for chr20 on a 12-core, 12 GB laptop
The problem

How do you know a variant caller is right?

A clinical genomics pipeline is only as good as its validation. Genome in a Bottle sequenced reference samples such as HG002 deeply with several technologies and published high-confidence truth calls, so a pipeline can be scored by recall and precision. But a single overall number hides where errors happen. GIAB’s genome stratifications split the genome into easy and difficult regions, so performance can be reported where it matters, in repeats and in regions that reads cannot map to uniquely.

Pipeline

From reads to a scored call set

  1. Reads and QC. The HG002 chr20 reads are converted back to paired FASTQ and trimmed with fastp.
  2. Alignment. BWA-MEM2, then samtools fixmate, sort and duplicate marking, with coverage from mosdepth.
  3. GATK. HaplotypeCaller scattered over eight windows, gathered and hard-filtered with the GATK best-practice thresholds for a single sample.
  4. DeepVariant. The WGS model on the same alignments.
  5. Benchmarking. hap.py against the GIAB v4.2.1 truth set within its benchmark regions, overall and in five stratifications.

Host tools are pinned in a conda environment. DeepVariant and hap.py run from versioned Docker images, and a test profile runs the whole pipeline on a 500 kb window in a few minutes.

Results

Both callers are excellent in easy sequence

CallerVariantRecallPrecisionF1
DeepVariantSNV0.99540.99830.9968
DeepVariantIndel0.99340.99750.9955
GATK HaplotypeCallerSNV0.98930.99210.9907
GATK HaplotypeCallerIndel0.99100.99250.9917

Outside GIAB’s difficult regions both callers are close to perfect, with SNV F1 0.9999 for DeepVariant and 0.9982 for GATK. The difference is concentrated in hard sequence. In low-mappability regions and segmental duplications, SNV F1 is 0.942 for DeepVariant and 0.863 for GATK. Overall, DeepVariant makes 124 false-positive SNVs against GATK’s 559. For a clinical pipeline, that is the case for reporting stratified performance rather than one headline number.

Notes

Limits worth knowing

  • The reads were pre-selected to chr20 by a whole-genome alignment and are re-aligned to chr20 only. This slightly favours both callers compared with a genome-wide run.
  • GATK is run without BQSR and with hard filters rather than VQSR, the usual choice for one sample. It is a fair baseline, not a tuned one.
  • hap.py receives PASS calls with the genotype field only, because its bundled htslib can crash on DeepVariant’s FORMAT fields; scoring compares genotypes, so results are unaffected.

Get in touch

I’m looking for bioinformatics roles in statistical genetics, transcriptomics and NGS analysis, especially where clinical or cell and gene therapy experience helps. Based in London, open to hybrid and remote.