Skip to main content

Table 2 Comparison of f ^ em and f ^ avg in terms of average relative error

From: A unified approach for allele frequency estimation, SNP detection and association studies based on pooled sequencing data using EM algorithms

  α= 0.05% α= 0.1% α= 0.5% α= 1%
f R E f ^ avg R E f ^ em R E f ^ avg R E f ^ em R E f ^ avg R E f ^ em R E f ^ avg R E f ^ em
0.1% 52.0 9.4 102.0 15.6 502.0 72.0 1000.0 146.0
0.5% 10.3 4.0 20.2 4.9 99.5 13.3 199.0 26.5
1% 5.0 3.3 10.0 3.7 49.2 5.7 98.3 9.5
5% 1.0 5.4 1.9 6.0 9.1 6.7 18.1 6.3
  1. The average RE of f ^ avg and f ^ em for different values of minor allele frequency f and sequencing error rate α.