Table 2 The resources consumption on different subsets of the entire autosome (chromosomes 1–22) of phase 1 as well as all of phase 3. Memory specified is the memory allocated to each executor
From: VariantSpark: population scale clustering of genotype information
Data | Portion | Pre-processing | Clustering | ||||
|---|---|---|---|---|---|---|---|
Executors | Memory | Time | Executors | Memory | Time | ||
Phase 1 | 20 % | 64 | 2 | 11 min 53 sec | 64 | 6 | 1 h 10 min |
40 % | 64 | 2 | 19 min 09 sec | 64 | 12 | 2 h 19 min | |
60 % | 64 | 2 | 26 min 34 sec | 64 | 17 | 3 h 33 min | |
100 % | 64 | 2 | 40 min 48 sec | 40 | 24 | 14 h 44 min | |
Phase 3 | 100 % | 64 | 2 | 3 h 54 min 24 sec | 40 | 24 | 27 h 46 min |