The effect of measurement error of phenotypes on genome wide association studies

It is usually assumed that a trait is measured with a degree of imprecision or error, but that the details of the measurement of a trait will not affect the results of a genome wide association study. Specifically, that 1) the particular details of how the trait is measured, 2) who measured it, and 3) whether the trait is subtly differently measured in a second sample will not i) materially affect the location of the most significant associations between a DNA marker and the trait of interest or ii) affect the confirmation of the results in a second sample, because it is assumed that the errors in measurement are not systematic. Genome wide association studies (GWAS) compare a large number of single nucleotide polymorphisms (SNPs) to a trait measurement and SNPs with strong associations are usually reported [1, 2]. The effect of genotypic error is well known and there is a substantial amount of error checking of genotypes that usually occurs before the GWAS is performed. Thereafter, a second sample is used for confirmation usually measured in the same way but often by a different set of phenotypers, and there may be subtle differences in the way the phenotypes were measured. The lack of confirmation of some of the associations is usually ascribed to differences in some aspect of the genetics of the trait or aspects of the sample, and lack of confirmation is most generally ascribed to false discovery due to the large number of tests performed or insufficient power in the original GWAS [3, 4]. For some traits, particularly ones that rely upon qualitative diagnosis, care is taken to ensure that the same diagnostic criteria are used [5], but there is always the possibility that different individuals will interpret the criteria in their own way, leading to heterogeneity in classifying individuals from one sample to the next. With quantitative traits, such as a height or weight measurement, this is usually not thought to be an important source of variability, but many quantitative traits are not automatically sampled by machine nor have precise and unambiguous guidelines for measurement, and alternative measurements of a trait may have correlations r < 0.5 [6].

Two measurements of subcutaneous fat thickness were collected on the same animals by two groups working independently [7] and this provided the opportunity to examine the effects of measurement error of phenotypes on the results of a GWAS. Of course, subcutaneous fat thickness is also of intrinsic interest in biological studies in many mammalian species because it is linked to the overall fatness of an animal or how well it is doing in a particular environment, to onset of puberty, and is easily measured [8, 9]. The thickness of the fat layer in these studies was measured manually using a ruler with 1 mm gradations adjacent to the crest of the 3^rd sacral vertebra, but as fat layers are not uniformly thick there is clearly scope for variation in measurements performed by different individuals. These animals had been genotyped for a GWAS of body composition [10] and one of the fat thickness measurements had been analysed along with estimates of growth and food efficiency. To determine how successful confirmation would be in future studies of the same trait, and to evaluate how successful others would be in confirming our findings, the likely importance of such imprecision was investigated by comparing the results of the GWAS for this trait to a GWAS performed using the unpublished second measurement of subcutaneous fat thickness.

This analysis took previously collected genotypic data, additional unpublished phenotypes and then reanalysed the entire data set. The expectation was that there would not be systematic differences in the two measurements, that the underlying genetic basis for fat thickness would be evident in both GWAS, and that the differences in trait values might merely affect the degree of significance of the most significant SNP associations, which would largely be in common. The data were trimmed in two ways to remove outliers to determine whether this would be successful in controlling the differences between the GWAS. The fat thickness data were also coded as a threshold trait to determine how important such measurement error might be for discrete traits, where individuals may be coded as affected or unaffected.

The error in measuring phenotypes had a strong effect on the profiles of significant SNP identified in the GWAS. There was a small overlap in the significant SNPs, the chromosomes with the largest number of significant SNPs were similar but not the same, and the region with the largest number of significant SNPs, occurring on BTA14 in both of the trait measurements, was shifted by at least 1 Mb towards the telomere. Follow up studies of this important QTL on BTA14 could have ended up wasting time studying a non-overlapping set of genes. Overall, the correlation in allele effects was moderate and was approximately equal to the square of the correlation between the trait measurements for all data sets. As the correlation between trait measurements declines below approximately r = 0.95 between duplicate trait measurements, the similarity between GWAS will decline rapidly. For qualitative traits using a threshold to determine whether an individual is affected, measurement error appeared to generate greater incompatibility between the GWAS, possibly because it is a more powerful statistical design. These results suggest that one would expect to find minimal overlap in the significant SNP from one study to the next if a different but correlated trait was used in a confirmation study, such as rib fat thickness or intramuscular fat percentage.

The attempts to control measurement error were not all equally successful, and even small amounts of measurement error still caused differences in the GWAS. Univariate trimming, which is generally used to remove obviously erroneous outliers, such as a 1,000 kg human, or a 90 kg peregrine falcon, made the comparison between GWAS worse. Nevertheless, despite the increased divergence between measurements, the QTL on BTA14 was still found in both GWAS, except that the disparity in the location of the QTL in each GWAS increased. Bivariate trimming, even at an equivalent of the removal of around 10% of the most divergent measurements, showed an improvement in similarity between the GWAS. At the most stringent level, the overall experimental statistics such as the FPR and the size of the p-values improved, so that smaller p-values were generated for the significant SNP despite the decrease in sample size. More importantly, for the QTL on BTA14, its location was clarified into two adjacent QTL, separated by 1 Mb. Using the mean of the two measurements helped to reduce phenotypic error, also showed two QTL separated by 1 Mb on BTA14, but the FPR was close to 100% and the p-values were substantially larger. This suggests that taking the average does not remove error, and it does not reduce error sufficiently that the errors will not affect a GWAS study. Nevertheless, even with the most stringent amounts of bivariate trimming, so that the trait measurements were very highly correlated, the list of significant SNP of the two GWAS did not coincide. This suggests that the deterministic methods used to analyse GWAS data are sensitive to small differences when mapping quantitative traits and every effort needs to be made to improve the precision of trait measurement, or to develop methods of analysis that are not so sensitive to error. It is therefore recommended that trait values in GWAS experiments be examined for repeatability before the experiment is performed. At present, duplicate, independent measures of the phenotype are not made for GWA studies, so routine bivariate trimming would not be possible for most data sets. For traits that do not have high repeatability (r < 0.95), two or more independent measurements of the same trait should be obtained for all samples, and individuals genotyped that have highly correlated trait measurements.

The major differences in the trait measurements in this study were that 1) a different group of individuals performed the measurement (accredited AUSMEAT inspectors versus trained meat scientists), and 2) the P8FAT measurement was performed about an hour after slaughter on a warm carcass whereas the CHILLP8 measurement was performed after the carcass had been refrigerated, and would have been measured within 24 hours of the P8FAT measurement. Although one cannot explicitly disentangle the effects of warm vs chiller and inspectors vs meat scientists, because the comparison was warm+inspectors vs cold+scientists, the differences between the mean CHILLP8 and P8FAT were ~ 1 mm and the regression coefficient of one measure on the other was nearly 1 in the entire sample (b = 0.90) with a small standard error of estimation (s.e. = 0.01), which suggests that overall, there was little shrinkage in the fat thickness. Furthermore, the plot of divergence suggests that most of the difference is due to strongly divergent measurements for a minority of samples, implicating operator error rather than systematic differences due to the chiller. Nevertheless, given these factors of when, where and by whom the trait was measured, these measurement differences may be at the extreme end of divergence of measurement of the same trait, and studies of other traits may not be as strongly affected. However, until other analyses of independently performed repeated measurements are obtained one cannot be sure that the results obtained here are atypical or that they will represent a general phenomenon.

More importantly, these results suggest that a wide variety of phenotypes may be subject to unreproducible results due to technical issues associated with phenotype measurement. In the measurement of fat thickness, fat layers are not uniformly thick and so slight differences in where an individual or group of scorers placed the ruler could potentially affect the measure obtained. Some phenotypes are more likely to be affected by such measurement problems. For example, metabolite or hormone concentrations may show diurnal or weekly cycles or may show different values in different assays [11]. A bone thickness or length will depend upon a landmark being identified and used consistently [12]. A waist measurement, defined as the region of smallest circumference, might be made at a different location in each individual, or if measured at a particular part of the abdomen might not be the smallest circumference [13]. These results also show that the effects of measurement accuracy are stronger on the coding of affected or unaffected status, such as when an individual passes a threshold. Traits that may be affected by such thresholds are schizophrenia or obesity.

These effects on GWAS studies yield sobering implications for genomic selection or phenotype prediction. Genomic selection or phenotype prediction uses LD between markers and traits in one study to predict the performance of a separate set of animals using their genotypes for the same panel of markers, usually a set of tens to hundreds of thousands of SNPs in a SNP array [14, 15]. There is the implicit assumption in those studies that if large enough numbers of SNPs and large enough sample sizes of animals are used then high predictive accuracy will be obtained. Genomic selection depends for its success not only on the significance of the most significant SNPs but includes loci with minimal evidence for significance and even SNPs with non-significant effects into these models [16–19]. The results in this study show that evidence for larger QTL can appear to shift by 1 Mb along the chromosome, and the evidence for smaller QTL can disappear altogether, which will affect the list of SNP used in these predictions. In addition, genomic selection analyses may be affected by trait measurement error because even after the majority of discordant data have been removed the allele effects in the two GWAS still showed clear differences. These results suggest that the prediction accuracy of such genomic selection or prediction studies will only improve once better phenotypic measures are collected, either by double scoring the phenotypes or by more accurate machine based collection of precisely defined phenotypes.

It is recommended that trait values in GWAS experiments be examined for repeatability before the experiment is performed. Wherever possible, independent scorers should collect the repeated data. For traits that do not have high repeatability (r < 0.95), two or more independent measurements of the same trait should be obtained for all samples. This threshold is suggested because the square of a decimal fraction begins to depart substantially from the original fraction below a value of 0.95, and the correlation between allele effects between two GWAS is approximately equal to the square of the correlation between trait values. For prospective individuals in a GWAS, only those with accurately measured trait values should be genotyped, such as those that have highly correlated trait measurements.