An analysis of intra array repeats: the good, the bad and the non informative
© Elbez et al; licensee BioMed Central Ltd. 2006
Received: 19 March 2006
Accepted: 05 June 2006
Published: 05 June 2006
On most common microarray platforms many genes are represented by multiple probes. Although this is quite common no one has systematically explored the concordance between probes mapped to the same gene.
Here we present an analysis of all the cases of multiple probe sets measuring the same gene on the Affymetrix U133a GeneChip and found that although in the majority of cases both measurements tend to agree there are a significant number of cases in which the two measurements differ from each other. In these cases the measurements can not be simply averaged but rather should be handled individually.
Our analysis allows us to provide a comprehensive list of the correlation between all pairs of probe sets that are mapped to the same gene and thus allows microarray users to sort out the cases that deserve further analysis. Comparison between the set of highly correlated pairs and the set of pairs that tend to differ from each other reveals potential factors that may affect it.
High density DNA microarrays are extensively used by biologists in order to gain information about the expression levels of thousands of genes simultaneously. The advantage of the genomic approach is that it allows the study of cell behavior on the systems level rather than on the gene specific level . One of the hurdles of this approach is that the confidence one has in each measurement is lower in comparison to the confidence gained by classical molecular biology techniques, thus extensive statistical analysis is needed in order to define the significance of the results. One of the approaches used to asses the quality of results is comparison between replicates . This is accomplished by both printing each DNA fragment several times on the array (internal replicates) and repeating the experiment multiple times (experimental replicates). While internal replicates usually give very similar results, experimental replicates may differ from one another due to variation in the biological samples, exact experimental conditions and array processing . There is yet another type of replicates – multiple array spots that are mapped to different regions of the same gene, thus allowing multiple measurements of the RNA levels in the exact same biological sample on the same array. We call these "intra array repeats" and not "intra array replicates" in order to highlight the fact that they are not identical. Although this is quite common, for example 35% of the genes on Affymetrix U133a array are represented by multiple probe sets, no systematic evaluation of the concordance between multiple representations of the same gene has been reported, thus leaving microarray users without clear guidelines how to deal with this problem.
A related issue which has been studied extensively in recent years is the question of cross platform comparisons in which researchers try to compare results obtained for the same gene on different microarray platforms. Overall there is only limited agreement between experiments performed on different platforms (reviewed by [5, 6]). Many factors can contribute to this cross platform inconsistency, including differences in RNA processing, noise levels , the statistical analysis required by each platform  and the sequences chosen to represent the gene by the different manufacturers [9, 10]. Recent studies suggest that the latter issue is very important since there is much more agreement between platforms when only spots with overlapping sequences are used [11–13]. In contrast to cross platform comparisons which are affected by many variables, intra platform comparisons may be affected only by the sequences chosen to represent the gene and thus may serve as an excellent test case for assessing the quality of microarrays results. In this study we identified all groups of probe sets mapped to the same gene that are present on the Affymetrix U133a GeneChip and examined the degree of agreement between those pairs. In most cases we observed significant correlations between the pairs suggesting that both probe sets are reporting similar results. However, we did find a large subset that differs significantly from one another. The identification of these pairs is important for analyzing expression profile results, since in these cases the investigator can not rely on the expression levels reported by those probes.
Gene expression data
Dataset accession number1
Number of samples2
Number of informative pairs3
Gene atlas of human protein-encoding transcriptome from 79 tissues obtained from various sources.
Asthma and atopy – Investigation of CD4+ lymphocytes from patients with and without atopy, in combination with asthma
Analysis of paired left ventricular apex samples harvested at implant of left ventricular assist device (LVAD) and at explant.
Expression profiling of HL-60 and acute promyelocytic leukemia cells treated with various drugs.
Since we are interested in comparing the changes in transcriptional level of both members of each pair of probe sets, we defined as informative only probe sets that showed changes in transcription along the different measurements in a dataset. Only those probe sets were included in our analysis (see methods).
Comparison of the correlation values obtained in the different datasets revealed a high degree of correlation between them (figure 2b,c), suggesting that the agreement between pairs of probe sets is characteristic of the probe sets and is not heavily dependent on the particular dataset from which the data was derived.
Statistical assessment of the results
One of the main purposes of our analysis was to guide microarray users in cases of genes that are represented by multiple probe sets. The users would like to know when the multiple measurements can be combined and when they should be handled with caution. In order to achieve this goal, we wanted to define a list of pairs which show overall good agreement between them. This list should be based on the combination of the correlation values calculated in each dataset. Thus for each pair, we combined the individual P values obtained for each dataset in which it was informative, resulting in a single combined P value for each pair. A list of all the pairs along with their combined P values is included in additional file 1. This list includes 6536 pairs of probe sets that were informative at least in one of the data sets and 77% of them showed significant correlation (FDR = 0.01).
Exploring the "good" and "bad" pairs
Recent studies have shown that when performing cross platform comparisons between Affymetrix arrays and cDNA arrays, restricting the analysis to sequence-matched probes produces a higher level of consistency between the platforms [11–14]. These studies suggest that one of the main reasons for inconsistencies between different microarray platforms is the differences in the gene regions chosen to be printed for each platform. Different sequences may report different values for a variety of reasons that can be sorted into two categories – the two sequences are reporting the levels of different transcripts (due to annotation mistakes or different splice variants) alternatively, both sequences may report the transcription of the same transcript at a different accuracy (due to the well documented 3'-5' degradation of the mRNA or due to differences in hybridization efficiency of the two regions).
Intra array repeats are printed on the same array and therefore they share all the external conditions (including the biological sample, growth conditions, probe labeling and data handling) and differ from one another only in the particular sequence chosen to be printed on the array. Thus multiple probe sets printed on the same array serve as an excellent case for exploring the contribution of sequence effects to the overall deviation of the transcription values reported by each probe set. To this end we defined two large sub groups of pairs (each containing approximately 1500 pairs) – pairs that were highly correlated (P < 10-15; herein "good") and pairs which did not show a significant concordance between them (P > 0.1; herein "bad").
Next we analyzed the affect of alternative splicing on probe accordance. If two probe sets are reporting the expression of two different splice variants of a gene, they may differ in the transcription values they are measuring. Thus another possible explanation for disagreement between pairs of probe sets is that each is reporting the transcription level of a different transcript of the same gene. In order to explore this idea, we looked at the number of known variant transcripts for each gene. Calculation of the average number of transcript variants in the "good" and in the "bad" pairs revealed that there are on average approximately 1.7 transcripts per gene and there is almost no difference in the distribution between the two groups (figure 4b). Moreover the frequency of the "bad" and of the "good" pairs is almost identical even in the group of genes with only a single known transcript, strongly suggesting that according to the current knowledge about alternative splicing, it can not be the reason for the deviation between pairs of probe sets in most cases.
Finally, we wanted to explore the contribution of cross hybridization between probes to the agreement between probe sets. To this end we used the Affymetrix suffix to the probe name as a measurement of the probe uniqueness. _at suffix designates a unique probe set, whereas _s_at and _x_at suffixes designate probe sets that can cross hybridize with multiple genes. To our surprise we found a very significant reverse correlation between the probe uniqueness and its agreement with another probe set. We found significant enrichment (P = 0.001) for the s_at and x_at suffixes in the "good" pairs (figure 4c), suggesting that cross hybridization between the genes is not the reason for the disagreement between probe sets.
Microarray users frequently encounter the problem of different values for the same gene resulting from multiple probe sets. Without clear guidelines from the manufacturers, each investigator deals with this problem in a different way, some are not aware of it and therefore either use the first appearance of the gene or use the probe set that fits their expectations better. Alternatively, users may use the average of all probe sets without validating that all indeed report similar transcription rates. In order to help microarray users to chose the right approach for multiple measurements of a gene in the Affymetrix U133a GeneChip we examined all such cases and developed a statistical method to asses the agreement between the measurements in each of these cases. Our method is based on the calculation of the Pearson correlation between all the measured values of a pair of probe sets along all the experiments in a data set. The Pearson correlation gives a good estimation of the overall agreement between such pairs without being sensitive to variability in the average expression level reported by the probe sets. We excluded from the analysis probes that were non informative either because their expression level was below detection or because their expression level did not change (>2 folds) along the experiments. As expected the correlation values obtained between probe sets from the same gene were much higher than those obtained from randomly chosen pairs of probe sets with 77% of the pairs reporting similar values. The pairs of probe sets where sorted according to the statistical significance of the Pearson correlation between them and a list of all the pairs together with their P value is provided [see additional file 2]. This list should serve as a guideline to the user: the lower the P value the higher chances that the two probes are reporting the same information whereas high P values suggest that the two observations should be treated individually (figure 3).
The division of the pairs of probe sets into those that tend to agree ("good") or disagree ("bad") allows us to examine the factors affecting it. The deviation between two probes that are mapped to the same gene may be due to several reasons including alternative splicing, mistakes in the annotation and differences in the efficiency each probe set reports the transcription. We found no evidence for the contribution of alternative splicing to the disagreement between probe sets (figure 4b). On the other hand by comparing the annotation in the sets of the "good" and the "bad" pairs we have shown that misannotation is obviously a main issue, with >25% of the "bad" pairs being misannotated (figure 4a) thus in cases of disagreement between probe sets one should first validate that both probe sets are indeed annotated correctly.
Recent studies about cross platform comparisons found that the best results are obtained if the comparison is done on the probe level rather than on the probe set level [12, 18, 19], suggesting that the actual sequence has a big contribution to the array results. In accordance with this idea we found that pairs of probe sets that share at least one common probe show almost always significant agreement (97.6% of 970 such cases passed the FDR of 0.01).
The observation that non unique probe sets perform better than unique ones (figure 4c) is surprising because two probe sets each cross hybridize with a different set of genes may cause disagreement. However our results suggest the contrary – probes that were designed to recognize multiple genes, are much more abundant in the "good" pairs suggesting that the cross hybridization may have a smoothing affect that reduces the difference between the probes. In other words if the reason for the dis accordance between pairs is that each recognizes a unique transcript, less specific probe sets will mask this difference and thus will be more abundant among the "good" pairs of probe sets.
Our analysis suggests that in most cases (77%) pairs of informative probe sets report similar transcription levels. However, it should be noted that our decision to concentrate only on informative probe sets may influence those results. By excluding from the analysis the most problematic cases in which one probe set is informative and the other is not we elevated the percentage of cases with good correlation [see additional file 3]. These cases were omitted from the analysis because of the limited usefulness of the Pearson correlation on non informative probe sets (see methods). Data about the concordance between these pairs is included in additional file 1.
The analysis of the differences between the "good" and the "bad" pairs suggests that biological differences such as alternative splicing have a very limited contribution to the disagreement between probe sets. This conclusion was derived from the current knowledge of the genome sequence and of splicing variants (using fairly strict criteria). Updates of the genome sequence and advancements in the technology of measuring splice variants (such as the use of exon microarrays) may change this conclusion. Although those changes may affect the reasons for the disagreements between probe sets it will not change the definition of "good" and "bad" pairs. Only updating the probe set definition according to the new sequence information  may solve this problem but it is beyond the scope of the current study.
An important internal control for microarray experiments is multiple array spots that are mapped to different regions of the same genes. In the ideal world these different probe sets should give similar measurements for expression of the gene. Our analysis tested whether this is true using four data sets that used the Affymetrix U133a platform. We found that in the majority of such cases both measurements agree and thus using the average value of the probe sets would be appropriate. However in a significant number of cases this approach is not suitable and our results can guide the user in deciding about each case. Further analysis is needed to define guidelines for choosing better regions in the gene that will represent the transcription level more accurately.
For our analysis we downloaded the data of four large experiment series from the gene expression omnibus database . We chose datasets with at least 29 experiments in order to have multiple measurements in each dataset. The GEO accession number, a short description of the experiments and the number of arrays used in each series are shown in table 1. All the expression values were log transformed.
In order to define intra array repeats, we used Affymetrix annotation and defined 11,722 pairs of probe sets in which both members were mapped to the same gene. In order to compare the RNA levels reported by the two probes in each pair we calculated the Pearson correlation between the expression values (log transformed) reported by each probe set in all the experiments in each dataset. We also performed Spearman correlation and the results [see additional file 4] are very similar [see additional file 5].
Since we were interested only in the correlations between probe sets of active genes we looked only on probe sets that passed two filtering criteria – they got a present call (P) in at least 10% of the experiments in a dataset and they showed at least 2 fold changes in their expression level over the average in at least 10% of the experiments. This filtering resulted in a different number of informative pair of probe sets for each dataset (table 1).
Our decision to include in the analysis only cases were both probe sets in a pair are informative was based on the observation that the transcription values reported by non informative probe sets frequently reflect random noise fluctuations and thus the meaning of the correlation between those values and transcription values from informative probes is not clear. Nevertheless this decision introduced a bias in our analysis since cases in which the two probe sets have clear different characteristics were excluded. Calculating the correlation of the group of pairs in which only one of the probe sets was informative revealed that in all data sets the average correlation of this group of pairs was much closer to the average correlation between random pairs than to the average correlation of the pairs in which both probe sets were informative [see additional file 3].
In order to calculate the statistical significance of the correlation between the probe sets in each pair we transformed the values into Fisher Z values and compared those values to the average correlation values obtained for random pairs of informative probe sets using a Z test. The P values obtained for each data set were combined using the chi square test for combined q probabilities .
In order to define a P value threshold while accounting for the multiple hypothesis problem we used the FDR approach .
Comparison between datasets
The fisher Z values obtained for each informative pair of probe sets in each dataset was used for comparing the results obtained in the different datasets. The comparison was done by calculating the Pearson correlation of those values. This procedure was repeated after shuffling the order of the pairs in order to obtain an estimation of the random correlation between the datasets.
Error rate estimation
Transcription values were converted to ratios by dividing the value reported in each experiment by the probe set mean value in all experiments in a dataset. Cases with ratios greater than |2| were identified. This was done for individual probe sets and for the average values of two probe sets mapped to the same gene. Cases without full agreement were counted.
Probe sets analysis
The accuracy of Affymetrix assignments of genes to probe sets was confirmed by comparing their assignments to those of the GeneAnnot database, which remapped Affymetrix probes to genes by performing blat searches with all individual probes forming the Affymetrix array . This allowed us to divide the probe sets pairs into two categories – those which were mapped to the same gene using updated sequence information (85%) and those in which the current sequence information does not support affymetrix original annotation (15%). The number of transcripts mapped to each probe set was taken from the U133a annotation file provided by Affymetrix using the "RefSeq Transcript ID" column. Sequence homology between probe sets was determined by counting the number of identical probes shared by both probe sets in a pair. For the probe set uniqueness analysis we used the Affymetrix suffix. Affymetrix annotate each probe set with a suffix depicting whether it is unique to a single gene (_at), all its probes cross hybridize with multiple genes (s_at) or there is inconsistency between the probes forming the probe set in terms of cross hybridization with different genes (x_at) http://www.affymetrix.com/support/technical/technotes/hgu133_design_technote.pdf. We limited our analysis only to cases in which both probe sets in a pair have the same suffix.
The statistic significance of the difference between the distribution of "bad" and "good" probes was assessed with the chi square test or Fisher exact test.
We are grateful to Norman Grover for his valuable statistical advices and to Ran Rubinstein for his help in programming. We thank Zahava Siegfried, Shifra Ben-Dor, Yoav Smith, Amos Grundwag and Ron Ophir for critical reading of the manuscript. The research was supported by grants from the Association for International Cancer Research (AICR), the F.I.R.S.T. (Bikura) program of the Israel Science Foundation (Grant No. 4103/03) and EC FP6 funding (contract 503576).
- Segal E, Friedman N, Kaminski N, Regev A, Koller D: From signatures to models: understanding cancer using microarrays. Nat Genet. 2005, 37 Suppl: S38-45. 10.1038/ng1561.PubMedView ArticleGoogle Scholar
- van Bakel H, Holstege FC: In control: systematic assessment of microarray performance. EMBO Rep. 2004, 5 (10): 964-969. 10.1038/sj.embor.7400253.PubMedPubMed CentralView ArticleGoogle Scholar
- Lee ML, Kuo FC, Whitmore GA, Sklar J: Importance of replication in microarray gene expression studies: statistical methods and evidence from repetitive cDNA hybridizations. Proc Natl Acad Sci U S A. 2000, 97 (18): 9834-9839. 10.1073/pnas.97.18.9834.PubMedPubMed CentralView ArticleGoogle Scholar
- Churchill GA: Fundamentals of experimental design for cDNA microarrays. Nat Genet. 2002, 32 Suppl: 490-495. 10.1038/ng1031.PubMedView ArticleGoogle Scholar
- Hardiman G: Microarray platforms--comparisons and contrasts. Pharmacogenomics. 2004, 5 (5): 487-502. 10.1517/146224220.127.116.117.PubMedView ArticleGoogle Scholar
- Draghici S, Khatri P, Eklund AC, Szallasi Z: Reliability and reproducibility issues in DNA microarray measurements. Trends Genet. 2005Google Scholar
- Shippy R, Sendera TJ, Lockner R, Palaniappan C, Kaysser-Kranich T, Watts G, Alsobrook J: Performance evaluation of commercial short-oligonucleotide microarrays and the impact of noise in making cross-platform correlations. BMC Genomics. 2004, 5 (1): 61-10.1186/1471-2164-5-61.PubMedPubMed CentralView ArticleGoogle Scholar
- Jarvinen AK, Hautaniemi S, Edgren H, Auvinen P, Saarela J, Kallioniemi OP, Monni O: Are data from different gene expression microarray platforms comparable?. Genomics. 2004, 83 (6): 1164-1168. 10.1016/j.ygeno.2004.01.004.PubMedView ArticleGoogle Scholar
- Kuo WP, Jenssen TK, Butte AJ, Ohno-Machado L, Kohane IS: Analysis of matched mRNA measurements from two different microarray technologies. Bioinformatics. 2002, 18 (3): 405-412. 10.1093/bioinformatics/18.3.405.PubMedView ArticleGoogle Scholar
- Larkin JE, Frank BC, Gavras H, Sultana R, Quackenbush J: Independence and reproducibility across microarray platforms. Nat Methods. 2005, 2 (5): 337-344. 10.1038/nmeth757.PubMedView ArticleGoogle Scholar
- Mecham BH, Klus GT, Strovel J, Augustus M, Byrne D, Bozso P, Wetmore DZ, Mariani TJ, Kohane IS, Szallasi Z: Sequence-matched probes produce increased cross-platform consistency and more reproducible biological results in microarray-based gene expression measurements. Nucleic Acids Res. 2004, 32 (9): e74-10.1093/nar/gnh071.PubMedPubMed CentralView ArticleGoogle Scholar
- Carter SL, Eklund AC, Mecham BH, Kohane IS, Szallasi Z: Redefinition of Affymetrix probe sets by sequence overlap with cDNA microarray probes reduces cross-platform inconsistencies in cancer-associated gene expression measurements. BMC Bioinformatics. 2005, 6 (1): 107-10.1186/1471-2105-6-107.PubMedPubMed CentralView ArticleGoogle Scholar
- Barnes M, Freudenberg J, Thompson S, Aronow B, Pavlidis P: Experimental comparison and cross-validation of the Affymetrix and Illumina gene expression analysis platforms. Nucleic Acids Res. 2005, 33 (18): 5914-5923. 10.1093/nar/gki890.PubMedPubMed CentralView ArticleGoogle Scholar
- Lee JK, Bussey KJ, Gwadry FG, Reinhold W, Riddick G, Pelletier SL, Nishizuka S, Szakacs G, Annereau JP, Shankavaram U, Lababidi S, Smith LH, Gottesman MM, Weinstein JN: Comparing cDNA and oligonucleotide array data: concordance of gene expression across platforms for the NCI-60 cancer cells. Genome Biol. 2003, 4 (12): R82-10.1186/gb-2003-4-12-r82.PubMedPubMed CentralView ArticleGoogle Scholar
- Chalifa-Caspi V, Yanai I, Ophir R, Rosen N, Shmoish M, Benjamin-Rodrig H, Shklar M, Stein TI, Shmueli O, Safran M, Lancet D: GeneAnnot: comprehensive two-way linking between oligonucleotide array probesets and GeneCards genes. Bioinformatics. 2004, 20 (9): 1457-1458. 10.1093/bioinformatics/bth081.PubMedView ArticleGoogle Scholar
- Harbig J, Sprinkle R, Enkemann SA: A sequence-based identification of the genes detected by probesets on the Affymetrix U133 plus 2.0 array. Nucleic Acids Res. 2005, 33 (3): e31-10.1093/nar/gni027.PubMedPubMed CentralView ArticleGoogle Scholar
- Zhang J, Finney RP, Clifford RJ, Derr LK, Buetow KH: Detecting false expression signals in high-density oligonucleotide arrays by an in silico approach. Genomics. 2005, 85 (3): 297-308. 10.1016/j.ygeno.2004.11.004.PubMedView ArticleGoogle Scholar
- Dai M, Wang P, Boyd AD, Kostov G, Athey B, Jones EG, Bunney WE, Myers RM, Speed TP, Akil H, Watson SJ, Meng F: Evolving gene/transcript definitions significantly alter the interpretation of GeneChip data. Nucleic Acids Res. 2005, 33 (20): e175-10.1093/nar/gni179.PubMedPubMed CentralView ArticleGoogle Scholar
- Elo LL, Lahti L, Skottman H, Kylaniemi M, Lahesmaa R, Aittokallio T: Integrating probe-level expression changes across generations of Affymetrix arrays. Nucleic Acids Res. 2005, 33 (22): e193-10.1093/nar/gni193.PubMedPubMed CentralView ArticleGoogle Scholar
- Barrett T, Suzek TO, Troup DB, Wilhite SE, Ngau WC, Ledoux P, Rudnev D, Lash AE, Fujibuchi W, Edgar R: NCBI GEO: mining millions of expression profiles--database and tools. Nucleic Acids Res. 2005, 33 (Database issue): D562-6. 10.1093/nar/gki022.PubMedPubMed CentralView ArticleGoogle Scholar
- D'Agostino RB, Stephens M: Goodness-Of-Fit Techniques. 1986, Marcel Dekker, 357-360.Google Scholar
- Benjamini Y, Hochberg Y: Controlling the False Discovery Rate: a practical and powerful approach to multiple testing. J Royal Stat Soc Ser B. 1995, 57 (1): 289-300.Google Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.