Can subtle changes in gene expression be consistently detected with different microarray platforms?

Microarray technologies are now commonly used for genome-wide surveying of gene expression. With the availability of an increasing amount of data from different studies, there is a growing need for comparison and combination of datasets. This would be helpful to increase statistical power and to compare biological processes. Comparisons across different studies are, however, complicated by the use of different platforms. Over the past years, many microarray platforms, based on different technologies, have been developed by commercial and academic institutions. How reliable and consistent the results from different platforms are is still a matter of debate [1–3]. Initially, platforms comparison studies were mainly focused on comparison between commercial chips (mainly Affymetrix) and in-house spotted microarrays [4–7]. In recent years, more comprehensive studies were done, some of them reporting agreement between platforms [8–13] and some of them not [14–20]. The largest comparison was performed within an FDA-initiated program for evaluation of the reproducibility, quality and consistency of microarray platforms (MicroArray Quality Control, MAQC). In general, a high agreement between platforms was reported [21–25]. Our study is an extension to previously published studies in several aspects: we investigated the capabilities of five microarray platforms with high technological diversity to identify differences in gene expression in a challenging and highly controlled biological condition, where the expected level of transcriptional regulation was low, the number of differentially expressed genes small, and the number of biological replicates small, but realistic.

The biological question addressed was the finding of differential gene expression in the hippocampus between transgenic mice overexpressing a splice-variant of the doublecortin-like kinase-1 gene, δC-doublecortin-like kinase (DCLK)-short, which makes the kinase constitutively active [26]. The DCLK gene has recently been implicated in crucial aspects of embryonic cortical development by controlling neurogenesis, neuronal migration and neuronal vesicle transport [27–30]. DCLK-short is not expressed during embryogenesis, is abundantly expressed in adult limbic brain structures, particularly in the hippocampus [26], and has mild kinase activity in vitro [26, 31]. The biological function of DCLK-short expression in the adult hippocampus is largely unknown and the transgenic mice have subtle phenotypes with no obvious differences in basal outcomes (Schenk et al, in preparation). Microarray-based expression profiling of the hippocampus tissues from δC-DCLK-short and controls should reveal the biological processes in which the gene is involved.

The main aim of this paper is to compare the performance of different microarray platforms to detect differences in gene expression in biologically related samples. The performance of and the consistency between the microarray platforms on the level of affected genes and gene sets are reported here. The biological findings will be discussed in more detail elsewhere (Schenk, in preparation).

The aims of the present study were to compare the ability of different microarray platforms to detect differences in gene expression, when levels of regulation and numbers of regulated genes are low, and to investigate the influence of the platform in the biological interpretation of the results.

We show that even when gene expression differences between groups are small, several microarray platforms are able to consistently detect them. This is an important point, since in most previously published microarray platform comparisons, including the toxicogenomics MAQC study where biological replicates were analyzed, differences between samples analyzed where much larger than in our study [12, 21, 23–25]. The MAQC papers conclude that the cross-platform correlation is higher for fold-changes than for t-statistics. This is not true for our study. This apparent contradiction is because high fold-changes, which we simply do not have in our study, are more likely to be measured consistently, and contribute most to the Pearson correlation coefficient. Cross platform consistency in our study may compare favorably to another platform comparison study within a biological setting: Tan et al. reported a low agreement between 3 platforms (Affymetrix, Agilent, Amersham) in the analysis of the effect of serum withdrawal [14]. In their case, the amount of interrogated genes shared by all platforms was low. In our study, the number of common probes is bigger (N~ 12,000) and allows for more reliable comparisons since a bigger and possibly more representative set of probes is taken into consideration.

In contrast to other papers, we did not apply any filter to our data. In the reanalysis of the Tan dataset by Shi and collaborators [41] the authors claimed that the use of the unfiltered dataset gave a poor agreement between platforms, while restricting the analyses to a small filtered subset gives highly reproducible results. Even if several filters are commonly used, strict investigation on the possible bias introduced in the data because of the exclusion of genes has not been done. Since filters of the data may affect individual datasets differently, we have avoided using them in order to reflect the true unbiased gene expression signatures. The drawback is that the correlation measures are more affected by biological and technical noise.

The choice of the type of cut-off is still a matter of debate, and several authors suggested using a mixed cut-off of p-values and Fold Changes (FCs) [21, 24]. However, even if a FC cut-off makes DEGs determination easier and from the technical point of view is more direct, it can eliminate the possibility of finding small differences in the data that are biologically interesting, as demonstrated in the current study (where only two genes showed a FC > 2). Furthermore, the FC statistics do not have the probabilistic characteristics guaranteed by theoretical conditions that allow to be sure about what the method does [42, 43].

The degree of overlap between DEGs can be influenced by the overlap in interrogated and detectable transcripts as well as the method for matching of the probes. The overlap in interrogated transcripts was >75%, as expected for these whole genome microarray platforms. The overlap in probes with signal above background was also in the same range. However, by adding the two effects, one can explain as much as 50% of the difference between two platforms and this can be even more for home-spotted arrays were the numbers of detectable transcripts are often reduced due to local background problems. The overlap may be further reduced due to the interrogation of different splice variants that are mapped to the same UG identifier. The Ensembl transcript mapping accounts for alternatively spliced transcripts. However, the correlation between platforms in the Ensembl transcript-mapped dataset was, in our case, not higher than in the UG dataset. This could be due to complications in the mapping process: AFF probe sets sometimes cover more than one transcript, and for ABI oligonucleotide sequences were not provided but only 380 bp regions in which the probes were designed. Furthermore, there is considerable redundancy in the Ensembl transcript dataset due to multiple splice variants from the same gene being detected by all platforms, which may introduce biases in the downstream analyses. In this respect, the use of the recently released whole genome exon arrays for gene expression probably provides an attractive alternative, coping with such a problem.

AGL selected a ten-fold higher number of DEGs and significant gene sets than all other platforms. This is partly attributable to the high signal to noise level of this platform, as evident from the number of probes with signal higher than background. Still, this huge difference was unexpected and we investigated the behavior of the AGL data in more detail, and compared this with AFF and LGTC data using different approaches [see Additional file 1]. Briefly, the AGL log ratios show a bigger variability than AFF log intensities, measured by the a posteriori standard deviations. This difference remains after multiplying the variance of AFF intensities by the square root 2 in order to calculate the variance in the ratio between two samples. To check whether the doubled number of observations on the AGL were the cause for finding many more differentially expressed genes, we left AGL arrays out one by one and repeated the EBLRM analysis. The number of DEGs decreased steadily from 3,051 (10 arrays, 20 samples) to 649 (5 arrays, 10 samples). This is on the same order of magnitude as the number of DEGs of AFF (10 arrays, 10 samples, 130 DEGs), but still five times larger.

This suggests that the direct comparison of the wild-type and transgenic mouse samples on the same array drives the better performance, which is accordance with previous observations [13, 33]. It argues against using either a common reference design or one-color protocols when comparing two groups of samples [21]. However, this does not explain the differences in performance between AGL and LGTC arrays. We found that AGL's technical replicates were much more reproducible than those of LGTC: Pearson correlation coefficients were 0.95–0.98 for AGL and 0.70–0.80 for LGTC, illustrating the differences in quality between commercial and home-spotted arrays. Overall, our study suggests that the differences in amount of DEGs found by the different platforms were mainly caused by differences in signal to noise ratios, and the numbers of observations between one and two-color platforms, when using the same number of arrays. Our qRT-PCR experiments validated differential gene expression in most cases, also for genes found by AGL only, indicating that these are not just false positives.

Our results illustrates once more that typical sample sizes used in microarray experiments, three samples per group, can be too small to enable reliable detection of subtle effects such as in this study. Even though using 5 samples per group still does not yield enough power for some platforms, it is possible to use our data as basis for estimation of sample size for the platforms considered. We are undergoing this work and the detailed analysis, beyond the scope of this paper, shall appear elsewhere. Our preliminary results confirm that AGL and AFF have comparable power, so the different outcomes observed by us are for the largest part due to the larger effective sample size involved in two-colour platforms design.

We investigated whether the power of the analysis could be enhanced by merging data from all five platforms in one statistical model. We applied an EBLRM on the UG subset and included samples, platforms and dye (only for the two-color arrays) as confounders. At an FDR of 0.1, 285 genes were selected (Table 1). Among these, most had been selected as DEGs by the individual platforms with the exception of 56 genes. However, we could not validate the differential expression of the top 5 of those genes by qRT-PCR, mainly due to large biological variation within groups. These genes seem to have been selected in the merged analysis due to the technical consistency on the microarray platforms allied to the larger pooled sample size.

This study also aims to elucidate the biological function of delta-DCLK-short expression in the hippocampus. Recent loss and gain of function studies strongly suggest involvement of the DCLK gene in neurogenesis, neuronal migration, vesicle transport, microtubule-directed retrograde transport, neurotransmission and apoptosis [28–30, 44–46]. Thus, DEGs identified in this study may be involved in these processes. The present study focuses on comparison of different array platforms and therefore the results of the biological function will be discussed more extensively elsewhere (Schenk in preparation). However, it is interesting to note that the DEGs and the significant gene sets revealed by the different microarrays are biologically meaningful. For example, numerous gene sets related to GABA-ergic neurotransmission emerged as highly significant in 4 out of 5 platforms. Intriguingly, similarly as the DCLK gene, excitatory GABA signalling has been shown to control neurogenesis, neuronal migration and differentiation of neuroblasts [47, 48]. DCLK-short expression starts postnatally around day 6, a timepoint that is characterized by a switch in excitatory GABAergic responses to inhibitory responses [49, 50]. The added value of the use of different microarray platforms lies in the prioritization of the pathways for follow-up experiments. When analyzing data from a single platform, many spurious gene sets apparently not related to the biological process under study (e.g. chemotaxis) ranked highly, probably due to the relatively small expression differences observed. By comparing platforms, a biologically meaningful consensus could be distilled.

The present study suggests that the choice of a platform can be mainly governed by practical and cost considerations. However, our data demonstrate that, given the much higher number of identified DEGs, commercial two-color platforms may be preferred when two groups with small differences in expression are to be compared. In these situations, a direct-comparison design helps to maximize signal-to-noise ratios in the ratios between the two groups through minimization of the array effect and the possibility for more replicates with the same number of arrays. Since we performed this study with a clear underlying biological question, we could demonstrate that there was agreement across platforms in the perturbed biological processes identified. Consistency between platforms helped to prioritize biological processes relevant for the biological question under study. The relevant gene sets were detected with an only partly overlapping set of genes. Our data indicate that when analyzing samples with only subtle differences in gene expression the use of two different platforms might be more rewarding than increasing the number of replicates on the same platform.