The pitfalls of platform comparison: DNA copy number array technologies assessed
- Christina Curtis†1, 2Email author,
- Andy G Lynch†1, 2Email author,
- Mark J Dunning2,
- Inmaculada Spiteri2,
- John C Marioni3,
- James Hadfield2,
- Suet-Feung Chin1, 2,
- James D Brenton1, 2,
- Simon Tavaré1, 2 and
- Carlos Caldas1, 2
© Curtis et al; licensee BioMed Central Ltd. 2009
Received: 18 June 2009
Accepted: 8 December 2009
Published: 8 December 2009
The accurate and high resolution mapping of DNA copy number aberrations has become an important tool by which to gain insight into the mechanisms of tumourigenesis. There are various commercially available platforms for such studies, but there remains no general consensus as to the optimal platform. There have been several previous platform comparison studies, but they have either described older technologies, used less-complex samples, or have not addressed the issue of the inherent biases in such comparisons. Here we describe a systematic comparison of data from four leading microarray technologies (the Affymetrix Genome-wide SNP 5.0 array, Agilent High-Density CGH Human 244A array, Illumina HumanCNV370-Duo DNA Analysis BeadChip, and the Nimblegen 385 K oligonucleotide array). We compare samples derived from primary breast tumours and their corresponding matched normals, well-established cancer cell lines, and HapMap individuals. By careful consideration and avoidance of potential sources of bias, we aim to provide a fair assessment of platform performance.
By performing a theoretical assessment of the reproducibility, noise, and sensitivity of each platform, notable differences were revealed. Nimblegen exhibited between-replicate array variances an order of magnitude greater than the other three platforms, with Agilent slightly outperforming the others, and a comparison of self-self hybridizations revealed similar patterns. An assessment of the single probe power revealed that Agilent exhibits the highest sensitivity. Additionally, we performed an in-depth visual assessment of the ability of each platform to detect aberrations of varying sizes. As expected, all platforms were able to identify large aberrations in a robust manner. However, some focal amplifications and deletions were only detected in a subset of the platforms.
Although there are substantial differences in the design, density, and number of replicate probes, the comparison indicates a generally high level of concordance between platforms, despite differences in the reproducibility, noise, and sensitivity. In general, Agilent tended to be the best aCGH platform and Affymetrix, the superior SNP-CGH platform, but for specific decisions the results described herein provide a guide for platform selection and study design, and the dataset a resource for more tailored comparisons.
The accurate and high-resolution mapping of DNA copy number aberrations (CNA) has become an important tool for biological and medical research. From understanding the extent of natural genetic variation , to associations with diseases such as HIV , to elucidating the mechanisms of tumourigenesis , such research is dependent on the quality of the data generated.
Numerous reports on the use and comparison of copy number profiling platforms have appeared [4–10] and more recently an approach to perform meta-analyses across such platforms has been described . Early studies  suggested a high level of concordance between BAC-based aCGH and SNP-based platforms (Affymetrix 10 K array) in detecting CNA, but did not formally compare them. Greshock et al.  performed the first systematic comparison of multiple platforms on melanoma cell lines and found that a high level of sensitivity and specificity was observed for the Agilent 185 K arrays and that the increased probe density of Affymetrix arrays (100 K and 500 K) results in increased confidence in detection for these platforms. These results were echoed by Gunnarsson et al.  who also examined the performance of several older copy number profiling platforms (a 32 K BAC array, the Affymetrix 250 K SNP array, the Agilent 185 K oligonucleotide array, and the Illumina 317 K SNP) array in 10 chronic lymphocyte leukaemia (CLL) samples. They concluded that all platforms performed reasonably well at detecting large alterations, but that BAC probes were too large to detect small alterations. While Agilent offered the highest sensitivity, the increased density of SNP-CGH platforms (Affymetrix and Illumina) compensated for their increased technical variability, with Affymetrix detecting a higher degree of CNA compared to Illumina. A further aCGH study did not compare platforms, but did investigate the influence of cellularity on copy number detection  and concluded that modern high-resolution arrays could cope with high levels of contamination.
To attempt a fair and formal comparison of copy-number profiling platforms in a general setting is an almost futile exercise. Quantification of performance is difficult even with idealized data, and while measurements have been proposed such as the theoretical power to discover a single copy loss or gain , or the 'functional resolution' of the platform , these tend either to measure a very specific aspect of the platform, or appear flawed under close examination. Such idealized data are, in any case, difficult to obtain, as one has to ask what is fair in terms of numbers entering the experimental design. Should one Illumina array be compared to one Nimblegen array or should the two-channel Nimblegen array be compared to two arrays from the single colour technology? Should the two-colour platform be penalized by an inefficient design to allow easier comparison, or the SNP-based platform credited for the additional information that it brings? If, as often is the case, the main experimental constraint is financial, then comparing $1000 of one technology to $1000 of another technology would seem sensible. However, the relative costs of platforms will vary from laboratory to laboratory and with time, and such an approach would foist the authors' view of microarray economics on the reader.
Additionally, the results from such an exercise are only as good as the analysis methods used and in that regard one has two options, both flawed. Naturally, the platforms will require different pre-processing strategies, but if different methods of analysis are also used for segmentation, then the performance of the technology will be confounded with the adequacy of the algorithm. This then punishes newer technologies for which analytical methodologies are not yet mature. The alternative, to use a common approach for the analysis of all platforms, is undesirable firstly because that approach is likely to have been developed for one of the technologies and may thus introduce bias, and secondly because the deliberate use of a sub-optimal analysis does not provide useful information to inform decisions in the real world. Nonetheless, informative qualitative comparisons can be made without performing segmentation that illuminate the relative strengths and weaknesses of each platform. We acknowledge that some users will be primarily interested in a comparison based on using existing analytical tools, rather than concerning themselves with the potential of each platform, but that is not the purpose of this study.
This study differs from previous comparative assessments of copy number profiling platforms in that we have attempted to characterize the strengths and weaknesses of various platforms in as unbiased a fashion as possible by avoiding measures that cannot be fairly computed, highlighting areas of potential bias, and emphasizing a graphical assessment of performance that provides insight about the underlying technology as well as the specific platform. Inevitably, despite considerable effort, these comparisons will be shaped by our own prejudices concerning copy number analysis, but we have made the raw data available for others to draw their own conclusions.
Due to the speed of platform development, it is typical for a platform to be superseded by one with a greater number of features before comparisons involving it are published. The generation of platforms described here have not yet been the subject of an in-depth comparison, but have indeed already been superseded since this study was performed. Nonetheless, the underlying technologies are similar and a comparison is still informative. Implications for the new generations are discussed in the New Platforms section.
Herein we describe a comparison based on the analysis of two cell lines, six primary breast tumours, including matched normal samples, and two HapMap individuals. The SUM159 and MT3 cell lines and HapMap samples were selected based on the presence of known chromosomal aberrations, while the tumours are highly heterogeneous and hence present additional complexity for copy number analysis, not least with regard to their varying degrees of cellularity.
Here we present an analysis of probe coverage on each of the microarray platforms and a technical description of their reproducibility, sensitivity, and noise. We also provide an in-depth visual assessment of the ability of the different platforms to identify a range of sizes of copy number aberration. Lastly, we provide a publicly available dataset resulting from the processing of a range of samples (chosen to evaluate different abilities) on each platform. This information will allow interested parties to make decisions based on their own circumstances, preferences, and constraints.
Theoretical and technical performance
Probe coverage and resolution
Basic summary of platform contents
We choose not to present the theoretical functional resolution of these platforms as calculated by ResCalc  for three reasons, each of which is, in itself, revealing with regard to the inter-platform differences. Firstly, the results presented in Coe et al.  obscure a large degree of inter- and intra- chromosome variability. As a proportion of their total, Illumina have more probes on chromosome 6 than do the other platforms, with the result that even though there are more probes in total on the Nimblegen platform, for this particular chromosome Illumina have 16% more probes than Nimblegen. On chromosome 19, Affymetrix put a noticeably higher proportion of probes on the q arm than do Agilent, a situation that is reversed on chromosome 7.
The second problem of comparing by ResCalc is that the tool allows the platforms to define their own range of coverage from telomere to centromere. This makes it possible for a platform to improve its functional resolution by removing probes (essentially by dropping peripheral loosely spaced probes, while retaining the central tightly spaced ones), which is undesirable. To take an example, on arm 7p, in the core region covered by all of the platforms, Affymetrix average a probe every 3 to 4 Kb. However on the telomeric side of that core region, they have two probes covering 80 Kb. Undoubtedly the functional resolution as calculated by ResCalc would improve if such probes were removed (indeed, in this example, the removal of a single telomeric probe improves the reported functional resolution by 140 bases). Taking a more extreme example, the p arm of chromosome 9 has 13,643 probes on the Affymetrix platform and has a reported single probe functional resolution of 222,000 bases, but by removing 6 extreme telomeric probes and 166 extreme centromeric probes that are more sparsely positioned, we can improve the reported resolution to 8,900 bases. In general, the SNP-based platforms cover a wider region, with Nimblegen coming third and Agilent, in effect, often defining the core region of common coverage.
Finally, the hypothesis of uniform occurrence of CNA is doubtful and some of the platforms have been designed to provide non-uniform coverage by tiling more probes in known regions of variation (see Methods section for further details), or in areas where variation would be of particular interest. For example, Nimblegen have chosen, for the second generation of the product featured here, to switch from a uniform spacing along the genome to a 'designed' layout. This move would appear detrimental using tools such as ResCalc, but is clearly done for a purpose.
Reasons that one might adopt a non-uniform spacing include the desire to incorporate prior knowledge of genomic structure (e.g. to target CNVs, promoter regions, genes etc. and avoid repetitive elements), empirical evidence of probe performance from previous array designs, and lastly to achieve uniformity of probe performance. We show in the Methods section that there are a number of probe properties (most notably GC content) that affect the consistency of probe performance. These trends were visible in our data for all four platforms. There may, of course, be effects that are less visible, from these data, such as saturation levels and dynamic ranges. Naturally, increased probe coverage can address issues of variation, but technical biases will not be salved by increasing the number of probes.
All of the platforms provide some replicate probes, by which we mean probes carrying the same sequence. For the SNP-CGH arrays, this is an integral aspect of the platforms and nearly all of the observations are actually averaged from replicate probes, 4 replicates for the Affymetrix SNP probes, and an average of 16 replicates for Illumina probes (although this ranges from 0 to over 40). With the Agilent and Nimblegen arrays, such probes are a rarity, and the majority of observations are based on only one probe. As such, for these two platforms, it makes sense to use the few probes with replicate information to characterize the performance of all observations. We can do this most informatively by calculating the variance of the replicate log-ratios between two samples.
Agilent provide, in addition to control probes, 916 60-mer probes for which there are three replicates. Nimblegen do not nominally provide any replication, but the coverage of the pseudoautosomal regions of the X and Y chromosomes results in 314 probes that are apparently replicated. However, we should note that these probes are treated as lying on different chromosomes, and thus if any within-chromosome normalization has taken place then their apparent reproducibility will be adversely affected. Neither Agilent nor Nimblegen show a strong association between the magnitude of log-ratio and variance of replicate observations (this is after all one of the reasons for analysing the log-ratio). To enable between-array comparisons, when we have resisted performing between-array normalizations, we summarize for the HapMap-HapMap comparisons the variance of replicate probes scaled by the mean difference in log-ratios observed in chromosomes X and 13, a difference that should be 1 for this comparison. Since this scaling does not share information between arrays, it is not a between-array normalization method.
For Agilent, the median variance of replicate probes is 0.042, 0.048, and 0.058 on three different arrays with third quartile values of 0.087, 0.111, and 0.120 respectively. In contrast, for Nimblegen, the median variance of replicate probes is 0.125, 0.142, and 0.144 with third quartile values of 0.309, 0.429, and 0.504, respectively. Thus Nimblegen exhibits 2-4 fold greater variability amongst replicate probes than Agilent. However, we note that the interpretation of the third quartile, in particular, should be tempered by our knowledge of the autocorrelation of probes along the genome.
Note that while the SNP-CGH platforms enable the quantification of allele-specific copy number [14–16], similar results cannot be obtained for the aCGH platforms. As such, we will focus strictly on the analysis of total copy number values. To quantify DNA abundance (or raw total copy number), the SNP-CGH platforms essentially sum the fluorescence intensities from the two alleles investigated for a given SNP. This involves, for each allele, averaging over the replicate probes and then summing.
Because of these replicate probes, for Affymetrix and Illumina estimating the variance of individual probes is of limited value, since the values of individual probes will not be reported. Yet, for Illumina we cannot provide a good estimate of the variance after averaging over the replicate probes and then summing over alleles because the covariance of the two channels is not estimable from the data provided by BeadStudio, but can be presumed not to be zero due to the array design.
Variance among three replicate HapMap-HapMap comparisons
The ability of a copy number profiling platform to detect aberrations is largely determined by the noise observed in the measurements from that platform. This is a measure not only of the variance of the noise (although this is important), but also the kurtosis of the noise (i.e., if the noise is relatively heavy tailed, then more false calls will be made) and the independence of neighbouring probes. Not only are there known autocorrelation effects along the genome , possibly driven or exacerbated by autocorrelation in the quality of probe design caused by regions of high GC content or highly repetitive elements, but if probes are too close then they may compete to register the same DNA fragments. In such a case, the lack of independence of measurements from the probes would detract from the benefits of having improved probe density.
The ideal test for such a comparison would be a set of log-ratios generated from two replicate normal samples, as any departure from a log-ratio of 0 for these platforms must be noise and can be easily quantified. Since for two platforms, one of the pooled normal samples intended for this task was of lower quality, instead we again use chromosome 13 from a comparison of the two HapMap samples. Not only does this have no known changes, but adds the benefit that again we can scale our observations so that the difference in log-ratios between chromosomes 13 and X is a standard 1.
Characteristics of a surrogate self-self hybridization
% z > 2
% z > 3
0.33, 0.34, 0.29
0.040, 0.039, 0.036
4.5, 4.8, 4.7
1.3, 1.4, 1.3
0.22, 0.24, 0.21
-0.001, 0.027, 0.019
4.4, 4.6, 4.9
0.5, 0.8, 0.7
0.28, 0.36, 0.31
0.086, 0.066, 0.076
5.2, 5.3, 5.3
1.4, 1.4, 1.2
0.81, 0.85, 0.60
0.009, 0.035, 0.026
4.3, 4.8, 4.5
0.5, 0.5, 0.5
These results indicate that Nimbelgen is noisy, exhibiting poor variance (2-4 fold greater than the other platforms). Additionally, Illumina has relatively poor autocorrelation for its probe density and has more outliers at a standard deviation of 2. Further, both SNP-CGH platforms have more outliers beyond a standard deviation of 3, which may be related to the autocorrelation. It is worth noting that Agilent has relatively few probes on chromosome 13 (see Table 1, Additional File 1), but based on other performance measures, this is unlikely to influence significantly its superior performance.
Male-Female comparisons based on X and Y chromosomes
For distinguishing between sites where both samples have two copies and sites where one sample has two copies while the other has one (13 versus X), Affymetrix and Agilent marginally outperform Illumina, while Nimblegen performs noticeably worse. In contrast, when distinguishing between sites where both samples have two copies and sites where one sample has no copies while the other has one (chromosome 13 versus Y), Agilent generally exhibits the highest sensitivity, although Illumina outperforms Agilent if very high specificity is sought. These are followed in performance by Nimblegen, with Affymetrix performing considerably worse.
Notably, the Affymetrix Human Mapping 100 K, 500 K, and SNP5 platforms include chromosome X SNPs but no chromosome Y or mitochondrial SNPs. With the SNP5 platform, copy number non-polymorphic (CN probes) were introduced and for the Y chromosome there are 996 such probes with sufficient genomic information (1994 in total) all of which map outside the pseudoautosomal region. As such, for the SNP5 platform, the Y chromosome is not representative of other chromosomes in that it does not include any SNP probes and contains 0.1% of all probes on the platform. The lack of SNP probes is one possible explanation for the poor discrimination of a single copy loss on the Y chromosome. As noted in the Methods section, the CN probes are generally unreplicated and while few in number, the actual number of probes is on par with the other platforms.
Qualitative assessment of copy number aberration detection
The platforms investigated in this study differ substantially in their design, the number of probes, and their experimental utility. To obtain an overview of platform performance, the ability to detect several types of common chromosomal changes was assessed. In particular, the following alterations were considered based on raw copy number changes: whole chromosome gains or losses, chromosome arm gains or losses, high amplitude focal amplifications as well as subchromosomal gains and losses, small regions of gain or loss as exemplified by normal copy number variation.
i.) Whole chromosome gains or losses
Also of note is the performance in terms of Y chromosome detection and the effect of normalization on the Illumina array. The performance of Illumina in detecting the absence of the Y chromosome in females is of concern. It is not unreasonable that what would ideally be an estimate of log2(0/0) should be unstable (although due to non-specific binding the extremes of this instability will not be observed). If the observed values are indicative of any bias in the probe design, then the apparently strong performance of Illumina in the chromosome Y versus chromosome 13 comparison may have been misleading.
ii.) Chromosome arm gains or losses
iii.) High amplitude focal amplifications and subchromosomal gains and losses
iv.) Small regions of gain/loss as exemplified by copy number variation between normal HapMap individuals
A total of 79 sites of copy number variation have been identified between the two HapMap individuals assayed in this study using an older technology, namely a custom whole-genome tiling path array developed at the Wellcome Trust Sanger Institute . These variants were validated across multiple hybridizations and also via PCR. For a full list of locations see Additional File 3. Examination with these higher resolution technologies suggests that some of the sites actually form one larger variant, but we shall treat them as separate sites for this analysis. Many of the sites showed no sign of variation with any of the platforms, and concordance amongst platforms was high. Due to the nature of these small changes, it is not uncommon for a platform simply to have no probes in the region of interest. This varies between platforms, with probe density being influential, but not the only factor.
CNVs observed between two HapMap samples
Notably, since some platforms (both Affymetrix and Illumina) have been designed to cover known CNVs and to target 'unSNPable' regions of the genome with copy-number non-polymorphic probes, this rate will by misleading if one is interested in identifying novel CNVs. Nimblegen has more probes than Agilent, and a similar number to Illumina, but does not attempt to target known interesting regions with this version of the array. Thus Nimblegen may well do relatively better with novel sites. That said, the evidence here is that even if novel sites have coverage, the platform may struggle to identify them as CNVs. Illumina cover more of the regions than do Affymetrix, but do not provide the clarity of change over these small intervals.
Detection of characterized copy number aberrations
Detection of anticipated aberrations across platforms for the 6 tumour samples
possibly 8q (all)
also 8q (all)
none, but gain on 8q (all but Nimb)
all but Nimb
and gain 16p (all)
and gain 16p (all)
all but Nimb
Affy and Agil
all but Nimb
The data set we present allows for the realistic comparison of platforms when considering copy-number changes in tumours. Tumour samples are often affected by stromal contamination  and to represent this, not only do we present 6 tumour samples of varying degrees of cellularity (see Additional File 5 for cellularity and clinical information for all samples), but a number of samples with simulated stromal contamination. Essentially, two of the tumour samples were diluted with their respective matched normal samples (7206: 30% tumour, 70% normal; 7207: 50% tumour, 50% normal) and two cancer cell lines were similarly treated (MT3 and SUM159: 30% tumour, 70% normal 7214).
Discussion of results
The ability of a platform to detect a particular aberration is a function of the distribution of probes in that region and the reliability of those probes. Of the two SNP-based platforms, there is little difference in terms of quality of individual probes, but those on the Affymetrix arrays are more numerous. That said, Illumina's strategy for locating probes means that there are locations where this platform offers greater coverage (cf. the known CNVs and the MHC-similar region 5q13, consistent with Illumina's stated design intent which also sees a greater focus on SNPs near RefSeq genes than does Affymetrix) but also some (such as the ERBB2 region) where they are lacking. The coverage of smaller features such as CNVs and genes is an important consideration in the choosing of a copy-number platform, as broadly speaking all of the platforms examined can identify large deletions and duplications.
A curiosity is that Illumina fails to identify robustly the chromosome 13 arm gain in the MT3 cell lines, suggesting an issue with the normalization applied by BeadStudio, but the main concern is the Nimblegen platform, which fails to spot some large aberrations in tumours T7195 and T7214. Of the two standard arrayCGH platforms, Agilent's performance is clearly superior. Not only is the Agilent data of high-enough quality to call aberrations from fewer probes than the other platforms, but also the ability of the Agilent platform to quantify aberrations appears to be superior. All of the platforms suffer from variation induced by probe design, related either to probe length, GC content or other aspects. Additionally, the quality of SNP and CGH probes on the Affymetrix and Illumina platforms may not be equivalent. Thus when choosing a platform one must consider not only the probe coverage in regions of interest, but also the quality of those probes.
Explanation of cellularity findings
In comparing microarray technologies, it is also important to keep in mind some of the more subtle differences between them in terms of the protocols, chemistry, and detection methods. For example, while both Affymetrix and Illumina are SNP-based copy number profiling platforms, there are important differences in their chemistries. The Affymetrix GenomeWide SNP 5.0 whole genome genotyping assay (as well as the newer SNP 6.0 array and older generations of this platform, namely the 10 K - 500 K arrays) all employ a complexity reduction procedure similar to that first described for representational oligonucleotide microarray analysis (ROMA)  in order to increase the signal-to-noise ratio. Essentially, the DNA is digested with the restriction enzymes Nsp I and Sty I, ligated to adaptors that recognize the cohesive four base-pair overhangs, and amplified using a universal primer that recognizes the adaptor sequence. The amplified DNA is subsequently fragmented, labelled, and hybridized to the oligonucleotide array. While the amplification of only the smaller restriction fragments improves the signal-to-noise ratio, these values still remain below that observed for BAC arrays, and the complexity reduction can potentially lead to the differential representation of certain genome regions and hence false positives. Also, since individuals vary in their restriction digestion profiles, certain probe ratio values may depend on differences in restriction fragment size rather that actual copy number variation .
In contrast, the Illumina whole genome genotyping protocol for the 370 HapMap Duo bead array (Infinium II technology) involves an isothermal genome amplification step (non-PCR based), fragmentation, hybridization to an oligonucleotide bead array, SNP detection based on a single-base extension reaction (SBE) on a single bead type with differentially-labelled terminators, and signal amplification. Thus the detection step, for the Illumina Infinium II assay is based on an enzymatic discrimination step (SBE for Infinium II, allele-specific extensions for Infinium I) rather than by hybridization as for Affymetrix. Illumina claims that the isothermal amplification step does not result in the preferential amplification of one allele .
All of the manufacturers now offer products with more features than those compared in this report: the Affymetrix GenomeWide SNP 6.0 array, the Illumina 1 M-Duo array, the Nimblegen Ultra-High Density CGH array with 2.1 million features, and the Agilent Human CGH 1 × 1 M array. All but the Affymetrix chip come available with fewer features but multiple arrays on the chip (the Illumina platform starting with 2 arrays on the chip); the ability to run multiple samples in parallel is of great potential value for sensitive experiments. Also worthy of note is that the Nimblegen and Agilent platforms offer full customization of content, while Illumina offer limited customization.
As the coverage of platforms increases, many of the subtleties that we have observed will have decreased impact on the conclusions. The Illumina coverage of ERBB2, for example, is satisfactory in the latest generation of chip. It remains to be seen whether the manufacturers have been able to maintain probe quality in the next generation of products. We have already commented that the second generation of the Nimblegen platform featured here has seen a revision of probes to improve performance.
Alternative analysis methods
It should be noted that we have made use of manufacturer-provided tools, where available, for pre-processing this dataset. This was intentional, as the choice of optimal tools is platform-specific, especially since older platforms will likely benefit from more mature analysis tools. For example, we employed the BeadStudio software to summarize the Illumina data as this is manufacturer-supplied. Likewise, Nimblegen supplied NimbleScan pre-processed data. In contrast, Affymetrix do not offer comprehensive support for copy number analysis of the GenomeWide SNP5.0 platform so we employed the open source aroma.affymetrix software, one of the few tools for pre-processing this relatively new SNP-CGH platform. Similarly, as Agilent do not provide free software for pre-processing of their aCGH data, standard open source methods were employed for this well-established platform. Although beyond the scope of this study, it is of interest to compare alternate pre-processing (and segmentation) methods for each of these platforms as this could influence the results obtained. In particular, the consideration of Illumina SNP data at the bead level could yield considerable improvements since this would enable calculation of the within-bead-type correlation or covariance  as well as more detailed quality assessment. For example, we were able to identify spatial artefacts in the Illumina data in this study (Additional File 7) that would benefit from the BASH tool  implemented in beadarray although this method has not yet been fully implemented for Illumina genotyping data. Additionally, there were some issues with small, localized failures of image registration that could only be addressed by bead-level pre-processing and that would undoubtedly improve the quality of the Illumina results if addressed successfully.
It is important to stress that there is no straightforward way to compare fairly copy number profiling platforms in a general manner. As such, the results presented here describe the detection and qualitative comparison of raw copy number alterations across four platforms in tumour samples for which both matched and pooled normal DNA were available and in two established cell-lines. Copy number variation in normal HapMap individuals was also compared using the same platforms. Whilst we have sought to avoid analytical techniques that are objective, but that we deem undesirable for the stated reasons, we have focused on graphical comparisons that are, of course, prone to subjectivity. In any case, the competing platforms have different merits, and users need to make subjective decisions based on their individual requirements.
Although there are substantial differences in the design, density, and replicate structure of the probes, the comparison indicates a generally high level of concordance between platforms. As expected, all platforms were able to detect large aberrations in a robust manner. However, some focal amplifications and deletions were only detected on a subset of the platforms. In particular, Nimblegen failed to detect numerous aberrations that were clear in the other platforms even when probes were tiled in the region of interest. This finding is perhaps not surprising given that this platform exhibits 2-4 fold greater variance amongst replicate probes and variances an order of magnitude greater for replicate array comparisons. In general, for the aCGH-based platform Agilent was the best performer and for the SNP-CGH platform, Affymetrix tended to outperform Illumina. An added bonus is that both Affymetrix and Agilent require only 0.5 μg DNA as starting material, thus removing this consideration from the platform decision. Another potential consideration is the quality or source of DNA (e.g. the use of paraffin-embedded samples ), for which some platforms may be more forgiving.
Our study differs from previously published ones in that we employ primary breast tumour samples rather then cell-lines. As noted previously, this introduces additional complexity due to the possibility of stromal contamination . Further to this, we have also made use of cell-line dilutions and well-characterized HapMap samples to evaluate copy number alterations across platforms. That we also conclude that Agilent performs best on a single-probe comparison is of interest because we are comparing newer platforms, yet we must keep in mind that the performance of platforms from generation to generation cannot be assumed to be constant.
In the new generation of arrays, Agilent have addressed their primary weakness by increasing probe coverage. Similarly, Nimblegen have modified their probe design in order to improve performance. Both Affymetrix and Illumina have increased probe coverage with Affymetrix introducing slight modifications to probe design. If Agilent have maintained probe quality, it seems likely they will remain the leader, but Nimblegen may close the observed gap. For the SNP-CGH arrays, it seems likely that Affymetrix will continue to perform well. The availability of data from these new platforms will enable comparisons with previous generations of arrays for the purposes of meta-analyses and the like.
Obtaining reproducible, high-resolution copy number data with high sensitivity and few false positives is the gold-standard objective for any such study. However, there are always tradeoffs and a critical assessment of the goals of the project and underpinning biological questions can help select the most suitable platform. For example, breakpoint precision, which is dependent on the local resolution, is likely more critical for mapping novel tumour suppressor genes and oncogenes, than for a more general survey of aberrations where little follow-up validation is planned. Additional considerations that might influence the choice of platform include probe coverage (whether gene-centric or uniformly spaced, targeting non-coding elements) and the ability to assay genotypic information, and hence allele-specific copy number and copy neutral loss of heterozygosity. If matched normal samples are available, it might be advantageous to exploit the direct comparison design offered by dual-channel technologies. In large-scale studies, it may also be useful to validate the higher-density SNP-CGH findings using a subset of samples on a lower-density, but more sensitive, platform. The results described here provide a guide for platform selection and study design, and the dataset a resource for more tailored comparisons.
The state of the art in terms of commercially available platforms for genome-wide CNA is constantly evolving. Here, four leading platforms were compared: the Affymetrix Genome-Wide Human SNP Array 5.0, the Agilent High-Density CGH Human 244A array, the Illumina HumanCNV370-Duo DNA Analysis BeadChip, and the Nimblegen 385 K oligonucleotide array. Several important differences exist between these platforms. Beyond the fact that the Affymetrix and Illumina employ a single-channel hybridization scheme, whereas Agilent and Nimblegen use a dual-channel competitive hybridization protocol, the former are also SNP-CGH platforms, while the latter are not. Other differences in the design of these platforms include the probe-length and probe-density. Whereas Nimblegen employs 45-mer to 85-mer probes, Agilent 60-mer probes and Illumina 50-mer probes, Affymetrix probes are considerably shorter at 25 nucleotides. In terms of probe-density, the Affymetrix SNP 5.0 array contains 500,568 SNP probes and an additional 420,000 non-polymorphic probes to facilitate studies of germline copy number variation in association studies. The Agilent 244A array contains computationally pre-selected probes that have been experimentally optimized for genomic hybridization with a bias towards gene-rich regions. The Illumina CNV370 array includes 318,000 SNP markers plus 52,000 markers targeting 14,000 additional CNV regions. Lastly, the Nimblegen 385 K array contains 386,165 isothermal oligonucleotide probes with relatively uniform genome coverage. Due to resource availability, two of the platforms (Agilent and Illumina) were processed in-house, whereas for the other platforms the samples were hybridized at a commercial vendor (Affymetrix and Nimblegen).
Two representative cell lines (MT3 and SUM159) were selected based on the presence of known chromosomal aberrations so as to provide markers of a platform's performance. The MT3 colorectal cell line contains a single copy gain of chromosome 7 and isochromosome 13 [27, 28]. The SUM159 breast carcinoma cell line is also reported to have several notable changes including a loss on chromosome 5q and gain on chromosome 8q24 [27, 28]. The ability of the various platforms to detect known focal amplifications was assessed using a panel of six tumour samples. To assess the effect of using a matched normal as compared to a pooled normal as the reference against tumour samples, a single replicate was included for each matched normal sample. Additionally, to ascertain the effect of cellular heterogeneity due to stromal contamination in detecting CNA, several dilution experiments were included for the two cell lines and two of the tumours such that a mixture of either 30% cell line (tumour) with 70% normal or a 1:1 ratio was hybridized to the arrays.
Two 'normal' samples (NA15510 and NA10851) from the Human HapMap study  were also selected to assess the detection of naturally occurring regions of copy number variation, as they have been characterized extensively [30–33] and are recommended for use as a standard control in all studies . Further, they provide an example in which gross abnormalities are not expected. Moreover, sample NA10851 is male, allowing for a controlled assessment of the platforms performances by examination of the sex chromosomes in the HapMap comparisons.
Each sample was hybridized to the single-channel platforms in triplicate, with the exception of the pooled normal samples, which were performed in duplicate. For the dual-channel platforms, tumours and cell-lines were hybridized against pooled normal tissue in duplicate, and the tumours were additionally hybridized against matched normal tissue. The HapMap samples were hybridized against each other in duplicate, as was a pool vs. pool hybridization. Additionally dye-swap hybridizations were performed for the HapMap samples and the MT3 cell-line. In all platforms, save for Nimblegen, some hybridizations were discarded under quality control procedures. Nimblegen only returned data for hybridizations that satisfied their quality control criteria.
Patient material and cell lines
Samples were collected in the year 2000 at Addenbrooke's Hospital, Cambridge, UK from female patients ranging from 41 to 83 years old. These samples correspond to fresh frozen biopsies and surgical resection samples and the resultant fresh breast tissue was stored in the Addenbrooke's Hospital tumour bank. Ethical consent was obtained for all patient samples. The MT3 cell line (with a single X chromosome, suggesting male origin) was obtained from its originators , and has been shown to be identical to the colorectal cancer cell line LS174T based on SNP analysis . This cell line exhibits an almost normal karyotype, apart from trisomy 7 and isochromosme 13. The SUM159 breast carcinoma cell line was obtained from the originators . SUM159 is a hyperdiploid cell line with a modal chromosome number of 47 and nine structural translocations.
All human samples used for this analysis were obtained with informed consent from patients and the study was performed with appropriate REC and NHS R&D approval.
DNA extraction and purification
Tumour DNA was extracted from 25 × 10 um sections manually using the DNAeasy kit (Qiagen, Valencia, CA). Matched normal DNA was obtained by homogenizing tissue in 180 μl of ATL buffer with Precellys, followed by extraction with the DNAeasy kit (Qiagen). For the cell lines, DNA was extracted using the proteinase-SDS method .
Array hybridization and analysis
The Agilent platform used is the Agilent Human Genome CGH Microarray Kit 244A This platform uses just under 240,000 unique 60-mer oligonucleotide probes across the genome, with tighter coverage in the region of RefSeq genes, and claims to emphasize other interesting genomic features (miRNAs, promoters, etc) also. Experiments were performed in-house using 0.5 μg of DNA and either the Agilent labelling kit or the Enzo labeling kit. After hybridization and washing, the slides were scanned on an Agilent Microarray Scanner and captured images were analysed with Feature Extraction Software v 9.1.
Arrays were considered for analysis using a guideline DLRS threshold of 0.3. This is higher than the threshold advised by Agilent, but that threshold does not allow for the large aberrations associated with tumour samples that will inevitably inflate this score. Where necessary (if multiple repeat hybridizations for a sample failed to bring the score down), hybridizations with a higher score were used to fill in gaps in the experimental design if they were judged to be acceptable. Similarly, some samples were not used despite passing the threshold if they were clearly problematic from a visual inspection. This resulted in 40 arrays remaining in the study (see Additional File 9). The Enzo protocol used for Agilent saw generally lower scores for this quality control measure, but saw an increase in the influence of probe length on the results from the array.
All analysis was performed in the R statistical programming language . The arrays described in this study been deposited in the Gene Expression Omnibus  with accession number GSE16400. Plots of each chromosome for each sample and platform are available in Additional File 12 and 13.
Where we have plotted relative copy number (log-ratio) against genomic location, we have used the best quality example for each platform. This may be the cause of a slight bias, as different platforms may have different numbers of replicates from which to choose, but since we are looking to establish the potential of the platforms, it is the appropriate approach. Replicates have not been averaged, as between-array standardization has not been performed, save for the case of CNV comparisons, where three replicates of each platform are comparable without standardization and the improved signal-to-noise allows for acceptable clarity with so few probes. Genomic location was taken from the supplied annotations for Agilent and Nimblegen, and likewise for Affymetrix and Illumina. For the different platforms the genomic location represents different properties (probe start, SNP location etc). However, on the scale on which we are plotting, this does not affect interpretation.
The scale for the y-axis for the plots is linear from -2 to 2, and linear also outside this region, but at a different rate. Most values lie in the -2 to 2 range and this needs to be our focus, but it is also important that we can depict more extreme cases. The discontinuity in the first derivative of the scale allows us to achieve this. As well as the log-ratios for the four platforms, we depict genes lying on the plus and minus strands, and a guide to the section of chromosome being illustrated. The information for these additional items was obtained from the GenomeGraphs package in Bioconductor.
Where CNV locations are plotted, the nominal location lies within the middle two-fifths of the x-axis, allowing for easy use of the provided axis coordinates to identify that region. Throughout the paper, we adopt a convention of colour-coding for platforms: Affymetrix are represented by magenta, Agilent by red, Illumina by black, and Nimblegen by blue.
We acknowledge the support of the University of Cambridge, Cancer Research UK, and Hutchinson Whampoa. ST is a Royal Society-Wolfson Research Merit Award holder. MJD was supported in part by a grant from the Medical Research Council. We thank Michelle Osborne and Sarah Moffatt for technical assistance with the Agilent and Illumina hybridizations. We also thank Andrew Teschendorff and Sergii Ivakhno for constructive discussions, Gulisa Turashvili for pathological expertise, and Matthew Hurles for access to CNV validation data.
- Redon R, Ishikawa S, Fitch KR, Feuk L, Perry GH, Andrews TD, Fiegler H, Shapero MH, Carson AR, Chen W, et al: Global variation in copy number in the human genome. Nature. 2006, 444 (7118): 444-454. 10.1038/nature05329.PubMed CentralView ArticlePubMedGoogle Scholar
- Schmutz J, Martin J, Terry A, Couronne O, Grimwood J, Lowry S, Gordon LA, Scott D, Xie G, Huang W, et al: The DNA sequence and comparative analysis of human chromosome 5. Nature. 2004, 431 (7006): 268-274. 10.1038/nature02919.View ArticlePubMedGoogle Scholar
- Comprehensive genomic characterization defines human glioblastoma genes and core pathways. Nature. 2008, 455 (7216): 1061-1068. 10.1038/nature07385.Google Scholar
- Gaasenbeek M, Howarth K, Rowan AJ, Gorman PA, Jones A, Chaplin T, Liu Y, Bicknell D, Davison EJ, Fiegler H, et al: Combined array-comparative genomic hybridization and single-nucleotide polymorphism-loss of heterozygosity analysis reveals complex changes and multiple forms of chromosomal instability in colorectal cancers. Cancer Res. 2006, 66 (7): 3471-3479. 10.1158/0008-5472.CAN-05-3285.View ArticlePubMedGoogle Scholar
- Greshock J, Feng B, Nogueira C, Ivanova E, Perna I, Nathanson K, Protopopov A, Weber BL, Chin L: A comparison of DNA copy number profiling platforms. Cancer Res. 2007, 67 (21): 10173-10180. 10.1158/0008-5472.CAN-07-2102.View ArticlePubMedGoogle Scholar
- Coe BP, Ylstra B, Carvalho B, Meijer GA, Macaulay C, Lam WL: Resolving the resolution of array CGH. Genomics. 2007, 89 (5): 647-653. 10.1016/j.ygeno.2006.12.012.View ArticlePubMedGoogle Scholar
- Hehir-Kwa JY, Egmont-Petersen M, Janssen IM, Smeets D, van Kessel AG, Veltman JA: Genome-wide copy number profiling on high-density bacterial artificial chromosomes, single-nucleotide polymorphisms, and oligonucleotide microarrays: a platform comparison based on statistical power analysis. DNA Res. 2007, 14 (1): 1-11. 10.1093/dnares/dsm002.PubMed CentralView ArticlePubMedGoogle Scholar
- Gunnarsson R, Staaf J, Jansson M, Ottesen AM, Goransson H, Liljedahl U, Ralfkiaer U, Mansouri M, Buhl AM, Smedby KE, et al: Screening for copy-number alterations and loss of heterozygosity in chronic lymphocytic leukemia--a comparative study of four differently designed, high resolution microarray platforms. Genes Chromosomes Cancer. 2008, 47 (8): 697-711. 10.1002/gcc.20575.View ArticlePubMedGoogle Scholar
- Baumbusch LO, Aaroe J, Johansen FE, Hicks J, Sun H, Bruhn L, Gunderson K, Naume B, Kristensen VN, Liestol K, et al: Comparison of the Agilent, ROMA/NimbleGen and Illumina platforms for classification of copy number alterations in human breast tumors. BMC Genomics. 2008, 9: 379-10.1186/1471-2164-9-379.PubMed CentralView ArticlePubMedGoogle Scholar
- Wicker N, Carles A, Mills IG, Wolf M, Veerakumarasivam A, Edgren H, Boileau F, Wasylyk B, Schalken JA, Neal DE, et al: A new look towards BAC-based array CGH through a comprehensive comparison with oligo-based array CGH. BMC Genomics. 2007, 8: 84-10.1186/1471-2164-8-84.PubMed CentralView ArticlePubMedGoogle Scholar
- Bengtsson H, Ray A, Spellman P, Speed TP: A single-sample method for normalizing and combining full-resolution copy numbers from multiple platforms, labs and analysis methods. Bioinformatics. 2009, 25 (7): 861-867. 10.1093/bioinformatics/btp074.PubMed CentralView ArticlePubMedGoogle Scholar
- Kloth JN, Oosting J, van Wezel T, Szuhai K, Knijnenburg J, Gorter A, Kenter GG, Fleuren GJ, Jordanova ES: Combined array-comparative genomic hybridization and single-nucleotide polymorphism-loss of heterozygosity analysis reveals complex genetic alterations in cervical cancer. BMC Genomics. 2007, 8: 53-10.1186/1471-2164-8-53.PubMed CentralView ArticlePubMedGoogle Scholar
- Garnis C, Coe BP, Lam SL, MacAulay C, Lam WL: High-resolution array CGH increases heterogeneity tolerance in the analysis of clinical samples. Genomics. 2005, 85 (6): 790-793. 10.1016/j.ygeno.2005.02.015.View ArticlePubMedGoogle Scholar
- Nannya Y, Sanada M, Nakazaki K, Hosoya N, Wang L, Hangaishi A, Kurokawa M, Chiba S, Bailey DK, Kennedy GC, et al: A robust algorithm for copy number detection using high-density oligonucleotide single nucleotide polymorphism genotyping arrays. Cancer Res. 2005, 65 (14): 6071-6079. 10.1158/0008-5472.CAN-05-0465.View ArticlePubMedGoogle Scholar
- Beroukhim R, Lin M, Park Y, Hao K, Zhao X, Garraway LA, Fox EA, Hochberg EP, Mellinghoff IK, Hofer MD, et al: Inferring loss-of-heterozygosity from unpaired tumors using high-density oligonucleotide SNP arrays. PLoS Comput Biol. 2006, 2 (5): e41-10.1371/journal.pcbi.0020041.PubMed CentralView ArticlePubMedGoogle Scholar
- Staaf J, Vallon-Christersson J, Lindgren D, Juliusson G, Rosenquist R, Hoglund M, Borg A, Ringner M: Normalization of Illumina Infinium whole-genome SNP data improves copy number estimates and allelic intensity ratios. BMC Bioinformatics. 2008, 9: 409-10.1186/1471-2105-9-409.PubMed CentralView ArticlePubMedGoogle Scholar
- Lynch AG, Dunning MJ, Iddawela M, Barbosa-Morais NL, Ritchie ME: Considerations for the processing and analysis of GoldenGate-based two-colour Illumina platforms. Stat Methods Med Res. 2009Google Scholar
- Marioni JC, Thorne NP, Valsesia A, Fitzgerald T, Redon R, Fiegler H, Andrews TD, Stranger BE, Lynch AG, Dermitzakis ET, et al: Breaking the waves: improved detection of copy number variation from microarray-based comparative genomic hybridization. Genome Biol. 2007, 8 (10): R228-10.1186/gb-2007-8-10-r228.PubMed CentralView ArticlePubMedGoogle Scholar
- Scherer SW, Lee C, Birney E, Altshuler DM, Eichler EE, Carter NP, Hurles ME, Feuk L: Challenges and standards in integrating surveys of structural variation. Nat Genet. 2007, 39 (7 Suppl): S7-15. 10.1038/ng2093.PubMed CentralView ArticlePubMedGoogle Scholar
- Borevitz JO, Liang D, Plouffe D, Chang HS, Zhu T, Weigel D, Berry CC, Winzeler E, Chory J: Large-scale identification of single-feature polymorphisms in complex genomes. Genome Res. 2003, 13 (3): 513-523. 10.1101/gr.541303.PubMed CentralView ArticlePubMedGoogle Scholar
- Carter NP: Methods and strategies for analyzing copy number variation using DNA microarrays. Nat Genet. 2007, 39 (7 Suppl): S16-21. 10.1038/ng2028.PubMed CentralView ArticlePubMedGoogle Scholar
- Illumina : Infinium II Assay Workflow Whitepaper, Illumina SNP Genotyping. 2006Google Scholar
- Chromosome X Titration Data Set. [http://www.affymetrix.com/support/technical/sample_data/genomewide_snp6_data.affx]
- Bengtsson H, Irizarry R, Carvalho B, Speed TP: Estimation and assessment of raw copy numbers at the single locus level. Bioinformatics. 2008, 24 (6): 759-767. 10.1093/bioinformatics/btn016.View ArticlePubMedGoogle Scholar
- Cairns JM, Dunning MJ, Ritchie ME, Russell R, Lynch AG: BASH: a tool for managing BeadArray spatial artefacts. Bioinformatics. 2008, 24 (24): 2921-2922. 10.1093/bioinformatics/btn557.PubMed CentralView ArticlePubMedGoogle Scholar
- Dunning MJ, Smith ML, Ritchie ME, Tavare S: beadarray: R classes and methods for Illumina bead-based data. Bioinformatics. 2007, 23 (16): 2183-2184. 10.1093/bioinformatics/btm311.View ArticlePubMedGoogle Scholar
- Forozan F, Mahlamaki EH, Monni O, Chen Y, Veldman R, Jiang Y, Gooden GC, Ethier SP, Kallioniemi A, Kallioniemi OP: Comparative genomic hybridization analysis of 38 breast cancer cell lines: a basis for interpreting complementary DNA microarray data. Cancer Res. 2000, 60 (16): 4519-4525.PubMedGoogle Scholar
- Davidson JM, Gorringe KL, Chin SF, Orsetti B, Besret C, Courtay-Cahen C, Roberts I, Theillet C, Caldas C, Edwards PA: Molecular cytogenetic analysis of breast cancer cell lines. Br J Cancer. 2000, 83 (10): 1309-1317. 10.1054/bjoc.2000.1458.PubMed CentralView ArticlePubMedGoogle Scholar
- Frazer KA, Ballinger DG, Cox DR, Hinds DA, Stuve LL, Gibbs RA, Belmont JW, Boudreau A, Hardenbol P, Leal SM, et al: A second generation human haplotype map of over 3.1 million SNPs. Nature. 2007, 449 (7164): 851-861. 10.1038/nature06258.View ArticlePubMedGoogle Scholar
- Tuzun E, Sharp AJ, Bailey JA, Kaul R, Morrison VA, Pertz LM, Haugen E, Hayden H, Albertson D, Pinkel D, et al: Fine-scale structural variation of the human genome. Nat Genet. 2005, 37 (7): 727-732. 10.1038/ng1562.View ArticlePubMedGoogle Scholar
- Fiegler H, Redon R, Andrews D, Scott C, Andrews R, Carder C, Clark R, Dovey O, Ellis P, Feuk L, et al: Accurate and reliable high-throughput detection of copy number variation in the human genome. Genome Res. 2006, 16 (12): 1566-1574. 10.1101/gr.5630906.PubMed CentralView ArticlePubMedGoogle Scholar
- Conrad DF, Andrews TD, Carter NP, Hurles ME, Pritchard JK: A high-resolution survey of deletion polymorphism in the human genome. Nat Genet. 2006, 38 (1): 75-81. 10.1038/ng1697.View ArticlePubMedGoogle Scholar
- McCarroll SA, Hadnott TN, Perry GH, Sabeti PC, Zody MC, Barrett JC, Dallaire S, Gabriel SB, Lee C, Daly MJ, et al: Common deletion polymorphisms in the human genome. Nat Genet. 2006, 38 (1): 86-92. 10.1038/ng1696.View ArticlePubMedGoogle Scholar
- Hambly RJ, Double JA, Thompson MJ, Bibby MC: Establishment and characterisation of new cell lines from human breast tumours initially established as tumour xenografts in NMRI nude mice. Breast Cancer Res Treat. 1997, 43 (3): 247-258. 10.1023/A:1005756632293.View ArticlePubMedGoogle Scholar
- Epithelial Cancer Cell Lines database. [http://www.path.cam.ac.uk/~pawefish/index.html]
- Maniatis T, Fritsch EF, Sambrook J: Molecular cloning: a laboratory manual. 1982, Cold Spring Harbor, N.Y.: Cold Spring Harbor LaboratoryGoogle Scholar
- Carvalho B, Bengtsson H, Speed TP, Irizarry RA: Exploration, normalization, and genotype calls of high-density oligonucleotide SNP array data. Biostatistics. 2007, 8 (2): 485-499. 10.1093/biostatistics/kxl042.View ArticlePubMedGoogle Scholar
- Ritchie ME, Silver J, Oshlack A, Holmes M, Diyagama D, Holloway A, Smyth GK: A comparison of background correction methods for two-colour microarrays. Bioinformatics. 2007, 23 (20): 2700-2707. 10.1093/bioinformatics/btm412.View ArticlePubMedGoogle Scholar
- Smyth GK: Linear models and empirical bayes methods for assessing differential expression in microarray experiments. Stat Appl Genet Mol Biol. 2004, 3: Article 3-Google Scholar
- R Development Core Team: R: A Language and Environment for Statistical Computing. 2008, R Foundation for Statistical Computing, Vienna, AustriaGoogle Scholar
- Edgar R, Domrachev M, Lash AE: Gene Expression Omnibus: NCBI gene expression and hybridization array data repository. Nucleic Acids Res. 2002, 30 (1): 207-210. 10.1093/nar/30.1.207.PubMed CentralView ArticlePubMedGoogle Scholar
- Durinck S, Bullard J, Spellman PT, Dudoit S: GenomeGraphs: integrated genomic data visualization with R. BMC Bioinformatics. 2009, 10: 2-10.1186/1471-2105-10-2.PubMed CentralView ArticlePubMedGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.