Volume 9 Supplement 2
Gene selection algorithm by combining reliefF and mRMR
© Zhang et al; licensee BioMed Central Ltd. 2008
Published: 16 September 2008
Gene expression data usually contains a large number of genes, but a small number of samples. Feature selection for gene expression data aims at finding a set of genes that best discriminate biological samples of different types. In this paper, we present a two-stage selection algorithm by combining ReliefF and mRMR: In the first stage, ReliefF is applied to find a candidate gene set; In the second stage, mRMR method is applied to directly and explicitly reduce redundancy for selecting a compact yet effective gene subset from the candidate set.
We perform comprehensive experiments to compare the mRMR-ReliefF selection algorithm with ReliefF, mRMR and other feature selection methods using two classifiers as SVM and Naive Bayes, on seven different datasets. And we also provide all source codes and datasets for sharing with others.
The experimental results show that the mRMR-ReliefF gene selection algorithm is very effective.
Gene expression refers to the level of production of protein molecules defined by a gene. Monitoring of gene expression is one of the most fundamental approach in genetics and molecular biology. The standard technique for measuring gene expression is to measure the mRNA instead of proteins, because mRNA sequences hybridize with their complementary RNA or DNA sequences while this property lacks in proteins. The DNA arrays, pioneered in [1, 2], are novel technologies that are designed to measure gene expression of tens of thousands of genes in a single experiment. The ability of measuring gene expression for a very large number of genes, covering the entire genome for some small organisms, raises the issue of characterizing cells in terms of gene expression, that is, using gene expression to determine the fate and functions of the cells. The most fundamental of the characterization problem is that of identifying a set of genes and its expression patterns that either characterize a certain cell state or predict a certain cell state in the future .
When the expression dataset contains multiple classes, the problem of classifying samples according to their gene expression becomes much more challenging, especially when the number of classes exceeds five . Moreover, the special characteristics of expression data adds more challenge to the classification problem. Expression data usually contains a large number of genes (in thousands) and a small number of experiments (in dozens). In machine learning terminology, these datasets are usually of very high dimensions with undersized samples. In microarray data analysis, many gene selection methods have been proposed to reduce the data dimensionality .
Gene selection aims to find a set of genes that best discriminate biological samples of different types. The selected genes are "biomarkers", and they form "marker panel" for analysis. In general, two types of gene selection methods have been studied in the literature: filter methods  and wrapper methods . As pointed out in , the essential differences between the two methods are:
(1) that a wrapper method makes use of the algorithm that will be used to build the final classifier while a filter method does not, and
(2) that a wrapper method uses cross validation to compare the performance of the final classifier and searches for an optimal subset while a filter method uses simple statistics computed from the empirical distribution to select attribute subset.
Wrapper methods could perform better but would require much more computational costs than filter methods. Most gene selection schemes are based on binary discrimination using rank-based schemes , such as information gain, which reduces the entropy of the class variables given the selected attributes. In expression data, many gene groups interact closely and gene interactions are important biologically and may contribute to class distinctions. However, the majority of the rank-based schemes assume the conditional independence of the attributes given the target variable and are thus not effective for problems involving much feature interaction .
In this paper, we present a two-stage selection algorithm by combining ReliefF  and mRMR . ReliefF, a general and successful attribute estimator, is able to effectively provide quality estimates of attributes in problems with dependencies between attributes. mRMR (minimal-redundancy-maximal-relevance) method selects genes that have the highest relevance with the target class and are also maximally dissimilar to each other. mRMR is computationally expensive. The integration of ReliefF and mRMR thus leads to an effective gene selection scheme. In the first stage, ReliefF is applied to find a candidate gene set. This filters out many unimportant genes and reduces the computational load for mRMR. In the second stage, mRMR method is applied to directly and explicitly reduce redundancy and select a compact yet effective gene subset from the candidate set. We perform comprehensive experiments to compare the mRMR-ReliefF selection algorithm with ReliefF, mRMR and other feature selection methods using two classifiers on seven different datasets. The experimental results show that the mRMR-ReliefF gene selection is very effective.
Result and discussion
In this section, we perform comprehensive experiments to compare the mRMR-ReliefF selection algorithm with ReliefF, mRMR and other feature selection methods using two classifiers (Support Vector Machine (SVM) and Naive Bayes (NB)) on seven different datasets.
The dataset description.
• ALL: The ALL dataset  is a dataset that covers six subtypes of acute lymphoblastic leukemia: BCR (15), E2A (27), Hyperdip (64), MLL (20), T (43), and TEL (79). Here the numbers in the parentheses are the numbers of samples. The dataset is available at .
• ARR: The Arrhythmia (ARR) dataset contains 420 samples and 278 features with two classes .
•GCM: The GCM dataset  consists of 198 human tumor samples of fifteen types. breast (12), prostate (14), lung (12), colorectal (12), lymphoma (22), bladder (11), melanoma (10), uterus (10), leukemia (10), renal (11), pancreas (11), ovary (120), mesothelioma (11), CNS (20), and MET (9). The prediction accuracy of 78% is reported in  using one-versus-the rest SVM with all the genes.
•MLL: The MLL-leukemia dataset consists of three classes and can be downloaded at .
•NCI60: The NCI60 dataset was first studied in . cDNA microarrays were used to examine the variation in gene expression among the 60 cell lines from the National Center Institute's anticancer drug screen. The dataset spans nine classes and can be downloaded at [17, 22].
Note that in these datasets, the samples in each class is generally small, and unevenly distributed. This, together with the large number of classes, especially for NCI60, GCM, makes the classification task more complex.
Compare ReliefF, mRMR and mRMR-ReliefF algorithm
Table 2 presents the detail of the accuracy values of applying SVM and NB classification on the top 30 selected genes, for some unavailable results which can not be computed by mRMR, we note them as "-". From the above comparative study, we observe that:
• The performance of mRMR algorithm is pulled down by its expensive computational cost, and it can not fulfill gene selection on the database with large features using the limited memory.
• Relief algorithm is not stable enough when only a small number of genes are selected. And when the number of selected genes is greater than 30, the variations of classification performance of both ReliefF and mRMR-ReliefF algorithms are generally small.
• The mRMR-ReliefF selection algorithm leads to significantly improved class predictions. With the same number of selected genes, the gene set obtained by the mRMR-ReliefF selection is more representative of the target class, therefore leading to better class prediction or generalization property.
The comparisons in ReliefF, mRMR and mRMR-ReliefF algorithms (gene number = 30)
Feature Selection Method
Comparison with other methods
The comparisons in seven gene selection methods (gene number = 30).
Feature Selection Method
No feature sel
• Gene selection improves class prediction. Note that the accuracy of SVM using feature selection generally outperforms that without feature selection. This implies that feature selection can effectively reduce the insignificant dimensions and noise to improve classification accuracy.
• The mRMR-ReliefF algorithm is shown to achieve better performance comparing with other gene selection algorithms on almost all datasets. The experimental comparisons demonstrate the effectiveness of the integration of ReliefF and mRMR.
• ReliefF achieves good performance on most of the data sets. Although its performance is not always as good as that of the mRMR-ReliefF algorithm. It outperforms mRMR, Maxrel, Sum Minority and partially wins information gain, twoing rule.
• Only a small number of genes are needed for classification purpose. In our experiments, the variations of the classification accuracy are small when the number of selected genes is greater than 30.
We have developed a software package for the above experiments, which includes: 1) The source codes for four feature selection algorithms including ReliefF, F-statistic, GNSR, and Relief-mRMR; 2) A MATLAB interface for Rankgene1.1  which contains another eight feature selection measures; 3) A MATLAB interface for two well-known classification tools (e.g., LIBSVM and WEKA); 4) Programs for converting data formats; 5) The collection of all datasets used in the experiments. We hope it is a useful tool in gene expression analysis and feature selection.
This package and all datasets can be downloaded from http://www.cis.fiu.edu/~yzhan004/genesel.html. All codes are implemented and tested in Matlab 7.0 and can be integrated into the Toolbox by adding its path to MATLAB search path.
Data structure and translation
This package supports consistent data formats. Each gene dataset is formatted as a MATLAB data structure file(.mat), in which a class label vector corresponds to a gene array. For any algorithm, the input is a .mat file, and the output is an index vector for the selected genes. Furthermore, a utility is provided for converting the data from .csv file to .mat file. The command line is as follows.
We also provide the function to convert .mat file to .csv file as:
mattocsv(X, y, Filename)
where X, y are the matrix defined in .mat file and Filename is the .csv file as output file.
Implementation of gene selection algorithms
MATLAB Command List For Gene Selection.
W = reliefF(X, y, n, K, typed, Topn)
W = Ftest(X, y, Topn)
W = Gsnr(X, y, Topn)
W = rm(X, y, n, K, Topn)
W = rankgene(X, y, T, Topn)
Assistant tools for classification
where Filename is the name of the output .csv file, X is a gene array, y is a label vector, and Classifier is the parameter for classification method, such as Naive Bayes and J4.5 tree.
In this paper, we present an mRMR-ReliefF selection algorithm by combining ReliefF and mRMR. ReliefF is able to effectively provide quality estimates of attributes in problems with dependencies between attributes and mRMR method selects genes that have the highest relevance with the target class and are also maximally dissimilar to each other. The integration of ReliefF and mRMR thus leads to an effective gene selection scheme: In the first stage, ReliefF is applied to find a candidate gene set; In the second stage, mRMR is applied to select a compact yet effective gene subset from the candidate set.
Comprehensive experiments are conducted to compare the mRMR-ReliefF selection algorithm with ReliefF, mRMR and other feature selection methods using two classifiers on seven different datasets. The experimental results show that the mRMR-ReliefF gene selection is very effective. In addition, we also developed a software package to help other researches explore gene expression.
In this part, firstly, ReliefF and mRMR algorithms are discussed, then mRMR-ReliefF selection algorithm is presented, and finally, other six gene selection algorithms used to compare with our mRMR-ReliefF algorithm are introduced.
ReliefF is a simple yet efficient procedure to estimate the quality of attributes in problems with strong dependencies between attributes . In practice, ReliefF is usually applied in data pre-processing as a feature subset selection method.
where mutual information between each pair of genes is taken into consideration. The minimum-redundancy maximum-relevance (mRMR) feature selection framework combines both optimization criteria of Eqs.(3, 4).
As we mentioned before, ReliefF is a general and successful attribute estimator and is able to effectively provide quality estimates of attributes in problems with dependencies between attributes. However, ReliefF does not explicitly reduce the redundancy in selected genes. mRMR selects genes that have the highest relevance with the target class and are also maximally dissimilar to each other. However, mRMR is computationally expensive. For example, using the mRMR program provided in , we could not obtain results for several datasets with a large number of genes, e.g., ALL and GCM. The integration of ReliefF and mRMR thus leads to an effective gene selection scheme.
where C(g j , g i ) is the Pearson correlation coefficient.
Our mRMR-ReliefF algorithm works as follows: In the first stage, ReliefF is applied to find a candidate gene set. This filters out many unimportant genes and reduces the computational load for mRMR. In the second stage, mRMR method is applied to directly and explicitly reduce redundancy and select a compact yet effective gene subset from the candidate set.
In our experiments, ReliefF is first used to choose 150 genes as the candidate set. from the all gene data. mRMR algorithm is then applied to select the final subset.
Other gene selection algorithms
In this part, we introduce six other gene selection algorithms which are mentioned in the chapter of "Result and discussion", which are named Max-Relevance, Information Gain, Sum Minority, Twoing Rule, F-statistic , and GSNR . These methods have been reported in previous work. The first four methods have been used either in machine learning (information gain) or in statistical learning theory (twoing rule and sum minority), and all of them measure the effectiveness of a feature by evaluating the strength of class prediction when the prediction is made by splitting it into two regions, the high region and the low region, by considering all possible split points . More detailed descriptions on these methods can be found in .
where C is the number of classes, is the mean of gene i variables, n c is the number of samples in class c, is the mean of gene i in class c, and g jic is sample j in gene i value in class c.
We would like to thank Ms D. Wang for assisting with the experiments on several gene selection algorithms. We are also grateful to the anonymous reviewers for their helpful comments.
This article has been published as part of BMC Genomics Volume 9 Supplement 2, 2008: IEEE 7th International Conference on Bioinformatics and Bioengineering at Harvard Medical School. The full contents of the supplement are available online at http://www.biomedcentral.com/1471-2164/9?issue=S2
- Chee M, Yang R, Hubbell E, Berno A, Huang X, Stern D, Winkler J, Lockhart D, Morris M, Fodor S: Accessing genetic information with high density DNA arrays. Science. 1996, 274: 610-614. 10.1126/science.274.5287.610.PubMedView Article
- Fodor S, Read J, Pirrung M, Stryer L, Lu A, Solas D: Light-directed, spatially addressable parallel chemical synthesis. Science. 1991, 251: 767-783. 10.1126/science.1990438.PubMedView Article
- Li T, Zhang C, Ogihara M: A comparative study of feature selection and multiclass classification methods for tissue classification based on gene expression. Bioinformatics. 2004, 20: 2429-2437. 10.1093/bioinformatics/bth267.PubMedView Article
- Ooi C, Tan P: Genetic algorithms applied to multi-class prediction for the analysis of gene expression data. Bioinformatics. 2003, 19: 37-44. 10.1093/bioinformatics/19.1.37.PubMedView Article
- Su Y, Muralin TM, Pavlovic V, Kasif S: Rankgene: Identification of diagnostic genes based on expression data. Bioinformatics. 2003, 19: 1578-1579. 10.1093/bioinformatics/btg179.PubMedView Article
- Langley P: Selection of relevant features in machine learning. AAAI Fall Symposium on Relevance. 1994, 140-144.
- Kohavi P, John GH: Wrappers for feature subset selection. Artificial Intelligence. 1997, 97: 273-324. 10.1016/S0004-3702(97)00043-X.View Article
- Xing EP, Jordan MI, Karp RM: Feature selection for high-dimensional genomic microarray data. Proc 18th International Conf on Machine Learning. 2001, 601-608.
- Dudoit S, Fridlyand J, Speed TP: Comparison of discrimination methods for the classification of tumors using gene expression data. Journal of the American Statistical Association. 2002, 97: 77-87. 10.1198/016214502753479248.View Article
- Marko RS, Igor K: Theoretical and empirical analysis of relief and rreliefF. Machine Learning Journal. 2003, 53: 23-69. 10.1023/A:1025667309714.View Article
- Peng H, Long F, Ding C: Feature selection based on mutual information: Criteria of max-dependency, max-relevance, and min-redundancy. IEEE Trans Pattern Anal and Mach Intell. 2005, 27: 1226-1238. 10.1109/TPAMI.2005.159.View Article
- Yeoh EJ, Ross ME, Shurtleff SA, Williams WK, Patel D, Mahrouz R, Behm FG, Raimondi SC, Relling MV, Patel A, Cheng C, Campana D, Wilkins D, Zhou X, Li J, Liu H, Pui CH, Evans WE, Naeve C, Wong L, Downing JR: Classification, subtype discovery, and prediction of outcome in pediatric lymphoblastic leukemia by gene expression profiling. Cancer Cell. 2002, 1: 133-143. 10.1016/S1535-6108(02)00032-6.PubMedView Article
- ALL Gene Expression Profiles. [http://www.stjuderesearch.org/data/ALL1/]
- Arr Gene Expression Profiles. [http://www.ics.uci.edu/mlearn/MLSummary.html]
- Ramaswamy S, Tamayo P, Rifkin R, Mukherjee S, Yeang CH, Angelo M, Ladd C, Reich M, Latulippe E, Mesirov JP, Poggio T, Gerald W, Loda M, Lander ES, Golub TR: Multiclass cancer diagnosis using tumor gene expression signatures. Proceeding of the National Academy of Sciences. 2001, 98: 15149-15154. 10.1073/pnas.211566398.View Article
- Hedenfalk I, Duggan D, Yidong C, Radmacher M, Bittner M, Simon R, Meltzer P, Gusterson B, Esteller M, Kallioniemi OP, B BW, Borg A, Trent J: Gene-expression profiles in hereditary breast cancer. The New England Journal of Medcine. 2001, 344: 539-548. 10.1056/NEJM200102223440801.View Article
- HBC Gene Expression Profiles. [http://www.columbia.edu/~xy56/project.htm]
- LYM Gene Expression Profiles. [http://genome-www.stanford.edu/lymphoma]
- Alizadeh AA, Eisen MBRE, Ma C, Lossos IS, Osenwald AR, Boldrick HC, Sabet H, Tran T, Yu X, Powell JI, Yang L, Martu GE, Moore T, Hudson J, Lu L, Lewis DB, Tibshirani R, Sherlock G, Chan WC, Greiner TC, Weisenburger DD, Armitage GP, Warnke R, Levy R, Wilson W, Grever MR, Byrd JC, Botsten D, Brown PO, Staudt LM: Distinct types of diffuse large B-cell lymphoma identified by gene expression profiling. Nature. 2000, 403: 503-511. 10.1038/35000501.PubMedView Article
- MLL Gene Expression Profiles. [http://research.dfci.harvard.edu/korsmeyer/MLL.htm]
- Ross DT, Scherf U, Eisen MB, Perou CM, Rees C, Spellmand P, Iyer V, Jeffrey SS, Van M, Waltham M, Pergamenschikov M, Lee JC, Lashkari D, Shalon D, Myers TG, Weinstein JN, Botstein D, Brown MPO: Systematic variation in gene expression patterns in human cancer cell lines. Nature Genetics. 2000, 24: 227-235. 10.1038/73432.PubMedView Article
- NCI60 Cancer Microaray Project. [http://genome-www.stanford.edu/nci60/]
- Ding C, Peng H: Minimum redundancy feature selection from microarray gene expression data. International Conference on Computational Systems Bioinformatics. 2003, 523-528.
- Zheng G: Statistical analysis of biomedical data with emphasis on data integration. phD thesis. 2006, Florida International University
- LIBSVM Software. [http://www.csie.ntu.edu.tw/~cjlin/libsvm/]
- Weka Software. [http://www.cs.waikato.ac.nz/ml/weka/]
- Sun Y, Li J: Iterative RELIEF for feature weighting: algorithms, theories and applications. Proceedings of the 23rd International Conference on Machine Learning. 2006, 29: 1035-1051.
- Cover T: The best two independent measurements are not the two best. IEEE Trans Systems, and Cybernetics. 1974, 4: 116-117.View Article
This article is published under license to BioMed Central Ltd. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.