Combining modularity, conservation, and interactions of proteins significantly increases precision and coverage of protein function prediction

Filtering coherent CCS

We first test all detected CCS for functional coherence using a functional similarity measure proposed by Couto et al. [61] that is based on semantic similarity. We compute, for each CCS, its average functional similarity within a species (Sim _neigh - similarity between neighbors) and across the species (Sim _ortho - similarity between orthologs). The formal definitions of both similarity measures are provided in the Additional File 1 (see Eq. S7 and S8 in Section S1.1).

We further only consider CCS which have (a) more than two proteins and (b) whose similarity score, either Sim _ortho or Sim _neigh , exceeds a given threshold. We applied three different thresholds (low: 0.3, medium: 0.5, high: 0.7) to study the performance of our method for different levels of functional coherence. This scheme is applied separately for each subontology of GO (molecular function (MF), biological process (BP), cellular component (CC)).

Prediction using orthology relationships

For inferring protein function from orthology relationships within a CCS, we determine orthologous groups that differ significantly in their individual functional similarity from the similarity score of the CCS by computing the standardized z-score (see Eq. S9). In groups with significant differences (p-value < 0.01) we transfer all known protein annotations to poorly annotated or uncharacterized orthologs. Note that an orthologous protein group might consist of more than one protein per species (orthologs and in-paralogs). Although all proteins within such a group in theory should be functionally highly similar, this is, probably due to missing or wrong annotations, not always reflected in the data (see Results). We define the consensus annotation of all proteins of one species in an orthologous group to be the set of all GO terms that are associated to more than half of the annotated proteins of that species in that group. When considering more than two species we combine the species-specific sets of consensus annotations and transfer them to the other proteins in the same group.

Prediction using neighboring proteins

Given a protein in a CCS, we decide for each GO term annotated to any of its direct neighbors whether it also should be annotated to the protein itself. Let G be the set of terms annotated to at least one neighbor of a protein p, and let N _g be the set of neighbors of p annotated with a term g ∈ G. We transfer g to p if there are more than f proteins in N _g whose functional similarity to p is higher than a given threshold t. For functional similarity between proteins, we again use the method from Couto et al. [61] (see Additional File 1, Eq. S5 in Section S1.1.2).

Because this approach cannot predict functions for proteins without any annotation (their computed similarity to other proteins is always zero), we also consider the pairwise functional relation between interaction partners, assuming that a high functional similarity between indirectly linked partners should also hold for the protein itself. Again, if the pairwise similarity scores exceed the threshold t we predict common GO annotations to the candidate protein.

Combined prediction method

We combine the two different methods to predict protein functions within a CCS (see Figure 1c). Proteins that are only weakly and incompletely characterized or not annotated at all are candidates for our prediction approach. For each candidate protein we infer novel protein function (a) within functionally coherent CCS by exploiting its (b) orthology relationship across other species as well as (c) the information shared by its neighboring proteins.

Processing large CCS

Comparing evolutionary close species (such as human and mouse) often results in very large CCS with up to several hundreds of proteins. However, biological processes typically involve only between 5 and 25 proteins [21]. Consequently, large CCS often encompass various functions (see Figure 3) which is reflected in a minor functional homogeneity. Our results confirm this fact, as large CCS always get low coherence scores (see Results). To adequately treat such CCS, we split CCS with more than 25 proteins into smaller, overlapping sub-subgraphs. Sub-subgraphs are built by considering each protein of the CCS as seed of a new, smaller CCS. Subsequently, we add all direct neighbors of this seed to the new CCS (see Additional File 1, Figure S1 for an example). Subgraphs with less than three proteins are removed. We then consider each of these subgraphs as an independent CCS.

Comparison to other methods

We compare our approach against a number of different techniques.

First, we use two baseline methods: The orthology baseline purely considers orthology ignoring structural network conservation. We randomly select one third of the orthologous protein groups, remove annotations from one protein in the group and predict their functions using only its orthologs. The link-based baseline takes only direct interaction partners into account, again independently of conservation of interactions. For each species we randomly choose one third of the proteins from the corresponding interaction network and exploit their direct neighbors for deriving new functions. We repeat this procedure 100 times for each baseline and compute average and standard deviation across all runs.

We also compare our results with three popular PPI-based function prediction methods. The Neighbor Counting Approach from Schwikowski et al. is a local prediction approach that derives new annotations for a protein based on the frequency of annotations within its direct interaction partners [19]. The χ² algorithm from Hishigaki et al. extended this idea by also considering the background frequency of a functional term [16]. Finally, the Functional Similarity Weighted Averaging method from Chua et al., a weighted averaging method to predict the function of a protein based on its direct and indirect interaction partner [27, 62]. Chua et al. demonstrate in [27] that the FS-Weighted Averaging significantly outperforms local and global network approaches, e.g. methods that are based on markov random field or functional flow [26, 29]. For comparisons, we adapted a script provided by Chua et al. that implements these three methods (see Additional File 1, Section S1.4 for details). To enable a valid direct comparison, we evaluate the three related predictions methods only on proteins that are involved in CCS. The individual performance of each method on the entire data set is shown for completeness in the Additional File 1.

Across Orthology Relationships within CCS

We use orthology relationships underpinned by interologs to infer novel functions from multiple species. Considering only orthology relationships for transferring functions to proteins within CCS results in predictions with medium to high precision. Additional File 1, Table S6 shows precision and recall estimated using cross-validation for the selected examples. Precision reaches 88% to 97% for yeast proteins when comparing hsa-dme-sce and 67% to 85% for mouse proteins when comparing mmu-hsa-dme-sce. Precision values increase considerably with a higher coherence threshold for CCS, but this improvement comes at the cost of lower coverage. Particularly low numbers of predictions are obtained for comparisons involving species with low PPI coverage. This is especially prominent for rno, where comparison of rno-hsa-sce result in only 8 predictions - but with a precision of 100%.

Besides the coherence threshold, also the number of species being compared has a strong impact on performance. Higher average precisions are achieved when analyzing multiple species compared to species pairs. For instance, the average precision for mmu-hsa-dme-sce is 79% at 0.3 in comparison to dme-sce with 54% at 0.3 and 69.5% at 0.7. This shows that using more species implicitly selects functions that are conserved more strongly, which underlines the impact of evolutionary functional conservation for protein function prediction. This fact also shows up when comparing to the orthology baseline (see Additional File 1, Table S4): Precision and per-protein recall using orthology within CCS are much higher, but the overall coverage is much lower. This means that CCS strongly restrict the number of proteins for which predictions are made, but this restriction is done in a very sensible way removing mostly false positive predictions.

Combining module, orthology and link-based PPI evidences

As mentioned before, one of the major drawbacks of using only CCS orthology relationships is the low number of predictions due to the restriction to orthologous proteins with at least one known function (see Additional File 1, Table S6). In contrast to orthology-only, the combined approach creates many more predictions (2- to 50-times more). It generates hundreds or even thousands of predictions also for those cases where the orthology-only method could not predict any function.

Comparing the combined method and CCS link-based only (see Additional File 1, Table S7) shows an increase within the amount of predictions (e.g. about 2-times for dme from dme-sce), although it is less steep than observed for orthology-only. This increase has mostly only minor influence on precision and recall. Precision reaches similar levels and the recall increases slightly. Note, for few combinations the combined method yields the same results as link-based-only because no predictions could be inferred through orthology relationships.

Overall, the impact of our combined approach is dominant, especially in terms of the number of predictions. Precision drops for some combinations compared to the single methods. However, the decrease of precision does not indicate a lower prediction quality. It rather indicates that the combined method derives many more novel predictions that can not be validated during cross-validation rather than successfully reproducing known function for well-characterized proteins (see Discussion of predictions). Precision is affected the least for the highest similarity threshold (0.7) fostering the most reliable precisions.

Overlap between orthology- and link-based predictions within CCS

We combined orthology- and link-based function prediction within CCS to benefit from the strengths of both methods. To study whether the predictions of the individual methods result in the same or complementary sets of predictions we determined the overlap of GO terms predicted by either strategy. For hsa-dme-sce, the respective numbers are shown as Venn diagrams in Additional File 1, Figure S4. In general, the major fraction of unique predictions is derived from neighboring proteins. The overlap between predictions is comparably small and decreases when increasing the similarity threshold. This shows that both methods complement each other very well as they predict rather different sets of functions. For hsa-dme-sce, the respective numbers are shown as Venn diagrams in Additional File 1, Figure S4. In general, the major fraction of unique predictions is derived from neighboring proteins. The overlap between predictions is comparably small and decreases when increasing the similarity threshold. This shows that both methods complement each other very well as they predict rather different sets of functions.

This behavior is also observable when predictions are analyzed separately per species (see Additional File 1, Figure S5). However, contrary to fly and yeast proteins (see Additional File 1 Figure S5(b) and S5(c)), the amount of orthology and link-based predictions is quite similar for human proteins (see Additional File 1, Figure S5(a)), which can be explained by the much denser PPI data available for the two model organisms (see Table 1). This observation clarifies that different species pro t differently from our method. Especially less characterized species, such as human, benefit strongly from the functional knowledge of model organisms.

Overlap between predictions derived from different species combinations

Large CCS

Large CCS naturally encompass various biological functions. In consequence, their functional homogeneity is often too low which excludes the entire CCS from function prediction. However, large CCS actually are strong indicators for conserved functions. For instance, Figure 3 shows the largest CCS from hsa-dme-cel-sce consisting of 61 proteins and 108 interologs with its different biological subprocesses. It clearly contains several functionally highly conserved clusters, probably forming discrete protein complexes. Considering such a large CCS as a whole is insufficient. Therefore, we modify our approach for large CCS by breaking them up into sub-subgraphs (see Methods). The impact on precision and recall is shown in Additional File 1, Table S8 (large CCS are split), which should be compared with entries of Table 3 (large CCS are ignored). As can be seen, processing large CCS creates many more predictions with mostly better precision. For example, the number of predictions almost triples for hsa-dme-sce at a similar or even better precision. When comparing split and non-split results from hsa-dme-cel-sce the precision decreases for human along a five-fold increase of the number of predictions, but increases for all the other species (at 0.7).

Comparison to Baselines

Compared to the orthology baseline, including CCS yields precisions up to 10-times higher, confirming that information on conserved interactions is a very effective filter for avoiding false positive predictions across orthology relationships. Compared to the neighbor baseline, considering CCS also leads to a clear and significant increase in precision. This effect can be explained by the fact that using interologs (strict or relaxed) instead of single interactions largely improves reliability of PPI data [66], since false positive PPIs are unlikely to be reproduced across multiple species.

Our results show that combining various evidences into a single and comprehensive method leads to improved results. Evidently, the predictions made by different methods using information from different species complement each other quite well instead of only predicting the same functions again and again. However, the concrete approach has to be chosen with care. We showed that good results can only be achieved when using a proper definition of interaction conservation and when treating large CCS in an adequate manner. Failing to do so either restricts coverage of the method or leads to a higher false positive rates.

Comparison to other Function Prediction Methods

We compared precision and recall of our approach to Neighbor Counting, χ² statistics and FS-Weighted Averaging (see Methods). Our combined CCS-based approach significantly outperforms Neighbor Counting and χ² statistics, especially in terms of precision. Moreover, we perform comparably well or better against FS-Weighted Averaging, mostly achieving much higher precision at higher recall. Notably, our method achieves better results especially for species without comprehensive PPI coverage, such as human. However, precision for Neighbor Counting and χ² is significantly lower (on the entire data sets, see Additional File 1, Figure S6, and the filtered protein sets) than reported in the respective original publications [16, 19]. There are three explanations for this drop (from ~70% to 15% precision). First, both methods originally were evaluated only on the functional classification scheme from YPD. This scheme covers, similar to GO, three categories of yeast protein function: biochemical function, cellular role and subcellular localization. However, categories have only 57, 41 and 22 members, respectively. Compared to our evaluation using GO, in which methods have to chose between up-to 17398 functional categories, this increases the chances to predict correct terms purely by chance. Furthermore, yeast is a particularly well-studied organism, while we applied the method also to less-well covered species. A similar performance drop was observed by Chua et al. [62], which also applied both methods to GO term prediction, with precision decreasing to 60% (NC) and 20% (χ²) for yeast and 20% (NC) and 16% (χ²) for fly. The second point concerns the amount of interaction data. For example, results from [19] are based on only 2,709 interactions among 2,039 proteins. In contrast, we integrated six different databases, leading to, for instance, almost 70,000 interactions for 6,500 proteins in yeast. Thus, we cover many more proteins and interactions which also increases the probability of false positives. Third, many prediction methods, including the two studies compared to here, consider only annotated proteins with at least one annotated interaction partner for their studies [16, 19, 26, 27, 62]. We did not exclude those proteins because we believe that especially weakly or un-annotated proteins must be a primary target for function prediction. In our combined approach, such proteins often receive functions from orthologous proteins in other species, an option missing in Neighbor Counting and χ². However, functions predicted for non-annotated proteins are necessarily counted as false positives although these are truly novel findings. Thus, disregarding such proteins results in higher precisions.

We did not compare to purely module-based prediction methods, as link-based techniques have been shown to outperform those [3, 37]. However, we evaluated the effect of requiring CCS to be "module-like", i.e., to exhibit a certain density of interactions between its members. CCS-density is defined as $\frac{2*|E|}{|V|\left(|V|-1\right)}$ where E presents the edges and V denotes the nodes within a CCS. As expected, filtering CCS according to their density considerably improves precision (see Additional File 1, Figure S9), e.g. in fly from 80% without filtering to 90% for a density of 0.7 and 95% for a density of 1, but this increase is at the cost of much fewer predictions (see Additional File 1, Section S2.2 for a detailed discussion).

Effects of Size of Data Sets

Results of our prediction method vary depending on the level of available annotations and PPIs for the species that are compared (see Additional File 1, Section S3.1 for a discussion of the data). They are better when well-studied species, such as yeast or fly, are involved. This is an inherent property of methods that transfer annotations, since better annotated species provide more source functions. This property underpins the importance of comparative genomics for elucidating the function of human proteins. It is also clearly visible that prediction precision is correlated to the threshold for functional conservation (see Figure 5a) and increases with the degree of evolutionary conservation of a CCS - from pairwise to multiple network comparisons (Figure 5b). Obviously, the functional conservation threshold is an important possibility to tune or method to the specific needs of an application. The higher the functional conservation, the higher is the precision of the predictions.

Note that in any gold standard evaluation as ours, new findings are always counted as false positives, independently of their real, biological truthfulness. Consequently, prediction methods perform better on well-studied organisms than on species that are functionally less well characterized. The precision values we report therefore should be considered as lower bounds on the true precision.

Performance on Weakly and Non-Annotated Proteins

An important goal of protein function prediction is to derive novel functions for proteins without any or with only very little functional information. Thus, we analyzed how our method performs on such proteins. We define as a weakly annotated protein (WAP) any protein which has at most two terms assigned a-priori in our data. For WAP, we count annotations as new if they are more specific than the existing ones or if they belong to another sub-branch in the subontology. Note that such annotations are counted as false positives in our evaluation as they cannot be validated from our gold standard data.

Results from comparing hsa-dme-sce are shown in Figure 6 and Additional File 1, Figure S10. As expected, the highest number of proteins without any annotation can be found in human. Annotation coverage of fly is not as good as for yeast but still much better than in human. For example, CCS at threshold 0.3 contain ~300 human proteins without any functional annotation in biological process. By means of our method, we predict 156 annotation for 52 of those proteins. Similarly, 20 fly proteins out of 72 are annotated with 67 GO annotation in biological process. But also well-studied species still contain many WAPs and benefit from our approach. For instance, about 380 yeast proteins are only weakly characterized for cellular component and for more than a quarter of them we predict about 200 functions. Note that the fraction of WAPs receiving new annotations decreases with the increase of the similarity threshold for each species.

MLH1 and PMS2

The DNA mismatch repair protein MLH1 and the mismatch repair endonuclease PMS2 belong to the main components of the post-replicative DNA mismatch repair (MMR) system (see Figure 7) [69]. The MMR system is required for correcting base mismatches and insertion or deletion loops resulting from DNA replication, DNA damage, or from recombination events between non-identical sequences during meiosis [70]. Curated annotation for MLH1 and PMS2 from UniProt and EntrezGene and newly inferred functions are listed in Additional File 1, Table S10 and S11.

The majority of our predictions (terms are set italics in the following) is directly related to the functionality of the MutLα complex which is formed by MLH1 and PMS2. Rich supporting evidence can be found from the respective orthologs in yeast and mouse. For instance, PMS1, the PMS2 ortholog in yeast, contributes to dinucleotide insertion or deletion binding, loop DNA binding[71]. Mlh1, the mouse ortholog of MLH1, is annotated to guanine/thymine mispair binding[72] and likely plays a role in the formation, stabilization and/or the resolution of Holliday junction intermediates (four-way junction DNA binding) [73]. High and low affinity ATP binding sites have been observed for MLH1 and PMS1 in yeast [74] which supports the ATP binding and ATPase activity predictions for their human orthologs [75]. Moreover, PMS2 contains a conserved metal-binding motif that constitutes part of the active site for the endonuclease activity of the protein and might enable magnesium ion binding[76]. Considering protein homodimerization activity, the dysregulated gene expression of PMS2, either as a monomer or homodimer, can disrupt MMR function in mammalian cells [77]. Note that, although we support our predictions by literature evidences that are mostly based on orthologs, our algorithm actually inferred them from conserved interaction partners as the orthologs in most cases do not carry the annotation we found in the literature.

Our algorithm also generates a number of predictions that are not as clearly supported by the existing literature, such as guanine/thymine mispair binding, single guanine and thymine insertion binding or oxidized DNA binding. Moreover, we associate both proteins to base-excision repair as well as postreplication repair and MLH1 to maintenance of DNA repeat elements. These are interesting hypotheses supported by recent findings from Erdeniz et al. who suggested that the endonuclease activity of PMS2 in MutLα is not only important in MMR-dependent mutation avoidance but also for suppression of homologous recombination, DNA damage signaling, and damage response functions [78]. Association of yeast PMS1 with meiotic mismatch repair and DNA recombination[79] further support these predictions. Regarding their cellular components both proteins are associated to the MutLα complex[67], an annotation predicted jointly from orthology and the CCS neighborhood. MutLα complex is a clearly sensible refinement of the existing annotation nucleus and only seven others genes are annotated to this term, which emphasizes the specificity of our method.

EPHB4

Ephrin type-B receptor 4 is a transmembrane receptor for the ephrin-B family. It belongs to the family of receptor tyrosine kinase (RTK) and is usually expressed in endothelial and neuronal cells. Known and predicted functional annotations are displayed in Additional File 1, Table S12.

Several predicted functions, such as protein, enzyme and ATP binding, SH3/SH2 adaptor and enzyme regulator activity and protein amino acid phosphorylation, derived both from conserved interactions and orthology, are evidently consistent with the characteristics of receptor tyrosine kinases.

Two functions inferred by orthology are transmembrane-ephrin receptor activity and transmembrane receptor protein tyrosine kinase signaling pathways. Both are supported by annotations from highly related receptors, such as Ephb1 in mouse and EPHB2 in human [80, 81]. Less evident predictions are, for instance, cell-cell signaling[82], cell migration[83], angiogenesis and behavior[84]. These functions were not predicted by orthology alone but only in combination with the conserved interaction neighborhood of EPHB4. EPHB4 participates in the axon guidance pathway and in this context predictions like axon guidance or axon guidance receptor activity can be integrated [85–87].