In this section, we introduce the generative mixture model of MixClone, which is an extension of our previous work on tumor purity estimation[11]. First, we introduce the notations for input data. Then, we describe the probabilistic models for sequence information of both SCNAs and allele frequencies. Finally, we combine these two types of data into a single likelihood model, and describe an algorithm to solve the model.

### Basic notations

The raw input data for MixClone are two aligned whole genome sequencing read sets of paired normal-tumor samples and a genome segmentation file based on the tumor sample. Following the notations from our previous work [11], we assume the tumor genome has been partitioned into *J* segments. We also assume there are *I*_{
j
} heterozygous SNP sites within segment *j* in the corresponding normal genome, and use (*i*, *j*) to index SNP site *i* within segment *j*. For each SNP site (*i*, *j*) we define the A allele to be the reference allele and the B to be the alternative allele, with respect to the reference genome. We also use a superscript N to denote data from normal samples and superscript T to denote data from tumor samples. Overall, the observed data are summarized in the following notations [13]:

{b}_{i\phantom{\rule{2.77695pt}{0ex}}j}^{N} = number of reads mapped to the B allele in the normal sample at site (*i*, *j*).

{d}_{i\phantom{\rule{2.77695pt}{0ex}}j}^{N} = reads depth of the normal sample at site (*i*, *j*).

{D}_{j}^{N} = total number of reads mapped to segment *j* of the normal sample.

The notations for the observed data from tumor samples are similarly defined, e.g. {D}_{j}^{T} denotes total number of reads mapped to segment *j* of the tumor sample.

### Modeling SCNAs

Next, we describe the probabilistic model for SCNAs data. For each segment *j*, we define an allelic configuration *H*_{
j
} to represent its underlying allele-specific copy number status. For example, if the absolute copy number of segment *j* is 2, then the compatible allelic configurations are PP, MM and PM, where P and M denotes the paternal and maternal allele of the tumor genome, respectively. Since PP and MM are not distinguishable based on sequence information alone as the reference human genome is not phased, we define the set of all possible allelic configuration as

{H}_{j}\in \mathcal{H}=\left\{\mathrm{0\u0338},\text{P/M},\text{PP/MM},\text{PM},\text{PPP/MMM},\text{PPM/PMM}\right\}

(1)

assuming the maximum copy number for each segment is 3. The corresponding copy number associated with each allelic configuration in \phantom{\rule{0.5em}{0ex}}\mathcal{H} is then

{n}_{h}=\left\{0,1,2,2,3,3\right\}

(2)

MixClone allows the user to specify the maximum copy number and the default value is 6 in the released package [11]. We further assume there are *K* subclonal populations within the tumor sample, each of which has an associated cellular prevalence *ϕ*_{
k
} ∈ 0[1]. The subclonal type of each segment *j* is denoted as

{Z}_{j}\in \mathcal{Z}=\left\{1,2,\cdot \phantom{\rule{2.77695pt}{0ex}}\cdot \phantom{\rule{2.77695pt}{0ex}}\cdot \phantom{\rule{2.77695pt}{0ex}},K\right\}

(3)

representing one of the *K* possible subclonal populations. Given the allelic configuration *H*_{
j
} = *h* and the subclonal type *Z*_{
j
} = *k*, the average copy number of segment *j* within the tumor sample, taking into account the subclonal cellular prevalence *ϕ*_{
k
}, is

{\stackrel{\u0304}{C}}_{j}={\varphi}_{k}{n}_{h}+\left(1-{\varphi}_{k}\right)2

(4)

Based on the Lander-Waterman model [14], the probability of sampling a read from a given segment *j* depends on three main factors: 1) its copy number, 2) its total genomic length, and 3) its mappability, which depends on factors such as repetitive sequence and GC content [9]. For each segment *j*, we associate a coefficient *θ*_{
j
} to account for the effect of its mappability and genomic length. Thus the expected read counts mapped to segment *j*, which is denoted as *λ*_{
j
}, is proportional to {\stackrel{\u0304}{C}}_{j}{\theta}_{j}. For example, for segment *x* and segment *y*, we have

\frac{{\lambda}_{x}}{{\lambda}_{y}}=\frac{{\stackrel{\u0304}{C}}_{x}{\theta}_{x}}{{\stackrel{\u0304}{C}}_{y}{\theta}_{y}}

(5)

Because the mappability coefficients (*θ*_{
j
}'s) matter only in a relative sense, we take {\theta}_{x}\text{/}{\theta}_{y}={D}_{x}^{N}\text{/}{D}_{y}^{N}, as these segments should have the same sequence properties between the normal and tumor samples.

Additionally, to determine the absolute value of *λ*_{
j
}, we curate a list of segments which contain no loss of heterozygosity according to their allele frequencies information. Based on the observed number of reads mapped to each segment, we further remove "outlier" segments from the list if their copy numbers are different from the bulk of the segments' copy numbers in the list. Finally, we call the remaining segments in the list as "baseline segments" and denote the set of these segments as *S*. We assume the allelic configurations of all the baseline segments are PM with copy number *n*_{
s
} = 2. Other possible allelic configurations for baseline segments, which have equal copy numbers for each allele (e.g. *φ*, PPMM), are likely to be rare, and currently we do not model them. Then based on *n*_{
s
}, we specify *λ*_{
j
} as follows

{\lambda}_{j}=\frac{1}{\left|S\right|}\phantom{\rule{2.77695pt}{0ex}}{\displaystyle \sum _{s\phantom{\rule{2.77695pt}{0ex}}\in \phantom{\rule{2.77695pt}{0ex}}S}}\frac{{\stackrel{\u0304}{C}}_{j}{\theta}_{j}}{{n}_{s}{\theta}_{s}}\phantom{\rule{2.77695pt}{0ex}}{D}_{s}^{T}

(6)

where {D}_{s}^{T} denotes the number of reads mapped to segment *s* of the tumor sample.

Finally, we model the number of reads mapped to segment *j* in the tumor sample as a Poisson distribution, given *H*_{
j
} and *Z*_{
j
}

{D}_{j}^{T}|{H}_{j},\phantom{\rule{2.77695pt}{0ex}}{Z}_{j}~\text{Poisson}\phantom{\rule{2.77695pt}{0ex}}\left({\lambda}_{j}\right)

(7)

Details on curating the baseline segments are given in Supplementary, Additional file 1.

### Modeling allele frequencies

Next, we describe the probabilistic model used for allele frequencies of heterozygous SNP data. For each SNP site *i* within segment *j*, we denote its tumor genotype as *G*_{
ij
}, which is selected from the set of all possible tumor genotypes up to a maximum copy number alteration, e.g.

\mathcal{G}=\left\{0\u0338,\text{A},\text{B},\text{AA},\text{AB},\text{BB},\text{AAA},\text{AAB},\text{ABB},\text{BBB}\right\}

(8)

assuming the maximum copy number is 3. The corresponding B allele frequencies (BAF) for all the genotypes in\phantom{\rule{0.5em}{0ex}}\mathcal{G} are

{\mu}_{g}=\left\{\frac{1}{2},\u03f5,1-\u03f5,\u03f5,\frac{1}{2},1-\u03f5,\u03f5,\frac{1}{3},\frac{2}{3},1-\u03f5\right\}

(9)

in which, *ε* ≪ 1 is a small random deviation accounting for general sequencing errors. We choose *E* = 0.01, which is equivalent to a Phred quality of 20 [15].

Given the tumor genotype *G*_{
ij
} = *g*, the allelic configuration *H*_{
j
} = *h*, and the subclonal type *Z*_{
j
} = *k*, the average BAF of site (*i*, *j*) within the tumor sample, taking into account the subclonal cellular prevalence *φ*_{
k
}, is

{\stackrel{\u0304}{\mu}}_{ij}=\frac{{\varphi}_{k}{n}_{h}{\mu}_{g}+\left(1-{\varphi}_{k}\right)2{\mu}_{0}}{{\varphi}_{k}{n}_{h}+\left(1-{\varphi}_{k}\right)2}

(10)

in which *µ*_{0} = 0.5 is the BAF of heterozygous SNP sites in the normal sample. Finally, we model the distribution of the B allele count {b}_{ij}^{T} at site (*i*, *j*) as a binomial distribution, given *G*_{
ij
} , *H*_{
j
} and *Z*_{
j
}

{b}_{ij}^{T}|{d}_{ij}^{T},{G}_{ij},{H}_{j},{Z}_{j}~\text{Binomial}\left({d}_{ij}^{T},{\stackrel{\u0304}{\mu}}_{ij}\right)

(11)

### Combining SCNAs and allele frequencies

Now, we combine sequence information from both SCNAs and heterozygous SNP sites. For all the heterozygous SNP sites within the same segment, their genotypes should be consistent with the underlying allelic configuration of the segment. We model this consistency through a predefined conditional probability {Q}_{gh}=\mathbb{P}\left({G}_{ij}=g|{H}_{j}=h\right). If the genotype *g* is inconsistent with the allelic configuration *h*, e.g. AA is inconsistent with PM, we assign a small probability *σ* as *Q*_{
gh
}, otherwise we assign equal probabilities to genotypes that are consistent with the allelic configuration.

Conditional on the underlying allelic configuration *H*_{
j
} and subclonal type *Z*_{
j
}, the probability of observing B allele read count {b}_{ij}^{T} at site (*i*, *j*) is given as

\mathbb{P}\left({b}_{ij}^{T}|{H}_{j}=h,{Z}_{j}=k\right)={\displaystyle \sum _{g\in \mathcal{G}}}{Q}_{gh}\mathbb{P}\left({b}_{ij}^{T}|{G}_{ij}=g,{H}_{j}=h,{Z}_{j}=k\right)

(12)

We assume that conditional on the allelic configuration *H*_{
j
} , the B allele read counts {\left\{{b}_{ij}^{T}\right\}}_{i=1}^{{I}_{j}} at different sites within the same segment *j* are independent of each other, and are also independent of the total read count {D}_{j}^{T} of the segment. Then, the joint probability of observing the two types of read counts information of segment *j* is

\begin{array}{c}\phantom{\rule{1.2em}{0ex}}\mathbb{P}\left({D}_{j}^{T},{\left\{{b}_{ij}^{T}\right\}}_{i=1}^{{I}_{j}}|{H}_{j}=h,{Z}_{j}=k\right)\hfill \\ =\mathbb{P}\left({D}_{j}^{T}|{H}_{j}=h,{Z}_{j}=k\right)\times {\displaystyle \prod _{i=1}^{{I}_{j}}}{\displaystyle \sum _{g\in \mathcal{G}}}Qgh\mathbb{P}\left({b}_{ij}^{T}|{G}_{ij}=g,{H}_{j}=h,{Z}_{j}=k\right)\hfill \end{array}

(13)

### Likelihood model

We have specified the joint distribution of the two types of read counts information of segment *j*. We then further model the allelic configuration *H*_{
j
} and the subclonal type *Z*_{
j
} of segment *j* as random variables that follow categorical distributions

{H}_{j}|{\rho}_{j}~\text{Categorical}\left({\rho}_{j}\right)

(14)

{Z}_{j}|\pi ~\text{Categorical}\left(\pi \right)

(15)

*ρ*_{
j
}= (*ρ*_{j∅}, ⋯, *ρ*_{jPPM/PMM}), where {\rho}_{jh}=\mathbb{P}\left({H}_{j}=h\right) is the probability of observing *h* as the allelic configuration of segment *j*. *π* = (*π*_{1}, ⋯, *π*_{
K
}), where {\pi}_{k}=\mathbb{P}\left({Z}_{j}=k\right) is the probability of observing subclonal type *k* for all the segments. The model parameters Θ is defined as

\mathrm{\Theta}=\left({\left\{{\rho}_{j}\right\}}_{j=1}^{J},{\left\{{\pi}_{k}\right\}}_{k=1}^{K},{\left\{{\varphi}_{k}\right\}}_{k=1}^{K}\right)

(16)

And the model likelihood of observing all the data is then

\begin{array}{c}\phantom{\rule{1.2em}{0ex}}\mathbb{P}\left({\left\{{D}_{j}^{T}\right\}}_{j=1}^{J},{\left\{{b}_{ij}^{T}\right\}}_{i=1,j=1}^{{I}_{j,}J}|\mathrm{\Theta}\right)\hfill \\ ={\displaystyle \prod _{j=1}^{J}}{\displaystyle \sum _{k=1}^{K}}{\displaystyle \sum _{h\in \mathcal{H}}}\mathbb{P}\left({Z}_{j}=k\right)\mathbb{P}\left({H}_{j}=h\right)\mathbb{P}\left({D}_{j}^{T}|{H}_{j}=h,{Z}_{j}=k\right)\hfill \\ \times {\displaystyle \prod _{i=1}^{Ij}}{\displaystyle \sum _{g\in \mathcal{G}}}{Q}_{gh}\mathbb{P}\left({b}_{ij}^{T}|{G}_{ij}=g,{H}_{j}=h,{Z}_{j}=k\right)\hfill \\ ={\displaystyle \prod _{j=1}^{J}}{\displaystyle \sum _{k=1}^{K}}{\displaystyle \sum _{h\in \mathcal{H}}}{\pi}_{k}{\rho}_{jh}\frac{{\lambda}_{j}^{{D}_{j}^{T}}{e}^{-{\lambda}_{j}}}{{D}_{j}^{T}!}\hfill \\ \times {\displaystyle \prod _{i=1}^{{I}_{j}}}{\displaystyle \sum _{g\in \mathcal{G}}}{Q}_{gh}\left(\begin{array}{c}\hfill {d}_{ij}^{T}\hfill \\ \hfill {b}_{ij}^{T}\hfill \end{array}\right){\stackrel{\u0304}{\mu}}_{ij}^{{b}_{ij}^{T}}\left(1-{\stackrel{\u0304}{\mu}}_{ij}\right){d}_{ij}^{T}-{b}_{ij}^{T}\hfill \end{array}

(17)

We use Expectation-Maximization (EM) algorithm [16] to find the maximum likelihood estimation of Θ. The complete details of the EM updates are given in Supplementary, Additional file 1.

### Model selection

One of the key issues in subclonal analysis is to determine the number of subclonal populations *K*. PyClone and PhyloSub use posterior sampling methods to estimate *K* [5, 7], while THetA requires users to specify *K* as an input [9]. Since the probabilistic model of MixClone is a generative mixture model, the model complexity and the corresponding log-likelihood increases as *K* increases. Therefore, we use a criterion based on the increase of the log-likelihood to select *K*. Practically, Mix-Clone allows the user to specify *K*. If *K* is not specified, MixClone runs the mixture model five times with different *K* in range of 1 to 5. We denote the log-likelihoods under the five different settings as {\left\{{L}_{K}\right\}}_{K=1}^{5}, and the total log-likelihood increase as

\mathrm{\Delta}={L}_{5}-{L}_{1}

(18)

If |Δ*/L*_{1}| < 0.01, which means the ratio of total log-likelihood increase is less than 0.01, MixClone predicts there is no subclonal event in the tumor sample and selects *K* = 1 as the number of subclonal populations. If |Δ */L*_{1}| ≥ 0.01, MixClone further calculates another quantity

{\delta}_{i}=|{L}_{i}-{L}_{1}|\mathrm{\Delta},i\in \left[2,5\right]

(19)

which is the cumulative log-likelihood increase from *K* = 1 to *K* = *i* as a percentage regarding to the total increase Δ. If *δ*_{
i
} ≥ 0.9 and *δ*_{i−1 }< 0.9, MixClone selects *K* = *i* as the number of subclonal populations.

In practice, we suggest users use this criterion as a heuristic guide when analyzing real data, and determine the number of subclonal populations in conjunction with regard to other external information.

### MixClone software package

Figure 1 is the general workflow of MixClone. MixClone is a comprehensive software package, including subclonal cellular prevalences estimation, allelic configuration estimation, absolute copy number estimation and a few visualization tools. This package is implemented in Python and is built on top of the PyLOH package, previously released by us [11]. It also utilizes some features from the software package JointSNVMix [13], which have been explicitly indicated in the source code.