Combinatorial drug optimization problem
Suppose we have N different types of drugs, where each drug can take one of pre-specified concentrations. Our goal is to predict the optimal drug cocktail, by mixing the available drugs, that maximizes the overall therapeutic effect. Let x
n
be the concentration of the n-th drug, where x
n
can take one of M
n
distinct concentrations in the set , where for all k. The drug cocktail is represented by an N-dimensional vector x = (x1, x2, . . . , x
N
), which consists of the N drug concentrations. Let f(x) be the normalized drug response that quantitatively measures the effectiveness of a given drug combination x. We assume the response has been normalized such that , where is the set of all possible drug combinations. We denote the number of all possible drug combinations as implies that the drug cocktail x is completely ineffective, and larger f (x) implies higher efficacy. In practical applications, f (x) may be obtained by monitoring the cell response to a drug intervention using fluorescent imaging, microarrays, or sequencing techniques. Under this setting, we aim to find the optimal drug combination x* that maximizes the normalized drug response:
As we can see, this is a combinatorial optimization problem, in which we have to find the optimal drug combination out of M1 M2 . . . M
N
possible combinations. The total number of distinct drug combinations quickly grows as the number of drugs increases. Considering the practical cost of experimentally measuring the normalized drug response function f (x), it is apparent that we cannot test all drug combinations to find the most effective one.
Stochastic search algorithms
Stochastic search algorithms [9, 11] aim to efficiently identify the potent drug combinations without exploring the entire combinatorial solution space. The basic idea is to randomly search through the solution space by iteratively updating the drug combination until an effective combination emerges. At each step, the current drug combination is incrementally updated towards the direction that is likely to improve the overall drug response. The updating decision is made in a stochastic manner, which allows the search to proceed towards directions that are deemed to be less likely to improve the response. This is an important characteristic of stochastic search algorithms, which is critical for keeping the search from being trapped in local maxima. Since a stochastic search algorithm tries to arrive at the optimal solution (i.e., the most effective drug combination) by performing iterative local searches, its overall performance depends on how it chooses the next solution state (i.e., an updated drug combination in , the set of all possible combinations) from a given state (i.e., the current drug combination). The two performance metrics of interest are: (i) the effectiveness of the predicted drug combination, in terms of how close its response is to the optimal response, and (ii) the number of search steps that the algorithm needs to take until an effective combination is found. Basically, we want to predict a potent drug cocktail by testing minimal number of drug combinations to minimize the actual experimental cost for measuring the cell response to combinatorial drugs. When choosing the next state, the search algorithm has to be as parsimonious as possible, in terms of the number of function evaluations, so that the overall experimental cost for identifying the optimal drug combination can be minimized. This has been one of the main design considerations of existing stochastic search algorithms that have been developed for combinatorial drug optimization [9, 11].
For example, the Gur Game algorithm adopted in [9] determines how to update the drug combination solely based on the current drug response. Suppose is the current drug combination with a normalized drug response of f (xc). The algorithm generates N random numbers r
n
∈ [0,1], one for each drug. Each r
n
is compared against the current drug response , which is used to either "reward" or "penalize" the n-th drug. For example, the drug is rewarded if f (xc) >r
n
. Otherwise, it is penalized. This update process is repeated for each of the N drugs. According to this scheme, drug combinations with higher response are more likely to be rewarded, while combinations with lower response are more likely to be penalized. In the long run, the algorithm is expected to drive the concentration of each drug towards the one that maximizes the response. Now, an important relevant question is what is the right way of rewarding (or penalizing) the current concentration of a given drug. The Gur Game algorithm used in [9] uses a predetermined finite state automaton (FSA) for this purpose. For example, let be the current concentration of the n-th drug. Suppose we have f (xc) >r
n
, hence the current concentration of the n-th drug should be rewarded. The FSA in [9] is designed such that the drug concentration is increased further if the current concentration is larger than a reference concentration , and decreased further if is smaller than . More specifically, if and f(xc) >r
n
, then the drug concentration is updated as follows.
(1)
The reference concentration is typically chosen as the median of the set . As shown in (1), the drug concentration remains unchanged if it cannot be increased (or decreased) further. Now, suppose that f (xc) <r
n
, and therefore the current concentration should be penalized. In this case, the concentration is updated in the opposite direction:
(2)
Note that penalization moves the current drug concentration closer to the reference concentration . As previously discussed in [11], one of the weaknesses of the Gur Game algorithm is that it uses a predetermined FSA for updating (i.e., rewarding/penalizing) the drug concentrations and does not adapt to the drug response function at hand, which is not known in advance. As a result, the algorithm may perform poorly unless the drug response function f(x) is properly normalized and the reference concentration is chosen adequately for each drug. For example, consider the one-dimensional drug response f(x) shown in Figure 1A. As we can see, the drug response has been over-normalized, hence f(x) < 0.5 for any allowed concentration x ∈ [cmin, cmax]. Since f(x) < 0.5, a uniformly distributed random number r ∈ [0,1] is more likely to be larger than f(x). This implies that the Gur Game algorithm always tends to penalize the current drug concentration (no matter what its value is), which will probabilistically drive the concentration towards cref although it is clearly not optimal. Figure 1B shows another drug response, for which the Gur Game algorithm will not work properly. In this example, we have f(x) > 0.5, hence the Gur Game algorithm is always more likely to reward the current drug concentration, which tends to drive the concentration away from the reference concentration cref. This will push the concentration either towards cmin or cmax, both of which are suboptimal.
The enhanced stochastic algorithm proposed in [11] addresses this problem by making the search algorithm adapt to a given drug response. The basic idea of this algorithm is to make an "informed-guess" about how to beneficially update a given drug concentration, instead of following a predetermined update rule. Unlike the Gur Game algorithm in [9], where all N drugs are simultaneously updated based on the (same) current drug response f(xc), the enhanced algorithm updates the concentration of one drug at a time. As an example, suppose during the last update of the n-th drug, the drug combination has been updated as
where x and are identical except for the concentration of the n-th drug, hence and . We assume that x
n
and differ only by a single concentration level, so that and, or and , for some k. The algorithm compares the two drug responses f (x) and , thereby determine wether it would be more beneficial to further increase or decrease the concentration of the n-th drug. For example, we may have the following four cases.
(3)
The above rules allow the algorithm to adaptively determine how to reward (or penalize) a given drug concentration based on the observed drug response values. However, the decision whether to reward or penalize the current drug is made in a probabilistic manner. For this purpose, we evaluate the following function
(4)
where α ∈ [0, 1] is a control parameter that adjusts the randomness of the algorithm [11]. This g(x, x') is compared with a uniformly distributed random number r
n
∈ [0, 1]. If g(x, x') > r
n
, the n-th drug is rewarded, i.e., updated in such a way that appears to be more beneficial for enhancing the drug response according to the rules shown in (3). Otherwise, the n-th drug is penalized, i.e., updated in a way that appears to be less beneficial based on the past observations. It is not difficult to see that this algorithm is always more likely to reward, or beneficially update, a given drug. Since the algorithm proposed in [11] adaptively determines how to update the drug concentration based on previous observations, it can also effectively deal with drug response functions shown in Figures 1A and 1B, for which the Gur Game algorithm does not perform well. Despite its merits, this algorithm also has its own shortcomings. For example, as the update rule for a given drug is determined only based on the two observations that correspond to its latest update, not on a longer-range trend, the algorithm may be sensitive to small variations in the drug response. As a result, it may not show satisfactory search performance for drug response functions that are similar to the one in Figure 1C. Furthermore, considering that f(x) has to be experimentally estimated, where a certain level of measurement noise and small variations due to a number of practical factors may not be avoidable, such sensitivity may adversely affect the overall performance of the algorithm. Another weakness of the algorithm is that it only utilizes a very small part of the past observations without fully utilizing them. In the following section, we introduce a novel stochastic search algorithm that can effectively address the aforementioned issues.
The adaptive reference update (ARU) stochastic search algorithm
In order to make the search algorithm robust to small variations in the observed drug response, the update rules have to be decided based on the general trend of the drug response change over a wide range of drug concentration, not just based on the response change resulting from a single-level concentration change. Based on this motivation, we propose a novel algorithm called the adaptive reference update (ARU) algorithm. In this algorithm, we compare the current drug response f(xc) with the response f(xref) of a reference drug combination xref, which is adaptively updated based on past observations. In the beginning, xref is set to the initial drug concentration, where we start the search process. During the search, when the algorithm encounters a local maximum, the current reference combination is replaced by the corresponding drug combination. As an example, let us consider the one-dimensional drug response function f(x) in Figure 2. For illustration, we consider the following hypothetical search process, where the drug concentration is constantly updated from left to right, starting from the lowest concentration cmin to the highest concentration cmax. Suppose the search begins at the concentration x = cmin. Initially the reference concentration is also set to this initial drug concentration xref ← cmin. As the search reaches the first local maximum, the reference concentration is updated to this local maximum solution xref ← xref1. As the search continues to the right, this reference concentration is used until we reach the next local maximum. After passing the second local maximum solution xref2, the reference point is updated to xref ← xref2. In a similar manner, as the search continues further to the right, the reference point is updated to xref ← xref3 after passing the third local maximum point.
At each iteration, the current drug response f(xc) is compared to the response f(xref) of the reference drug combination, based on which the drug update rule is determined. For example, let and , and assume that we want to update the concentration of the n-th drug by comparing the two drug response values f(xc) and f(xref). As before, we have the following four possible cases.
(5)
Conceptually, we can view the above as estimating the "virtual" slope between two points and as follows
(6)
based on which we determine how to update the concentration x
n
of the n-th drug to increase the drug response f(). Given the update rules in (5), the actual update decision is made by evaluating the following function
(7)
and comparing it with a random number r
n
∈ [0, 1]. If g(xc, xref) >r
n
, the drug concentration x
n
is updated by a single level, following the beneficial update direction predicted by (5). Otherwise, the concentration is updated in the opposite direction. As briefly mentioned before, the parameter α ∈ [0, 1] controls the randomness of the algorithm. For example, α = 0 will make the search process completely random, regardless of the observed drug responses. Using a larger α means that we are giving a larger weight to the past observations when deciding how to update the drug concentrations. The value of this control parameter is limited to α ≤ 1 such that g(xc, xref) ≤ 1. Also note that we always have g(xc, xref) ≥ 0.5, which implies that at any drug update step, the update is always more likely to take place in accordance with the rules in (5), which have been derived based on past observations of the drug response. In other words, the ARU algorithm tries to effectively utilize the past response data to beneficially update the drug concentrations, and ultimately, to identify a potent drug combination, while keeping the search still stochastic. For illustration, let us again consider the drug response function in Figure 2, where the hypothetical search process proceeds from the lowest drug concentration to the highest concentration. The black solid arrows below the graph shows the drug update direction that gets higher probability according to the new algorithm, described above. For example, in region-A (cmin < × < xref1), the algorithm tends to increase the drug concentration x further, as the response f(x) is larger than f(cmin) of the initial reference concentration (i.e., cmin). As x continues to increase and passes the first local maximum point xref1, the reference is updated to xref ← xref1. In region-B (xref1 < × < xref2), the search algorithm tends to drive the concentration towards xref1 by decreasing the concentration. Suppose the search continues to increase the drug concentration x beyond xref2, the second local maximum point, despite the tendency of the algorithm to decrease x back to xref1. After passing xref2, the reference is updated to xref ← xref2. In region-C, the search algorithms assigns higher probability to the update rule that tries to bring the concentration down to xref2, since f(x) < f(xref2) in the given region. However, once x enters region-D, where f(x) > f(xref2), the algorithm begins to drive the drug concentration x further to the right until it passes the third local maximum point xref3. The reference concentration is updated to xref ← xref3, once the search continues to the right and the drug concentration x gets larger than xref3. Since f(xref3) is larger than f(x) in region-D (xref3 < × < cmax), the search algorithm will tend to bring the concentration down to the current reference concentration, namely, xref = xref3.
Choosing a local maximum solution as a reference combination has a number of practical advantages. First of all, it allows the algorithm to adjust the drug update rules based on a long-range trend of the given drug response function, which makes the algorithm robust to small variations in the observed response. Another advantage of using a long-range trend is that the search process will become also less sensitive to random fluctuations that may exist in the observed drug response. Considering that the drug response function f(x) has to be experimentally estimated through actual biological experiments, where random factors (e.g., measurement noise) that affect the estimation results cannot be completely ruled out, such robustness is critical for the algorithm to be used in practical drug optimization applications. It is also beneficial to use the drug combination that corresponds to the most recent local maximum response, instead of the drug combination that has yielded the highest response among all past combinations, as the reference point. This prevents the search process from dwelling too much on past observations, while keeping it robust to variations and random fluctuations.
Drug response functions
In order to evaluate the overall performance of the ARU algorithm, we used the algorithm to search for the optimal drug cocktail for several different drug response functions.
Two-dimensional drug response functions For performance assessment, we first used the four two-dimensional drug response functions that are shown in Figure 3. The first drug response function f2a(x1, x2) shown in Figure 3A is the normalized HIV inhibitor response obtained from [17], where x1 ∈ {0, 0.01, 0.03, 0.09, 0.27, 0.82, 2.47, 7.41, 22.22, 66.67}(nM) was considered for Maraviroc and x2 ∈ {0, 0.09, 0.27, 0.8, 2.41, 7.22, 21.67, 65}(nM) for ROAb14. The second drug response f2b(x1, x2) shown in Figure 3B is the normalized second De Jong function (Rosenbrock's saddle) [18]. We considered x1, x2 ∈ {c0, c1, ... , c20}, where c
k
= 4(k/ 20 - 0.5), obtained by evenly dividing the range [-2, 2] into 21 distinct values. The third drug response function f2c(x1, x2) in Figure 3C is the normalized lung cancer inhibition response obtained from [1], where x1 ∈ {0, 1, 2, 4, 6, 8, 12, 16, 20, 22}(μM ) was considered for Chlorpromazine and x2 ∈ {0, 0.25, 0.4, 0.6, 0.8, 1, 1.5, 2, 4, 6.8}(μM ) for Pentamidine. Finally, Figure 3D shows the fourth response function f2d(x1, x2), the normalized bacterial (S. aureus) inhibition response reported in [6]. x1 ∈ {0, 0.08, 0.16, 0.32, 0.63, 1.25, 2.5, 5, 10} was considered for Trimethoprim and x2 ∈ {0, 0.31, 0.62, 1.25, 2.5, 5, 10, 20, 40} was considered for Sulfamethoxazole. All four drug response functions were normalized to span the entire range [0, 1], such that the minimum response is 0 and the maximum response is 1.
Multi-dimensional drug response functions To evaluate the performance for optimizing multi-drug cocktails, we defined several hypothetical drug response functions with up to six drugs. First, we defined two 3-dimensional drug functions f3a(x1, x2, x3) and f3b(x1, x2, x3). The first function is defined as
(8)
where each of x1, x2, x3 can take one of the 11 discrete concentrations that evenly divide the range [-2.5, 2.5]. The second function is defined as follows
(9)
using the Matlab peaks(x1, x2) function. We assume that each drug can take one of the 11 discrete values that evenly divide [-3, 3]. Next, we defined two 4-dimensional drug functions
(10)
and
(11)
We assume that x1 and x2 in the first function f4a(x1, x2, x3, x4) can take one of the 11 discrete values that evenly divide the range [-2, 2], while x3 and x4 can take one of the 11 discrete values that evenly divide the range [-3, 3]. For the second drug response function f4b(x1, x2, x3, x4), we assume that each drug can take one of the 11 distinct values that evenly divide [-3, 3]. In addition, we also defined the following 5-dimensional drug response functions
(12)
and
(13)
For the first function f5a(x1, x2, x3, x4, x5), x1 and x2 are allowed to take any value from the set of values obtained by evenly dividing the range [-2, 2] into 11 discrete concentrations. The remaining drug concentrations (x3, x4, and x5) can take any of the 11 concentrations that evenly divide [-4.5, 4.5]. For the second drug response function f5b(x1, x2, x3, x4, x5), we assume that each drug can take one of the 11 discrete values that evenly divide [-3, 3]. Finally, we also defined two 6-dimensional drug response functions
(14)
and
(15)
where every drug concentration can take its value from one of the 11 discrete concentrations that evenly divide the range [-2.5, 2.5].