Tsetlin Machine for Solving Contextual Bandit Problems
Abstract
This paper introduces an interpretable contextual bandit algorithm using Tsetlin
Machines, which is an inherently interpretable machine learning method that solves complex pattern recognition tasks using propositional (Boolean) logic.
The proposed bandit learning algorithm relies on straightforward bit manipulation, thus simplifying computation and interpretation.
We then present a mechanism for performing Thompson sampling with Tsetlin Machine, given its non-parametric nature.
Our empirical analysis shows that Tsetlin Machine as a base contextual bandit learner outperforms other popular comparable base learners on eight out of nine datasets.
We further analyze the interpretability of our learner, investigating how arms are selected based on propositional expressions that model the context 1 .
Introduction
Contextual bandits play a fundamental role in many applications involving sequential decision making, ranging from personalized recommendations [19] to designing effective treatment allocation strategies in clinical trials [11, 7].
Algorithms for contextual bandits have additionally gained significant interest because of their theoretical elegance.
In brief, a decision-maker selects one of multiple bandit arms over a sequence of rounds, taking into account an observed context.
Each round, the arm chosen elicits feedback in the form of a reward signal associated with the success of selecting that arm (such as a user purchasing a recommended product).
The contextual bandit problem is particularly intriguing because the decision-maker must maximize the expected reward in as few rounds as possible, trading exploitation against exploration to identify the optimal arm.
In this paper we recast the Tsetlin Machine (TM) [14] as a contextual bandit algorithm and study the resulting scheme empirically.
TM is a recent machine learning approach to pattern recognition that employs a team of non-contextual bandit algorithms, in the form of Tsetlin automata [27], to learn patterns expressed in propositional logic.
TMs have been shown to obtain competitive accuracy, memory footprint, and learning speed on several benchmark datasets [3, 18].
They have been particularly successful in natural language processing, including explainable aspect-based sentiment analysis [28] and robust text classification with ANDed word negations [29].
Being based on finite state automata, they further support Markov chain-based convergence analysis [30].
Leveraging the non-linear pattern recognition capability of TMs, our proposed scheme thus addresses the contextual bandit problem using a team of non-contextual bandit algorithms.
Although existing algorithms for contextual bandits provide theoretical guarantees, they are either difficult to interpret or make assumptions that limit their usability in real-world settings.
Motivated by these limitations, our work contributes as follows:
• We investigate TM as a base learner for contextual bandit problems and empirically demonstrate its effectiveness compared to other popular comparable base learners.
• We propose how the Thompson sampling scheme can be leveraged by TMs through bootstrapping [12].
• We provide interpretability analysis that shows how algorithms using TMs select arms based on propositional expressions of context features.
To the best of our knowledge, this is the first reported work on learning arm selection strategies for contextual bandits expressed in propositional logic.
The paper is organized as follows.
Section 2 and 3 present the problem formulation and introduces the TM algorithm, respectively.
In Section 4, we show how TM can be used as a base learner for contextual bandits.
The numerical results and the interpretability analysis are then presented in Section 5 and 7, respectively.
We conclude the paper and provide directions for future work in Section 9.
A literature overview on contextual bandits and different algorithms for solving them can be found in Appendix A.
Problem Formulation
We consider an online stochastic contextual bandit setup where at time t a context-reward pair denoted by ( s t , r t ) is sampled independently from past data distribution D .
Here, s t ∈ S represents an M dimensional context vector and r t = (( r t (1) , . . .
, r t ( K )) ∈ { 0 , 1 } K is the reward vector for K possible actions.
The learner chooses an arm u t ∈ { 1 , . .
, K } after observing the context s t and receives a reward r t ( u t ) for the chosen arm.
The objective of the learner is to perform a sequence of actions in order to minimize the cumulative expected regret given by
[Equation]
Here, π ∗ = arg max
[Equation]
E ( s,r ) ∼D [ r ( π ( s ))] , where Π denotes the set of large (possibly infinite) policies and π : S (cid:55)→ { 1 , . . .
, K } .
For our analysis we consider maximizing the expected total reward which is equivalent to minimizing regret.
[Figure]
Tsetlin Machine
Structure.
A TM in its simplest form takes a feature vector x = [ x 1 , x 2 , . . .
, x o ] ∈ { 0 , 1 } o of o propositional values as input and assigns the vector a class ˆ y ∈ { 0 , 1 } .
To minimize classification error, the TM produces n self-contained patterns.
In brief, the input vector x provides the literal set
L = { l 1 , l 2 , . . .
, l 2 o } = { x 1 , x 2 , . . .
, x o , ¬ x 1 , ¬ x 2 , . . .
, ¬ x o } , consisting of the input features and their negations.
By selecting subsets L j ⊆ L of the literals, the TM can build arbitrarily complex patterns, ANDing the selected literals to form conjunctive clauses: [Equation]
[Figure]
include exclude include exclude
N N + 2 N
N N + 2 N
[Figure]
Above, j ∈ { 1 , 2 , . .
. , n } refers to a particular clause C j and k ∈ { 1 , 2 , . .
, 2 o } refers to a particular literal l k .
As an example, the clause C j ( x ) = x 1 ∧ ¬ x 2 consists of the literals L j = { x 1 , ¬ x 2 } and evaluates to 1 when x 1 = 1 and x 2 = 0 .
The TM assigns one Tsetlin Automata (TA) per literal l k per clause C j to build the clauses.
The TA assigned to literal l k of clause C j decides whether l k is Excluded or Included in C j .
Figure 1 depicts a two-action TA with 2 N states.
For states 1 to N , the TA performs action Exclude (Action 1), while for states N + 1 to 2 N it performs action Include (Action 2).
[-for states N + 1 to 
-]
As feedback to the action performed, the environment responds with either a Reward or a Penalty.
If the TA receives a Reward, it moves deeper into the side of the action.
If it receives a Penalty, it moves towards the middle and eventually switches action.
With n clauses and 2 o literals, we get n × 2 o TAs.
We organize the states of these in an n × 2 o matrix A = [ a jk ] ∈ { 1 , 2 , . .
. , 2 N } n × 2 o .
We will use the function g ( · ) to map the automaton state a jk to Action 0 ( Exclude ) for states 1 to N and to Action 1 ( Include ) for states N + 1 to 2 N : g ( a jk ) =
a jk > N .
We can connect the states a jk of the TAs assigned to clause C j with its composition as follows:
[Equation]
Here, l k is one of the literals and a jk is the state of its TA in clause C j .
The logical imply operator ⇒ implements the Exclude / Include action.
That is, the imply operator is always 1 if g ( a jk ) =
( Exclude ), while if g ( a jk ) = 1 ( Include ) the truth value is decided by the truth value of the literal.
The complexity of the algorithm is proportional (linear) to the number of features (input bits).
E.g., doubling the number of features doubles the number of computations[1].
Classification.
Classification is performed as a majority vote.
The odd-numbered half of the clauses vote for class[-ˆ y = 
-]
ˆ y = 1 and the even-numbered half vote for ˆ y = 0 :
[Equation]
As such, the odd-numbered clauses have positive polarity, while the even-numbered ones have negative polarity.
As an example, consider the input vector x
=
[0 , 1] in the lower part of Figure 2.
The figure depicts two clauses of positive polarity, C 1 ( x ) =
x 1 ∧ ¬ x 2 and C 3 ( x ) =
¬ x 1 ∧ ¬ x(the negative polarity clauses are not shown).
Both of the clauses evaluate to zero, leading to class prediction ˆ y = 1 .
I NPUT
C LAUSE
L ITERAL
I NCLUDE L ITERAL
P(R EWARD ) s − 1 s
NA
P(I NACTION ) 1 s
NA s − 1 s s − 1 s
P(P ENALTY )
NA 1 s 1 s
E XCLUDE L ITERAL
P(R EWARD )
1 s
s 1 s
P(I NACTION ) 1 s s − 1 s s −
[Table]
s s − 1 s
[-C LAUSE 
-]
P(P ENALTY )[-
-]
s
[-P(I NACTION ) 1.0 NA 1.0 1. P(P ENALTY ) 
-][Table]
s
[-E XCLUDE L ITERAL P(R EWARD ) 
 P(I NACTION ) 1.01.0 1.-][Table]
[Table] [Table]
Learning.
The upper part of Figure 2 illustrates learning.
A TM learns online, processing one training example ( x , y ) at a time.
Based on ( x , y ) , the TM rewards and penalizes its TAs, which amounts to incrementing and decrementing their states.
There are two kinds of feedback: Type
I Feedback produces frequent patterns and Type II Feedback increases the discrimination power of the patterns.
Type I feedback is given stochastically to clauses with positive polarity when y = 1 and to clauses with negative polarity when y = 0 .
Conversely, Type II Feedback is given stochastically to clauses with positive polarity when y = 0 and to clauses with negative polarity when y = [-
-]1 .
The probability of a clause being updated is based on the vote sum v : v = (cid:80) n − 1 j =1 , 3 ,... (cid:86)
2 ok =1 (cid:104) g
( a jk ) ⇒ l k (cid:105) − (cid:80) nj =2 , 4 ,... (cid:86) 2 ok
=1 (cid:104) g
( a jk ) ⇒ l k (cid:105) .
The voting error is calculated as:
[Equation]
Here, T is a user-configurable voting margin yielding an ensemble effect.
The probability of updating each clause is P (Feedback) =
ϵ 2 T .
After random sampling from P (Feedback) has decided which clauses to update, the following TA state updates can be formulated as matrix additions, subdividing
Type I Feedback into feedback Type Ia and Type Ib:
[Equation]
Here, A t = [ a jk ] ∈ { 1 , 2 , . . .
, 2 N } n × 2 o contains the states of the TAs at time step t and A ∗ t + contains the updated state for time step t + 1 (before clipping).
The matrices F Ia ∈ { 0 , 1 } n × 2 o and
F Ib ∈ { 0 , 1 } n × 2 o contains Type I Feedback.
A zero-element means no feedback and a one-element means feedback.
As shown in Table 1, two rules govern Type I feedback:
• Type Ia Feedback is given with probability s − 1 s whenever both clause and literal are 1 -valued.
It penalizes Exclude actions and rewards Include actions.
The purpose is to remember and refine the patterns manifested in the current input x .
This is achieved by increasing selected TA states.
The user-configurable parameter s controls pattern frequency, i.e., a higher s produces less frequent patterns.
• Type Ib Feedback is given with probability 1 s whenever either clause or literal is 0 -valued.
This feedback rewards Exclude actions and penalizes Include actions to coarsen patterns, combating overfitting.
Thus, the selected TA states are decreased.
The matrix F II ∈ { 0 , 1 } n × 2 o contains Type II Feedback to the TAs, given per Table 2.
• Type II Feedback penalizes Exclude actions to make the clauses more discriminative, combating false positives.
That is, if the literal is 0 -valued and the clause is 1 -valued, TA states below N + 1 are increased.
Eventually the clause becomes 0 -valued for that particular input, upon inclusion of the 0 -valued literal.
The final updating step for training example ( x , y ) is to clip the state values to make sure that they stay within value 1 and 2 N :
[Equation]
For example, both of the clauses in Figure 2 receives Type I Feedback over several training examples, making them resemble the input associated with y = 1 .
Weighted Tsetlin Machine:
The learning of weights is based on increasing the weight of clauses that receive Type Ia feedback (due to true positive output) and decreasing the weight of clauses that receive Type II feedback (due to false positive output).
The overall rationale is to determine which clauses are inaccurate and thus must team up to obtain high accuracy as a team (low weight clauses), and which clauses are sufficiently accurate to operate more independently (high weight clauses).
The weight updating procedure is summarized in Algorithm 1.
Here, w i is the weight of clause
C i at the n th training round (ignoring polarity to simplify notation).
The first step of a training round is to calculate the clause output as per Equation 2.
The weight of a clause is only updated if the clause output C i is 1 and the clause has been selected for feedback ( P i = 1).
Then the polarity of the clause and the class label y decide the type of feedback given.
That is, like a regular TM, positive polarity clauses receive Type Ia feedback if the clause output is a true positive and Type II feedback if the clause output is a false positive.
For clauses with negative polarity, the feedback types switch roles.
When clauses receive Type Ia or Type II feedback, their weights are updated accordingly.
We use the stochastic searching on the line (SSL) automaton to learn appropriate weights.
SSL is an optimization scheme for unknown stochastic environments pioneered by Oommen [22].
The goal is to find an unknown location λ ∗ within a search interval [0 , 1] .
In order to find λ ∗ , the only available information for the Learning Mechanism (LM) is the possibly faulty feedback from its attached environment E .
In SSL, the search space λ is discretized into N points, { 0 , 1 /N, 2 /N, ..., ( N − 1) /N, 1 } with N being the discretization resolution.
During the search, the LM has a location λ ∈ { 0 , 1 /N, 2 /N, ..., ( N − 1) /N, 1 } , and can freely move to the left or to the right from its current location.
The environment E provides two types of feedback: E = 1 is the environment suggestion to increase the value of λ by one step, and E = 0 is the environment suggestion to decrease the value of λ by one step.
The next location of λ , i.e. λ n +1 , can thus be expressed as follows:
[Equation]
Asymptotically, the learning mechanics is able to find a value arbitrarily close to λ ∗ when N → ∞ and n → ∞ .
In our case, the search space of clause weights is [0 , ∞ ] , so we use resolution N = 1 , with no upper bound for λ .
Accordingly, we operate with integer weights.
As described in Algorithm 1, if the clause output is a true positive, we simply increase the weight by 1 .
Conversely, if the clause output is a false positive, we decrease the weight by 1 .
By following the above procedure, the goal is to make low precision clauses team up by giving them low weights, so that they together can reach the summation target T .
By teaming up, precision increases due to the resulting ensemble effect.
Clauses with high precision, however, gets a higher weight, allowing them to operate more independently.
The above weighting scheme has several advantages.
First of all, increment and decrement operations on integers are computationally less costly than multiplication based updates of real-valued weights.
Additionally, a clause with an integer weight can be seen as multiple copies of the same clause, making it more interpretable than real-valued weighting, as studied in the next section.
Additionally, clauses can be turned completely off by setting their weights to 0 if they do not contribute positively to the classification task.
Contextual Bandits with Tsetlin Machines
We use TM as a contextual bandit learner that learns a mapping from context to actions incrementally with streaming data.
Since both inputs, patterns and outputs of TM are represented as bits [14], each M dimensional context s t is binarized with appropriate number of bits.
This results in a B dimensional binarized context with B ≥ M which is fed to each TM learner.
We outline appropriate choices of bits for binarization for different datasets in the next section.
The TM learner, being non-parametric, has the advantage that it makes few or no assumption about the underlying functions to be learned; hence it is adaptive and exhibit high degree of flexibility.
Another popular nonparametric learner for contextual bandits is the decision tree, which has been thoroughly studied in the literature [12, 26, 13].
The TM, in contrast, learns a linear combination of conjunctive clauses do p 5: if
6: if c i = 1 then 7: w i ← w i 8: for k =
2 o do 9: then 10:
Type Ia Feedback 11: else : 12:
Type Ib Feedback 13: end if 14: end for 15: else : 16: w i ← w i
▷ [No Change] 17:
Type Ib Feedback 18: end if 19: else : ( y =
and i is even) or ( y =
and i is odd) 20: if c i = 1 then 21: if w i > 0 then 22: w i ← w i −23: end if 24: for feature k = 1 , ..., 2 o do 25: if l k =
then 26:
Type II Feedback 27: else : 28:
Inaction 29: end if 30: end for 31: else : 32: w i ← w i
▷
[No Change] 33:
Inaction 34: end if 35: end if 36: end for in propositional logic by producing decision rules similar to the branches in decision trees [1], with the added advantage of being memory and energy efficient, computationally simple and not having a tendency to overfit the training data [18].
We consider two contextual bandit learning algorithms using TM: (i) TM with epsilon greedy arm selection; and (ii) TM with Thompson sampling.
Epsilon greedy TM:
The exploration-exploitation trade-off is a fundamental problem in learning to make decisions under uncertainty.
In a multi-armed bandit setting, the ε − greedy algorithm is one of the simplest ones.
The learner either chooses the empirically best arm with probability (1 − ε ) (exploitation) or a random arm with probability ε (exploration).
In the contextual bandit setting, given a set of contexts, the learner chooses to select the current empirically best arm with some high probability, maximizing immediate rewards.
Otherwise, it selects a randomized arm with the hope of improving future rewards given the context.
Variations of epsilon greedy algorithms have been explored in the literature, including decaying the ε parameter and eventually dropping the probability of choosing a random arm to zero.
In our setting, we consider a fixed small value of ε for each TM learner associated with the arm.
TM with Thompson sampling: We now show how Thompson sampling can be achieved with TM.
Our approach is similar to that presented in the Tree Bootstrap Algorithm in [12].
Since a TM learner is non-parametric, we use bootstrapping to simulate the behavior of sampling from a posterior distribution.
At each time instant t , N context reward pairs are bootstrapped from the D t,u with replacement, where D t,u represents the set of observations (context reward pairs) for arm u and
N = | D t,u | .
A TM learner is fitted to each of these bootstrapped datasets and at each time t , the bootstrapping algorithm selects the arm u t that has the maximum probability of success ˆ p .
For ease of exposition, we outline Thompson Sampling with bootstrapping for TMs in Algorithm 2.
Algorithm 2 Thompson t =
Empirical Analysis
Given the interactive aspect of contextual bandit algorithms, it is often difficult to evaluate them on real datasets except for a handful number of tasks [6].
Therefore, we use standard supervised learning classification datasets for our evaluation.
We consider both binary as well as multiclass classification datasets.
Each unique label in these datasets is considered as an arm of the equivalent contextual bandit problem, where a separate learner associated with each arm is trained independently.
Our numerical analysis contrasts the performance of TM with ε − greedy arm selection and TM with
Thompson sampling against Tree Bootstrap [12], Linear UCB [8], Neutral Network ( ε − greedy) and
Logistic regression ( ε − greedy).
For our analysis we consider two scenarios described as follows.
Scenario 1:
In this scenario, we consider five standard supervised learning classification datasets from
UCI machine learning repository [10]: Iris, Breast Cancer Wisconsin (Diagnostic), Adult, Covertype , and Statlog (Shuttle) .
The response variables for each of these datasets are, Iris types, Diagnosis,
Occupation , and Covertypes , respectively, where as for the Statlog(Shuttle) dataset, we consider the last column as the response variable.
Additionally we considered two other classification datasets:
MNIST [9] and Noisy XOR , where the respective response variables are digits and the XOR output .
The noisy XOR dataset contains the XOR operation of 12 bit input, where the output is flipped with probability 0 .
These classification datasets are converted into a contextual bandit problem where the learner receives a reward of +1 for correctly identifying the target value and 0 for incorrect classification.
The datasets are processed by removing entries with missing values.
Scenario 2:
In this scenario, we consider two datasets:
Movielens 100 K [15] and Simulated
Article [23].
Both these datasets simulate a recommender system.
The reward function for the
Movielens dataset is the user’s rating for a particular movie while for Simulated Article dataset, a reward of +1 is received if the recommended article is clicked by the user.
The Movielens dataset is preprocessed, so that top 10 rated movies are selected.
One important takeaway from these two datasets is that only the reward function for correct recommendation is provided.
For instance the
Simulated Article dataset provides recommended articles that has been clicked by the user.
Similarly, the Movielens dataset provides user movie ratings, where the users only rated the movies they have watched.
In such circumstances, the rewards are partial and sparse since there are no information on the ratings of all movies or user click information for each article.
To circumvent this, we perform singular value decomposition (SVD) for both the datasets which is a popular approach in collaborative filtering [17].
For the Movielens dataset, let S i,j be the rating of user i on movie j .
Low rank SVD of this matrix results in two matrices W and X , where S ≈ W ∗ X .
The i t h row of W represents the context features for each user i and the rows of X represents the actions or the movie to be recommended.
The reward for recommending a movie j to user i is then the dot product of the corresponding row of W i and X j .
We perform SVD for Simulated Article dataset in a similar manner.
In order to obtain binarized rewards for the actions, the maximum reward corresponding to an arm that yields a reward of 1 , and the rest a reward of 0 .
For our analysis we consider rank 10 for both the
Movielens and the Simulated Article dataset.
For both the scenarios we use appropriate binarization before fitting the TM learners.
Details of binarization and parameter choices are provided in
Appendix B.
The results for both the scenarios are presented in Figures 3.
These results show that for eight datasets, TM with Thompson sampling outperforms all other algorithms except for Linear UCB on Covertype.
Further, TM with ε − greedy [Figure] provides competitive performance when compared with popular contextual bandit algorithms in the literature.
Also, the TM learner learns at a faster rate compared to other learners.
These experiments were performed with 10 independent runs for each dataset.
The average result across the independent runs are reported.
Choice of Binarization Parameters
The working principle of TM relies on binarization of the context features.
We use gap-based thresholding as described in [2], each threshold yielding one bit.
The binarization is performed with the parameter which controls the maximum number of bits allowed to represent each feature.
We varied the number of bits allowed per feature for binarization and performed an ablation study across these datasets, in order to observe the effects of sub-optimal binarization.
Our analysis shows that
TM as a base learner is robust to the choice of maximum bits per feature allowable for binarization.
Detailed evaluation results of TM with epsilon greedy and TM with Thompson sampling are provided in Appendix C.
Interpretability
Unlike neural networks and some other complex machine learning approaches, one of the advantages of TMs is that they produce propositional logic expressions.
These are in flat AND-form, which have proven to be human interpretable [21].
As explained in Section 3, each propositional expression is a conjunctive clause, consisting of features, in their original or negated forms, interacting with each other using logical AND operations.
These clauses can form a simplified representation of the arm selection policy by combining them into a single Disjunctive Normal Form (DNF) expression.
Since clauses are assigned to each arm of the multi-armed contextual bandit problem, we produce a single
DNF expression for each arm.
These DNF expressions are propositional logic expressions made up of the context.
The TM is able to produce these interpretations demonstrating how it interprets the context with respect to each arm.
Here, we show the simplified propositional expressions for each arm, obtained from TMs trained with Thompson Sampling on the Iris dataset:
Arm-1: x 10 ∨ x 14 ∨ x 15 ∨ xArm-2: ¬ x 1 ∨ x 12 ∨ ¬ x 13 ∨ ¬ x 14 ∨ x 16 ∨ x 8 ∨ ¬ x 9 ∨ ( x 10 ∧ x 11 ∧ x 15 ∧ x 2 ∧ ¬ x 4 ∧ x 5 ∧ x 6 ∧ ¬ x 7 ) ∨ ( x 10 ∧ x 11 ∧ x 15 ∧ x 3 ∧ ¬ x 4 ∧ x 5 ∧ ¬ x 7 ) Arm-3: ¬ x 10 ∨ ¬ x 11 ∨ ¬ x 15 ∨ ( x 1 ∧ ¬ x 12 ∧ x 13 ∧ x 14 ∧ ¬ x 16 ∧ x 2 ∧ x 3 ∧ x 5 ∧ ¬ x 8 ∧ x 9 ) ∨ ( ¬ x 12 ∧ ¬ x 16 ∧ x 4 )
The next set of propositional expressions represent the arms for the Simulated Article dataset:
Arm-1: ¬ x 4 ∨ ( x 1 ∧ ¬ x 10 ∧ ¬ x 11 ∧ x 12 ∧ x 13 ∧ x 14 ∧ x 15 ∧ x 16 ∧ x 17 ∧ x 18 ∧ x 19 ∧ x 2 ∧ x 20 ∧ ¬ x 21 ∧ x 3 ∧ x 5 ∧ x 6 ∧ x 7 ∧ x 8 ∧ x 9 ) Arm-2: ( x 1 ∧ ¬ x 10 ∧ ¬ x 11 ∧ x 12 ∧ x 13 ∧ x 14 ∧ x 15 ∧ x 16 ∧ x 17 ∧ x 18 ∧ x 19 ∧ x 2 ∧ x 20 ∧ ¬ x 21 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 6 ∧ x 7 ∧ x 8 ∧ x 9 ) ∨ ( ¬ x 1 ∧ x 10 ∧ x 11 ∧ x 12 ∧ x 13 ∧ x 14 ∧ x 15 ∧ x 16 ∧ x 17 ∧ x 18 ∧ x 19 ∧ x 2 ∧ ¬ x 20 ∧ ¬ x 21 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 6 ∧ ¬ x 7 ∧ x 8 ∧ x 9 )
Arm-3: ( x 1 ∧ ¬ x 10 ∧ ¬
[Table]
11 ∧
[Table]
12 ∧ x 13 ∧ x 14 ∧ x
[Table]
∧ x 16 ∧ x
[Table]
x
[Table]
∧
[Table]
∧
[Table]
∧
[Table]
∧ [Table] ∧
[Table] [Table]
[Table]
[Table]
[Table]
[Table]
-4: x
[Table]
x
[Table]
x
[Table]
x
[Table]
x
[Table]
[-Arm-4: x 10 ∧ x 11 ∧ x 12 ∧ x 13 ∧ x 14 ∧ x 15 ∧ x 16 ∧ x 17 ∧ x 18 ∧ x 19 ∧ x 2 ∧ x 3 ∧ x 4 ∧ x 5 ∧ x 6 ∧x 8 ∧ x 9 ∧ ¬ x 1 ∧ ¬ x 20 ∧ ¬ x 21 ∧ ¬ x 
-]
The propositional expressions shown above exhibit how the TM learns interactions between propositional input features (literals x i ).
As seen, the interactions are learned with logical AND operations between original and negated features.
These learnt interactions (clauses) are combined with the logical OR operation (or by addition for T > 1 ).
From the propositional expressions, we can see which arm is selected by TM by simply plugging in the input values.
An arm is selected if its propositional expression evaluates to True (or it obtains the largest net sum for T > 1 ).
This is arguably an important advantage of TMs — a trained TM model can be reduced down to simple propositional expressions.
Of course, as such, any machine learning algorithm can be converted into logical form.
However, the complexity of such representations can be immense for the competitive classic machine learning models, which the TM still outperforms by a large margin [24].
Note that, although the length of expressions can increase with the increase in dimensionality of data, there are simple ways of reducing the length of expressions in O ( n ) time [20].
Limitations
One of the limitations of using TM is that it requires the contexts to be binarized.
While such binarization leads to a loss of information, our empirical analysis shows that TM with Thompson sampling performs substantially better than the other evaluated learners on the majority of the datasets and TM learners are quite robust to sub-optimal binarization.
While our work does not provide any theoretical guarantees on performance or regret bounds for TM learners, we believe the latter will form an interesting direction of research.
Conclusion
In this paper, we presented an interpretable and practical contextual bandit learner using Tsetlin
Machine (TM).
Our analysis showed that TM as a contextual bandit learner provides competitive performance compared to other popular contextual bandit algorithms.
We then presented how
Thompson sampling can be implemented using TM, where our approach is derived from the Tree
Bootstrap algorithm.
Finally, we perform interpretability analysis where the arm selection strategy can be characterized by a propositional function of the contexts..
Having such promising empirical results, we aim at providing theoretical performance guarantees for algorithms with TM as future work.
A Literature Overview
Existing works on contextual bandits can be broadly categorized into two categories [12]. The first category includes algorithms that focus on exploration problems while the second category pertains to using different base learners for contextual bandits. Literature focusing on the exploration problems provide theoretical guarantees on regret bounds, however, the bounds are often tied with very strong underlying assumptions.
In [8], for instance, the authors provide a regret bound that holds with probability 1 − δ and is of the order O (cid:16)(cid:113)
Td
ln 3 ( KT ln( T ) /δ
(cid:17) for d dimensional context vector with T rounds and K actions. The underlying assumption for obtaining such regret bounds is that the payoff (or reward) is a linear function of the context features. Similar assumptions underlie the work in [4], where the authors present Thompson sampling for contextual bandits with linear payoffs. For the second category, different base learners for contextual bandits have been investigated.
In [5], the authors employ neural networks to model the value of rewards given the contexts. Subsequently deep learning for contextual bandits have also been explored in [31, 25, 16].
In [13], the authors use random forest as a base learner for contextual bandits. The proposed learner is optimal up to a logarithmic factor where the computational cost of the algorithm is linear with respect to the time horizon. In [12], the authors use decision tree learners for contextual bandits, and then propose Thompson sampling for such non-parametric learners.
Our work pertains to the second category, where we use TM as a base learner for contextual bandits. Our TM learner supports incremental training with streaming data. In contrast to popular baseline learners such as artificial neural networks, both the learned arm-context model and the process of learning are easy to follow and explain. The interpretability is attained using propositional functions of context features used by TM for arm selection.
The Noisy XOR dataset was already binarized and for MNIST , we binarize the features having value ≥ 75 to be 1 and 0 otherwise. For other datasets, the maximum number of bits for binarization per feature is given by Table 3. [Table]
[Table]
Table 4 shows the configuration of the contextual bandit learners using TM.
The same TM configurations are used for both TM with ε − greedy arm selection and TM with Thompson sampling.
C Empirical analysis of sub-optimal binarization in TM
[Figure]
a n
[Figure]
w a r d
[Figure]
M e a n
R e w a r d
[Figure]
M e a n
R e w a r d
[Figure]