Original: L EARNING FOR E DGE -W EIGHTED O NLINE B IPARTITE M ATCHING WITH R OBUSTNESS G UARANTEES
Revised: L EARNING FOR E DGE -W EIGHTED O NLINE B IPARTITE M ATCHING WITH R OBUSTNESS G UARANTEES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Many real-world problems, such as online ad display, can be formulated as online bipartite matching.
Revised: Many real-world problems, such as online ad display, can be formulated as online bipartite matching.

Original: The crucial challenge lies in the nature of sequentially-revealed online item information, based on which we make irreversible matching decisions at each step.
Revised: The crucial challenge lies in the nature of sequentially-revealed online item information, based on which we make irreversible matching decisions at each step.

Original: While numerous expert online algorithms have been proposed with bounded worst-case competitive ratios, they may not offer satisfactory performance in average cases.
Revised: While numerous expert online algorithms have been proposed with bounded worst-case competitive ratios, they may not offer satisfactory performance in average cases.

Original: On the other hand, reinforcement learning (RL) has been applied to improve the average performance, but they lack robustness and can perform arbitrarily badly.
Revised: On the other hand, reinforcement learning (RL) has been applied to improve the average performance, but they lack robustness and can perform arbitrarily badly.

Original: In this paper, we propose a novel RL-based approach to edgeweighted online bipartite matching with robustness guarantees ( LOMAR ), achieving both good average-case and good worst-case performance.
Revised: In this paper, we propose a novel RL-based approach to edgeweighted online bipartite matching with robustness guarantees ( LOMAR ), achieving both good average-case and good worst-case performance.

Original: The key novelty of
Revised: The key novelty of

Original: LOMAR is a new online switching operation which, based on a judiciously-designed condition to hedge against future uncertainties, decides whether to follow the expert’s decision or the RL decision for each online item arrival.
Revised: LOMAR is a new online switching operation which, based on a judiciously-designed condition to hedge against future uncertainties, decides whether to follow the expert’s decision or the RL decision for each online item arrival.

Original: We prove that for any ρ ∈ [0 , 1] , LOMAR is ρ -competitive against any given expert online algorithm.
Revised: We prove that for any ρ ∈ [0 , 1] , LOMAR is ρ -competitive against any given expert online algorithm.

Original: To improve the average performance, we train the RL policy by explicitly considering the online switching operation.
Revised: To improve the average performance, we train the RL policy by explicitly considering the online switching operation.

Original: Finally, we run empirical experiments to demonstrate the advantages of LOMAR compared to existing baselines.
Revised: Finally, we run empirical experiments to demonstrate the advantages of LOMAR compared to existing baselines.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Online bipartite matching is a classic online problem of practical importance (Mehta, 2013; Kim & Moon, 2020; Fahrbach et al., 2020;
Revised: Online bipartite matching is a classic online problem of practical importance (Mehta, 2013; Kim & Moon, 2020; Fahrbach et al., 2020;

Original: Antoniadis et al., 2020; Huang & Shu, 2021; Gupta &
Revised: Antoniadis et al., 2020b; Huang & Shu, 2021; Gupta &

Original: Roughgarden, 2020).
Revised: Roughgarden, 2020).

Original: In a nutshell, online bipartite matching assigns online items to offline items in two separate sets: when an online item arrives, we need to match it to an offline item given applicable constraints (e.g., capacity constraint), with the goal of maximizing the total rewards collected (Mehta, 2013).
Revised: In a nutshell, online bipartite matching assigns online items to offline items in two separate sets: when an online item arrives, we need to match it to an offline item given applicable constraints (e.g., capacity constraint), with the goal of maximizing the total rewards collected (Mehta, 2013).

Original: For example, numerous applications, including scheduling tasks to servers, displaying advertisements to online users, recommending articles/movies/products, among many others, can all be modeled as online bipartite matching or its variants.
Revised: For example, numerous applications, including scheduling tasks to servers, displaying advertisements to online users, recommending articles/movies/products, among many others, can all be modeled as online bipartite matching or its variants.

Original: The practical importance, along with substantial algorithmic challenges, of online bipartite matching has received extensive attention in the last few decades (Karp et al., 1990; Fahrbach et al., 2020).
Revised: The practical importance, along with substantial algorithmic challenges, of online bipartite matching has received extensive attention in the last few decades (Karp et al., 1990; Fahrbach et al., 2020).

Original: Concretely, many algorithms have been proposed and studied for various settings of online bipartite matching, ranging from simple yet effective greedy algorithms to sophisticated ranking-based algorithms (Karp et al., 1990; Kim & Moon, 2020; Fahrbach et al., 2020;
Revised: Concretely, many algorithms have been proposed and studied for various settings of online bipartite matching, ranging from simple yet effective greedy algorithms to sophisticated ranking-based algorithms (Karp et al., 1990; Kim & Moon, 2020; Fahrbach et al., 2020;

Original: Aggarwal et al., 2011;
Revised: Aggarwal et al., 2011;

Original: Devanur et al., 2013).
Revised: Devanur et al., 2013).

Original: These expert algorithms typically have robustness guarantees in terms of the competitive ratio — the ratio of the total reward obtained by an online algorithm to the reward of another baseline algorithm (commonly the optimal offline algorithm) — even under adversarial settings given arbitrarily bad problem inputs (Karp et al., 1990; Huang & Shu, 2021).
Revised: These expert algorithms typically have robustness guarantees in terms of the competitive ratio — the ratio of the total reward obtained by an online algorithm to the reward of another baseline algorithm (commonly the optimal offline algorithm) — even under adversarial settings given arbitrarily bad problem inputs (Karp et al., 1990; Huang & Shu, 2021).

Original: In some settings, even the optimal competitive ratio for adversarial inputs has been derived (readers are referred to (Mehta, 2013) for an excellent tutorial).
Revised: In some settings, even the optimal competitive ratio for adversarial inputs has been derived (readers are referred to (Mehta, 2013) for an excellent tutorial).

Original: The abundance of competitive online algorithms has clearly demonstrated the importance of performance robustness in terms of the competitive ratio, especially in safety-sensitive applications such as matching mission-critical items or under contractual obligations (Fahrbach et al., 2020).
Revised: The abundance of competitive online algorithms has clearly demonstrated the importance of performance robustness in terms of the competitive ratio, especially in safety-sensitive applications such as matching mission-critical items or under contractual obligations (Fahrbach et al., 2020).

Original: Nonetheless,  the good worst-case competitive ratio does not necessarily lead to good average-case performance (Boyar et al., 2016).
Revised: Nonetheless, as commonly known in the literature, the necessity of conservativeness to address the worst-case adversarial input means that the average performance is typically not optimal (see, e.g., (Christianson et al., 2022; Zeynali et al., 2021) for discussions in other general online problems).

Original: More recently, online optimizers based on reinforcement learning (RL) (Chen et al., 2022; Georgiev & Lió, 2020;
Revised: More recently, online optimizers based on reinforcement learning (RL) (Chen et al., 2022; Georgiev & Lió, 2020;

Original: Wang et al., 2019; Alomrani et al., 2021; Du et al., 2019; Zuzic et al., 2020) have been proposed in the context of online bipartite matching as well as other online problems.
Revised: Wang et al., 2019; Alomrani et al., 2021; Du et al., 2019; Zuzic et al., 2020) have been proposed in the context of online bipartite matching as well as other online problems.

Original: Specifically, by exploiting statistical information of problem inputs, RL models are trained offline and then applied online to produce decisions given unseen problem inputs.
Revised: Specifically, by exploiting statistical information of problem inputs, RL models are trained offline and then applied online to produce decisions given unseen problem inputs.

Original: These RL-based optimizers can often achieve high average rewards in many typical cases.
Revised: These RL-based optimizers can often achieve high average rewards in many typical cases.

Original: Nonetheless, they may not have any performance robustness guarantees in terms of the competitive ratio.
Revised: Nonetheless, they may not have any performance robustness guarantees in terms of the competitive ratio.

Original: In fact, a crucial pain point is that the worst-case performance of many RL-based optimizers can be arbitrarily bad, due to, e.g., testing distribution shifts, inevitable model generalization errors, finite samples, and/or even adversarial inputs.
Revised: In fact, a crucial pain point is that the worst-case performance of many RL-based optimizers can be arbitrarily bad, due to, e.g., testing distribution shifts, inevitable model generalization errors, finite samples, and/or even adversarial inputs.

Original: Consequently, the lack of robustness guarantees has become a key roadblock for wide deployment of RL-based optimizers in real-world applications.
Revised: Consequently, the lack of robustness guarantees has become a key roadblock for wide deployment of RL-based optimizers in real-world applications.

Original: In this paper,by integrating an expert algorithm with RL, we achieve both good average performance and  robustness guarantees forthe problem of edge-weighted online bipartite matching (Fahrbach et al., 2020; Kim & Moon, 2020).
Revised: In this paper, we focus on an important and novel objective — achieving both good average performance and guaranteed worst-case robustness — for edge-weighted online bipartite matching (Fahrbach et al., 2020; Kim & Moon, 2020).

Original: Our algorithm is called LOMAR (Learning-based approach to edge-weighted Online bipartite MAtching with Robustness guarantees).
Revised: More specifically, our algorithm, called LOMAR (Learning-based approach to edge-weighted Online bipartite MAtching with Robustness guarantees), integrates an expert algorithm with RL.

Original: The key novelty of LOMAR lies in an online switching step that dynamically switches between the RL decision and the expert decision online, as well as a switching-aware training algorithm.
Revised: The key novelty of LOMAR lies in a carefully-designed online switching step that dynamically switches between the RL decision and the expert decision online, as well as a switching-aware training algorithm.

Original: For both no-free-disposal and free-disposal settings, we design novel switching conditions as to when the RL decisions can be safely followed while still guaranteeing robustness of being ρ -competitive against any given expert online algorithms for any ρ ∈ [0 , 1] .
Revised: For both no-free-disposal and free-disposal settings, we design novel switching conditions as to when the RL decisions can be safely followed while still guaranteeing robustness of being ρ -competitive against any given expert online algorithms for any ρ ∈ [0 , 1] .

Original: Furthermore, if the expert itself has a competitive ratio of λ ≤ 1 against the optimal offline algorithm (OPT), then it will naturally translate into LOMAR being ρλ -competitive against OPT.
Revised: Furthermore, if the expert itself has a competitive ratio of λ ≤ 1 against the optimal offline algorithm (OPT), then it will naturally translate into LOMAR being ρλ -competitive against OPT.

Original: To improve the average performance of LOMAR , we train the RL policy in LOMAR by explicitly taking into account the introduced switching operation.
Revised: To improve the average performance of LOMAR , we train the RL policy in LOMAR by explicitly taking into account the introduced switching operation.

Original: Importantly, to avoid the “no supervision” trap during the initial RL policy training, we propose to approximate the switching operation probabilistically.
Revised: Importantly, to avoid the “no supervision” trap during the initial RL policy training, we propose to approximate the switching operation probabilistically.

Original: Finally, we offer empirical experiments to demonstrate that LOMAR can improve the average cost (compared to existing expert algorithms) as well as lower the competitive 
Revised: Finally, we offer empirical experiments to demonstrate that LOMAR can improve the average cost (compared to existing expert algorithms) as well as lower the competitive ratio (compared to pure RL-based optimizers).

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Online bipartite matching has been traditionally approached by expert algorithms (Mehta, 2013;
Revised: Online bipartite matching has been traditionally approached by expert algorithms (Mehta, 2013;

Original: Karande et al., 2011; Huang et al., 2019; Devanur et al., 2013).
Revised: Karande et al., 2011; Huang et al., 2019; Devanur et al., 2013).

Original: A simple but widely-used algorithm is the (deterministic) greedy algorithm (Mehta, 2013), achieving reasonably-good competitive ratios and empirical performance (Alomrani et al., 2021).
Revised: A simple but widely-used algorithm is the (deterministic) greedy algorithm (Mehta, 2013), achieving reasonably-good competitive ratios and empirical performance (Alomrani et al., 2021).

Original: Randomized algorithms have also been proposed to improve the competitive ratio (Ting & Xiang, 2014).
Revised: Randomized algorithms have also been proposed to improve the competitive ratio (Ting & Xiang, 2014;

Original: For example, RANKING (Aggarwal et al., 2011; Devanur et al., 2013) is a well-known randomized algorithm achieving a (1 − 1 /e ) -competitive ratio.
Revised: Aggarwal et al., 2011).

Original: In addition, competitive algorithms based on the primal-dual framework have also been proposed (Mehta, 2013; Buchbinder et al., 2009).
Revised: In addition, competitive algorithms based on the primal-dual framework have also been proposed (Mehta, 2013; Buchbinder et al., 2009).

Original: More recently, multi-phase information andmachine learning (ML) predictions have been leveraged to  improve the algorithm performance (Kesselheim et al., 2013).
Revised: More recently, multi-phase information and predictions have been leveraged to exploit stochasticity within each problem instance and improve the algorithm performance (Kesselheim et al., 2013).

Original: For example, (Korula & Pál, 2009) designs a secretary matching algorithm based on a threshold obtained using the information of phase one, and exploits the threshold for matching in phase two.
Revised: For example, (Korula & Pál, 2009) designs a secretary matching algorithm based on a threshold obtained using the information of phase one, and exploits the threshold for matching in phase two.

Original: In (Antoniadis et al., 2020), a two-phase algorithm (Kesselheim et al., 2013) is first applied and then ML advice is exploited in phase three.
Revised: Note that stochastic settingsconsidered by expert algorithms (Mehta, 2013; Karande et al., 2011) mean that the arrival orders and/or rewards of different online items within each problem instance are stochastic.

Original: Nonetheless, the ML advice does not provide any specific decision as  in our RL policy, and how to train the ML advice model is unaddressed.
Revised: By contrast, as shown in equation 2, we focus on an unknown distribution of problem instances whereas the inputs within each instance can still be arbitrary.

Original: Another line of algorithms utilize RLmodels to improve the average performance (Wang et al., 2019;
Revised: Another line of algorithms utilize RL to improve the average performance (Wang et al., 2019;

Original: Georgiev & Lió, 2020; Chen et al., 2022; Alomrani et al., 2021).
Revised: Georgiev & Lió, 2020; Chen et al., 2022; Alomrani et al., 2021).

Original: Even though heuristic methods (such as using adversarial training samples (Zuzic et al., 2020; Du et al., 2022)) have been considered to empirically improve the robustness, they do not provide any theoretically-proved robustness guarantees.
Revised: Even though heuristic methods (such as using adversarial training samples (Zuzic et al., 2020; Du et al., 2022)) are used to empirically improve the robustness, they do not provide any theoretically-proved robustness guarantees.

Original: ML-augmented algorithms have been recently considered for various problems (Chł˛edowski et al., 2021), (Boyar et al., 2016; Lykouris & Vassilvitskii, 2021; Gupta & Roughgarden, 2017).
Revised: ML-augmented algorithms have been recently considered for various problems (Rutten et al., 2022; Christianson et al., 2022; Chł˛edowski et al., 2021; Lykouris & Vassilvitskii, 2021; Gupta & Roughgarden, 2017).

Original: By viewing the ML prediction as blackbox advice, these algorithms strive to provide good competitive ratios when the ML predictions are nearly perfect, and also bounded competitive ratios when ML predictions are bad.
Revised: By viewing the ML prediction as blackbox advice, these algorithms strive to provide good competitive ratios when the ML predictions are nearly perfect, and also bounded competitive ratios when ML predictions are bad.

Original: But, they still focus on the worst case without addressing the average performance or how the ML model is trained.
Revised: But, they still focus on the worst case without addressing the average performance or how the ML model is trained.

Original: By contrast, the RL model in LOMAR is trained by taking into account the switching operation and performs inference based on the actual state (rather than its own independently-maintained state as a blackbox).
Revised: By contrast, the RL model in LOMAR is trained by taking into account the switching operation and performs inference based on the actual state (rather than its own independently-maintained state as a blackbox).

Original: Assuming a given downstream algorithm, (Wang et al., 2021; Liu & Grigas, 2021; Wilder et al., 2019; Elmachtoub & Grigas, 2017;Du et al., 2021;
Revised: Assuming a given downstream algorithm, (Wang et al., 2021; Liu & Grigas, 2021; Wilder et al., 2019; Elmachtoub & Grigas, 2017;

Original:  Anand et al., 2021) focus on learning the ML model to better serve the end goal in completely different (sometimes, offline optimization) problems.
Revised: Du et al., 2021; Anand et al., 2021) focus on learning the ML model to better serve the end goal in completely different (sometimes, offline optimization) problems.

Original: LOMAR is relevant to conservative bandits/RL (Wu et al., 2016; Kazerouni et al., 2017;
Revised: LOMAR is relevant to conservative bandits/RL (Wu et al., 2016; Kazerouni et al., 2017;

Original: Yang et al., 2022; Garcelon et al., 2020).
Revised: Yang et al., 2022; Garcelon et al., 2020).

Original: With unknown reward functions (as well as transition models if applicable), conservative bandits/RL leverages an existing policy to safeguard the exploration process.
Revised: With unknown reward functions (as well as transition models if applicable), conservative bandits/RL leverages an existing policy to safeguard the exploration process.

Original: But, they only consider the cumulative reward without addressing future uncertainties when deciding exploration vs. rolling back to an existing policy.
Revised: But, they only consider the cumulative reward without addressing future uncertainties when deciding exploration vs. rolling back to an existing policy.

Original: Thus, as shown in Section 4, this cannot guarantee robustness in our problem.
Revised: Thus, as shown in Section 4, this cannot guarantee robustness in our problem.

Original: Also, constrained policy optimization (Yang et al., 2020; Kumar et al., 2020; Schulman et al., 2015; Achiam et al., 2017; Thomas et al., 2021; Berkenkamp et al., 2017) focuses on average (cost) constraints in the long run, whereas LOMAR achieves stronger robustness (relative to an expert algorithm) for each episode with even adversarial inputs.
Revised: Also, constrained policy optimization (Yang et al., 2020; Kumar et al., 2020; Schulman et al., 2015; Achiam et al., 2017; Thomas et al., 2021; Berkenkamp et al., 2017) focuses on average (cost) constraints in the long run, whereas LOMAR achieves stronger robustness (relative to an expert algorithm) for each episode with even adversarial inputs.

Original:  P ROBLEM F ORMULATION
Revised:  P ROBLEM F ORMULATION

Original: We focus on edge-weighted online bipartite matching, which includes un-weighted and vertexweighted matching as special cases (Fahrbach et al., 2020; Kim & Moon, 2020).
Revised: We focus on edge-weighted online bipartite matching, which includes un-weighted and vertexweighted matching as special cases (Fahrbach et al., 2020; Kim & Moon, 2020).

Original: In the following, we also drop “edge-weighted” if applicable when referring to our problem.
Revised: In the following, we also drop “edge-weighted” if applicable when referring to our problem.

Original: The goal of the agent is to match items (a.k.a. vertices) between two sets U and V to gain as high total rewards as possible.
Revised: The goal of the agent is to match items (a.k.a. vertices) between two sets U and V to gain as high total rewards as possible.

Original: Suppose that U is fixed and contains offline items u ∈ U , and that the online items v ∈ V arrive sequentially: in each time slot, an online item v ∈ V arrives and the weight/reward information { w uv | w u, min ≤ w uv ≤ w u, max , u ∈ U} is revealed, where w uv represents the reward when the online item v is matched to each offline u ∈ U . For online decisions, the agent does not know the reward of future online items.
Revised: Suppose that U is fixed and contains offline items u ∈ U , and that the online items v ∈ V arrive sequentially: in each time slot, an online item v ∈ V arrives and the weight/reward information { w uv | w u, min ≤ w uv ≤ w u, max , u ∈ U} is revealed, where w uv represents the reward when the online item v is matched to each offline u ∈ U . We denote one problem instance by G = {U , V , W} , where W = { w uv | u ∈ U , v ∈ V} .

Original:  Also, any offline item u ∈ U can be matched up to c u times, where c u is essentially the capacity for offline item u known to the agent.We denote x uv ∈ { 0 , 1 } as the matching decision indicating whether u is matched to v .
Revised: We denote x uv ∈ { 0 , 1 } as the matching decision indicating whether u is matched to v . Also, any offline item u ∈ U can be matched up to c u times, where c u is essentially the capacity for offline item u known to the agent.

Original: The objective is to maximize the total collected reward (cid:80) v ∈V ,u ∈U x uv w uv .
Revised: The objective is to maximize the total collected reward (cid:80) v ∈V ,u ∈U x uv w uv .

Original: With a slight abuse of notations, we denote x v ∈ U as the index of item in U that is matched to item v ∈ V .
Revised: With a slight abuse of notations, we denote x v ∈ U as the index of item in U that is matched to item v ∈ V .

Original: The set of online items matched to u ∈ U is denoted as V u = { v ∈ V | x uv = 1 } .
Revised: The set of online items matched to u ∈ U is denoted as V u = { v ∈ V | x uv = 1 } .

Original: The edge-weighted online bipartite matching problem has been mostly studied under two different settings: no free disposal and with free disposal (Mehta, 2013).
Revised: The edge-weighted online bipartite matching problem has been mostly studied under two different settings: no free disposal and with free disposal (Mehta, 2013).

Original: In the no-free-disposal case, each offline item u ∈ U can only be matched strictly up to c u times; in the free-disposal case, each offline item u ∈ U can be matched more than c u times, but only the top c u rewards are counted when more than c u online items are matched to u .
Revised: In the no-free-disposal case, each offline item u ∈ U can only be matched strictly up to c u times; in the free-disposal case, each offline item u ∈ U can be matched more than c u times, but only the top c u rewards are counted when more than c u online items are matched to u .

Original: Compared to the free-disposal case, the no-free-disposal case is significantly more challenging with the optimal competitive ratio being 0 in the strong adversarial setting unless additional assumptions are made (e.g., w u, min > 0 for each u ∈ U (Kim & Moon, 2020) and/or random-order of online arrivals) (Fahrbach et al., 2020; Mehta, 2013).
Revised: Compared to the free-disposal case, the no-free-disposal case is significantly more challenging with the optimal competitive ratio being 0 in the strong adversarial setting unless additional assumptions are made (e.g., w u, min > 0 for each u ∈ U (Kim & Moon, 2020) and/or random-order of online arrivals) (Fahrbach et al., 2020; Mehta, 2013).

Original: The free-disposal setting not only makes the problem more tractable, but also is practically motivated by the display ad application where the advertisers (i.e., offline items u ∈ U ) will not be unhappy if they receive more impressions (i.e., online items v ∈ V ) than their budgets c u , even though only the top c u items count.
Revised: The free-disposal setting not only makes the problem more tractable, but also is practically motivated by the display ad application where the advertisers (i.e., offline items u ∈ U ) will not be unhappy if they receive more impressions (i.e., online items v ∈ V ) than their budgets c u , even though only the top c u items count.

Original: LOMAR can handle both no-free-disposal and free-disposal settings.
Revised: LOMAR can handle both no-free-disposal and free-disposal settings.

Original: For better presentation of our key novelty and page limits, we focus on the no-free-disposal setting in the body of the paper, while deferring the free-disposal setting to Appendix B .
Revised: For better presentation of our key novelty and page limits, we focus on the no-free-disposal setting in the body of the paper, while deferring the free-disposal setting to Appendix B .

Original: Specifically, the offline problem with no free disposal can be expressed as:
Revised: Specifically, the offline problem with no free disposal can be expressed as:

Original: [Equation]
Revised: [Equation]

Original: where the constraints specify the offline item capacity limit and each online item v ∈ V can only be matched up to one offline item u ∈ U .
Revised: where the constraints specify the offline item capacity limit and each online item v ∈ V can only be matched up to one offline item u ∈ U .

Original: For the convenience of notations, wealso use f u to denote the total rewards collected for offline item u ∈ U : f u = f u ( V u ) =(cid:80) v ∈V u w uv . Additionally, we use
Revised: Given a problem instance G and an online algorithm α , we use f αu ( G ) to denote the total reward collected for offline item u ∈ U , and R α ( G ) =

Original: R = (cid:80) u ∈U f u to denote the total collected reward.
Revised: (cid:80) u ∈U f αu ( G ) to denote the total collected reward.

Original: Even with all the reward information available in advance, finding the offline optimal matching is non-trivial because of the combinatorial nature (Dai et al., 2017; Bengio et al., 2021;
Revised: We will also drop the superscript α for notational convenience wherever applicable.

Original: Shao et al., 2022; Bello et al., 2017).
Revised: Objective.

Original: In the online case,  the agent has to make irreversible decisions without knowing the future online item arrivals, adding significantly more challenges.
Revised: Solving the problem in equation 1 is very challenging in the online case, where the agent has to make irreversible decisions without knowing the future online item arrivals.

Original: To measure the decision robustness, atypical performance metric for online optimization is the competitive ratio , which is defined as below.
Revised: Next, we first define the following generalized competitiveness as a metric of robustness.

Original: Definition 1 (Competitive ratio) .
Revised: Definition 1 (Competitiveness) .

Original: An online bipartite matching algorithm  is said to have a competitive ratio ρ ≥ 0 against the algorithm π if for any problem input { w uv | w u, min ≤ w uv ≤ w u, max , u ∈ U , v ∈ V} , its total collected reward R  satisfies R  ≥ ρR π  − B , where B ≥ 0 is a constant independent of the problem input, and R π is the total reward of the algorithm π .
Revised: An online bipartite matching algorithm α is said to be ρ − competitive with ρ ≥ 0 against the algorithm π if for any problem instance G , its total collected reward R α ( G ) satisfies R α ( G ) ≥ ρR π ( G ) − B , where B ≥ 0 is a constant independent of the problem input, and R π is the total reward of the algorithm π .

Original: A commonly-considered competitive ratio is against the optimal offline algorithm(denoted as OPT) by setting π as OPT, for which we have ρ ∈ [0 , 1] . Naturally, the greater ρ ≥ 0 , the better robustness of the online algorithm, although the average rewards may not necessarily be better.
Revised: Competitiveness against a given (online) algorithm π is common in the literature (Christianson et al., 2022): the greater ρ ≥ 0 , the better robustness of the online algorithm, although the average rewards can be worse.

Original: The constant B ≥relaxes the strict competitive ratio by allowing an additive regret .
Revised: Additionally, the constant B ≥ 0 relaxes the strict competitive ratio by allowing an additive regret (Antoniadis et al., 2020a). When B = 0 , the competitive ratio becomes the strict one. In this paper, we focus on a setting where the problem instance G = {U , V , W} follows an unknown distribution, whereas both the rewards W and online arrival order within each instance G can be adversarial. We consider both average performance and worst-case robustness guarantees as formalized below:

Original: When B = 0 , the competitive ratio becomes the strict one (Boyar et al., 2016).
Revised: [Equation] where the expectation E G [ R α ( G )] is over the randomness G = {U , V , W} .

Original: Also, while we focus on the deterministic competitive ratio to provide deterministic guarantees, our results still apply to the randomized setting in which R is the expected reward by averaging out randomness over the coin flips of  the algorithm (Section 4.3).
Revised: Note carefully that some manually-designed algorithms focus on a stochastic setting where the arrival order is random and/or the rewards { w uv | w u, min ≤ w uv ≤ w u, max , u ∈ U} of each online item is independently and identically distributed (i.i.d.) within each problem instance G Mehta (2013). By contrast, our settings are significantly different — we only assume an unknown distribution for the entire problem instance G = {U , V , W} while both the rewards W and online arrival order within each instance G can be adversarial in our problem.

Original:  O NLINE S WITCHING FOR R OBUSTNESS G UARANTEES
Revised:  O NLINE S WITCHING FOR R OBUSTNESS G UARANTEES

Original: We present LOMAR , which includes an online switching operation to dynamically decide to follow the ML decision or the expert decision, to achieve robustness guarantees with respect to the expert.
Revised: We present LOMAR , which includes an online switching operation to dynamically decide to follow the ML decision or the expert decision, to achieve robustness guarantees with respect to the expert.

Original: 4. O NLINE S WITCHING
Revised: 4. O NLINE S WITCHING

Original: RL-based optimizers (Alomrani et al., 2021) trained offline can exploit the statistical information hidden in offline samples and achieve high average rewards. But, the performance of conventional
Revised: While switching is common in online algorithms, “ how to switch ”is highly non-trivial and a key merit for algorithm designs Antoniadis et al. (2020a)

Original: RL-based optimizers are highly volatile and lacks guaranteed robustness, e.g., when the testing problem inputs have different distributions than the training samples.
Revised: ; Christianson et al.

Original: Even using adversarial training and without distributional shifts, the focus on average performance means that RL-based optimizers may not guarantee good rewards for every problem instance due to the finite model capacity. On the other hand, algorithms designed by experts (Mehta, 2013; Kim & Moon, 2020; Fahrbach et al., 2020; Huang & Shu, 2021) offer guaranteed performance robustness (typically, in terms of the competitive ratio). For example, the simple greedy algorithm is 12 -competitive against the optimal offline algorithm for the free-disposal setting, which can be further enhanced by more sophisticated algorithms and/or making additional assumptions (Fahrbach et al., 2020; Mehta, 2013). Nonetheless, the expert algorithms may not provide as high average rewards as their RL counterparts, especially when the training and testing distributions are consistent.
Revised: ; Rutten et al. (2022).

Original: To guarantee robustness (i.e., ρ -competitive against a given expert for any ρ ∈ [0 , 1] ), we propose a novel online algorithm
Revised: To guarantee robustness (i.e., ρ -competitive against a given expert for any ρ ∈ [0 , 1] ), we propose a novel online algorithm

Original: (Algorithm 1).
Revised: (Algorithm 1).

Original: In the algorithm, we independently run an expert online algorithm π — the cumulative reward and item matching decisions are all maintained virtually for the expert, but not used as the actual decisions.
Revised: In the algorithm, we independently run an expert online algorithm π — the cumulative reward and item matching decisions are all maintained virtually for the expert, but not used as the actual decisions.

Original: Meanwhile, instead of being independently executed to provide blackbox advice based on its own virtual state (like in the prior ML-augmented online algorithm (Christianson et al., 2022)), the RL model in LOMAR makes online decisions based on the actual state at each step.
Revised: Meanwhile, instead of being independently executed to provide blackbox advice based on its own virtual state (like in the prior ML-augmented online algorithm (Christianson et al., 2022)), the RL model in LOMAR makes online decisions based on the actual state at each step.

Original: The most crucial step for safeguarding RL decisions is online switching: Lines 13–19 in Algorithm 1.
Revised: The most crucial step for safeguarding RL decisions is online switching: Lines 13–19 in Algorithm 1.

Original: The key idea for this step is to switch between the expert decision x πv and the RL decision ˜ x v in order to ensure that the actual online decision x v meets the ρ -competitive requirement (against the expert π ).
Revised: The key idea for this step is to switch between the expert decision x πv and the RL decision ˜ x v in order to ensure that the actual online decision x v meets the ρ -competitive requirement (against the expert π ).

Original: Specifically, we follow the RL decision ˜ x v only if it can safely hedge against any future uncertainties (i.e., the expert’s future reward increase); otherwise, we need to roll back to the expert’s decision x πv to stay on track for robustness.
Revised: Specifically, we follow the RL decision ˜ x v only if it can safely hedge against any future uncertainties (i.e., the expert’s future reward increase); otherwise, we need to roll back to the expert’s decision x πv to stay on track for robustness.

Original: Note that naive switching conditions, e.g., only ensuring that the actual cumulative reward is at least ρ times of the expert’s cumulative reward at each step (Wu et al., 2016; Yang et al., 2022), can fail to meet the competitive ratio requirement in the end.
Revised: Note that naive switching conditions, e.g., only ensuring that the actual cumulative reward is at least ρ times of the expert’s cumulative reward at each step (Wu et al., 2016; Yang et al., 2022), can fail to meet the competitive ratio requirement in the end.

Original: The reason is that, even though the competitive ratio requirement is met (i.e., R v ≥ ρR πv − B ) at the current step v , the expert can possibly obtain much higher rewards from future online items v + 1 , v + 2 , · · · if it has additional offline item capacity that the actual algorithm LOMAR does not have.
Revised: The reason is that, even though the competitive ratio requirement is met (i.e., R v ≥ ρR πv − B ) at the current step v , the expert can possibly obtain much higher rewards from future online items v + 1 , v + 2 , · · · if it has additional offline item capacity that the actual algorithm LOMAR does not have.

Original: Thus, we must carefully design the switching conditions to hedge against future risks.
Revised: Thus, we must carefully design the switching conditions to hedge against future risks.

Original: The no-free-disposal and free-disposal settings
Revised: The no-free-disposal and free-disposal settings

Original: 1: Initialization: The actual set of items matched to u ∈ U is V u,v after sequentially-arriving item v ’s assignment with V u, 0 = ∅ , the actual remaining capacity is b u = c u for u ∈ U , and the actual cumulative reward is R 

Revised: 1: Initialization: The actual set of items matched to u ∈ U is V u,v after sequentially-arriving item v ’s assignment with V u, 0 = ∅ , the actual remaining capacity is b u = c u for u ∈ U , and the actual cumulative reward is R

Original: = (cid:80) u ∈U f u
Revised: = (cid:80) u ∈U f u

Original: ( V u, 0 )
Revised: ( V u, 0 )

Original: = 0 .
Revised: = 0 .

Original: The same notations apply to the expert algorithm π by adding the superscript π .
Revised: The same notations apply to the expert algorithm π by adding the superscript π .

Original: Competitive ratio requirement ρ ∈ [0 , 1] and slackness
Revised: Competitive ratio requirement ρ ∈ [0 , 1] and slackness

Original: B ≥ 0 with respect to the expert algorithm π .
Revised: B ≥ 0 with respect to the expert algorithm π .

Original: 2: for v = 1 to |V| do 3:
Revised: 2: for v = 1 to |V| do 3:

Original: Run the algorithm π and match the item v to u ∈ U based on the expert’s decision u = x πv .
Revised: Run the algorithm π and match the item v to u ∈ U based on the expert’s decision u = x πv .

Original: Update the expert’s decision set and reward for offline item u = x πv :
Revised: Update the expert’s decision set and reward for offline item u = x πv :

Original: V πx πv ,v = V πx πv ,v − 1 (cid:83) { v } and f x πv =
Revised: V πx πv ,v = V πx πv ,v − 1 (cid:83) { v } and f x πv =

Original: f x πv ( V πx πv ,v ) .
Revised: f x πv ( V πx πv ,v ) .

Original: Update the expert’s cumulative reward R πv
Revised: Update the expert’s cumulative reward R πv

Original: =
Revised: =

Original: (cid:80) u ∈U f u 6:
Revised: (cid:80) u ∈U f u 6:

Original: Get the set of available items U a = { u ∈ U | |V u,v − 1 | < c u } 7: for u in U a do 8:
Revised: Get the set of available items U a = { u ∈ U | |V u,v − 1 | < c u } 7: for u in U a do 8:

Original: Collect the available history information I u about item u 9:
Revised: Collect the available history information I u about item u 9:

Original: Run the RL model to get score: s u = w uv − h
Revised: Run the RL model to get score: s u = w uv − h

Original: θ ( I u , w uv ) where θ is the network weight 10: end for 11:
Revised: θ ( I u , w uv ) where θ is the network weight 10: end for 11:

Original: Calculate the probability of choosing each available item or skip :
Revised: Calculate the probability of choosing each available item or skip :

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: a , 0 } .
Revised: a , 0 } .

Original: Obtain RL decision: ˜ x v = arg max u ∈U a (cid:83) { skip } {{ ˜ s u } u ∈U a , ˜ s skip } .
Revised: Obtain RL decision: ˜ x v = arg max u ∈U a (cid:83) { skip } {{ ˜ s u } u ∈U a , ˜ s skip } .

Original: 13: if
Revised: 13: if

Original: [Equation]
Revised: [Equation]

Original: ∈U (cid:0) |V u,v − 1 | − | V πu,v | + I u =˜ x v (cid:1) + · w u, max (cid:17) − B then 14:
Revised: ∈U (cid:0) |V u,v − 1 | − | V πu,v | + I u =˜ x v (cid:1) + · w u, max (cid:17) − B then 14:

Original: Select x v = ˜ x v .
Revised: Select x v = ˜ x v .

Original: //Follow the RL decision 15: else if x πv is available for matching (i.e., |V x πv ,v − 1 | < c x πv ) then 16:
Revised: //Follow the RL decision 15: else if x πv is available for matching (i.e., |V x πv ,v − 1 | < c x πv ) then 16:

Original: Select x v = x πv .
Revised: Select x v = x πv .

Original: //Follow
Revised: //Follow

Original: the expert 17: else 18:
Revised: the expert 17: else 18:

Original: Select x v = skip .
Revised: Select x v = skip .

Original: 19: end if 20:
Revised: 19: end if 20:

Original: Update assignment and reward: V x v ,v = V x v ,v − 1 (cid:83) { v } and R v =
Revised: Update assignment and reward: V x v ,v = V x v ,v − 1 (cid:83) { v } and R v =

Original: R v − 1 + w x v ,v 21: end for require different switching conditions.
Revised: R v − 1 + w x v ,v 21: end for require different switching conditions.

Original: Due to the page limit, we focus on the no-free-disposal setting below, while referring readers to Appendix B for more details about the free-disposal setting.
Revised: Due to the page limit, we focus on the no-free-disposal setting below, while referring readers to Appendix B for more details about the free-disposal setting.

Original: 4. R OBUSTNESS C ONSTRAINT
Revised: 4. R OBUSTNESS C ONSTRAINT

Original: In the no-free-disposal case, an offline item u ∈ U cannot receive any additional online items if it has been matched for c u times up to its capacity.
Revised: In the no-free-disposal case, an offline item u ∈ U cannot receive any additional online items if it has been matched for c u times up to its capacity.

Original: By assigning more online items to u ∈ U than the expert algorithm at step v , LOMAR can possibly receive a higher cumulative reward than the expert’s cumulative reward.
Revised: By assigning more online items to u ∈ U than the expert algorithm at step v , LOMAR can possibly receive a higher cumulative reward than the expert’s cumulative reward.

Original: But, such advantages are just temporary , because the expert may receive an even higher reward in the future by filling up the unused capacity of item u .
Revised: But, such advantages are just temporary , because the expert may receive an even higher reward in the future by filling up the unused capacity of item u .

Original: Thus, to hedge against the future uncertainties, LOMAR chooses the RL decisions only when the following condition is satified:
Revised: Thus, to hedge against the future uncertainties, LOMAR chooses the RL decisions only when the following condition is satified:

Original: [Equation]
Revised: [Equation]

Original: where I u =
Revised: where I u =

Original: ˜ x v = 1 if and only if u = ˜ x v and 0 otherwise, ( · ) + = max( · , 0) , ρ ∈ [0 , 1] and B ≥are the hyperparameters indicating the desired robustness with respect to the expert algorithm π .
Revised: ˜ x v = 1 if and only if u = ˜ x v and 0 otherwise, ( · ) + = max( · , 0) , ρ ∈ [0 , 1] and B ≥are the hyperparameters indicating the desired robustness with respect to the expert algorithm π .

Original: The interpretation of equation 2 is as follows.
Revised: The interpretation of equation 3 is as follows.

Original: The left-hand side is the total reward of LOMAR after assigning the online item v based on the RL decision (i.e. ˜ x t ).
Revised: The left-hand side is the total reward of LOMAR after assigning the online item v based on the RL decision (i.e. ˜ x t ).

Original: The right-hand side is the expert’s cumulative cost R πv , plus the term (cid:80) u ∈U (cid:0) |V u,v − 1 | − | V πu,v |
Revised: The right-hand side is the expert’s cumulative cost R πv , plus the term (cid:80) u ∈U (cid:0) |V u,v − 1 | − | V πu,v |

Original: + I u =
Revised: + I u =

Original: ˜ x v (cid:1) + · w u, max which indicates the maximum reward that can be possibly received by the expert in the future.
Revised: ˜ x v (cid:1) + · w u, max which indicates the maximum reward that can be possibly received by the expert in the future.

Original: This reservation term is crucial, especially when the expert has more unused capacity than LOMAR .
Revised: This reservation term is crucial, especially when the expert has more unused capacity than LOMAR .

Original: Specifically, |V u,v − 1 | is the number of online items (after assigning v − 1 items) already assigned to the offline item u ∈ U , and hence (cid:0) |V u,v − 1 | − | V πu,v |
Revised: Specifically, |V u,v − 1 | is the number of online items (after assigning v − 1 items) already assigned to the offline item u ∈ U , and hence (cid:0) |V u,v − 1 | − | V πu,v |

Original: + I u =
Revised: + I u =

Original: ˜ x v (cid:1) + represents the number of more online items that LOMAR has assigned to u than the expert if LOMAR follows the RL decision at step v .
Revised: ˜ x v (cid:1) + represents the number of more online items that LOMAR has assigned to u than the expert if LOMAR follows the RL decision at step v .

Original: If LOMAR assigns fewer items than the expert for an offline item u ∈ U , there is no need for any hedging because LOMAR is guaranteed to receive more rewards by filling up the item u
Revised: If LOMAR assigns fewer items than the expert for an offline item u ∈ U , there is no need for any hedging because LOMAR is guaranteed to receive more rewards by filling up the item u

Original: up to the expert’s assignment level.
Revised: up to the expert’s assignment level.

Original: The term w u, max in equation 2 is the set as the maximum possible reward for each decision.
Revised: The term w u, max in equation 3 is the set as the maximum possible reward for each decision.

Original: Even when w u, max is unknown in advance, LOMAR still applies by simply setting w u, max = ∞ .
Revised: Even when w u, max is unknown in advance, LOMAR still applies by simply setting w u, max = ∞ .

Original: In this case, LOMAR will be less “greedy” than the expert and never use more resources than the expert at any step for any u ∈ U .
Revised: In this case, LOMAR will be less “greedy” than the expert and never use more resources than the expert at any step for any u ∈ U .

Original: 4. R OBUSTNESS A NALYSIS
Revised: 4. R OBUSTNESS A NALYSIS

Original: We now formally show the competitive ratio of LOMAR .
Revised: We now formally show the competitive ratio of LOMAR .

Original: The proof is available in the appendix.
Revised: The proof is available in the appendix.

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: For any 0 ≤ ρ ≤ 1 and B ≥ 0 and any expert algorithm π , LOMAR achieves a competitive ratio of ρ against the algorithm π , i.e., R ≥ ρR π − B for any problem input.
Revised: For any 0 ≤ ρ ≤ 1 and B ≥ 0 and any expert algorithm π , LOMAR achieves a competitive ratio of ρ against the algorithm π , i.e., R ≥ ρR π − B for any problem input.

Original: The hyperparameters 0 ≤ ρ ≤ 1 and B ≥ 0 govern the level of robustness we would like to achieve, at the potential expense of average reward performance.
Revised: The hyperparameters 0 ≤ ρ ≤ 1 and B ≥ 0 govern the level of robustness we would like to achieve, at the potential expense of average reward performance.

Original: For example, by setting ρ = 1 and B = 0 , we achieve the same robustness as the expert but leave little to none freedom for RL decisions.
Revised: For example, by setting ρ = 1 and B = 0 , we achieve the same robustness as the expert but leave little to none freedom for RL decisions.

Original: On the other hand, by setting a small ρ > 0 and/or large B , we provide higher flexibility to RL decisions for better average performance, while potentially decreasing the robustness.
Revised: On the other hand, by setting a small ρ > 0 and/or large B , we provide higher flexibility to RL decisions for better average performance, while potentially decreasing the robustness.

Original: In fact, designing an algorithm that is guaranteed to simultaneously outperform RL and the expert is very challenging, if not impossible, and such tradeoff is necessary in the broad context of ML-augmented online algorithms (Rutten et al., 2022; Christianson et al., 2022).
Revised: In fact, designing an algorithm that is guaranteed to simultaneously outperform RL and the expert is very challenging, if not impossible, and such tradeoff is necessary in the broad context of ML-augmented online algorithms (Rutten et al., 2022; Christianson et al., 2022).

Original: Additionally, in case of multiple experts, we can first combine these experts into a single expert and then apply LOMAR as if it works with a single combined expert.
Revised: Additionally, in case of multiple experts, we can first combine these experts into a single expert and then apply LOMAR as if it works with a single combined expert.

Original: While the competitive ratio of all online algorithms against the optimal offline algorithm is zero in the no-free-disposal and general adversarial setting, there exist provably competitive online expert algorithms under some mild assumptions and other settings (Mehta, 2013).
Revised: While the competitive ratio of all online algorithms against the optimal offline algorithm is zero in the no-free-disposal and general adversarial setting, there exist provably competitive online expert algorithms under some mild assumptions and other settings (Mehta, 2013).

Original: For example, the simple greedy algorithm achieves (cid:16) 1 + max u ∈U w u, max w u, min (cid:17) − 1 under bounded weights assumptions for the adversarial no-free-disposal setting (Kim & Moon, 2020), and 12 for the free-disposal setting (Fahrbach et al., 2020), and there also exist 1 /e -competitive algorithms against the optimal offline algorithm for the random-order setting (Mehta, 2013).
Revised: For example, the simple greedy algorithm achieves (cid:16) 1 + max u ∈U w u, max w u, min (cid:17) − 1 under bounded weights assumptions for the adversarial no-free-disposal setting (Kim & Moon, 2020), and 12 for the free-disposal setting (Fahrbach et al., 2020), and there also exist 1 /e -competitive algorithms against the optimal offline algorithm for the random-order setting (Mehta, 2013).

Original: Thus, an immediate result follows.
Revised: Thus, an immediate result follows.

Original: Corollary 4.1.1.
Revised: Corollary 4.1.1.

Original: For any 0 ≤ ρ ≤ 1 and B ≥ 0 , by using Algorithm 1 and an expert online algorithm π that is λ -competitive against the optimal offline algorithm OPT, then under the same assumptions for π to be λ -competitive, LOMAR is ρλ -competitive against OPT.
Revised: For any 0 ≤ ρ ≤ 1 and B ≥ 0 , by using Algorithm 1 and an expert online algorithm π that is λ -competitive against the optimal offline algorithm OPT, then under the same assumptions for π to be λ -competitive, LOMAR is ρλ -competitive against OPT.

Original: Corollary 4.1.1 provides a general result that applies to any λ -competitive expert algorithm π under its respective required assumptions.
Revised: Corollary 4.1.1 provides a general result that applies to any λ -competitive expert algorithm π under its respective required assumptions.

Original: For example, if the expert π assumes an adversarial or random-order setting, then Corollary 4.1.1 also holds true under the same adversarial or random-order setting.
Revised: For example, if the expert π assumes an adversarial or random-order setting, then Corollary 4.1.1 also holds true under the same adversarial or random-order setting.

Original: Finally, we comment on the randomized setting where randomization is over the algorithm choice and potentially increases the competitive ratio (Gupta & Roughgarden, 2020; Mehta, 2013).
Revised: Finally, we comment on the randomized setting where randomization is over the algorithm choice and potentially increases the competitive ratio (Gupta & Roughgarden, 2020; Mehta, 2013).

Original: For the randomized setting, the competitive ratio is modified as E ( R ) ≥ ρR OPT − B , where E is the expectation and R OPT is the optimal reward of the optimal offline algorithm (Mehta, 2013).
Revised: For the randomized setting, the competitive ratio is modified as E ( R ) ≥ ρR OPT − B , where E is the expectation and R OPT is the optimal reward of the optimal offline algorithm (Mehta, 2013).

Original: We make no assumptions on the expert π in Algorithm 1.
Revised: We make no assumptions on the expert π in Algorithm 1.

Original: Thus, if the expert π itself is randomized in
Revised: Thus, if the expert π itself is randomized in

Original: Algorithm 1, then LOMAR will also be randomized.
Revised: Algorithm 1, then LOMAR will also be randomized.

Original: Also, for any ρ ∈ [0 , 1] and B ≥ 0 , by directly applying Theorem 4.1 and Corollary 4.1.1 to the randomized setting, the competitive ratio of LOMAR will be ρλ -competitive against OPT if the randomized expert π itself is λ -competitive against OPT.
Revised: Also, for any ρ ∈ [0 , 1] and B ≥ 0 , by directly applying Theorem 4.1 and Corollary 4.1.1 to the randomized setting, the competitive ratio of LOMAR will be ρλ -competitive against OPT if the randomized expert π itself is λ -competitive against OPT.

Original:  RL P OLICY T RAINING WITH O NLINE S WITCHING
Revised:  RL P OLICY T RAINING WITH O NLINE S WITCHING

Original: The existing ML-augmented online algorithms typically assume a pre-trained standalone RL model (Christianson et al., 2022; Rutten et al., 2022).
Revised: The existing ML-augmented online algorithms typically assume a pre-trained standalone RL model (Christianson et al., 2022; Rutten et al., 2022).

Original: While the standalone RL model may perform well on its own, some already good actions can be replaced by expert’s action due to switching for robustness in inference.
Revised: While the standalone RL model may perform well on its own, some already good actions can be replaced by expert’s action due to switching for robustness in inference.

Original: In other words, there will be a objective mismatch between training and testing.
Revised: In other words, there will be a objective mismatch between training and testing.

Original: To rectify the mismatch, we propose a novel approach to train the RL model in LOMAR by explicitly considering the switching operation.
Revised: To rectify the mismatch, we propose a novel approach to train the RL model in LOMAR by explicitly considering the switching operation.

Original: RL architecture.
Revised: RL architecture.

Original: For solving online bipartite matching, there exist various network architectures, e.g., fully-connected networks and scalable invariant network for arbitrary graph sizes.
Revised: For solving online bipartite matching, there exist various network architectures, e.g., fully-connected networks and scalable invariant network for arbitrary graph sizes.

Original: The prior study (Mehta, 2013) has shown using extensive empirical experiments that the invariant network architecture, where each offline-online item pair runs through a separate neural network with shared weights among all the item pairs, is empirically advantageous, due to its scalability to large graph sizes and high average performance.
Revised: The prior study (Mehta, 2013) has shown using extensive empirical experiments that the invariant network architecture, where each offline-online item pair runs through a separate neural network with shared weights among all the item pairs, is empirically advantageous, due to its scalability to large graph sizes and high average performance.

Original: We denote the RL model as h θ ( I u , w uv ) where θ is the network parameter.
Revised: We denote the RL model as h θ ( I u , w uv ) where θ is the network parameter.

Original: By feeding the item weight w uv and applicable history information I u for each offline-online item pair ( u, v ) , we can use the RL model to output a threshold for possible item assignment, following threshold-based algorithms (Huang et al., 2019; Mehta, 2013).
Revised: By feeding the item weight w uv and applicable history information I u for each offline-online item pair ( u, v ) , we can use the RL model to output a threshold for possible item assignment, following threshold-based algorithms (Huang et al., 2019; Mehta, 2013).

Original: The history information I u includes, but is not limited to, the average value and variance of weights assigned to u , average in-degree of u , and maximum weight for the already matched items.
Revised: The history information I u includes, but is not limited to, the average value and variance of weights assigned to u , average in-degree of u , and maximum weight for the already matched items.

Original: More details about the information can be found in the appendix.
Revised: More details about the information can be found in the appendix.

Original: Then, with the RL output, we can obtain a score s u = w uv − h
Revised: Then, with the RL output, we can obtain a score s u = w uv − h

Original: θ ( I u , w uv ) , for each possible assignment, and the RL uses the offline item u ∈ U (plus “skip” with s skip = 0 in the no-free-disposal setting) with the largest s u as its candidate action ˜ x v when checking the switching condition in Algorithm 1.
Revised: θ ( I u , w uv ) , for each possible assignment, and the RL uses the offline item u ∈ U (plus “skip” with s skip = 0 in the no-free-disposal setting) with the largest s u as its candidate action ˜ x v when checking the switching condition in Algorithm 1.

Original: Policy training.
Revised: Policy training.

Original: Training the RL model by considering switching in Algorithm 1 is highly non-trivial.
Revised: Training the RL model by considering switching in Algorithm 1 is highly non-trivial.

Original: Most critically, the initial RL decisions can perform arbitrarily badly upon policy initialization, which means that the initial RL decisions are almost always overridden by the expert’s decisions for robustness.
Revised: Most critically, the initial RL decisions can perform arbitrarily badly upon policy initialization, which means that the initial RL decisions are almost always overridden by the expert’s decisions for robustness.

Original: Due to following the expert’s decisions, the RL agent almost always receive a good reward, which actually has nothing to do with the RL’s own decisions and hence provides little to no supervision to improve the RL policy.
Revised: Due to following the expert’s decisions, the RL agent almost always receive a good reward, which actually has nothing to do with the RL’s own decisions and hence provides little to no supervision to improve the RL policy.

Original: Consequently, this creates a gridlock for RL policy training.
Revised: Consequently, this creates a gridlock for RL policy training.

Original: While using an offline pre-trained standalone RL model without considering online switching (e.g., (Alomrani et al., 2021)) as an initial policy may partially address this gridlock, this is certainly inefficient as we have to spend resources for training another RL model, let alone the likelihood of being trapped into the standalone RL model’s suboptimal policies (e.g. local minimums).
Revised: While using an offline pre-trained standalone RL model without considering online switching (e.g., (Alomrani et al., 2021)) as an initial policy may partially address this gridlock, this is certainly inefficient as we have to spend resources for training another RL model, let alone the likelihood of being trapped into the standalone RL model’s suboptimal policies (e.g. local minimums).

Original: To address these issues, we introduce another softmax probability with temperature t to approximate the hard switching process during training.
Revised: To address these issues, we introduce another softmax probability with temperature t to approximate the hard switching process during training.

Original: The switching probability depends on the cumulative reward difference R diff in the switching condition, which is
Revised: The switching probability depends on the cumulative reward difference R diff in the switching condition, which is

Original: [Equation]
Revised: [Equation]

Original: Then the probability of following RL is p os = e Rdiff/t 1+ e Rdiff/t , where t is the temperature of softmax function.
Revised: Then the probability of following RL is p os = e Rdiff/t 1+ e Rdiff/t , where t is the temperature of softmax function.

Original: This softmax probability is differentiable and hence allows backpropagation to supervise the training of the RL model weight θ .
Revised: This softmax probability is differentiable and hence allows backpropagation to supervise the training of the RL model weight θ .

Original: We train the RL agent by applying REINFORCE (Williams, 1992) to optimize the policy parameter θ .
Revised: We train the RL agent by applying REINFORCE (Williams, 1992) to optimize the policy parameter θ .

Original: Denote τ = { x 1 , · · · , x v } as an action trajectory sample and p θ ( τ ) as the possibility of the trajectory given the RL policy, where p θ is calculated based on the selection probability of RL model and expert.
Revised: Denote τ = { x 1 , · · · , x v } as an action trajectory sample and p θ ( τ ) as the possibility of the trajectory given the RL policy, where p θ is calculated based on the selection probability of RL model and expert.

Original: Our goal is to maximize the expected total reward R θ =
Revised: Our goal is to maximize the expected total reward R θ =

Original: E τ ∼ p θ [ w x v ,v ] .
Revised: E τ ∼ p θ [ w x v ,v ] .

Original: Thus, at each training step, given an RL policy with parameter θ , we sample n action trajectories { τ i = { x 1 ,i , · · · , x v,i } , i ∈ [ n ] } and record the corresponding rewards.
Revised: Thus, at each training step, given an RL policy with parameter θ , we sample n action trajectories { τ i = { x 1 ,i , · · · , x v,i } , i ∈ [ n ] } and record the corresponding rewards.

Original: We can get the approximated average reward as ˆ R θ = 1 n (cid:80) ni =1 w ix
Revised: We can get the approximated average reward as ˆ R θ = 1 n (cid:80) ni =1 w ix

Original: i,v ,v .
Revised: i,v ,v .

Original: Then, we calculate the gradient of the RL policy parameter as ∇ θ ˆ R θ = (cid:80) ni
Revised: Then, we calculate the gradient of the RL policy parameter as ∇ θ ˆ R θ = (cid:80) ni

Original: =1 (cid:0)(cid:80) v ∈V ∇ θ log p θ ( x v,i | I u,i ) (cid:1) (cid:16)(cid:80) v ∈V w ix v,i ,v (cid:17) .
Revised: =1 (cid:0)(cid:80) v ∈V ∇ θ log p θ ( x v,i | I u,i ) (cid:1) (cid:16)(cid:80) v ∈V w ix v,i ,v (cid:17) .

Original: Then, we update the parameter θ by θ = θ + α ∇ θ ˆ R θ , where α controls the training step size.
Revised: Then, we update the parameter θ by θ = θ + α ∇ θ ˆ R θ , where α controls the training step size.

Original: By changing the temperature t for softmax, we are able to balance exploration and exploitation.
Revised: By changing the temperature t for softmax, we are able to balance exploration and exploitation.

Original: Specifically, at the beginning of the policy training, we can set a high temperature to encourage the
Revised: Specifically, at the beginning of the policy training, we can set a high temperature to encourage the

Original: RL model to explore more aggressively, instead of sticking to the expert’s decisions.
Revised: RL model to explore more aggressively, instead of sticking to the expert’s decisions.

Original: As the RL model performance continuously improves, we can reduce the temperature in order to make the RL agent more aware of the downstream switching operation.
Revised: As the RL model performance continuously improves, we can reduce the temperature in order to make the RL agent more aware of the downstream switching operation.

Original: The training process is performed offline as in the existing RL-based optimizers (Alomrani et al., 2021; Du et al., 2022) and described in
Revised: The training process is performed offline as in the existing RL-based optimizers (Alomrani et al., 2021; Du et al., 2022) and described in

Original: Algorithm 3.
Revised: Algorithm 3.

Original:  E XPERIMENT 6. S ETUP
Revised:  E XPERIMENT 6. S ETUP

Original: To validate the effectiveness of LOMAR , we conduct experiments based on the movie recommendation application.
Revised: To validate the effectiveness of LOMAR , we conduct experiments based on the movie recommendation application.

Original: Specifically, when an user (i.e., online item v ) arrives, we recommend a movie (i.e., offline item u ) to this user and receive a reward based on the user-movie preference information.
Revised: Specifically, when an user (i.e., online item v ) arrives, we recommend a movie (i.e., offline item u ) to this user and receive a reward based on the user-movie preference information.

Original: We choose the MovieLens dataset (Harper & Konstan, 2015), which provides a total of 3952 movies, 6040 users and 100209 ratings.
Revised: We choose the MovieLens dataset (Harper & Konstan, 2015), which provides a total of 3952 movies, 6040 users and 100209 ratings.

Original: We preprocess the dataset to sample movies and users randomly from the dataset to generate subgraphs, following the same steps as used by (Dickerson et al., 2019) and (Alomrani et al., 2021).
Revised: We preprocess the dataset to sample movies and users randomly from the dataset to generate subgraphs, following the same steps as used by (Dickerson et al., 2019) and (Alomrani et al., 2021).

Original: In testing dataset, we empirically evaluate each algorithm using average reward (AVG) and competitive ratio (CR, against OPT), which represents the average performance and worst case performance, respectively.
Revised: In testing dataset, we empirically evaluate each algorithm using average reward (AVG) and competitive ratio (CR, against OPT), which represents the average performance and worst case performance, respectively.

Original: Thus, the value of CR is the empirically worst reward ratio in the testing dataset.
Revised: Thus, the value of CR is the empirically worst reward ratio in the testing dataset.

Original: For fair comparison, all the experimental settings like capacity c u follow those used in (Alomrani et al., 2021).
Revised: For fair comparison, all the experimental settings like capacity c u follow those used in (Alomrani et al., 2021).

Original: More details about the problem setup and training details are deferred to Appendix A.
Revised: More details about the problem setup and training details are deferred to Appendix A.

Original: Baseline Algorithms.
Revised: Baseline Algorithms.

Original: We consider the following baselines.
Revised: We consider the following baselines.

Original: All the RL policies are trained offline with the same architecture and applicable hyperparameters.
Revised: All the RL policies are trained offline with the same architecture and applicable hyperparameters.

Original: • OPT: The offline optimal oracle has the complete information about the bipartite graph.
Revised: • OPT: The offline optimal oracle has the complete information about the bipartite graph.

Original: We use the Gurobi optimizer to find the optimal offline solution.
Revised: We use the Gurobi optimizer to find the optimal offline solution.

Original: Greedy: At each step, Greedy selects the available offline item with highest weight.
Revised: Greedy: At each step, Greedy selects the available offline item with highest weight.

Original: • DRL: It uses the same architecture as in LOMAR , but does not consider online switching for training or inference.
Revised: • DRL: It uses the same architecture as in LOMAR , but does not consider online switching for training or inference.

Original: That is, the RL model is both trained and tested with ρ = 0 .
Revised: That is, the RL model is both trained and tested with ρ = 0 .

Original: More specifically, our RL architecture has 3 fully connected layers, each with 100 hidden nodes.
Revised: More specifically, our RL architecture has 3 fully connected layers, each with 100 hidden nodes.

Original: • DRL-OS
Revised: • DRL-OS

Original: (DRL-OnlineSwitching):
Revised: (DRL-OnlineSwitching):

Original: We apply online switching to the same RL policy used by DRL during inference.
Revised: We apply online switching to the same RL policy used by DRL during inference.

Original: That is, the RL model is trained with ρ = 0 , but tested with a different ρ > 0 .
Revised: That is, the RL model is trained with ρ = 0 , but tested with a different ρ > 0 .

Original: The choice of baselines include all those considered in (Alomrani et al., 2021).
Revised: The choice of baselines include all those considered in (Alomrani et al., 2021).

Original: In the no-free-disposal setting, the best competitive ratio is 0 in general adversarial cases (Mehta, 2013).
Revised: In the no-free-disposal setting, the best competitive ratio is 0 in general adversarial cases (Mehta, 2013).

Original: Here, we use
Revised: Here, we use

Original: Greedy as the expert algorithm, because the recent study (Alomrani et al., 2021) has shown that
Revised: Greedy as the expert algorithm, because the recent study (Alomrani et al., 2021) has shown that

Original: Greedy performs better than other alternatives and is a strong baseline.
Revised: Greedy performs better than other alternatives and is a strong baseline.

Original: 6. R ESULTS
Revised: 6. R ESULTS

Original: [Table]
Revised: [Table]

Original: We show the comparison of LOMAR with baseline algorithms in Table 1.
Revised: We show the comparison of LOMAR with baseline algorithms in Table 1.

Original: First, we see that DRL has the highest average reward
Revised: First, we see that DRL has the highest average reward

Original: [Table]
Revised: [Table]

Original: but its empirical competitive ratio is the lowest.
Revised: but its empirical competitive ratio is the lowest.

Original: The expert algorithm
Revised: The expert algorithm

Original: Greedy is fairly robust, but has a lower average award than RL-based policies.
Revised: Greedy is fairly robust, but has a lower average award than RL-based policies.

Original: Second, DRL-OS can improve the competitive ratio compared to DRL.
Revised: Second, DRL-OS can improve the competitive ratio compared to DRL.

Original: But, its RL policy is trained alone without being aware of the online switching.
Revised: But, its RL policy is trained alone without being aware of the online switching.

Original: Thus, by making the RL policy aware of online switching, LOMAR can improve the average reward compared to DRL-OS.
Revised: Thus, by making the RL policy aware of online switching, LOMAR can improve the average reward compared to DRL-OS.

Original: Specifically, by training LOMAR using the same ρ as testing it, we can obtain both the highest average cost and the highest competitive ratio.
Revised: Specifically, by training LOMAR using the same ρ as testing it, we can obtain both the highest average cost and the highest competitive ratio.

Original: One exception is the minor decrease of competitive ratio when ρ = 0 . 8 for testing.
Revised: One exception is the minor decrease of competitive ratio when ρ = 0 . 8 for testing.

Original: This is likely due to the dataset and a few hard instances can affect the empirical competitive ratio, which also explains why the empirical competitive ratio is not necessarily monotonically increasing in the ρ ∈ [0 , 1] .
Revised: This is likely due to the dataset and a few hard instances can affect the empirical competitive ratio, which also explains why the empirical competitive ratio is not necessarily monotonically increasing in the ρ ∈ [0 , 1] .

Original: Nonetheless, unlike DRL that may only work well empirically without guarantees, LOMAR offers provable robustness guarantees while exploiting the power of RL to improve the average performance.
Revised: Nonetheless, unlike DRL that may only work well empirically without guarantees, LOMAR offers provable robustness guarantees while exploiting the power of RL to improve the average performance.

Original: The boxplots in Fig.
Revised: The boxplots in Fig.

Original: 1 visualizes the reward ratio distribution of LOMAR , which further validates the importance of switching-aware training.
Revised: 1 visualizes the reward ratio distribution of LOMAR , which further validates the importance of switching-aware training.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: L R e w a r d
Revised: L R e w a r d

Original: R a t i o     (c) ρ = 0 .0.8 0.9 1.0 1.1 1.2 1.3 DRL-OS / GREEDY Reward Ratio 0.
Revised: R a t i o     (c) ρ = 0 .0.8 0.9 1.0 1.1 1.2 1.3 DRL-OS / GREEDY Reward Ratio 0.

Original: D R L - O S / D R L R e w a r d
Revised: D R L - O S / D R L R e w a r d

Original: R a t i o     (d) ρ = 0 .
Revised: R a t i o     (d) ρ = 0 .

Original: To show the effect of switching with different ρ , we calculate the bi-competitive reward ratios.
Revised: To show the effect of switching with different ρ , we calculate the bi-competitive reward ratios.

Original: Specifically, for each problem instance, the bi-competitive ratio compares the actual reward against those of Greedy and RL model, respectively.
Revised: Specifically, for each problem instance, the bi-competitive ratio compares the actual reward against those of Greedy and RL model, respectively.

Original: To highlight the effect of online switching, we focus on DRL-OS (i.e., training the RL with ρ = 0 ) whose training process of RL model is not affected by ρ , because the RL model trained with ρ > 0 in LOMAR does not necessarily perform well on its own and the reward ratio of LOMAR to its RL model is not meaningful.
Revised: To highlight the effect of online switching, we focus on DRL-OS (i.e., training the RL with ρ = 0 ) whose training process of RL model is not affected by ρ , because the RL model trained with ρ > 0 in LOMAR does not necessarily perform well on its own and the reward ratio of LOMAR to its RL model is not meaningful.

Original: The histogram of the bi-competitive ratios are visualized in Fig.
Revised: The histogram of the bi-competitive ratios are visualized in Fig.

Original: When ρ = 0 , the ratio of DRL-OS/ DRL is alwaysunsurprisingly, since DRL-OS are essentially the same as DRL in this case (i.e., both trained and tested with ρ = 0 ).
Revised: When ρ = 0 , the ratio of DRL-OS/ DRL is alwaysunsurprisingly, since DRL-OS are essentially the same as DRL in this case (i.e., both trained and tested with ρ = 0 ).

Original: With a large ρ (e.g. 0.8) for testing, the reward ratios of DRL-OS/Greedy for most samples are around 1, which means the robustness is achieved, as proven by our theoretical analysis.
Revised: With a large ρ (e.g. 0.8) for testing, the reward ratios of DRL-OS/Greedy for most samples are around 1, which means the robustness is achieved, as proven by our theoretical analysis.

Original: But on the other hand, DRL-OS has limited flexibility and can less exploit the good average performance of DRL.
Revised: But on the other hand, DRL-OS has limited flexibility and can less exploit the good average performance of DRL.

Original: Thus, the hyperparameter ρ ∈ [0 , 1] governs the tradeoff between average performance and robustness relative to the expert and, like other hyperparameters, can be tuned to maximize the average performance subject to the robustness requirement.
Revised: Thus, the hyperparameter ρ ∈ [0 , 1] governs the tradeoff between average performance and robustness relative to the expert and, like other hyperparameters, can be tuned to maximize the average performance subject to the robustness requirement.

Original: We also consider a crowdsourcing application, as provided by the gMission dataset (Chen et al., 2014).
Revised: We also consider a crowdsourcing application, as provided by the gMission dataset (Chen et al., 2014).

Original: Additional results for LOMAR and baselines in gMission are deferred to Appendix A.
Revised: Additional results for LOMAR and baselines in gMission are deferred to Appendix A.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we propose LOMAR to achieve both good average-case and good worst-case performance for edge-weighted online bipartite matching.
Revised: In this paper, we propose LOMAR to achieve both good average-case and good worst-case performance for edge-weighted online bipartite matching.

Original: LOMAR includes a novel online switching operation to decide whether to follow the expert’s decision or the RL decision for each online item arrival.
Revised: LOMAR includes a novel online switching operation to decide whether to follow the expert’s decision or the RL decision for each online item arrival.

Original: We prove that for any ρ ∈ [0 , 1] , LOMAR is ρ -competitive against any expert online algorithms, which directly translates a bounded competitive ratio against OPT if the expert algorithm itself has one.
Revised: We prove that for any ρ ∈ [0 , 1] , LOMAR is ρ -competitive against any expert online algorithms, which directly translates a bounded competitive ratio against OPT if the expert algorithm itself has one.

Original: We also train the RL policy by explicitly considering the online switching operation so as to improve the average performance.
Revised: We also train the RL policy by explicitly considering the online switching operation so as to improve the average performance.

Original: Finally, we run empirical experiments to validate LOMAR .
Revised: Finally, we run empirical experiments to validate LOMAR .

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: The details of proving Theorem 4.1 are included in Appendix C.
Revised: The details of proving Theorem 4.1 are included in Appendix C.

Original: The experimental codes and settings are based on the open-sourced resources in Alomrani et al.
Revised: The experimental codes and settings are based on the open-sourced resources in Alomrani et al.

Original: The implementation of
Revised: The implementation of

Original: LOMAR mainly includes adding the switching condition for training and testing based on the standard
Revised: LOMAR mainly includes adding the switching condition for training and testing based on the standard

Original: DRL-based algorithm Alomrani et al.
Revised: DRL-based algorithm Alomrani et al.

Original: and will be released upon publication.
Revised: and will be released upon publication.

Original: A PPENDIX
Revised: A PPENDIX

Original: In the appendix, we show the experimental setup and additional results (Appendix A), algorithm details for the free-disposal setting (Appendix B), and finally the proof of Theorem 4.1 (Appendix C).
Revised: In the appendix, we show the experimental setup and additional results (Appendix A), algorithm details for the free-disposal setting (Appendix B), and finally the proof of Theorem 4.1 (Appendix C).

Original: A E XPERIMENTAL S ETTINGS AND A DDITIONAL R ESULTS
Revised: A E XPERIMENTAL S ETTINGS AND A DDITIONAL R ESULTS

Original: Our implementation of all the considered algorithms, including LOMAR , is based on the source codes provided by Alomrani et al.
Revised: Our implementation of all the considered algorithms, including LOMAR , is based on the source codes provided by Alomrani et al.

Original: (2021), which includes codes for training the RL model, data pre-proposing and performance evaluation.
Revised: (2021), which includes codes for training the RL model, data pre-proposing and performance evaluation.

Original: We conduct experiments on two real-world datasets:
Revised: We conduct experiments on two real-world datasets:

Original: MovieLens Harper & Konstan (2015) and gMission Chen et al.
Revised: MovieLens Harper & Konstan (2015) and gMission Chen et al.

Original: A. M OVIE L ENS A.1. S ETUP AND T RAINING
Revised: A. M OVIE L ENS A.1. S ETUP AND T RAINING

Original: We first sample u 0 movies from the original MovieLens dataset Harper & Konstan (2015).
Revised: We first sample u 0 movies from the original MovieLens dataset Harper & Konstan (2015).

Original: We then sample v 0 users and make sure each user can get at least one movie; otherwise, we remove the users that have no matched movies, and resample new users.
Revised: We then sample v 0 users and make sure each user can get at least one movie; otherwise, we remove the users that have no matched movies, and resample new users.

Original: After getting the topology graph, we use Gurobi to find the optimal matching decision.
Revised: After getting the topology graph, we use Gurobi to find the optimal matching decision.

Original: In our experiment, we set u 0 = 10 and v 0 = 60 to generate the training and testing datasets.
Revised: In our experiment, we set u 0 = 10 and v 0 = 60 to generate the training and testing datasets.

Original: The number of graph instances in the training and testing datasets are 20000 and 1000, respectively.
Revised: The number of graph instances in the training and testing datasets are 20000 and 1000, respectively.

Original: For the sake of reproducibility and fair comparision, our settings follows the same setup of Alomrani et al.
Revised: For the sake of reproducibility and fair comparision, our settings follows the same setup of Alomrani et al.

Original: In particular, the general movie recommendation problem belongs to online submodular optimization, but it can actually be equivalently mapped to edge-weighted online bipartite matching with no free disposal under the setting considered in Alomrani et al. (2021).
Revised: In particular, the general movie recommendation problem belongs to online submodular optimization, but it can actually be equivalently mapped to edge-weighted online bipartite matching with no free disposal under the setting considered in Alomrani et al. (2021).

Original: So by default, the capacity c u for each offline node is set as 1 and w u, max = 5 .
Revised: So by default, the capacity c u for each offline node is set as 1 and w u, max = 5 .

Original: While LOMAR can use any RL architecture, we follow the design of inv-ff-hist proposed by Alomrani et al.
Revised: While LOMAR can use any RL architecture, we follow the design of inv-ff-hist proposed by Alomrani et al.

Original: (2021), which empirically demonstrates the best performance among all the considered architectures.
Revised: (2021), which empirically demonstrates the best performance among all the considered architectures.

Original: The input to our considered RL model is the edge weights w uv revealed by the online items plus some historical information, which includes: Mean and variances of each offline node’s weights; Average degree of each offline nodes; Normalized step size; Percentage of offline nodes connected to the current node; Statistical information of these already matched nodes’ weights (maximum, minimum, mean and variance); Ratio of matched offline node; Ratio of skips up to now; Normalized reward with respect to the offline node number.
Revised: The input to our considered RL model is the edge weights w uv revealed by the online items plus some historical information, which includes: Mean and variances of each offline node’s weights; Average degree of each offline nodes; Normalized step size; Percentage of offline nodes connected to the current node; Statistical information of these already matched nodes’ weights (maximum, minimum, mean and variance); Ratio of matched offline node; Ratio of skips up to now; Normalized reward with respect to the offline node number.

Original: For more details of the historical information, readers are referred to Table 1 in Alomrani et al.
Revised: For more details of the historical information, readers are referred to Table 1 in Alomrani et al.

Original: [Figure]
Revised: [Figure]

Original: For applicable algorithms (i.e., DRL, DRL-OS, and
Revised: For applicable algorithms (i.e., DRL, DRL-OS, and

Original: LOMAR ), we train the RL model for 300 epochs in the training dataset with a batch size of 100.
Revised: LOMAR ), we train the RL model for 300 epochs in the training dataset with a batch size of 100.

Original: In LOMAR , the parameter B = 0 is used to follow the strict definition of competitive ratio.
Revised: In LOMAR , the parameter B = 0 is used to follow the strict definition of competitive ratio.

Original: We test the algorithms on the testing dataset to obtain the average reward and the worst-case competitive ratio empirically.
Revised: We test the algorithms on the testing dataset to obtain the average reward and the worst-case competitive ratio empirically.

Original: By setting ρ = 0 for training, LOMAR is equivalent to the vanilla inv-ff-hist RL model (i.e., DRL) used in Alomrani et al.
Revised: By setting ρ = 0 for training, LOMAR is equivalent to the vanilla inv-ff-hist RL model (i.e., DRL) used in Alomrani et al.

Original: Using the same problem setup, we can reproduce the same results shown in Alomrani et al.
Revised: Using the same problem setup, we can reproduce the same results shown in Alomrani et al.

Original: (2021), which reaffirms the correctness of our data generation and training process.
Revised: (2021), which reaffirms the correctness of our data generation and training process.

Original: Additionally, training the RL model in LOMAR usually takes less than 8 hours on a shared research cluster with one NVIDIA K80 GPU, which is almost the same as the training the model for DRL in a standalone manner (i.e., setting ρ = 0 without considering online switching).
Revised: Additionally, training the RL model in LOMAR usually takes less than 8 hours on a shared research cluster with one NVIDIA K80 GPU, which is almost the same as the training the model for DRL in a standalone manner (i.e., setting ρ = 0 without considering online switching).

Original: A.1. A DDITIONAL R ESULTS
Revised: A.1. A DDITIONAL R ESULTS

Original: In Table 1, we have empirically demonstrated that LOMAR achieves the best tradeoff between the average reward and competitive ratio.
Revised: In Table 1, we have empirically demonstrated that LOMAR achieves the best tradeoff between the average reward and competitive ratio.

Original: In Fig 3, we further demonstrate that LOMAR not only achieves a better worst-case competitive ratio (at 100.0%).
Revised: In Fig 3, we further demonstrate that LOMAR not only achieves a better worst-case competitive ratio (at 100.0%).

Original: The tail reward ratio of LOMAR is also good compared to the baseline algorithms.
Revised: The tail reward ratio of LOMAR is also good compared to the baseline algorithms.

Original: Specifically, we show the percentile of reward ratios (compared to the optimal offline algorithm) — the 100% means the worst-case empirical reward ratio (i.e., competitive ratio).
Revised: Specifically, we show the percentile of reward ratios (compared to the optimal offline algorithm) — the 100% means the worst-case empirical reward ratio (i.e., competitive ratio).

Original: We see that DRL has a bad high-percentile reward ratio and lacks performance robustness, although its lower-percentile cost ratio is better.
Revised: We see that DRL has a bad high-percentile reward ratio and lacks performance robustness, although its lower-percentile cost ratio is better.

Original: This is consistent with the good average performance of LOMAR .
Revised: This is consistent with the good average performance of LOMAR .

Original: Because of online switching, both DRL-OS and LOMAR achieve better robustness, and LOMAR is even better due to its awareness of the online switching operation during its training process.
Revised: Because of online switching, both DRL-OS and LOMAR achieve better robustness, and LOMAR is even better due to its awareness of the online switching operation during its training process.

Original: The expert Greedy has a fairly stable competitive ratio, showing its good robustness.
Revised: The expert Greedy has a fairly stable competitive ratio, showing its good robustness.

Original: But, it can be outperformed by other algorithms when we look at lower-percentile reward ratio.
Revised: But, it can be outperformed by other algorithms when we look at lower-percentile reward ratio.

Original: A. G M ISSION
Revised: A.1. R ESULTS FOR ANOTHER EXPERT ALGORITHM

Revised: Optimally competitive expert algorithms have been developed under the assumptions of random oder and/or i.i.d. rewards of different online items. In particular, by considering the random order setting,

Revised: OSM (online secretary matching) has the optimal competitive ratio of 1 /e (Kesselheim et al., 2013).

Revised: Note that the competitive ratio for OSM is average over the random order of online items, while the rewards can be adversarially chosen.

Revised: We show the empirical results in Fig.

Revised: As OSM skips the first |V| /e online items, it actually does not perform (in terms of the empirical worst-case cost ratio) as well as the default expert Greedy in our experiments despite its guaranteed competitive ratio against

Revised: OPT.

Revised: That said, we still observe the same trend as using Greedy for the expert: by tuning ρ ∈ [0 , 1] ,

Revised: LOMAR achieves a good average performance while guaranteeing the competitiveness against the expert OSM (and against OPT as OSM itself is optimally competitive against OPT).

Revised: [Figure]

Revised: Fig.

Revised: 4 shows the empirical results in our testing dataset, which does not strictly satisfy the random order assumption required by OSM.

Revised: Next, to satisfy the random order assumption, we select a typical problem instance and randomly vary the arrival orders of online items.

Revised: We show the cost ratio averaged over the random arrival order in Table 2.

Revised: Specifically, we calculate each cost ratio bydifferent random orders, and repeat this process 100 times.

Revised: We show the mean and stand deviation of the average cost ratios (each averaged over 100 different random orders).

Revised: We see that LOMAR improves the average cost ratio compared to OSM under the random order assumption.

Revised: While DRL has a better average cost for this particular instance, it does not provide any guaranteed worst-case robustness as LOMAR .

Revised: [Table]

Revised: A. G M ISSION

Original: The gMission dataset Chen et al.
Revised: The gMission dataset Chen et al.

Original: (2014) considers a crowdsourcing application, where the goal is to assign the tasks (online items) to workers (offline items).
Revised: (2014) considers a crowdsourcing application, where the goal is to assign the tasks (online items) to workers (offline items).

Original: The edge weight between a certain online task and each worker can be calculated by the product of the task reward and the worker’s success probability, which is determined by the physical location of workers and the type of tasks.
Revised: The edge weight between a certain online task and each worker can be calculated by the product of the task reward and the worker’s success probability, which is determined by the physical location of workers and the type of tasks.

Original: Our goal is to maximize the total reward given the capacity of each worker, which perfectly fits into our formulation in Eqn.
Revised: Our goal is to maximize the total reward given the capacity of each worker, which perfectly fits into our formulation in Eqn.

Original: equation 1.
Revised: equation 1.

Original: We use the same data processing and RL architecture design as introduced in Section A.1.1.
Revised: We use the same data processing and RL architecture design as introduced in Section A.1.1.

Original: We train
Revised: We train

Original: LOMAR with different ρ in the gMission dataset by setting u 0 = 10 , v 0 = 60 , w u, max = 1 .
Revised: LOMAR with different ρ in the gMission dataset by setting u 0 = 10 , v 0 = 60 , w u, max = 1 .

Original: Again, we use Greedy as the expert, which is an empirically strong baseline algorithm as shown in Alomrani et al.
Revised: Again, we use Greedy as the expert, which is an empirically strong baseline algorithm as shown in Alomrani et al.

Original: Our results are all consistent with those presented in Alomrani et al.
Revised: Our results are all consistent with those presented in Alomrani et al.

Original: A.2. T ESTING ON 10 ×
Revised: A.2. T ESTING ON 10 ×

Original: In our the first result, we generate a testing dataset with u 0 = 10 and v 0 = 60 , which is the same setting as our training dataset.
Revised: In our the first result, we generate a testing dataset with u 0 = 10 and v 0 = 60 , which is the same setting as our training dataset.

Original: In other words, the training and testing datasets have similar distributions.
Revised: In other words, the training and testing datasets have similar distributions.

Original: Specifically, Greedy’s average reward and competitive ratio are 4.508 and 0.432, while these two values for DRL are 5.819 and 0.604, respectively.
Revised: Specifically, Greedy’s average reward and competitive ratio are 4.508 and 0.432, while these two values for DRL are 5.819 and 0.604, respectively.

Original: Thus, DRL performs outperforms
Revised: Thus, DRL performs outperforms

Original: Greedy in both average performance and the worst-case performance.
Revised: Greedy in both average performance and the worst-case performance.

Original: [Table]
Revised: [Table]

Original: Next [Table] we show the results for LOMAR and DRL-OS under different ρ ∈ [0 , 1] in Table 2.
Revised: Next, we show the results for LOMAR and DRL-OS under different ρ ∈ [0 , 1] in Table 3.

Original: In general, by setting a larger ρ for inference, both LOMAR and DRL-OS are closer to the expert algorithm Greedy, because there is less freedom for the RL decisions.
Revised: In general, by setting a larger ρ for inference, both LOMAR and DRL-OS are closer to the expert algorithm Greedy, because there is less freedom for the RL decisions.

Original: As a result, when ρ increases for inference, the average rewards of both DRL-OS and LOMAR decrease, although they have guaranteed robustness whereas DRL does not.
Revised: As a result, when ρ increases for inference, the average rewards of both DRL-OS and LOMAR decrease, although they have guaranteed robustness whereas DRL does not.

Original: Moreover, by training the RL model with explicit awareness of online switching, LOMAR can have a higher average cost than DRL-OS, which reconfirms the benefits of training the RL model by considering its downstream operation.
Revised: Moreover, by training the RL model with explicit awareness of online switching, LOMAR can have a higher average cost than DRL-OS, which reconfirms the benefits of training the RL model by considering its downstream operation.

Original: Interestingly, by setting ρ identical for both training and testing, the average reward may not always be the highest for LOMAR .
Revised: Interestingly, by setting ρ identical for both training and testing, the average reward may not always be the highest for LOMAR .

Original: This is partially because of the empirical testing dataset.
Revised: This is partially because of the empirical testing dataset.

Original: Another reason is that, in this test, DRL alone already performs the best (both on average and in the worst case).
Revised: Another reason is that, in this test, DRL alone already performs the best (both on average and in the worst case).

Original: Hence, by setting a smaller ρ for inference, LOMAR works better empirically though it is trained under a different ρ .
Revised: Hence, by setting a smaller ρ for inference, LOMAR works better empirically though it is trained under a different ρ .

Original: Nonetheless, this does not void the benefits of guaranteed robustness in LOMAR .
Revised: Nonetheless, this does not void the benefits of guaranteed robustness in LOMAR .

Original: The empirically better performance of
Revised: The empirically better performance of

Original: DRL lacks guarantees, which we show as follows.
Revised: DRL lacks guarantees, which we show as follows.

Original: A.2. T ESTING ON 100 ×
Revised: A.2. T ESTING ON 100 ×

Original: In our second test [Table] we consider an opposite case compared to the first one.
Revised: In our second test, we consider an opposite case compared to the first one.

Original: We generate a testing dataset with u 0 = 100 and v 0 = 100 , which is different from the training dataset setting.
Revised: We generate a testing dataset with u 0 = 100 and v 0 = 100 , which is different from the training dataset setting.

Original: As a result, the training and testing datasets have very different distributions, making DRL perform very badly.
Revised: As a result, the training and testing datasets have very different distributions, making DRL perform very badly.

Original: Specifically, Greedy’s average reward and competitive ratio are 40.830 and 0.824, and these two values for DRL are 32.938 and 0.576, respectively.
Revised: Specifically, Greedy’s average reward and competitive ratio are 40.830 and 0.824, and these two values for DRL are 32.938 and 0.576, respectively.

Original: DRL has an even lower average reward than
Revised: DRL has an even lower average reward than

Original: Greedy, showing its lack of performance robustness
Revised: Greedy, showing its lack of performance robustness.

Original: [Table] We show the results for LOMAR and DRL-OS under different ρ ∈ [0 , 1] in Table 3.
Revised: We show the results for LOMAR and DRL-OS under different ρ ∈ [0 , 1] in Table 4.

Original: In general, by setting a larger ρ for inference, both LOMAR and DRL-OS are closer to the expert algorithm Greedy.
Revised: In general, by setting a larger ρ for inference, both LOMAR and DRL-OS are closer to the expert algorithm Greedy.

Original: As Greedy works empirically much better than DRL in terms of the average performance and the worst-case performance, both LOMAR and DRL-OS have better performances when we increase ρ to let Greedy safeguard the RL decisions more aggressively.
Revised: As Greedy works empirically much better than DRL in terms of the average performance and the worst-case performance, both LOMAR and DRL-OS have better performances when we increase ρ to let Greedy safeguard the RL decisions more aggressively.

Original: Moreover, by training the RL model with explicit awareness of online switching, LOMAR can have a higher average cost than DRL-OS, which further demonstrates the benefits of training the RL model by considering its downstream operation.
Revised: Moreover, by training the RL model with explicit awareness of online switching, LOMAR can have a higher average cost than DRL-OS, which further demonstrates the benefits of training the RL model by considering its downstream operation.

Original: Also, interestingly, by setting ρ identical for both training and testing, the average reward may not be the highest for LOMAR , partially because of the empirical testing dataset.
Revised: Also, interestingly, by setting ρ identical for both training and testing, the average reward may not be the highest for LOMAR , partially because of the empirical testing dataset.

Original: Another reason is that, in this test, DRL alone already performs very badly (both on average and in the worst case)  due to the significant training-testing distributional discrepancy.
Revised: Another reason is that, in this test, DRL alone already performs very badly (both on average and in the worst case) [Table] due to the significant training-testing distributional discrepancy.

Original: Hence, by setting a higher ρ , LOMAR works better empirically though it is tested under a different ρ .
Revised: Hence, by setting a higher ρ [Table] LOMAR works better empirically though it is tested under a different ρ .

Original: An exception is when testing LOMAR with ρ = 0 .
Revised: An exception is when testing LOMAR with ρ = 0 .

Original: 9 : setting ρ = 0 .
Revised: 9 : setting ρ = 0 .

Original: 8 for training makes LOMAR perform slightly better in terms of the average performance and worst-case performance, respectively.
Revised: 8 for training makes LOMAR perform slightly better in terms of the average performance and worst-case performance, respectively.

Original: But, setting ρ = 0 .
Revised: But, setting ρ = 0 .

Original: 9 for training still brings benefits to LOMAR compared to DRL-OS that does not consider the downstream online switching operation. 
Revised: 9 for training still brings benefits to LOMAR compared to DRL-OS that does not consider the downstream online switching operation. To sum up

Original: To sum up , our experimental results under different settings demonstrate: LOMAR ’s empirical improvement in terms of the average reward compared to DRL-OS; the improved competitive ratio of LOMAR and DRL-OS compared to DRL, especially when the training-testing distributions differ significantly; and the improved average reward of LOMAR compared to Greedy when RL is good.
Revised: [Table] our experimental results under different settings demonstrate: LOMAR ’s empirical improvement in terms of the average reward compared to DRL-OS; the improved competitive ratio of LOMAR and DRL-OS compared to DRL, especially when the training-testing distributions differ significantly; and the improved average reward of LOMAR compared to Greedy when RL is good.

Original: Therefore,
Revised: Therefore,

Original: LOMAR can exploit the power of RL while provably guaranteeing the performance robustness.
Revised: LOMAR can exploit the power of RL while provably guaranteeing the performance robustness.

Original: B F REE D ISPOSAL
Revised: B F REE D ISPOSAL

Original: The offline version of bipartite matching with free disposal can be expressed as:
Revised: The offline version of bipartite matching with free disposal can be expressed as:

Original: [Equation]

Original: Disposal
Revised: With Free Disposal:

Original: [Equation]
Revised: [Equation]

Original: where V u = { v ∈ V | x uv = 1 } is the set of online items matched to u ∈ U and the objective max S∈V u , |S|≤
Revised: where V u = { v ∈ V | x uv = 1 } is the set of online items matched to u ∈ U and the objective max S∈V u , |S|≤

Original: c u (cid:80) v ∈S x uv w uv indicates that only up to top c u rewards are counted for u ∈ U .
Revised: c u (cid:80) v ∈S x uv w uv indicates that only up to top c u rewards are counted for u ∈ U .

Original: In the free-disposal setting, it is more challenging to design the switching conditions to guarantee the robustness.
Revised: In the free-disposal setting, it is more challenging to design the switching conditions to guarantee the robustness.

Original: The reason is the additional flexibility allowed for matching decisions — each offline item u ∈ U is allowed to be matched more than c u times although only up to top c u rewards actually count
Revised: The reason is the additional flexibility allowed for matching decisions — each offline item u ∈ U is allowed to be matched more than c u times although only up to top c u rewards actually count

Original: Mehta (2013); Fahrbach et al. (2020).
Revised: Mehta (2013); Fahrbach et al. (2020).

Original: For example, even though LOMAR and the expert assign the 
Revised: For example, even though LOMAR and the expert assign the same number of online items to an offline item u ∈ U and LOMAR is better than the expert at a certain step, future high-reward online items can still be assigned to u ∈ U , increasing the expert’s total reward or even equalizing the rewards of LOMAR and the expert (i.e., high-reward future online items become the top c u items for u ∈ U for both LOMAR and the expert). Thus, the temporarily “higher” rewards received by LOMAR must be hedged against such future uncertainties. Before designing our switching condition for the free-disposal setting, we first define the set containing the top c u online items for u ∈ U after assignment of v : [Equation]

Revised: where V u,v is the set of all online items matched to u ∈ U so far after assignment of item v ∈ V .

Revised: When there are fewer than c u items in V uv , we will simply add null items with reward 0 to E u,v such that |E u,v |

Revised: = c u .

Revised: We also sort the online items denoted as e u,i , for i = 1 , · · · , c u , contained in E u,v according to their weights in an increasing order such that w u,e u, 1 ≤ · · · ≤ w u,e u,cu .

Revised: Similarly, we define the same top-

Revised: c u item set for the expert algorithm π by adding the superscript π .

Revised: Next, we define the following value which indicates the maximum possible additional reward for the expert algorithm π if LOMAR simply switches to the expert and follows it for all the future steps

Original: 1: Initialization: The actual set of items matched to u ∈ U is V u,v after sequentially-arriving item v ’s assignment with V u, 0 = ∅ , the actual remaining capacity is b u = c u for u ∈ U , and the actual cumulative reward is R 

Revised: 1: Initialization: The actual set of items matched to u ∈ U is V u,v after sequentially-arriving item v ’s assignment with V u, 0 = ∅ , the actual remaining capacity is b u = c u for u ∈ U , and the actual cumulative reward is R

Original: = (cid:80) u ∈U f u
Revised: = (cid:80) u ∈U f u

Original: ( V u, 0 )
Revised: ( V u, 0 )

Original: = 0 .
Revised: = 0 .

Original: The same notations apply to the expert algorithm π by adding the superscript π .
Revised: The same notations apply to the expert algorithm π by adding the superscript π .

Original: Competitive ratio requirement ρ ∈ [0 , 1] and slackness
Revised: Competitive ratio requirement ρ ∈ [0 , 1] and slackness

Original: B ≥ 0 with respect to the expert algorithm π .
Revised: B ≥ 0 with respect to the expert algorithm π .

Original: 2: for v = 1 to |V| do 3:
Revised: 2: for v = 1 to |V| do 3:

Original: Run the algorithm π and match the item v to u ∈ U based on the expert’s decision u = x πv .
Revised: Run the algorithm π and match the item v to u ∈ U based on the expert’s decision u = x πv .

Original: Update the expert’s decision set and reward for offline item u = x πv :
Revised: Update the expert’s decision set and reward for offline item u = x πv :

Original: V πx πv ,v = V πx πv ,v − 1 (cid:83) { v } and f x πv =
Revised: V πx πv ,v = V πx πv ,v − 1 (cid:83) { v } and f x πv =

Original: f x πv ( V πx πv ,v ) .
Revised: f x πv ( V πx πv ,v ) .

Original: Update the expert’s cumulative reward R πv
Revised: Update the expert’s cumulative reward R πv

Original: =
Revised: =

Original: (cid:80) u ∈U f u 6: for u in U do 7:
Revised: (cid:80) u ∈U f u 6: for u in U do 7:

Original: Collect the available history information I u about item u 8:
Revised: Collect the available history information I u about item u 8:

Original: Run the RL model to get score: s u = w uv − h
Revised: Run the RL model to get score: s u = w uv − h

Original: θ ( I u , w uv ) where θ is the network weight 9: end for 10:
Revised: θ ( I u , w uv ) where θ is the network weight 9: end for 10:

Original: Calculate the probability of choosing each item u : {{ ˜ s u } u ∈U } = softmax {{ s u } u ∈U } .
Revised: Calculate the probability of choosing each item u : {{ ˜ s u } u ∈U } = softmax {{ s u } u ∈U } .

Original: Obtain RL decision: ˜ x v = arg max u ∈U (cid:83) { skip } {{ ˜ s u } u ∈U } .
Revised: Obtain RL decision: ˜ x v = arg max u ∈U (cid:83) { skip } {{ ˜ s u } u ∈U } .

Original: Find ∆ f ˜ x v in Eqn.
Revised: Find ∆ f ˜ x v in Eqn.

Original: equation 7 and G (cid:0) ˜ x v , {V u,v − 1 } u ∈U , {V π u,v } u ∈ U (cid:1) in Eqn. equation13: if
Revised: equation 8 and G (cid:0) ˜ x v , {V u,v − 1 } u ∈U , {V π u,v } u ∈ U (cid:1) in Eqn. equation13: if

Original: [Equation]
Revised: [Equation]

Original: G (cid:0) ˜ x v , {V u,v − 1 } u ∈U , {V πu,v } u ∈U (cid:1)(cid:1) − B then 14:
Revised: G (cid:0) ˜ x v , {V u,v − 1 } u ∈U , {V πu,v } u ∈U (cid:1)(cid:1) − B then 14:

Original: Select x v = ˜ x v .
Revised: Select x v = ˜ x v .

Original: //Follow
Revised: //Follow

Original: the ML action 15: else 16:
Revised: the ML action 15: else 16:

Original: Select x v = x πv .
Revised: Select x v = x πv .

Original: //Follow
Revised: //Follow

Original: the expert 17: end if 18:
Revised: the expert 17: end if 18:

Original: Update assignment and reward: V x v ,v = V x v ,v − 1 (cid:83) { v } and R v = (cid:80) u ∈U f u ( V u,v ) 19: end for same number of online items to an offline item u ∈ U and LOMAR is better than the expert at a certain step, future high-reward online items can still be assigned to u ∈ U ,increasing the expert’s total reward or even equalizing the rewards of LOMAR and the expert (i.e., high-reward future online items become the top c u items for u ∈ U for both LOMAR and the expert). Thus, the temporarily “higher” rewards received by LOMAR must be hedged against such future uncertainties. Before designing our switching condition for the free-disposal setting, we first define the set containing the top c u online items for u ∈ U after assignment of v  :
Revised: Update assignment and reward: V x v ,v = V x v ,v − 1 (cid:83) { v } and R v = (cid:80) u ∈U f u ( V u,v ) 19: end for v + 1 , v + 2 , · · · :

Original: [Equation]
Revised: [Equation]

Original: where V u,v is the set of all online items matched to u ∈ U so far after assignment of item v ∈ V . When there are fewer than c u items in V uv , we will simply add null items with reward 0 to E u,v such that |E u,v | = c u .
Revised: 

Original: We also sort the online items denoted as e u,i , for i = 1 , · · · , c u , contained in E u,v according to their weights in an increasing order such that w u,e u, 1 ≤ · · · ≤ w u,e u,cu . Similarly, we define the same top- c u item set for the expert algorithm π by adding the superscript π . Next, we define the following value which indicates the maximum possible additional reward for the expert algorithm π if LOMAR simply switches to the expert and follows it for all the future steps v + 1 , v + 2 , · · · :
Revised: 

Original: [Equation]
Revised: [Equation]

Original:  where e πu,j ∈ E πu ( V πu,v ) , and e u,j ∈ E u (˜ V u,v ) in which ˜ V u,v = V u,v − 1 if ˜ x v ̸ = u and ˜ V u,v =
Revised: (7) where e πu,j ∈ E πu ( V πu,v ) , and e u,j ∈ E u (˜ V u,v ) in which ˜ V u,v = V u,v − 1 if ˜ x v ̸ = u and ˜ V u,v =

Original: V u,v − 1 (cid:83) { v } if ˜ x v = u .
Revised: V u,v − 1 (cid:83) { v } if ˜ x v = u .

Original: The interpretation is as follows.
Revised: The interpretation is as follows.

Original: Suppose that LOMAR follows the RL decision for online item v .
Revised: Suppose that LOMAR follows the RL decision for online item v .

Original: If it has a higher cumulative reward for the j -th item in the top- c u item set E u,v than the expert algorithm π , then the expert can still possibly offset the reward difference w u,e u,j − w u,e πu,j by receiving a highreward future online item that replaces the j -th item for both LOMAR and the expert.
Revised: If it has a higher cumulative reward for the j -th item in the top- c u item set E u,v than the expert algorithm π , then the expert can still possibly offset the reward difference w u,e u,j − w u,e πu,j by receiving a highreward future online item that replaces the j -th item for both LOMAR and the expert.

Original: Nonetheless, in the free-disposal model, the items in the top- c u set E u,v are removed sequentially — the lowest-reward R diff = R v − 1 + w ˜ x v ,v − ρ (cid:16) R πv + (cid:80) u ∈U (cid:0) |V u,v − 1 | − | V πu,v | + I u =˜ x v (cid:1) + · w u, max (cid:17) + B 7: The probability of each action is (cid:110) {
Revised: Nonetheless, in the free-disposal model, the items in the top- c u set E u,v are removed sequentially — the lowest-reward item will be first removed from the sorted set E u,v , followed by the next lowest-reward item, and so on.

Original: ˜ s πu } u ∈U a , ˜ s π skip (cid:111) = {{ 1 } u = x πv , { 0 } u ̸ = x πv , skip } 9: else 10:

Original: The probability of each action is (cid:110) {

Original: ˜ s πu } u ∈U a , ˜ s π skip (cid:111) = {{ 0 } u ∈U a , 1 } 11: end if 12: With online switching, the probabilities of following RL or expert are { ˜ s OS ,

Original: ˜ s πOS } =

Original: softmax { R diff /t

Original: [Equation]

Original: } 13: Calibrated probabilities of choosing an available or are [Equation] item will be first removed from the sorted set E u,v , followed by the next lowest-reward item, and so on. Thus, in order for a high-reward item to replace the i -th item in the sorted set E u,v , the first ( i − 1) items have to be removed first by other high-reward online items.
Revised: Thus, in order for a high-reward item to replace the i -th item in the sorted set E u,v , the first ( i − 1) items have to be removed first by other high-reward online items.

Original: As a result, if LOMAR has a lower reward for the j -th item (for j ≤ i ) in the top- c u item set E u,v than the expert algorithm π , then it will negatively impact the expert’s additional reward gain in the future.
Revised: As a result, if LOMAR has a lower reward for the j -th item (for j ≤ i ) in the top- c u item set E u,v than the expert algorithm π , then it will negatively impact the expert’s additional reward gain in the future.

Original: Therefore, for item u ∈ U we only need to find the highest total reward difference, (cid:16) max i =1 , ··· ,c u (cid:80) ij =1 ( w u,e u,j − w u,e πu,j ) (cid:17) + , that can be offset for the expert algorithm π by considering that i items are replaced by future high-reward online items for i
Revised: Therefore, for item u ∈ U we only need to find the highest total reward difference, (cid:16) max i =1 , ··· ,c u (cid:80) ij =1 ( w u,e u,j − w u,e πu,j ) (cid:17) + , that can be offset for the expert algorithm π by considering that i items are replaced by future high-reward online items for i

Original: = 1 , · · · , c u .
Revised: = 1 , · · · , c u .

Original: If max i =1 , ··· ,c u (cid:80) ij =1 ( w u,e u,j − w u,e πu,j ) is negative (i.e., the expert algorithm cannot possibly gain higher rewards than LOMAR by receiving high-reward online items to replace its existing ones), then we use 0 as the hedging reward.
Revised: If max i =1 , ··· ,c u (cid:80) ij =1 ( w u,e u,j − w u,e πu,j ) is negative (i.e., the expert algorithm cannot possibly gain higher rewards than LOMAR by receiving high-reward online items to replace its existing ones), then we use 0 as the hedging reward.

Original: Finally, by summing up the hedging rewards for all the offline items u ∈ U , we obtain the total hedging reward in Eqn.
Revised: Finally, by summing up the hedging rewards for all the offline items u ∈ U , we obtain the total hedging reward in Eqn.

Original: equation 6.
Revised: equation 7.

Original: Based on this hedging reward, we have the condition (Line 28 in Algorithm 1 for LOMAR to follow the RL decision: R v − 1 + ∆ f ˜ x v ≥ ρ (cid:0) R πv [Equation] G 
Revised: Based on this hedging reward, we have the condition (Line 28 in Algorithm 1 for LOMAR to follow the RL decision: R v − 1 + ∆ f ˜ x v ≥ ρ (cid:0) R πv + G (cid:0)

Original: [Equation] [Equation] [Equation] πu,
Revised: ˜ x v , {V u,v − 1 } u ∈U , {V πu,v } u ∈U (cid:1)(cid:1) − B , where ∆ f ˜ x v defined below is the additional reward that would be obtained by following the RL decision:

Original: [Equation]
Revised: [Equation]

Original:  − B , where ∆ f ˜ x v defined below is the additional reward that would be obtained by following the RL decision:
Revised: R diff = R v − 1 + w ˜ x v ,v − ρ (cid:16) R πv + (cid:80) u ∈U (cid:0) |V u,v − 1 | − | V πu,v | + I u =˜ x v (cid:1) + · w u, max (cid:17) + B 7: The probability of each action is (cid:110) { ˜ s πu } u ∈U a , ˜ s π skip (cid:111) = {{ 1 } u = x πv , { 0 } u ̸ = x πv , skip } 9: else 10:

Original:  [Equation]
Revised: The probability of each action is (cid:110) { ˜ s πu } u ∈U a , ˜ s π skip (cid:111) = {{ 0 } u ∈U a , 1 } 11: end if 12: With online switching, the probabilities of following RL or expert are { ˜ s OS , ˜ s πOS } = softmax { R diff /t, 0 } 13: Calibrated probabilities of choosing an available offline item u or skip are [Equation]

Original: in which f u = f u ( V ′ )
Revised: in which f u = f u ( V ′ )

Original: =
Revised: =

Original: max S∈V ′ , |S|≤ c u (cid:80) v ∈S w uv is the reward function for an offline item u ∈ U in the free-disposal model.
Revised: max S∈V ′ , |S|≤ c u (cid:80) v ∈S w uv is the reward function for an offline item u ∈ U in the free-disposal model.

Original: The condition means that if LOMAR can maintain the competitive ratio ρ against the expert algorithm π by being able to hedge against any future uncertainties even in the worst case, then it can safely follow the RL decision ˜ x v at step v .
Revised: The condition means that if LOMAR can maintain the competitive ratio ρ against the expert algorithm π by being able to hedge against any future uncertainties even in the worst case, then it can safely follow the RL decision ˜ x v at step v .

Original: Training with free disposal.
Revised: Training with free disposal.

Original: The training process for the free-disposal setting is the same as that for the no-free-disposal setting, except for Line 6 of Algorithm 3 in which we need to modify R diff based on the switching condition (i.e., Line 13 of Algorithm 2) for the free-disposal setting.
Revised: The training process for the free-disposal setting is the same as that for the no-free-disposal setting, except for Line 6 of Algorithm 3 in which we need to modify R diff based on the switching condition (i.e., Line 13 of Algorithm 2) for the free-disposal setting.

Original: C P ROOF OF T HEOREM 4. 
Revised: C P ROOF OF T HEOREM 4.

Original: The key idea of proving Theorem 4.1 is to show that there always exist feasible actions (either following the expert or skip) while being able to guarantee the robustness if we follow the switch- ing condition.
Revised: The key idea of proving Theorem 4.1 is to show that there always exist feasible actions (either following the expert or skip) while being able to guarantee the robustness if we follow the switching condition.

Original: Next, we prove Theorem 4.1 for the no-free-disposal and free-disposal settings, respectively.
Revised: Next, we prove Theorem 4.1 for the no-free-disposal and free-disposal settings, respectively.

Original: C. N O F REE D ISPOSAL
Revised: C. N O F REE D ISPOSAL

Original: Denote V u,v as the actual set of items matched to u ∈ U after making decision for v .
Revised: Denote V u,v as the actual set of items matched to u ∈ U after making decision for v .

Original: Denote V πu,v as the expert’s set of items matched to u ∈ U .
Revised: Denote V πu,v as the expert’s set of items matched to u ∈ U .

Original: We first prove a technical lemma.
Revised: We first prove a technical lemma.

Original: Lemma C.1.
Revised: Lemma C.1.

Original: Assuming that the robustness condition is met after making the decision for v − 1 , i.e. R [Equation] + [Equation]
Revised: Assuming that the robustness condition is met after making the decision for v − 1 , i.e. R v − 1 ≥ ρ (cid:16) R πv − 1 + (cid:80) u ∈U (cid:0) |V u,v − 1 | − | V πu,v − 1 | (cid:1) + · w u, max (cid:17) − B .

Original: u [Equation] max (cid:17) − B . If at the step when v arrives and the expert’s decision x πv is not available for matching, then x v = skip always satisfies
Revised: If at the step when v arrives and the expert’s decision x πv is not available for matching, then x v = skip always satisfies

Original: [Equation]
Revised: R v ≥ ρ (cid:16) R π v + (cid:80) u ∈U (cid:0) |V u,v | − | V π u,v | (cid:1) + · w u, max (cid:17) − B .

Original: Proof.
Revised: Proof.

Original: If the item x πv is not available for matching, it must have been consumed before v arrives, which means |V x πv ,v − 1 | − | V πx πv ,v − 1 | ≥ 1 (since otherwise the expert cannot choose x piv either).
Revised: If the item x πv is not available for matching, it must have been consumed before v arrives, which means |V x πv ,v − 1 | − | V πx πv ,v − 1 | ≥ 1 (since otherwise the expert cannot choose x piv either).

Original: Since x v = skip , we have R v =
Revised: Since x v = skip , we have R v =

Original: R v − 1 and V u,v = V u,v − 1 , ∀ u ∈ U .
Revised: R v − 1 and V u,v = V u,v − 1 , ∀ u ∈ U .

Original: Then, by the robustness assumption of the previous step, we have
Revised: Then, by the robustness assumption of the previous step, we have

Original: [Equation]
Revised: [Equation]

Original: where the last equality holds because ( |V u,v | − |V πu,v | )
Revised: where the last equality holds because ( |V u,v | − |V πu,v | )

Original: + − ( |V u,v − 1 | − |V πu,v − 1 | ) + = − 1 if u = x πv , and ( |V u,v | − |V πu,v | ) + − ( |V u,v − 1 | − |V πu,v − 1 | ) + = 0 otherwise.
Revised: + − ( |V u,v − 1 | − |V πu,v − 1 | ) + = − 1 if u = x πv , and ( |V u,v | − |V πu,v | ) + − ( |V u,v − 1 | − |V πu,v − 1 | ) + = 0 otherwise.

Original: We next prove by induction that the condition
Revised: We next prove by induction that the condition

Original: [Equation]
Revised: [Equation]

Original: holds for all steps by Algorithm 1.
Revised: holds for all steps by Algorithm 1.

Original: At the first step, if ˜ x v is not the same as x πv and w ˜ x v ,v ≥ ρ (cid:0) w x πv ,v + w u, max (cid:1) − B , we select the RL decision x v = ˜ x v , and the robustness condition equation 9 is satisfied.
Revised: At the first step, if ˜ x v is not the same as x πv and w ˜ x v ,v ≥ ρ (cid:0) w x πv ,v + w u, max (cid:1) − B , we select the RL decision x v = ˜ x v , and the robustness condition equation 10 is satisfied.

Original: Otherwise, we select the expert action x v = x π v and the condition still holds since A v = C v and w x v ,v ≥ ρw x πv ,v − B holds when ρ ≤ 1 and B ≥ 0 .
Revised: Otherwise, we select the expert action x v = x πv and the condition still holds since A v = C v and w x v ,v ≥ ρw x πv ,v − B holds when ρ ≤ 1 and B ≥ 0 .

Original: Then, assuming that the robustness condition in equation 9 is satisfied after making the decision for v − 1 , we need to prove it is also satisfied after making the decision for v .
Revised: Then, assuming that the robustness condition in equation 10 is satisfied after making the decision for v − 1 , we need to prove it is also satisfied after making the decision for v .

Original: If the condition in equation 2 in Algorithm 1 is satisfied, then x v = ˜ x v and so equation 9 holds naturally.
Revised: If the condition in equation 3 in Algorithm 1 is satisfied, then x v = ˜ x v and so equation 10 holds naturally.

Original: Otherwise, if the expert action x πv is available for matching, then we select expert action x v = x πv .
Revised: Otherwise, if the expert action x πv is available for matching, then we select expert action x v = x πv .

Original: Then, we have w x v ,v ≥ ρw x πv ,v − B and |V u,v | − | V πu,v | =
Revised: Then, we have w x v ,v ≥ ρw x πv ,v − B and |V u,v | − | V πu,v | =

Original: |V u,v − 1 | − | V πu,v − 1 | , ∀ u ∈ U , and hence the condition equation 9 still holds.
Revised: |V u,v − 1 | − | V πu,v − 1 | , ∀ u ∈ U , and hence the condition equation 10 still holds.

Original: Other than these two cases, we also have the option to “skip”, i.e. x v = skip .
Revised: Other than these two cases, we also have the option to “skip”, i.e. x v = skip .

Original: By Lemma C.1, the condition equation 9 still holds.
Revised: By Lemma C.1, the condition equation 10 still holds.

Original: Therefore, we prove that the condition equationholds for every step.
Revised: Therefore, we prove that the condition equation 10 holds for every step.

Original: After the last step v = |V| , we must have
Revised: After the last step v = |V| , we must have

Original: [Equation]
Revised: [Equation]

Original: where R v and R πv are the total rewards of LOMAR and the expert algorithm π after the last step v = |V| , respectively.
Revised: where R v and R πv are the total rewards of LOMAR and the expert algorithm π after the last step v = |V| , respectively.

Original: This completes the proof for the no-free-disposal case.
Revised: This completes the proof for the no-free-disposal case.

Original: C. W ITH F REE D ISPOSAL
Revised: C. W ITH F REE D ISPOSAL

Original: We now turn to the free-disposal setting which is more challenging than the no-free-disposal setting because of the possibility of using future high-reward items to replace existing low-reward ones.
Revised: We now turn to the free-disposal setting which is more challenging than the no-free-disposal setting because of the possibility of using future high-reward items to replace existing low-reward ones.

Original: We first denote ∆ f x πv as the actual additional reward obtained by following the expert’s decision x πv ,
Revised: We first denote ∆ f x πv as the actual additional reward obtained by following the expert’s decision x πv ,

Original: [Equation]
Revised: [Equation]

Original: Additionally, we denote ∆ f πx πv as the expert’s additional reward of choosing x πv , where
Revised: Additionally, we denote ∆ f πx πv as the expert’s additional reward of choosing x πv , where

Original: [Equation]
Revised: [Equation]

Original: For presentation convenience, we rewrite the hedging reward as ˜ G (cid:0) {V u,v } u ∈U , {V πu,v } u ∈U (cid:1) as˜ G [Equation] =
Revised: For presentation convenience, we rewrite the hedging reward as ˜ G (cid:0) {V u,v } u ∈U , {V π u,v } u ∈U (cid:1) as

Original: [Equation]
Revised: [Equation]

Original: where e πu,j ∈ E πu ( V πu,v ) , e u,j ∈ E u ( V u,v ) , and E u is defined in Eqn.
Revised: where e π u,j ∈ E πu ( V πu,v ) , e u,j ∈ E u ( V u,v ) , and E u is defined in Eqn.

Original: equation 5.
Revised: equation 6.

Original: Lemma C.2.
Revised: Lemma C.2.

Original: Assuming that the robustness condition is met after making the decision for v − 1 , i.e.
Revised: Assuming that the robustness condition is met after making the decision for v − 1 , i.e.

Original: [Equation]
Revised: R v − 1 ≥ ρ (cid:16) R πv − 1 + ˜ G (cid:0) {V u,v − 1 } u ∈U , {V πu,v − 1 } u ∈U (cid:1)(cid:17) − B .

Original:  we [Equation] − ∆ f πx πv ≥
Revised: At step v , we have ∆ f x πv − ∆ f πx πv ≥

Original: G (cid:0) x πv , {V u,v − 1 } u ∈U , {V πu,v } u ∈U (cid:1) − ˜ G (cid:0) {V u,v − 1 } u ∈U , {V πu,v − 1 } u ∈U (cid:1) .
Revised: G (cid:0) x πv , {V u,v − 1 } u ∈U , {V πu,v } u ∈U (cid:1) − ˜ G (cid:0) {V u,v − 1 } u ∈U , {V πu,v − 1 } u ∈U (cid:1) .

Original: Proof.
Revised: Proof.

Original: We begin with “ G (cid:0) x πv , {V u,v − 1 } u ∈U , {V πu,v } u ∈U (cid:1) − ˜ G (cid:0) {V u,v − 1 } u ∈U , {V πu,v − 1 } u ∈U (cid:1) ” in
Revised: We begin with “ G (cid:0) x πv , {V u,v − 1 } u ∈U , {V πu,v } u ∈U (cid:1) − ˜ G (cid:0) {V u,v − 1 } u ∈U , {V πu,v − 1 } u ∈U (cid:1) ” in

Original: Lemma C.2.
Revised: Lemma C.2.

Original: By definition, it can be written as
Revised: By definition, it can be written as

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: where u = x πv , ˆ e πu,j ∈ E πu ( V πu,v − 1 (cid:83) { v } ) , and ˆ e u,j ∈ E u ( V u,v − 1 (cid:83) { v } ) .
Revised: where u = x πv , ˆ e πu,j ∈ E πu ( V πu,v − 1 (cid:83) { v } ) , and ˆ e u,j ∈ E u ( V u,v − 1 (cid:83) { v } ) .

Original: Besides, e πu,j ∈
Revised: Besides, e πu,j ∈

Original: E πu ( V πu,v − 1 ) , and e u,j ∈ E u ( V u,v − 1 ) .
Revised: E πu ( V πu,v − 1 ) , and e u,j ∈ E u ( V u,v − 1 ) .

Original: To prove the lemma, we consider four possible cases for w u,v to cover all the cases.
Revised: To prove the lemma, we consider four possible cases for w u,v to cover all the cases.

Original: Case 1 : If the reward for v is small enough such that w u,v < w u,e u, 1 and w u,v < w u,e πu, 1 , then v / ∈ E u ( V u,v − 1 (cid:83) { v } ) and v / ∈
Revised: Case 1 : If the reward for v is small enough such that w u,v < w u,e u, 1 and w u,v < w u,e πu, 1 , then v / ∈ E u ( V u,v − 1 (cid:83) { v } ) and v / ∈

Original: E u ( V πu,v − 1 (cid:83) { v } ) .
Revised: E u ( V πu,v − 1 (cid:83) { v } ) .

Original: Then we have ∆ f x πv =
Revised: Then we have ∆ f x πv =

Original: ∆ f πx πv = 0 , since both the expert and LOMAR cannot gain any reward from the online item v .
Revised: ∆ f πx πv = 0 , since both the expert and LOMAR cannot gain any reward from the online item v .

Original: From Eqn.
Revised: From Eqn.

Original: equation 14, we can find that the right-hand side is also 0.
Revised: equation 15, we can find that the right-hand side is also 0.

Original: Therefore, the conclusion in Lemma C.2 holds with the equality activated.
Revised: Therefore, the conclusion in Lemma C.2 holds with the equality activated.

Original: Case 2 : If the reward for v is large enough such that w u,v > w u,e u, 1 and w u,v > w u,e πu, 1 , then v ∈ E u ( V u,v − 1 (cid:83) { v } ) and v
Revised: Case 2 : If the reward for v is large enough such that w u,v > w u,e u, 1 and w u,v > w u,e πu, 1 , then v ∈ E u ( V u,v − 1 (cid:83) { v } ) and v

Original: ∈ E u ( V πu,v − 1 (cid:83) { v } ) .
Revised: ∈ E u ( V πu,v − 1 (cid:83) { v } ) .

Original: In other words, we will remove the smallestreward item e u, 1 / ∈ E u ( V u,v − 1 (cid:83) { v } ) and e πu, 1 / ∈ E u ( V πu,v − 1 (cid:83) { v } ) .
Revised: In other words, we will remove the smallestreward item e u, 1 / ∈ E u ( V u,v − 1 (cid:83) { v } ) and e πu, 1 / ∈ E u ( V πu,v − 1 (cid:83) { v } ) .

Original: Then
Revised: Then

Original: G (cid:0) x πv , {V u,v − 1 } u ∈U , {V πu,v } u ∈U (cid:1) − ˜ G (cid:0) {V u,v − 1 } u ∈U , {V πu,v − 1 } u ∈U (cid:1) ≤ − w u,e u, 1 + w u,e πu,
Revised: G (cid:0) x πv , {V u,v − 1 } u ∈U , {V πu,v } u ∈U (cid:1) − ˜ G (cid:0) {V u,v − 1 } u ∈U , {V πu,v − 1 } u ∈U (cid:1) ≤ − w u,e u, 1 + w u,e πu

Revised: [Equation]

Original: The inequality holds because ( w u,e u, 1 − w u,e πu, 1 ) + ≥ w u,e u, 1 − w u,e πu, 1 .
Revised: The inequality holds because ( w u,e u, 1 − w u,e πu, 1 ) + ≥ w u,e u, 1 − w u,e πu, 1 .

Original: In this case, ∆ f x πv = w u,v − w u,e u, 1 and ∆ f πx πv = w u,v − w u,e πu, 1 .
Revised: In this case, ∆ f x πv = w u,v − w u,e u, 1 and ∆ f πx πv = w u,v − w u,e πu, 1 .

Original: Therefore, the conclusion in Lemma C.2 holds.
Revised: Therefore, the conclusion in Lemma C.2 holds.

Original: Case 3 : If the reward for v satisfies w u,v ≥ w u,e u, 1 and w u,v ≤ w u,e πu, 1 , then v ∈ E u ( V u,v − 1 (cid:83) { v } ) and v / ∈ E u ( V πu,v − 1 (cid:83) { v } ) .
Revised: Case 3 : If the reward for v satisfies w u,v ≥ w u,e u, 1 and w u,v ≤ w u,e πu, 1 , then v ∈ E u ( V u,v − 1 (cid:83) { v } ) and v / ∈ E u ( V πu,v − 1 (cid:83) { v } ) .

Original: In other words, even if v ∈ E u ( V u,v − 1 (cid:83) { v } ) (i.e., the online item v produces additional rewards for LOMAR ), the reward of v is still smaller than the smallest reward for the expert.
Revised: In other words, even if v ∈ E u ( V u,v − 1 (cid:83) { v } ) (i.e., the online item v produces additional rewards for LOMAR ), the reward of v is still smaller than the smallest reward for the expert.

Original: Then, we have G (cid:0) x πv , {V u,v − 1 } u ∈U , {V πu,v } u ∈U (cid:1) − ˜ G (cid:0) {V u,v − 1 } u ∈U , {V πu,v − 1 } u ∈U (cid:1) =
Revised: Then, we have G (cid:0) x πv , {V u,v − 1 } u ∈U , {V πu,v } u ∈U (cid:1) − ˜ G (cid:0) {V u,v − 1 } u ∈U , {V πu,v − 1 } u ∈U (cid:1) =

Original: In this case, ∆ f x πv = w u,v − w u,e u, 1 ≥ 0 and ∆ f πx πv = 0 .
Revised: In this case, ∆ f x πv = w u,v − w u,e u, 1 ≥ 0 and ∆ f πx πv = 0 .

Original: Therefore, the conclusion in Lemma C. still holds.
Revised: Therefore, the conclusion in Lemma C. still holds.

Original: Case 4 : If the reward for v satisfies w u,v ≤ w u,e u, 1 and w u,v ≥ w u,e πu, 1 , then in this case, only the current smallest-reward item is replaced with v for the expert, while the reward of LOMAR remains unchanged.
Revised: Case 4 : If the reward for v satisfies w u,v ≤ w u,e u, 1 and w u,v ≥ w u,e πu, 1 , then in this case, only the current smallest-reward item is replaced with v for the expert, while the reward of LOMAR remains unchanged.

Original: Thus, we have
Revised: Thus, we have

Revised: [Equation]

Revised: ∈U (cid:1)

Revised: [Equation]

Revised: w

Original: [Equation]
Revised: [Equation]

Revised: πu

Revised: [Equation]

Original:  In this case, ∆ f x πv = 0 and ∆ f πx πv = w u,v − w u,e πu, 1 .
Revised: [Equation] w u [Equation] In this case, ∆ f x πv = 0 and ∆ f πx πv = w u,v − w u,e πu, 1 .

Original: Then the conclusion in Lemma C.2 still holds with the equality activated.
Revised: Then the conclusion in Lemma C.2 still holds with the equality activated.

Original: We next prove by induction that the condition
Revised: We next prove by induction that the condition

Original: [Equation]
Revised: [Equation]

Original: holds for all steps by Algorithm 1.
Revised: holds for all steps by Algorithm 1.

Original: At the first step, by using x v = x πv , we have R v = R πv and ˜ G (cid:0) {V u,v } u ∈U , {V πu,v } u ∈U (cid:1) = 0 , and it is obvious that the condition in equation 15 is satisfied.
Revised: At the first step, by using x v = x πv , we have R v = R πv and ˜ G (cid:0) {V u,v } u ∈U , {V π u,v } u ∈U (cid:1) = 0 , and it is obvious that the condition in equation 16 is satisfied.

Original: Thus, there is at least one solution x v = x πv for our robustness condition in equation 15.
Revised: Thus, there is at least one solution x v = x πv for our robustness condition in equation 16.

Original: Starting from the second step, assume that after the step v − 1 , we already have
Revised: Starting from the second step, assume that after the step v − 1 , we already have

Original: [Equation]
Revised: [Equation]

Original: If the condition in Line 28 of Algorithm 1 is already satisfied, we can just use x v = ˜ x v , which directly satisfies equation 15.
Revised: If the condition in Line 28 of Algorithm 1 is already satisfied, we can just use x v = ˜ x v , which directly satisfies equation 16.

Original: Otherwise, we need to follow the expert by setting x v = x πv .
Revised: Otherwise, we need to follow the expert by setting x v = x πv .

Original: We prove x v = x πv satisfies the robustness condition at any step v .
Revised: We prove x v = x πv satisfies the robustness condition at any step v .

Original: From Lemma C.2, since 0 ≤ ρ ≤ 1 and ∆ f x πv ≥ 0 we have
Revised: From Lemma C.2, since 0 ≤ ρ ≤ 1 and ∆ f x πv ≥ 0 we have

Original: [Equation]
Revised: [Equation]

Original: Then, by substituting it back to Eqn.
Revised: Then, by substituting it back to Eqn.

Original: equation
Revised: equation

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Therefore, after the last step v , LOMAR must satisfy
Revised: Therefore, after the last step v , LOMAR must satisfy

Original: [Equation]
Revised: [Equation]

Original: where R v and R πv are the total rewards of LOMAR and the expert algorithm π after the last step v = |V| , respectively.
Revised: where R v and R πv are the total rewards of LOMAR and the expert algorithm π after the last step v = |V| , respectively.

Original: Thus, we complete the proof for the free-disposal setting.
Revised: Thus, we complete the proof for the free-disposal setting.