Differentiable Multi-Agent Actor-Critic for Multi-Step Radiology Report Summarization (2022)

Abstract: The IMPRESSIONS section of a radiology report about an imaging study is a summary of the radiologist’s reasoning and conclusions, and it also aids the referring physician in confirming or excluding certain diagnoses. A cascade of tasks are required to automatically generate an abstractive summary of the typical information-rich radiology report. These tasks include acquisition of salient content from the report and generation of a concise, easily consumable IMPRESSIONS section. Prior research on radiology report summarization has focused on single-step end-to-end models – which subsume the task of salient content acquisition. To fully explore the cascade structure and explainability of radiology report summarization, we introduce two innovations. First, we design a two-step approach: extractive summarization followed by abstractive summarization. Second, we additionally break down the extractive part into two independent tasks: extraction of salient (1) sentences and (2) keywords. Experiments on English radiology reports from two clinical sites show our novel approach leads to a more precise summary compared to single-step and to two-step-with-single-extractive-process baselines with an overall improvement in F1 score of 3-4%.

## introduction a diagnostic radiology report about an examination includes findings in which the radiologist describes normal and abnormal imaging results of their analysis @xcite . it also includes impressions or a summary that communicates conclusions about the findings and suggestions for the referring physician; a sample report is shown in impressions 1) no definite acute intracranial process. impressions as it helps them evaluate differential diagnoses while avoiding additional conversations with the radiologist or the need for repeat procedures. a well known end-to-end method for text summarization is two-step: extractive summarization followed by abstractive summarization. for instance, @xcite initially train extractive and abstractive systems separately and then use the extractive system as an agent in a singleagent reinforcement learning (rl) setup with the abstractive system as part of the environment. their extractive system extracts salient sentences and the abstractive system paraphrases these sentences to produce a summary. this summary is in turn used to compute the reward for rl training. however, this single-agent setup often fails to extract some salient sentences or it extracts irrelevant ones, leading to the generation of incomplete/incorrect im-pressions. we hypothesize that granular categories of core concepts (e.g., abnormalities, procedures) can be leveraged for generating more comprehensive summaries. thus, a separate rl agent is dedicated to the task of extracting salient keywords (core concepts) in the two-step system. the novelty in this approach is that the new, second agent can now collaborate with the first one and the two can influence each other in their extraction decisions. multiagent reinforcement learning (marl) requires that an agent coordinate with the other agents to achieve the desired goal. marl often has centralized training and decentralized execution @xcite . there are several protocols for marl training, such as sharing parameters between agents and explicit @xcite @xcite or implicit @xcite communication between agents by using an actor-critic policy gradient with a centralized critic for all agents @xcite . the aim of these protocols is to correctly assign credits so that an agent can deduce its contribution to the team's success. to train our cooperative agents that extract salient sentences and keywords, we propose a novel differentiable multiagent actor-critic (dimac) rl learning method. we learn independent agents in an actor-critic setup and use a communication channel to allow agents to coordinate by passing real-valued messages. as gradients are pushed through the communication channel, dimac is end-to-end trainable across agents. the novelties in the paper are threefold: the remaining paper is structured as follows. in section 2, we provide a detailed description of our notations general mle rl f findings e encoder network a agent (actors) i impressions d pointer network c critic k keywords w2w word lstm u action w word s2s sentence lstm m message value s sentence α attention score r reward p position c context vector g discounted reward m total words v trainable vector v value function n total sentences q switch value q action value function h hidden vector w trainable matrix a advantage function y train label conv cnn network in the first step of the framework, the two extractors independently select words and sentences from findings f but also coordinate such that the selection of salient words is followed by the selection of the sentence comprising these words. in the next step, a seq2seq abstractor paraphrases the selected sentences to generate impressions i. figure 1 illustrates the proposed framework. we refer to ## dimac as extractor and abstractor are separately trained in a two-step framework, @xcite proposed using rl training with the extractor assuming the agent role and the abstractor as part of the environment to address the separation. furthermore, as rl loss is computed out of final summary and ground-truth impressions, rl training addresses the error due to heuristic labels in the pretrained networks. unlike @xcite , our setup involves multiple extractors, so we use marl for the coordination. in other words, the word and sentence extractors d w2w and d s2s operate as rl agents a w and a s (figure 1 , right sidie). in @xcite , an actor-critic marl has a centralized critic and parameter-sharing actors. in contrast, our extractors have different characteristics, e.g., amount of selection (salient words greater than sentences) and size of source representations; therefore, we exclude parameter sharing between actors. additionally, to not have actors influence each other's policies, we have a critic that estimates the value function by not conditioning on the actions of other agents, thereby ensuring actor independence. furthermore, we introduce a communicator (m) that coordinates actors through message passing. the dedicated channel m addresses the issue of the environment appearing nonstationary due to independent agents; see @xcite @xcite . the channel allows gradients to flow between actors, transforming the setup into an end-to-end differentiable multi-agent actor critic (dimac). the actors and the communicator are initialized with the maximum-likelihood (ml) trained extractors and switch network, respectively. actions. actors a w and a s have action spaces of source words {w 1 , • • • , w m } and sentences {s 1 , • • • , s n }, respectively. at any decoding step j, actors choose actions (i.e., source selection) u aw j and u as j by using policy networks π aw and π as and hidden states h aw j and h as j . due to the communication between actors in dimac training, we intuitively expect some correlation in the actions. reward. for any decoding step j, if the communicator indicates sentence selection (@xmath0, a sentence reward r as j is computed using r 1 (rouge unigram recall) between the abstract summary ŝas j of selected sentence s as j (out of action u as j ) from the abstractor and a ground-truth impressions sentence. we sequentially match summary and im-pressions sentences such that a s learns to select relevant sentences sequentially. similarly, word reward r aw j for selected word w aw j out of action u aw j is 1 if the word is in the subset of keywords in findings, k f , else it is 0. again, we match selected and findings keywords sequentially. when an agent selects extra items, the reward for those selections is 0, and thus, the agent learns to select only relevant sentences and keywords. in addition, joint actions of actors eventually generate a global reward in a multi-agent cooperative setting as: action value functions q aw j and q as j for actions u aw j and u as j are estimated as e u aw critic. like the actors, the critic c is an lstm based network. it runs for the same number of steps as the actors and estimates gradients to train them. as the critic is used only in training, at each step j, the critic conditions on the actors' ground-truth selection indices, y s j and y w j , as the actions and uses these indices to obtain word and sentence encoder representations. in addition to source representations, it uses its state, h c j , and attends to all encoder states, {h e w2w 1 , • • • } and {h e s2s 1 , • • • }) to estimate a value function v j . v j is then used to compute advantage functions a aw j 1545 algorithm 1 differentiable multi-agent actor critic 1: procedure train-dimac 2: initialize parameters of actors (aw and as), critic (c) & communicator (m) as θ as := ds2s, θ aw := dw2w, θ c := dw2w & θ m := switch 3: for each training episode i do 4: step j ← 1 5: while action u as @xmath1: compute actors & critic states 7: sample actions u as j & u aw j 8: compute rewards r aw j & r as j for u as j & u aw j 9: compute message mj & value function vj 10: j ← j + 1 11: compute global reward r g 12: for @xmath2: compute discounted reward g as j and g aw j 14: estimate action-value functions q aw j & q as j 15: compute advantages a as j & a aw j 16: accumulate critic gradient ∆θ c 17: accumulate actor gradients ∆θ as &∆θ aw 18: update critic θ c i+1 = θ c i -α∆θ c 19: update actors as θ as i+1 = θ as i + α∆θ as & θ aw i+1 = θ aw i + α∆θ aw 20: return θ as , θ aw & θ m and a as j for actors as q aw j -v j and q as j -v j . at any step, one of the two ground-truth actions y s j / y w j is empty. therefore, the computed value and action-value functions v j and q j at that step intuitively become agent-specific, resulting in independent agent learning. finally, agent specific advantage functions are used to compute actor gradients as ∇ θ aw log π aw j a aw j and ∇ θ as log π as j a as j . importantly, value, action-value and advantage can be calculated in a single forward pass of the actor and critic for each agent. see appendix for details and proofs. communication. the communicator m (figure 1 , red circles) passes messages between the actors. actor previous hidden states and contexts, h as j , h aw j , c s j and c w j , are fed to m and a sigmoidal m j is obtained. value m j is fed to a s while 1 -m j is fed to a w . the gradient of m j flows between actors during backpropagation and provides rich training signal that minimizes the learning effort. see algorithm 1 for dimac training algorithm details. ## results in this section, we compare results from our model and various baselines using both automatic and human evaluation. ## conclusion in this work, we introduce a novel extractive approach into a two-step rl-based summarization task (extractive-then-abstractive). this approach is a marl (rather than the traditional single-agent rl) which includes a new agent that extracts salient keywords from the source text and collaborates with an agent that extracts salient sentences. we also present a differentiable multi-agent actor-critic (dimac) learning method, a novel yet simple marl training for independent agents communicating via a dedicated channel. we