Teacher:In this task, you will be presented with a context from an academic paper and a question separated with a
. You have to answer the question based on the context.
Teacher: Now, understand the problem? Solve this instance: Proposed Fusion Techniques ::: Step-Wise Decoder Fusion
Our first proposed technique is the step-wise decoder fusion of visual features during every prediction step i.e. we concatenate the visual encoding as context at each step of the decoding process. Proposed Fusion Techniques ::: Multimodal Attention Modulation
Similar to general attention BIBREF8, wherein a variable-length alignment vector $a_{th}(s)$, whose size equals the number of time steps on the source side, is derived by comparing the current target hidden state $h_{t}$ with each source hidden state $\overline{h_{s}}$; we consider a variant wherein the visual encoding $v_{t}$ is used to calculate an attention distribution $a_{tv}(s)$ over the source encodings as well. Proposed Fusion Techniques ::: Visual-Semantic (VS) Regularizer
In terms of leveraging the visual modality for supervision, BIBREF1 use multi-task learning to learn grounded representations through image representation prediction.
Question: What are 3 novel fusion techniques that are proposed?
Student:
Step-Wise Decoder Fusion Multimodal Attention Modulation Visual-Semantic (VS) Regularizer