ELASTIC: Numerical Reasoning with Adaptive Symbolic Compiler
Abstract
Numerical reasoning over text is a challenging task of Artificial Intelligence (AI),requiring reading comprehension and numerical reasoning abilities.
Previousapproaches use numerical reasoning programs to represent the reasoning process.
However, most works do not separate the generation of operators and operands,which are key components of a numerical reasoning program, thus limiting theirability to generate such programs for complicated tasks.
In this paper, we introducethe num E rica L re AS oning with adap T ive symbol I c C ompiler (ELASTIC) model,which is constituted of the RoBERTa as the Encoder and a Compiler with fourmodules: Reasoning Manager, Operator Generator, Operands Generator, and
Memory Register.
ELASTIC is robust when conducting complicated reasoning.
Also, it is domain agnostic by supporting the expansion of diverse operators withoutcaring about the number of operands it contains.
Experiments show that ELASTICachieves 68.96 and 65.21 of execution accuracy and program accuracy on the
FinQA dataset and 83.00 program accuracy on the MathQA dataset, outperformingprevious state-of-the-art models significantly.
Introduction
Recently, Pre-trained Language Models (PLMs) [1, 2, 3, 4, 5] show astonishing performance overreading comprehension tasks like SQuAD [6].
However, PLMs fall short of numerical reasoning overtext [7], which requires conducting numerical reasoning based on understanding the text.
Hence,numerical reasoning over text is more challenging than reading comprehension [8] and attracts theinterest of the AI community.
Previous approaches adopt the sequence-to-sequence architecture togenerate the sequential format of numerical reasoning programs (see (b) in Table 1) [9, 10].
However,the sequential format could produce invalid expressions such as " 3 − ((2) " because of the wrongposition of parentheses [11].
To avoid this, some methods convert the reasoning program to the binarytree, then use the tree-decoder to generate the pre/post-order traversal sequence (see (c) in Table 1)[12, 13, 14].
Alternatively, FinQANet [15] represents the reasoning program in a flattened formatand generates the right parentheses forcibly after generating two consecutive operands.
To increasethe scalability, NeRd [7] introduces the symbolic operations and generates the reasoning program asthe nested compositional format (see (e) in Table 1).
Researchers also investigate to capture valuableinformation between entities and numbers to improve numerical reasoning ability.
Some works use
PLMs [8, 7, 16], while others, like Li et al.
[17] and Ran et al.
[18], adopt graph neural network toencode the text.
Currently, proposed approaches struggle with two significant problems.
Firstly, they are vulnerable tocomplicated numerical reasoning problems.
The complicated numerical reasoning problems usuallycontain a long reasoning program, in which the types of operators are diverse, and the number ofoperands is dynamic.
Since most works do not separate the generation of operators and operands,their performance is hindered by cascading errors when encountering complicated tasks.
Secondly,previous works lack extensibility for the operators, which arises from either the flaw of the modelarchitecture or the representation format of the program, making them hard to apply to different datadomains.
[Table]
Hence, we present the num E rica L re AS oning with adap T ive symbol I c C ompiler (ELASTIC) model.
ELASTIC separates the generation of operators and operands, allowing it to be less influencedby the cascading error from the complicated reasoning.
Moreover, ELASTIC is adaptable to thenumber of operands following an operator, making it domain-agnostic to support diverse operators.
Specifically, ELASTIC contains an Encoder part extracting the contextual representations of thepassage and question and a Compiler part generating the numerical reasoning program.
The Compilerconsists of four modules: Reasoning Manager, Operator Generator, Operands Generator, and Memory
Register.
We conduct experiments on two challenging datasets: FinQA [15], and MathQA [19].
Since FinQA and MathQA are collected from different domains: annual financial reports and
GRE/GMAT, ELASTIC demonstrates its adaptability by achieving state-of-the-art results on bothdatasets.
Furthermore, our ablation studies investigate how the length of the numerical reasoningprogram influences the model’s numerical reasoning ability, which shows that ELASTIC is lessliable to being influenced by the cascading error.
In addition, we introduce the maximum Memory
Departing Distance (M-MDD), which measures how difficult for the mode to use the executableresults from the previous sub-program.
We use M-MDD to demonstrate the necessity of the Memory
Register in ELASTIC.
The contributions of our work are: (1) we present a numerical reasoningmodel ELASTIC with good adaptability and elasticity, which separates the generation of operatorsand operands.
ELASTIC achieves state-of-the-art results on two challenging datasets:
FinQA and
MathQA; (2) we introduce the design of separate modules and Memory Register, making ELASTICperform stably on complicated numerical reasoning problems; (3) the proposed ELASTIC is domainagnostic because it supports diverse operators.
Related Work
Making models to conduct numerical reasoning has attracted the AI community since the last century[20].
Previous research has investigated making the model do numerical reasoning over text by usingstatistical learning methods to find a similar equation pattern [21, 22, 23, 24].
Since deep learninghas recently achieved great success in many tasks, Wang et al.
[25] propose DNS, which is the firstdeep learning model for solving math word problems as far as we know.
After their work, researcherstry to find a better way to represent the numerical reasoning program.
For example, Wang et al.
[26], and Wang et al.
[27] use the expression tree to represent the reasoning program.
Sun et al.
[11] createtree-decoder GTS, which generates prefix traverse sequence of the tree.
Chiang et al.
[28], and Qin etal.
[13] extract the semantic information from the question and passage texts and want to connectthem with the reasoning steps.
In addition, Li et al.
[29], and Zhang et al.
[30] introduce the graphencoder to capture the structural information or syntactic information to capture the relation betweennumbers and entities.
Shen et al.
[31] propose a unified model, which uses both sequential and graphas the encoder, then uses seq2seq and tree decoder to generate the reasoning program.
Furthermore, several datasets are proposed to evaluate the model’s numerical reasoning ability, suchas Math23K [32] and HWMP [13].
There are also more challenging datasets, like ASdiv [33], and
MathQA [19].
At the same time, Dua et al.
propose DROP [34], which requires more than arithmeticoperations to conduct numerical reasoning.
State-of-the-art models like NeRd [7] and NumNet [18]are introduced to solve the DROP dataset.
More recently, a dataset called FinQA [15] has beenproposed, which is constructed from the annual financial report.
Despite the considerable success achieved by these approaches, Patel et al.
[35] argues that somestate-of-the-art models, like GTS [11] and Graph2Tree [30], only learn the statistical relation insteadof numerical reasoning ability.
Unlike previous works, our model ELASTIC separates the generationof operators and operands, allowing it to conduct complicated numerical reasoning.
Moreover,
ELASTIC is adaptable to the number of operands following an operator, making it domain agnostic.
Approach
Figure 1 shows the architecture of our ELASTIC model.
ELASTIC consists of an Encoder partencoding the question text and problem text into contextual vectors and a Compiler part producingthe numerical reasoning programs.
The Compiler part consists of four modules: Reasoning Manager,
Operator Generator, Operands Generator, and Memory Register.
The Reasoning Manager leveragesother modules to produce the numerical reasoning program.
Since a complete numerical reasoningprogram usually contains several sub-programs, the generation steps between operators and operandsare interchangeable.
To help the following sub-programs use executable results from the previoussub-programs, Memory Register stores the sub-programs executable results into correspondingpre-defined cache tokens embeddings.
[Table]
Task Definition Given the problem text P and question text
Q , the task is to generate a numerical reasoning program R .
Both problem text P and question text Q consist of words andnumbers (denoted by NUM ).
The Numerical reasoning program R represents the numerical reasoning process, which is asequence of symbols (denoted by s ) from mathematical operators (denoted by OP ) and operands (denoted by OE ).Operands
OE are from either constant numbers (denoted by CONS ) defined in Domain Specific Language (DSL) or NUM .
CONSare the special numbers that do not exist in either the problem text P and question text Q , such as const_pi( π ).
Finally,the pattern of the numerical reasoning program R is definedas R = (cid:110) op
i (cid:2) oe ij (cid:3) m − 1 j =0 (cid:111) n −i =0 , where op i ∈ OP , it is the i thoperator in R , and op i contains several operands oe ij .
In addition, we regard a group of one operator and its operands as thesub-program r .
For example, op i (cid:2) oe i j (cid:3) is the i th sub-programr i , which can be executed since it is a complete arithmeticprogram.
3.1 Encoder Part
[Figure] As shown in Figure 1 (Encoder), the Encoder takes the concatenated sequence of Q and P as input.
The Encoder encodes the input sequence and outputs the contextual vectors h enc .
Next, h enc is usedfor the Compiler to produce the numerical reasoning program R .
In this work, we use RoBERTa asthe Encoder.
The outputs from the final layer of RoBERTa is used as h enc ∈ R h ∗ s , where s is themaximum input length of the RoBERTa, and h is the hidden size of RoBERTa.
Note that ELASTICis not dependent on the specific type of encoder.
Any model providing contextual vectors of thesequence can be used.
3.2 Compiler Part
Decoding Vocabulary and Token
Embedding We first describe the decoding vocabulary.
Thedecoding vocabulary consists of OP and OE , where OE can be further categorized into NUM and
CONS .
The embedding e s of symbol s of the decoding vocabulary is represented by the embedding
E op , cons , num ( s ) , which is the embedding look-up function.
Hence, the embedding for symbol s isdefined as:
[Equation]
The symbols embeddings of OP and CONS are two trainable embedding matrices E op ∈ R h ∗ n op and
E cons ∈
R h ∗ n cons ( n op and n cons refers sizes of OP and CONS respectively).
The embedding for thesymbol of NUM is h enc
i ∈
R h , where i denotes the index position in the sequence of Q and P .
Reasoning Manager As shown in Figure 1 (Reasoning Manager), the Reasoning Manager outputsthe vector g , which guides the Operator Generator and the Operands Generator to produce op andoe .
The inputs for the Reasoning Manager are contextual vectors h enc ( h enc q for generating operators)from the Encoder and embedding of the previously generated symbol s t − 1 .
The Reasoning Managerfirst calculates the context vector c by the normalized vectors of h enci and the attention weights a i :
[Equation]
[Equation]
where W 1 ∈ R h ∗ h and W 2 ∈ R h ∗ h , and both are trainable parameters.
The c summarizes theencoded information from the Encoder according to the previous generated symbol s .
Next, the Reasoning Manager adopts the GRU [36] network to generate the guidance output g :
[Equation]
where " : " represents concatenation.
W 3 ∈ R h ∗ 2 h is trainable parameter, and Relu is the activationfunction.
h t − 1 ∈ R h is the hidden state of GRU from the previous step, and h 0 is 0 .
Operator Generator As shown in Figure 1 (Operator Generator).
Firstly, the Operator Generatorreceives the guidance vector g opt from the Reasoning Manager by inputting: contextual vectors h enc qof tokens from the question Q , and embedding E op ( op t − 1 ) of the previously generated operator.
Next, the Operator Generator calculates the probabilities of i -th operator (denoted as i - op ) of the OP:
[Equation]
where W op ∈ R h ∗ h is trainable parameter.
The Operator Generator selects the operator with thehighest probability as the predicted op .
Next, unlike other models, the Reasoning Manager suspendsthe generation of operators and starts to generate operands { oe t } through the Operands Generator.
Operands Generator As shown in Figure 1 (Operands Generator).
The inputs from Operands
Generator to the Reasoning Manager are different from Operator Generator’s.
Because oe could bea number in Q or P , the contextual vectors h enc of all tokens are used.
Furthermore, the Operands
Generator initializes the embedding of the initial operand e ( oe t 0 ) as Relu ( W 4 [ E op ( op t ) : g t ]) ( W 4 ∈
R h ∗ 2 h ), leveraging information of op t to produce oe t .
Next, the Reasoning Manager outputs g oen forn -th step generation of operand oe tn .
Finally, the probability of i -th operand (denoted as i - oe ) of the OE:
[Equation]
where W oe ∈ R h ∗ h is trainable parameter.
The Operands Generator selects the operand withthe highest probability as the predicted oe .
After one operand has been generated, the Operands Generator continues producing operands for the sub-program r t .
The decoding process for theoperands terminates when the token none is produced.
Memory Register When generating sub-program r i , its operands could be the executable resultsfrom the previous sub-program r p ( p < i ) .
To make the Operands Generator be able to use the resultsfrom previous sub-programs.
Inspired by Chen et al.
[15], we introduce a cache token # n to the
CONS of DSL, which is used for storing the information of executable results.
Unlike other constants,# n does not point to a static value.
It is different according to the different sub-program r n .
As theresults, ELASTIC needs to update the representation of # n after the sub-program r n is generated.
Specifically, the Memory and Register module update the cache # n by replacing its embedding withoutput g oen , which is the guidance vector from Reasoning Manager to guide the generation of the lastoperands belonging to the sub-program r n .
Training Objective Given the data D with size of N containing P i , Q i , ˆ op
i , ˆ oe i , where P i and Q irefer to the passage and question in the the i th training data, likewise, ˆ op
i and ˆ
oe i are the goldenoperators and operands.
Our training goal is to minimize the sum of the negative log-likelihood overthe entire data, so the training loss is (cid:80) Ni =1 − (cid:8) log P ( OP i | P i , Q i )
+ log P
( OE i | P i , Q i ) (cid:9) .
Experimental Set-up
Datasets We conduct evaluation experiments on two datasets: FinQA [15] and MathQA [19].
FinQA: FinQA is a dataset created from the annual financial reports.
It contains 8,281 data, splitinto train, eval, and test parts with 6,251, 883, and 1,147 examples.
We adopt the evaluation metricsfrom the original FinQA paper: execution accuracy (Exe Acc) and program accuracy (Prog Acc).
The program accuracy calculates the accuracy of the operators and operands between the predictedprogram and the golden program.
The execution accuracy calculates the accuracy between thegolden executable result and the result from the predicted program.
Since the FinQA dataset onlycontains operators with two operands, we extend it by creating questions required to be solved bythe operators with more than two operands.
We use the extended FinQA dataset to evaluate ourmodels’ adaptability to the number of operands (See Appendix A).
MathQA: MathQA is created from GRE/GMAT examinations, containing 37,200 math wordproblems.
The dataset is split into 80% , 12% , and 8% of train, dev, and test data.
Comparedwith the FinQA dataset, the examples of MathQA require more advanced reasoning ability, whichchallenges the model to conduct advanced numerical reasoning (see Appendix B).
A significantdifference with FinQA is that the number of operands following an operator is not explicit in the
MathQA dataset.
Each MathQA question contains one correct of several answer options, calculatedby the reasoning program with the knowledge of the operation semantics.
Since we do not havethis kind of knowledge, we adopted the same way as NeRd [7], by only using program accuracy toevaluate models’ performances.
Note that program accuracy is stricter than execution accuracybecause the model could find the correct answer by spurious reasoning programs.
Baselines We compare our ELASTIC model with several state-of-the-art models.
(1) FinQANet[15]: It adopts the Encoder-Decoder architecture with a cache updating mechanism to generate theprogram.
Since FinQANet only supports generating operators with exact two operands, we manageto train and evaluate FinQANet on the MathQA dataset by discarding the operators containing morethan two operands.
(2) NeRd [7]: it uses the BERT and a pointer-generator-based model to generatethe symbolic nested program.
(3) Graph2Tree [17]: It models the dependency information of thetext sequence by the GraphSAGE [38] like model, and generates the program in a tree-structuredway.
(4) NumNet [18]: NumNet models the numeracy information by a GNN network.
We also trainthe NumNet+ , which replaces the Encoder of the NumNet by RoBERTa-large.
5 Note that programaccuracy does not apply to NumNet, since NumNet does not generate compositional reasoningprograms.
(5) Human Performance : We also report the human performance of both experts andnon-experts in the FinQA dataset.
The results are taken from the original FinQA paper [15].
Implementation Details The model is implemented by Pytorch [39] and Transformer [40], thentrained on a server with an NVIDIA Tesla A100 GPU of 40G memory.
Training epochs are set to50 and 100 for FinQA and MathQA, respectively.
The batch size for all datasets is set to 10.
Weuse Adam as optimizer [41] to update the parameters of the models.
The initial learning rate is setto 1e-5 equally, and it would be halved in every 25 epochs and 50 epochs for FinQA and MathQA.
During training, the dropout rate and the weight decay are set to 0.1 and 1e-5 to prevent over-fitting.
The parameters of the RoBERTa are fine-tuned during training.
For the GRU cell in the decoder, thehidden size is the same as the RoBERTa, and the GRU layers number is 4.
During inference, we usegreedy decoding to generate the reasoning program.
Results
[Table]
Overall Results Table 3 shows the performances of our ELASTIC model and baselines on FinQAand MathQA.
Overall, ELASTIC (RoBERTa-large) achieves the highest scores on both datasets.
Inthe FinQA dataset, we see a significant lead in our ELASTIC (RoBERTa-large) model compared to thebest baseline FinQANet (RoBERTa-large), with 3.91 points higher execution accuracy and 1.69 pointshigher program accuracy.
When we change the Encoder part of ELASTIC from RoBERTa-largeto RoBERTa-base, it still achieves better results than FinQANet using the same size of RoBERTa.
Since both ELASTIC and FinQANet use the RoBERTa as the encoder, the results demonstrate theimprovements brought by separating the generation procedures for operators and operands.
Both
ELASTIC models outperform the NeRd with a large margin.
It is worth mentioning that NeRd definesexternal rules for different operators in their model [7], which is not the case with the ELASTIC.
ELASTIC also outperforms the NumNet and NumNet+ by a considerable margin.
This could be dueto the internal structure of these models limiting their scalability in generating reasoning programs,thus struggling to produce reasoning steps in a systematic manner [42].
Finally, Graph2Tree achievesonly 0.37 accuracy on the FinQA test dataset, which is much lower compared to its 69.96 programaccuracy on the MathQA dataset.
We suspect that this is because of the data leak problem existing in
FinQA train and eval data (see detailed explanation in Appendix D).
Although ELASTIC surpassesthe non-expert performance, we can still find a large gap between our ELASTIC model and Human
Expert.
[Table] [Table]
For the MathQA dataset, ELASTIC (RoBERTa-large) is the best performing model, with 3.3 higherprogram accuracy than NeRd and 13.04 points higher than Graph2Tree.
We further investigate theperformance of ELASTIC using RoBERTa-base, which still achieves higher accuracy of 82.27 than79.7 of NeRd.
The slight performance difference between ELASTIC (RoBERTa-large) and ELASTIC(RoBERTa-base) suggests that the extracted contextual semantic information of passage and questionis sufficient.
Finally, FinQANet achieves promising results on MathQA, 79.20% program accuracyby FinQANet (RoBERTa-large) and 74.12% program accuracy by FinQANet (RoBERTa-base).
Notethat we discarded the data of MathQA containing more than two operands for the FinQANet, sothat performance of FinQANet on MathQA is the overestimation of its numerical reasoning ability.
Even with such consideration, ELASTIC outperforms FinQANet significantly, demonstrating that
ELASTIC is more adaptable by supporting diverse operators than FinQANet.
Performance Breakdown
To demonstrate the strength of ELASTIC, we investigate the importanceof the Memory Register.
Also, we show ELASTIC performance when generating different lengths ofprogram steps.
Necessity of Memory Register As discussed in the Section Memory Register, ELASTIC storesthe executable results of each sub-program into a special cache token # n, and updates its embeddingafter n -th sub-program is generated.
The longer the reasoning program is, the higher the probabilityof the generating process using the previous sub-program result.
This section investigates the effectof the Memory Register on improving numerical reasoning performance.
[Table]
First, we present an ablation study of using and not using the Memory Register for ELASTIC.
From
Table 4, we find that ELASTIC with Memory Register performs slightly better than it without.
Similarobservations can be found for the MathQA dataset.
This observation demonstrates the value of the
Memory Register.
Next, since ELASTIC and FinQANet store the executable results from the previoussub-program in a different way, we conduct a comparison between the two models.
The results from
Table 4 show that the ELASTIC with Memory Register achieves significantly higher scores than
FinQANet on both datasets.
[Figure]
Next, given two sub-program belonging to the same R : r i , r j ( i < j ).
where the executable resultof r i is used as the operand for r j .
Then, we introduce the Memory Departing Distance (MDD) forr i and r j as j − i , and the maximum Memory Departing Distance (M-MDD) as the longest MDDbetween all { r } of R .
The bigger M-MDD is, the more challenging to select the correct previoussub-program result, since the model tends to forget the information passing from long steps before.
As the result, we investigate how models perform when dealing with different M-MDD.
From Figure 2a and Figure 2b, the ELASTIC with Memory Register performs better than ELASTICwithout it at each M-MDD on FinQA and MathQA datasets.
Particularly in the MathQA dataset,when M-MDD is larger than 5, ELASTIC with Memory Register can achieve better results thanthe ELASTIC without it.
This demonstrates the importance of the Memory Register when usingexecutable results from long steps before.
Worth mentioning that ELASTIC performs better than
FinQANet on both datasets, even without the Memory Register.
Performance on Different Program Steps When producing the long numerical reasoning program,
ELASTIC is less influenced by the cascading error.
To demonstrate this superiority of ELASTIC, weinvestigate how different lengths of programs influence models’ performances.
[Table]
Table 5 displays the models’ performances when generating programs with different steps.
ELASTIC(RoBERTa-large) performs better than FinQANet (RoBERTa-large) † when the program step is either1 or 2, indicating ELASTIC also performs well on the shorter program steps.
Surprisingly, with theprogram step increasing from 3, the accuracy for both ELASTIC and FinQANet tumbles by halfcompared with the performances on program steps equal 2.
We suspect that the FinQA dataset lackssufficient training examples for the data with more than 3 program steps.
Table 5 shows that thenumber of training data with more than 2 program steps is 594 compared to the numbers of dataavailable for program steps equal to 1 (4240) or 2 (2300).
For a fair comparison, we retrained the
FinQANet (RoBERTa-large) on the FinQA dataset, but ELASTIC still outperforms it in executionaccuracy and program accuracy.
From Figure 2c, ELASTIC (RoBERTa-large) surpasses the FinQANet (RoBERTa-large) on MathQAdataset almost on every program step.
Meanwhile, although MathQA is challenging, ELASTIC(RoBERTa-large) still achieves program accuracy over 80.0 when program steps are less than orequal to 8.
The model’s performance drops when the program steps are equal to 9 and 10, but startsto soar when the program steps are bigger than 10.
This demonstrates that ELASTIC performs wellwhen generating longer program steps.
As shown in Figure 2c, we plot the accuracy for the operatorgeneration, which ignores the correctness of operands generation.
We could find that the operationaccuracy is always higher than the program accuracy regarding different program steps (except forprogram steps equal to 12).
This finding demonstrates the advantage of separating the generationprocedure for operators and operands.
This finding also reveals that the wrong predictions are because
ELASTIC selects the wrong operands.
We suspect this is due to too much noise from the context.
Conclusion and Future Work
This paper presents the num E rica L re AS oning with adap T ive symbol I c C ompiler
(ELASTIC) modelaiming to solve the numerical reasoning over text problem.
ELASTIC separates the generationof operators and operands, allowing the model to generate the long and complicated reasoningprogram.
Also, ELASTIC is domain agnostic and supports diverse operators, increasing adaptability.
In addition, we introduce the Memory Register and improve the performance of the model by usingexecutable results from the preceding sub-programs.
We evaluated the performance of the ELASTICmodel on FinQA and MathQA datasets and conducted an extensive comparison with state-of-theart models.
The results show ELASTIC gained significant improvement over the state-of-the-artbaselines.
Furthermore, We investigated the model’s performance in terms of different M-MDD,demonstrating the necessity of the Memory Register.
Finally, we compared models’ performancesregarding different lengths of numerical reasoning programs, showing ELASTIC is adept at producinglong numerical reasoning programs.
In the future, we plan to improve the accuracy of matchingnumbers and entities of the text.
In addition, ELASTIC requires annotated reasoning programs, whichis labor intensive.
It is worth investigating how to generate reasoning programs from the trainedmodel.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We provide theURL of our code in 1 of the paper.(b)
Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Section Implementation Details.(c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] We think the error bars are not related to the core result ofour experiments.
(d) Did you include the total amount of compute and the type of resources used (e.g.,type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section Implementation
Details.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section Datasets
and Section Baselines.
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]