The successful application of neural methods to machine translation has realized huge quality advances for the community. With these improvements, many have noted outstanding challenges, including the modeling and treatment of gendered language. Where previous studies have identified concerns using manually-curated synthetic examples, we develop a novel technique to leverage real world data to explore challenges for deployed systems. We use our new method to compile an evaluation benchmark spanning examples relating to four languages from three language families, which we will publicly release to facilitate research. The examples in our benchmark expose the ways in which gender is represented in a model and the unintended consequences these gendered representations can have in downstream applications .
The successful application of neural methods to machine translation has realized huge quality advances for the community. With these improvements, many have noted outstanding challenges, including the modeling and treatment of gendered language. While previous studies have identified issues using synthetic examples, we develop a novel technique to mine examples from real world data to explore challenges for deployed systems. We use our method to compile an evaluation benchmark spanning examples for four languages from three language families, which we publicly release to facilitate research. The examples in our benchmark expose where model representations are gendered, and the unintended consequences these gendered representations can have in downstream application .
R
Where
While
R
concerns using manually-curated
issues using
R
leverage
mine examples from
D
new
R
relating to
for
D
will
R
the ways in which gender is represented in a model
where model representations are gendered,
R
applications
application