  Many crowdsourced NLP datasets contain systematic gaps and biases that are
identified only after data collection is complete. Identifying these issues
from early data samples during crowdsourcing should make mitigation more
efficient, especially when done iteratively. We take natural language inference
as a test case and ask whether it is beneficial to put a linguist `in the loop'
during data collection to dynamically identify and address gaps in the data by
introducing novel constraints on the task. We directly compare three data
collection protocols: (i) a baseline protocol, (ii) a linguist-in-the-loop
intervention with iteratively-updated constraints on the task, and (iii) an
extension of linguist-in-the-loop that provides direct interaction between
linguists and crowdworkers via a chatroom. The datasets collected with linguist
involvement are more reliably challenging than baseline, without loss of
quality. But we see no evidence that using this data in training leads to
better out-of-domain model performance, and the addition of a chat platform has
no measurable effect on the resulting dataset. We suggest integrating expert
analysis \textit{during} data collection so that the expert can dynamically
address gaps and biases in the dataset.
