The brain is the locus of our language ability , and so brain images can be used to ground linguistic theories . Here we introduce Brain-Bench , a lightweight system for testing distributional models of word semantics . We compare the performance of several models , and show that the performance on brain-image tasks differs from the performance on behavioral tasks . We release our benchmark test as part of a web service . There is active debate over how we should test semantic models . In fact , in 2016 there was an entire workshop dedicated to the testing of semantic representations ( RepEval , 2016 ) . Several before us have argued for the usage of brain data to test semantic models ( Anderson et al . , 2013 ; Murphy et al . , 2012 ; Anderson et al . , 2015 ) , as a brain image represents a snapshot of one person 's own semantic representation . Still , testing semantic models against brain imaging data is rarely done by those not intimately involved in psycholinguistics or neurolinguistics . This may be due to a lack of familiarity with neuroimaging methods and publicly available datasets . We present the first iteration of BrainBench , a new system that makes it easy to test semantic models using brain imaging data ( Available at http://www.langlearnlab.cs.uvic . ca / brainbench/ ) . Our system has methodology that is similar to popular tests based on behavioral * Corresponding Author data ( see Section 2.2 ) , and has the additional benefit of being fast enough to offer as a web service . We have presented our new system , BrainBench , which is a fast and lightweight alternative to previous methods for comparing DS models to brain images . Our proposed methodology is more similar to well-known behavioral tasks , as BrainBench also uses the similarity of words as a proxy for meaning . We hope that this contribution will bring brain imaging tests " to the masses " and encourage discussion around the testing of DS models against brain imaging data .

Challenge: While brain data can be used for evaluating semantic models, it is limited in accessibility because of familiarity and available datasets.
Approach: They introduce a benchmark system hosted on the web to facilitate the evaluation of semantic models with brain data images.
Outcome: Benchmark experiments on the proposed evaluation framework show the difference performances of different models.