gitignore

refactor metric to support evaluating `all-2020` split

test folder

fall back to GPT2TokenizerFast for models which don't have a fast tokenizer (like OPT)

update evaluate dependency

don't mix up condition indexing in suites where items have different order of content by condition (e.g. number_prep in syntaxgym2020)

batch surprisal computation, now GPU friendly

get_region_edges: fix edge case (ha) where empty initial regions cause the count to be off

document accuracy property

interface change: use namedtuple for metric result so that we can add an `accuracy` property and have dot-access

readme example fixes

update metric to support new dataset repr, and evaluating multiple suites in one call

partial readme draft

minor code cleanup

disable truncation. longer sequences are handled just fine, AFAICS

add tokenizer config from perplexity metric. truncation breaks tests

update evaluate dependency

add requirements

add torch, transformers requirements

remove useless detach call

move metric and tests from dataset repo

stage template files

initial commit