BSL - H ansard: A parallel, multimodal corpus of E nglish and interpreted B ritish S ign L anguage data from parliamentary proceedings (2023)

Abstract: BSL-Hansard is a novel open source and multimodal resource composed by combining Sign Language video data in BSL and English text from the official transcription of British parliamentary sessions. This paper describes the method followed to compile BSL-Hansard including time alignment of text using the MAUS (Schiel, 2015) segmentation system, gives some statistics about this dataset, and suggests experiments. These primarily include end-to-end Sign Language-to-text translation, but is also relevant for broader machine translation, and speech and language processing tasks.

## introduction in the united kingdom (uk), there are an estimated 151,000 british sign language (bsl) signers according to the british deaf association 1 many of whom constitute the d/deaf and hard-of-hearing (dhh) community in that country. bsl is a flourishing language, and has seen a 40% increase in the number of people who identify their main language as bsl in the ten years between the 2011 and 2021 census in england and wales 2 . in recent years, there has been marked progress in the provision of information and services for bsl signers. for example, a growing proportion of public service television broadcasting is available with bsl interpretation and members of the dhh community are becoming more prominent in the national media foot_0 . the recent british sign language act 2022 has also enshrined in law bsl's status as an official language of england, scotland, and wales. however, there remains a comparatively small amount of data available to develop language technology resources for bsl. the bsl-hansard dataset intends to make a large amount of parallel english-bsl data available to researchers. section 2 explores resources available for bsl, before sections 3 and 4 introduce and describe the parallel bsl-hansard dataset of english-bsl parliamentary utterances. section 5 then discusses possible uses and experiments with the dataset, and offers concluding remarks. ## bsl resources there is already a body of extant resources available for slt research using bsl. perhaps the most prominent is the bsl corpus @xcite which is the first digitised corpus of continuous bsl. it contains an impressive amount of vari-ation from elicited and natural conversation, with 249 signers across eight british cities and is intended for a broad range of research tasks. annotation is currently incomplete, so it is important to pursue other data collection projects. there exist other resources for bsl including the echo corpus @xcite with sign video and extensive linguistic annotation, and dicta-sign foot_1 which contains isolated sign videos. another resource is the bobsl @xcite parallel english-bsl dataset. similar to bsl-hansard, bobsl was created by collating 1, 400h of a broad range of bbc television programmes and their companion bsl interpretation. it is a valuable resource which is large enough to conduct machine learning research, shown by the researchers' experiments on slt, sign language recognition (slr), and sentence alignment. however it seems that although the corpus is free to use, each researcher must request access individually which makes it impractical to leverage its data in large, commercial projects @xcite . other resources may be generated through data augmentation, transfer learning, and bootstrapping techniques from better-resourced sls such as american sign language or from spoken languages @xcite . this type of data may be suboptimal (yin and read, 2020) as they are not a genuine representation of a sl, and the same may be said about data from sl interpretation @xcite . however, these are currently frequently-utilised ways of obtaining sufficient quantities of data for datahungry machine learning approaches. ## dataset compilation videos in .mp4 format are manually downloaded from the parliamentlive.tv web service, and the official transcripts are manually downloaded from the hansard web page. the videos and texts are then processed predominantly using the functionality of the munich automatic segmentation system (maus) @xcite . maus is a hidden markov model-based statistical forced aligner which first predicts the phonetic label based on an input transcript, and then aligns the predicted phones with an input audio signal. this service is available on the web @xcite , and can be used with other functionalities such as pre-processing, graphemeto-phoneme conversion, and subtitle generation. figure 2 provides an overview of the processing tasks and file types involved. a given input video with a maximum duration of no more than nine minutes is matched with the appropriate hansard transcript, and converted to .wav format using the ffmpeg library. the input text is pre-processed by removing all content inside parentheses and square brackets, as well as the first three lines of procedure in each hansard document. the resulting .txt and .wav files are input into the 'webmaus basic' foot_4 web service where the british english language model is chosen, and output format is set to .bpf -a file type which allows for time alignment between the phonetic transcription and the audio signal. the text file containing the original transcription and the aligned .bpf file are subsequently input into the bas 'subtitle' foot_5 web tool which maps the alignment with the original transcript in order to generate sentence-type utterances. in order to preserve full phrases as well as possible, the parameters are set to split subtitles on punctuation marks, or otherwise at a maximum length of 20 wordsthe result is output to .vtt file format. these files may be converted to a researcher-friendly .csv or .json formats by means of straightforward, freelyavailable conversion scripts foot_6 .