  Recurrent Neural Networks (RNNs) have achieved remarkable performance on a
range of tasks. A key step to further empowering RNN-based approaches is
improving their explainability and interpretability. In this work we present
MEME: a model extraction approach capable of approximating RNNs with
interpretable models represented by human-understandable concepts and their
interactions. We demonstrate how MEME can be applied to two multivariate,
continuous data case studies: Room Occupation Prediction, and In-Hospital
Mortality Prediction. Using these case-studies, we show how our extracted
models can be used to interpret RNNs both locally and globally, by
approximating RNN decision-making via interpretable concept interactions.
