Using Captum to Explain Generative Language Models

Great work! I previously used Captum to compute Shapley Values (SV) to explain natural language models in classification tasks and it is great to see new features for explaining generation. In our work (forgive me for shameless self-plug: https://arxiv.org/pdf/2305.19998.pdf), we find that 1) random seed choices can influence the explanation results a bit, and 2) computing SV for a large language model is costly if you want a stable explanation with large sample size. Casting generation as a consecutive classification task, I think that is still the case. We developed an amortized model to achieve a better stability-efficiency trade-off even allowing online computation of SV for LLMs. Would love to chat about incorporating my work into Captum or extending my work to generation setting :-)