app

LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

README.md

@@ -0,0 +1,43 @@
+---
+title: Systematic Error Analysis and Labeling
+emoji: 🦭
+colorFrom: yellow
+colorTo: pink
+sdk: streamlit
+sdk_version: 1.10.0
+app_file: app.py
+pinned: false
+license: apache-2.0
+---
+# SEAL
+Systematic Error Analysis and Labeling (SEAL) is an interactive tool for discovering systematic errors in NLP models via clustering on high-loss example groups and semantic labeling for interpretability of those error-groups. It supports fine-grained analytical visualization for interactively zooming into potential systematic bugs and features for crafting prompts to label those bugs semantically.
+
+🎥 [Demo screencast](https://vimeo.com/736659216)
+
+<p>
+    <img src="./assets/website/seal.gif" alt="Demo gif"/>
+</p>
+
+## Table of Contents
+- [Installation](#installation)
+- [Quickstart](#quickstart)
+- [Running Locally](#running-locally)
+- [Citation](#citation)
+
+## Installation
+Please use python>=3.8 since some dependencies require that for installation.
+```shell
+git clone https://huggingface.co/spaces/nazneen/seal
+cd seal
+pip install --upgrade pip
+pip install -r requirements.txt
+```
+
+## Quickstart
+```
+streamlit run app.py
+```
+
+## Running Locally
+
+## Citation

app.py

@@ -0,0 +1,403 @@
+## LIBRARIES ###
+## Data
+import numpy as np
+from numpy.core.numeric import outer
+import pandas as pd
+import torch
+import pickle
+from tqdm import tqdm
+from math import floor
+from collections import defaultdict
+from transformers import AutoTokenizer
+#pd.set_option('precision', 2)
+#pd.options.display.float_format = '${:,.2f}'.format
+
+# Analysis
+# from gensim.models.doc2vec import Doc2Vec
+# from sklearn.metrics import accuracy_score, f1_score, precision_score, recall_score
+import nltk
+from nltk.cluster import KMeansClusterer
+import scipy.spatial.distance as sdist
+from scipy.spatial import distance_matrix
+# nltk.download('punkt') #make sure that punkt is downloaded
+
+# App & Visualization
+import streamlit as st
+import altair as alt
+import plotly.graph_objects as go
+from streamlit_vega_lite import altair_component
+
+
+# utils
+from random import sample
+from seal import utils as ut
+
+
+def down_samp(embedding):
+    """Down sample a data frame for altiar visualization """
+    # total number of positive and negative sentiments in the class
+    #embedding = embedding.groupby('slice').apply(lambda x: x.sample(frac=0.3))
+    total_size = embedding.groupby(['slice', 'label'], as_index=False).count()
+
+    user_data = 0
+    # if 'Your Sentences' in str(total_size['slice']):
+    #     tmp = embedding.groupby(['slice'], as_index=False).count()
+    #     val = int(tmp[tmp['slice'] == "Your Sentences"]['source'])
+    #     user_data = val
+
+    max_sample = total_size.groupby('slice').max()['content']
+
+    # # down sample to meeting altair's max values
+    # # but keep the proportional representation of groups
+    down_samp = 1/(sum(max_sample.astype(float))/(1000-user_data))
+
+    max_samp = max_sample.apply(lambda x: floor(
+        x*down_samp)).astype(int).to_dict()
+    max_samp['Your Sentences'] = user_data
+
+    # # sample down for each group in the data frame
+    embedding = embedding.groupby('slice').apply(
+        lambda x: x.sample(n=max_samp.get(x.name))).reset_index(drop=True)
+
+    # # order the embedding
+    return(embedding)
+
+#down sample low loss points only so misclassified examples are not down sampled in viz
+
+
+def down_samp_ll(embedding):
+    df_ll = embedding[embedding['slice'] == 'low-loss']
+    #if(len(df_ll)<5000):
+    #    return embedding
+    #else:
+    df_hl = embedding[embedding['slice'] == 'high-loss']
+    down_samp = len(df_ll) - (1000-len(df_hl))
+    df_ll.sample(n=down_samp)
+    embedding.drop(df_ll.index)
+    return embedding
+
+
+def data_comparison(df):
+    selection = alt.selection_multi(fields=['cluster', 'label'])
+    color = alt.condition(alt.datum.slice == 'high-loss', alt.Color('cluster:N', scale=alt.Scale(
+        domain=df.cluster.unique().tolist()), legend=None), alt.value("lightgray"))
+    opacity = alt.condition(selection, alt.value(0.7), alt.value(0.25))
+
+    # basic chart
+    scatter = alt.Chart(df).mark_point(size=100, filled=True).encode(
+        x=alt.X('x:Q', axis=None),
+        y=alt.Y('y:Q', axis=None),
+        color=color,
+        shape=alt.Shape('label:N', scale=alt.Scale(
+            range=['circle', 'diamond'])),
+        tooltip=['cluster:N', 'slice:N', 'content:N', 'label:N', 'pred:N'],
+        opacity=opacity
+    ).properties(
+        width=1000,
+        height=800
+    ).interactive()
+
+    legend = alt.Chart(df).mark_point(size=100, filled=True).encode(
+        x=alt.X("label:N"),
+        y=alt.Y('cluster:N', axis=alt.Axis(
+            orient='right'), sort='ascending', title=''),
+        shape=alt.Shape('label:N', scale=alt.Scale(
+            range=['circle', 'diamond']), legend=None),
+        color=color,
+    ).add_selection(
+        selection
+    )
+    layered = scatter | legend
+    layered = layered.configure_axis(
+        grid=False
+    ).configure_view(
+        strokeOpacity=0
+    )
+
+    content = legend.encode(text='content:N')
+
+    return layered
+
+
+def viz_panel(embedding_df):
+    """ Visualization Panel Layout"""
+    all_metrics = {}
+    st.warning("**Error group visualization**")
+    with st.expander("How to read this chart:"):
+        st.markdown("* Each **point** is an input example.")
+        st.markdown("* Gray points have low-loss and the colored have high-loss. High-loss instances are clustered using **kmeans** and each color represents a cluster.")
+        st.markdown(
+            "* The **shape** of each point reflects the label category --  positive (diamond) or negative sentiment (circle).")
+    #st.altair_chart(data_comparison(down_samp(embedding_df)), use_container_width=True)
+    viz = data_comparison(embedding_df)
+    st.altair_chart(viz, use_container_width=True)
+
+@st.cache()
+def frequent_tokens(data, tokenizer, loss_quantile=0.95, top_k=200, smoothing=0.005):
+    unique_tokens = []
+    tokens = []
+    for row in tqdm(data['content']):
+        tokenized = tokenizer(row, padding=True, truncation=True, return_tensors='pt')
+        tokens.append(tokenized['input_ids'].flatten())
+        unique_tokens.append(torch.unique(tokenized['input_ids']))
+    losses = data['loss'].astype(float)
+    high_loss = losses.quantile(loss_quantile)
+    loss_weights = np.where(losses > high_loss,losses,0.0)
+    loss_weights = loss_weights / loss_weights.sum()
+
+    token_frequencies = defaultdict(float)
+    token_frequencies_error = defaultdict(float)
+    weights_uniform = np.full_like(loss_weights, 1 / len(loss_weights))
+
+    for i in tqdm(range(len(data))):
+        for token in unique_tokens[i]:
+            token_frequencies[token.item()] += weights_uniform[i]
+            token_frequencies_error[token.item()] += loss_weights[i]
+
+    token_lrs = {k: (smoothing+token_frequencies_error[k]) / (
+        smoothing+token_frequencies[k]) for k in token_frequencies}
+    tokens_sorted = list(map(lambda x: x[0], sorted(
+        token_lrs.items(), key=lambda x: x[1])[::-1]))
+
+    top_tokens = []
+    for i, (token) in enumerate(tokens_sorted[:top_k]):
+        top_tokens.append(['%10s' % (tokenizer.decode(token)), '%.4f' % (token_frequencies[token]), '%.4f' % (
+            token_frequencies_error[token]), '%4.2f' % (token_lrs[token])])
+    return pd.DataFrame(top_tokens, columns=['token', 'freq', 'error-freq', 'ratio'])
+
+
+def load_precached_groups(data_ll, df_list, num_clusters, group_dict_path, group_idx_path, num_points=1000):
+    merged = dynamic_groups(df_list, num_clusters)
+    down_samp = len(data_ll) - (num_points-len(merged))
+    sample_idx = data_ll.sample(n=down_samp)
+    data_ll = data_ll.drop(sample_idx.index)
+    # put all the low loss data in one bigger cluster
+    data_ll['cluster'] = merged.loc[merged['cluster'].idxmax()].cluster + 1
+    merged = pd.concat([merged, data_ll])
+    # merged['cluster'] = merged['cluster'].astype('str')
+    # with open(group_dict_path, 'rb') as f:
+    #     group_dict = pickle.load(f)
+    # with open(group_idx_path, 'rb') as f:
+    #     group_idx_dict = pickle.load(f)
+    # for k,v in group_idx_dict.items():
+    #     label = group_dict.get(k)
+    #     merged.loc[merged.index.isin(v), ['cluster']] = label
+    return merged
+
+
+def dynamic_groups(df_list, num_clusters):
+    merged = pd.DataFrame()
+    ind = 0
+    for df in df_list:
+        kmeans_df, assigned_clusters = kmeans(df, num_clusters=num_clusters)
+        kmeans_df['cluster'] = kmeans_df['cluster'] + ind*num_clusters
+        ind = ind+1
+        merged = pd.concat([merged, kmeans_df])
+    return merged
+
+
+@st.cache(ttl=600)
+def get_data(inference, emb):
+    preds = inference.outputs.numpy()
+    losses = inference.losses.numpy()
+    embeddings = pd.DataFrame(emb, columns=['x', 'y'])
+    num_examples = len(losses)
+    # dataset_labels = [dataset[i]['label'] for i in range(num_examples)]
+    return pd.concat([pd.DataFrame(np.transpose(np.vstack([dataset[:num_examples]['content'],
+                                                           dataset[:num_examples]['label'], preds, losses])), columns=['content', 'label', 'pred', 'loss']), embeddings], axis=1)
+
+
+def kmeans(data, num_clusters=3):
+    X = np.array(data['embedding'].to_list())
+    kclusterer = KMeansClusterer(
+        num_clusters, distance=nltk.cluster.util.cosine_distance,
+        repeats=25, avoid_empty_clusters=True)
+    assigned_clusters = kclusterer.cluster(X, assign_clusters=True)
+    data['cluster'] = pd.Series(
+        assigned_clusters, index=data.index).astype('int')
+    data['centroid'] = data['cluster'].apply(lambda x: kclusterer.means()[x])
+    return data, assigned_clusters
+
+
+def distance_from_centroid(row):
+    return sdist.norm(row['embedding'] - row['centroid'].tolist())
+
+
+@st.cache(ttl=600)
+def craft_prompt(cluster_df):
+    instruction = "In this task, we'll assign a short and precise label to a cluster of documents based on the topics or concepts most relevant to these documents. The documents are all subsets of a sentiment classification dataset.\n"
+    if len(cluster_df) > 10:
+        content = cluster_df['content'].str[:600].tolist()
+    else:
+        content = cluster_df['content'].str[:1000].tolist()
+    examples = '\n - '.join(content)
+    text = instruction + '- ' + examples + '\n Cluster label:'
+    return text.strip()
+
+
+@st.cache(ttl=600)
+def topic_distribution(weights, smoothing=0.01):
+    topic_frequencies = defaultdict(float)
+    topic_frequencies_error = defaultdict(float)
+    weights_uniform = np.full_like(weights, 1 / len(weights))
+    num_examples = len(weights)
+    for i in range(num_examples):
+        example = dataset[i]
+        category = example['title']
+        topic_frequencies[category] += weights_uniform[i]
+        topic_frequencies_error[category] += weights[i]
+
+    topic_ratios = {c: (smoothing + topic_frequencies_error[c]) / (
+        smoothing + topic_frequencies[c]) for c in topic_frequencies}
+
+    categories_sorted = map(lambda x: x[0], sorted(
+        topic_ratios.items(), key=lambda x: x[1], reverse=True))
+
+    topic_distr = []
+    for category in categories_sorted:
+        topic_distr.append(['%.3f' % topic_frequencies[category], '%.3f' %
+                           topic_frequencies_error[category], '%.2f' % topic_ratios[category], '%s' % category])
+
+    return pd.DataFrame(topic_distr, columns=['Overall frequency', 'Error frequency', 'Ratio', 'Category'])
+
+
+def populate_session(dataset, model):
+    data_df = read_file_to_df(
+        './assets/data/'+dataset + '_' + model+'.parquet')
+    if model == 'albert-base-v2-yelp-polarity':
+        tokenizer = AutoTokenizer.from_pretrained('textattack/'+model)
+    else:
+        tokenizer = AutoTokenizer.from_pretrained(model)
+    # if "user_data" not in st.session_state:
+    #     st.session_state["user_data"] = data_df
+    # if "selected_slice" not in st.session_state:
+    #     st.session_state["selected_slice"] = None
+    return tokenizer
+
+
+@st.cache(allow_output_mutation=True)
+def read_file_to_df(file):
+   return pd.read_parquet(file)
+
+
+if __name__ == "__main__":
+    ### STREAMLIT APP CONGFIG ###
+    st.set_page_config(layout="wide", page_title="Interactive Error Analysis")
+
+    ut.init_style()
+
+    lcol, rcol = st.columns([5, 2])
+    # ******* loading the mode and the data
+    #st.sidebar.mardown("<h4>Interactive Error Analysis</h4>", unsafe_allow_html=True)
+
+    dataset = st.sidebar.selectbox(
+        "Dataset",
+        ["amazon_polarity", "yelp_polarity", "imdb"],
+        index=1
+    )
+
+    model = st.sidebar.selectbox(
+        "Model",
+        ["distilbert-base-uncased-finetuned-sst-2-english",
+            "albert-base-v2-yelp-polarity", "distilbert-imdb"],
+    )
+
+    ### LOAD DATA AND TOKENIZER VARIABLES ###
+    ##uncomment the next next line to run dynamically and not from file
+    #tokenizer = populate_session(dataset, model)
+    if dataset == 'imdb':
+        data_df = read_file_to_df('./assets/data/imdb_distilbert.parquet')
+    else:
+        data_df = read_file_to_df(
+            './assets/data/'+dataset + '_' + model+'.parquet')
+        data_df = data_df[:20000]
+
+    loss_quantile = st.sidebar.slider(
+        "Loss Quantile", min_value=0.9, max_value=1.0, step=0.01, value=0.98
+    )
+
+    data_df['loss'] = data_df['loss'].astype(float)
+    data_df['pred'] = data_df['pred'].astype(int)
+    losses = data_df['loss']
+    high_loss = losses.quantile(loss_quantile)
+    data_df['slice'] = np.where(data_df['loss'] >= high_loss, 'high-loss', 'low-loss')
+    # drop rows that are not hl
+    data_hl = pd.DataFrame(data_df[data_df['slice'] == 'high-loss'])
+    #data_hl = data_hl.drop(data_hl[data_hl.pred==data_hl.label].index)
+    data_ll = pd.DataFrame(data_df[data_df['slice'] == 'low-loss'])
+    # this is to allow clustering over each error type. fp, fn for binary classification
+    df_list = [d for _, d in data_hl.groupby(['label'])]
+
+    run_kmeans = st.sidebar.radio(
+        "Cluster error group?", ('True', 'False'), index=0)
+
+    num_clusters = st.sidebar.slider(
+        "# clusters", min_value=1, max_value=60, step=1, value=3)
+
+    num_points = st.sidebar.slider(
+        "# data points to visualize", min_value=1000, max_value=5000, step=100, value=1000)
+
+    selected_cluster = st.sidebar.number_input(
+        label='Cluster #:', max_value=num_clusters-1, min_value=0)
+
+    if run_kmeans == 'True':
+        with st.spinner(text='running kmeans...'):
+            group_dict_path = './assets/data/cluster-labels/'+dataset+'.pkl'
+            group_idx_path = './assets/data/cluster-labels/'+dataset+'_idx.pkl'
+            #data_hl_path = './assets/data/high-loss/'+dataset+'.parquet'
+            merged = load_precached_groups(data_ll, df_list, int(
+                (num_clusters/2)), group_dict_path, group_idx_path, num_points=num_points)
+            #dynamic_groups(df_list,)
+            #tmp = pd.concat([data_ll, merged], axis =0, ignore_index=True)
+
+    cluster_content = craft_prompt(
+        merged.loc[merged['cluster'] == selected_cluster])
+
+    with lcol:
+        st.markdown('<h5>Error Groups</h5>', unsafe_allow_html=True)
+        with st.expander("How to read this table:"):
+            st.markdown(
+                "* *Error groups* refers to the subset of evaluation dataset the model performs poorly on.")
+            st.markdown(
+                "* The table displays model error groups on the evaluation dataset, sorted by loss.")
+            st.markdown(
+                "* Each row is an input example that includes the label, model pred, loss, and error group.")
+        with st.spinner(text='loading error groups...'):
+            #dataframe=read_file_to_df('./assets/data/'+dataset+ '_'+ model+'_error-slices.parquet')
+            #uncomment the next next line to run dynamically and not from file
+            dataframe = merged[['content', 'label', 'pred', 'loss', 'cluster']].sort_values(
+                by=['loss'], ascending=False)
+            #table_html = dataframe.to_html(columns=['content', 'label', 'pred', 'loss', 'cluster'], max_rows=50)
+            #table_html = table_html.replace("<th>", '<th align="left">')  # left-align the headers
+            st.write(dataframe.style.format(
+                {'loss': '{:.2f}'}), width=1000, height=300)
+
+    with rcol:
+        with st.spinner(text='loading...'):
+            st.markdown('<h5>Word Distribution in Error Groups</h5>',
+                        unsafe_allow_html=True)
+            #uncomment the next two lines to run dynamically and not from file
+            # if model == 'albert-base-v2-yelp-polarity':
+            #     tokenizer = AutoTokenizer.from_pretrained('textattack/'+model)
+            # else:
+            #     tokenizer = AutoTokenizer.from_pretrained(model)
+            # commontokens = frequent_tokens(data_df, tokenizer, loss_quantile=loss_quantile)
+            if dataset == 'imdb':
+                commontokens = read_file_to_df('./assets/data/imdb_distilbert_commontokens.parquet')
+            else:
+                commontokens = read_file_to_df(
+                    './assets/data/'+dataset + '_' + model+'_commontokens.parquet')
+            with st.expander("How to read this table:"):
+                st.markdown(
+                    "* The table displays the most frequent tokens in error groups, relative to their frequencies in the val set.")
+
+            st.write(commontokens)
+
+    with st.spinner(text='loading visualization...'):
+        viz_panel(merged)
+
+    st.sidebar.download_button(
+        data=cluster_content,
+        label="Build prompt from data",
+        file_name='prompt'
+    )

requirements.txt

@@ -0,0 +1,139 @@
+# This file may be used to create an environment using:
+# $ conda create --name <env> --file <this file>
+# platform: osx-arm64
+absl-py==1.0.0; python_version >= '3.6'
+aiohttp==3.8.0
+aiosignal==1.2.0; python_version >= '3.6'
+altair==4.1.0
+antlr4-python3-runtime==4.8
+appnope==0.1.2; sys_platform == 'darwin' and platform_system == 'Darwin'
+argon2-cffi==21.1.0; python_version >= '3.5'
+astor==0.8.1; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'
+async-timeout==4.0.1; python_version >= '3.6'
+attrs==21.2.0; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'
+backcall==0.2.0
+backports.zoneinfo==0.2.1; python_version >= '3.6' and python_version < '3.9'
+base58==2.1.1; python_version >= '3.5'
+bleach==4.1.0; python_version >= '3.6'
+blinker==1.4
+cachetools==4.2.4; python_version ~= '3.5'
+certifi==2021.10.8
+cffi==1.15.0
+charset-normalizer==2.0.7; python_version >= '3'
+click==7.1.2; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'
+cython==0.29.24; python_version >= '2.6' and python_version not in '3.0, 3.1, 3.2, 3.3'
+cytoolz==0.11.2; python_version >= '3.5'
+dataclasses==0.6
+datasets==1.15.1
+debugpy==1.5.1; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'
+decorator==5.1.0; python_version >= '3.5'
+defusedxml==0.7.1; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'
+dill==0.3.4; python_version >= '2.7' and python_version != '3.0'
+entrypoints==0.3; python_version >= '2.7'
+fastbpe==0.1.0
+filelock==3.3.2; python_version >= '3.6'
+frozenlist==1.2.0; python_version >= '3.6'
+fsspec[http]==2021.11.0; python_version >= '3.6'
+future==0.18.2; python_version >= '2.6' and python_version not in '3.0, 3.1, 3.2, 3.3'
+fuzzywuzzy==0.18.0
+gitdb==4.0.9; python_version >= '3.6'
+gitpython==3.1.24; python_version >= '3.7'
+google-auth-oauthlib==0.4.6; python_version >= '3.6'
+google-auth==2.3.3; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4, 3.5'
+grpcio==1.41.1
+idna==3.3; python_version >= '3'
+importlib-resources==5.4.0; python_version < '3.9'
+kaleido==0.2.1
+markdown==3.3.4; python_version >= '3.6'
+markupsafe==2.0.1; python_version >= '3.6'
+matplotlib-inline==0.1.3; python_version >= '3.5'
+meerkat-ml==0.1.2; python_version >= '3.7'
+mistune==0.8.4
+multidict==5.2.0; python_version >= '3.6'
+multiprocess==0.70.12.2
+nbclient==0.5.8; python_full_version >= '3.6.1'
+nbconvert==6.3.0; python_version >= '3.7'
+nbformat==5.1.3; python_version >= '3.5'
+nest-asyncio==1.5.1; python_version >= '3.5'
+nltk==3.6.5
+notebook==6.4.5; python_version >= '3.6'
+numpy==1.21.4
+oauthlib==3.1.1; python_version >= '3.6'
+omegaconf==2.1.1; python_version >= '3.6'
+packaging==21.2; python_version >= '3.6'
+pandas==1.3.4
+pandocfilters==1.5.0; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'
+parso==0.8.2; python_version >= '3.6'
+pexpect==4.8.0; sys_platform != 'win32'
+pickleshare==0.7.5
+pillow==8.4.0; python_version >= '3.6'
+plotly==5.3.1
+progressbar==2.5
+prometheus-client==0.12.0; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'
+prompt-toolkit==3.0.22; python_full_version >= '3.6.2'
+protobuf==3.19.1; python_version >= '3.5'
+ptyprocess==0.7.0; os_name != 'nt'
+pyahocorasick==1.4.2
+pyarrow==6.0.0; python_version >= '3.6'
+pyasn1-modules==0.2.8
+pyasn1==0.4.8
+pycparser==2.21; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'
+pydeck==0.7.1; python_version >= '3.7'
+pydeprecate==0.3.1; python_version >= '3.6'
+pygments==2.10.0; python_version >= '3.5'
+pympler==0.9
+pyparsing==2.4.7; python_version >= '2.6' and python_version not in '3.0, 3.1, 3.2, 3.3'
+pyrsistent==0.18.0; python_version >= '3.6'
+python-dateutil==2.8.2; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'
+python-levenshtein==0.12.2
+pytorch-lightning==1.5.1; python_version >= '3.6'
+pytz-deprecation-shim==0.1.0.post0; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4, 3.5'
+pytz==2021.3
+pyyaml==6.0; python_version >= '3.6'
+pyzmq==22.3.0; python_version >= '3.6'
+regex==2021.11.10
+requests-oauthlib==1.3.0
+requests==2.26.0; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4, 3.5'
+robustnessgym==0.1.3
+rsa==4.7.2; python_version >= '3.6'
+sacremoses==0.0.46
+scikit-learn==1.0.1; python_version >= '3.7'
+scipy==1.7.2; python_version < '3.11' and python_version >= '3.7'
+semver==2.13.0; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'
+send2trash==1.8.0
+six==1.16.0; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'
+sklearn==0.0
+smart-open==5.2.1; python_version >= '3.6' and python_version < '4'
+smmap==5.0.0; python_version >= '3.6'
+streamlit-vega-lite==0.1.0
+streamlit==1.2.0
+tenacity==8.0.1; python_version >= '3.6'
+tensorboard-data-server==0.6.1; python_version >= '3.6'
+tensorboard-plugin-wit==1.8.0
+tensorboard==2.7.0; python_version >= '3.6'
+terminado==0.12.1; python_version >= '3.6'
+testpath==0.5.0; python_version >= '3.5'
+threadpoolctl==3.0.0; python_version >= '3.6'
+tokenizers==0.10.3
+toml==0.10.2; python_version >= '2.6' and python_version not in '3.0, 3.1, 3.2, 3.3'
+toolz==0.11.2; python_version >= '3.5'
+torch==1.10.0; python_full_version >= '3.6.2'
+torchmetrics==0.6.0; python_version >= '3.6'
+tornado==6.1; python_version >= '3.5'
+tqdm==4.62.3; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'
+traitlets==5.1.1; python_version >= '3.7'
+transformers==4.12.3; python_version >= '3.6'
+typing-extensions==3.10.0.2; python_version < '3.10'
+tzdata==2021.5; python_version >= '3.6'
+tzlocal==4.1; python_version >= '3.6'
+ujson==4.2.0; python_version >= '3.6'
+urllib3==1.26.7; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4' and python_version < '4'
+validators==0.18.2; python_version >= '3.4'
+wcwidth==0.2.5
+webencodings==0.5.1
+werkzeug==2.0.2; python_version >= '3.6'
+wheel==0.37.0; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4'
+widgetsnbextension==3.5.2
+xxhash==2.0.2; python_version >= '2.6' and python_version not in '3.0, 3.1, 3.2, 3.3'
+yarl==1.7.2; python_version >= '3.6'
+zipp==3.6.0; python_version < '3.10'