updated readme with Milestone 4 breakdown

README.md

@@ -11,6 +11,376 @@ pinned: false
 
 # cs-gy-6613-project-rk2546
 
+## Milestone 4
+
+### **Code Breakdown**
+
+The USPTO application is divided into several directories. Overall, the important files are present in the application as such:
+
+````
+- data/
+    - train.json
+    - val.json
+- src/
+    - main.py
+    - train.ipynb
+````
+
+Both `train.json` and `val.json` contain the original USPTO data, sized down to contain only the relevant data from each recorded patent and split between training and validation data. The validation data `val.json` is used in the online USPTO application as a set of pre-set patents that a user can select when using the USPTO patent prediction function.
+
+The primary code back-end is stored in `main.py`, which runs the application on the HuggingFace space UI. The application uses `Streamlit` to render UI elements on the screen. All models run off of Transformers and Tokenizers from HuggingFace.
+
+The application has two features: Sentiment Analysis (for Milestone #2) and USPTO Patent Acceptance Prediction (Milestone #3). Both run on `main.py`. Sentiment Analysis relies on pre-trained [models](https://huggingface.co/models) from HuggingFace's public [datasets](https://huggingface.co/datasets) - particularly 4 models:
+
+- [cardiffnlp/twitter-roberta-base-sentiment](https://huggingface.co/cardiffnlp/twitter-roberta-base-sentiment)
+- [finiteautomata/beto-sentiment-analysis](https://huggingface.co/finiteautomata/beto-sentiment-analysis)
+- [bhadresh-savani/distilbert-base-uncased-emotion](https://huggingface.co/bhadresh-savani/distilbert-base-uncased-emotion)
+- [siebert/sentiment-roberta-large-english](https://huggingface.co/siebert/sentiment-roberta-large-english)
+
+The Patent Acceptance Prediction uses two fine-tuned models, which are built off of a pre-existing model named [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) and fine-tuned off of the USPTO dataset. The tokenizer used to parse text uses the same [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) model but is left unmodified.
+
+### **Shared Code**
+
+In this section, I describe some code back-end used universally across both the Sentiment Analysis and Patent Accceptance Prediction functions.
+
+Starting from `main.py`, we load in the necessary tokenizers and pipelines needed to run the model.
+
+````python
+import streamlit as st
+from transformers import TextClassificationPipeline, pipeline
+from transformers import AutoTokenizer, AutoModelForSequenceClassification, DistilBertTokenizerFast, DistilBertForSequenceClassification
+````
+
+These transformers are crucial to the functionality of both Sentiment Analysis and Patent Acceptance Prediction. Particularly, Sentiment Analysis uses `AutoModelForSequenceClassification` as our model class, `AutoTokenizer` as our string pre-processor, and `pipeline` to combine the two. Patent Acceptance Prediction uses `DistilBertForSequenceClassification` as our model class, `DistilBertTokenizerFast` as our string pre-processor, and `TextClassificationPipeline` to combine the two.
+
+Because everything runs on one `main.py` page and both the Sentiment Analysis and Patent Acceptance Prediction function similarly, I developed a custom python class called `ModelImplementation` that's used by both and allows for switching between different models:
+
+````python
+class ModelImplementation(object):
+    def __init__(
+            self, 
+            transformer_model_name, 
+            model_transformer, 
+            tokenizer_model_name,
+            tokenizer_func, 
+            pipeline_func, 
+            parser_func,
+            classifier_args={},
+            placeholders=[""]
+    ):
+        self.transformer_model_name = transformer_model_name
+        self.tokenizer_model_name = tokenizer_model_name
+        self.placeholders = placeholders
+
+        self.model = model_transformer.from_pretrained(self.transformer_model_name)
+        self.tokenizer = tokenizer_func.from_pretrained(self.tokenizer_model_name)
+        self.classifier = pipeline_func(model=self.model, tokenizer=self.tokenizer, padding=True, truncation=True, **classifier_args)
+        self.parser = parser_func
+
+        self.history = []
+    
+    def predict(self, val):
+        result = self.classifier(val)
+        return self.parser(self, result)
+````
+
+The main idea is that for every model that's needed, we create a new instance of this class. In each case, we can store a reference to the tokenizer, model, and pipeline; the model will then use that tokenizer, model, and pipeline in the `predict()` call. If the output of a model needs to be curated in some way (ex. we need to post-process the output of a model so that it's more human-readable), we can also pass a custom method alongside the other parameters too. This is useful when we are switching between models in the Sentiment Analysis page or between the Sentiment Analysis and Patent Acceptance Prediction page - we merely have to create or modify an instance of the `ModelImplementation` class with the proper tokenizer, model, pipeline, and post-process method (if needed). Placeholder text for any inputs can also be stored as well in an array.
+
+The Sentiment Analysis and Patent Acceptance Prediction pages are both stored on one interface, with a sidebar menu allowing a user to switch between the two. The page has a simple title, subtitle, and sidebar implementation through `Streamlit`:
+
+````python
+# Title
+st.title("CSGY-6613 Project")
+# Subtitle
+st.markdown("_**Ryan Kim (rk2546)**_")
+st.markdown("---")
+
+def PageToHome():
+    st.session_state.page = "home"
+def PageToEmotion():
+    st.session_state.page = "emotion"
+def PageToPatent():
+    st.session_state.page = "patent"
+
+with st.sidebar:
+    st.subheader("Toolbox")
+    home_selected = st.button("Home", on_click=PageToHome)
+    emotion_selected = st.button(
+        "Emotion Analysis [Milestone #2]", 
+        on_click=PageToEmotion
+    )
+    patent_selected = st.button(
+        "Patent Prediction [Milestone #3]", 
+        on_click=PageToPatent
+    )
+````
+
+We store the current page of the user inside an `st.session_state` dictionary, which persists every time the page loads or changes. Because `Streamlit` will only re-render the page every time a change is made to the interface - this means that variables not stored in a session will be re-set. Alongside the current page, we also store models and user inputs inside of the session as well, which allows them to persist between `Streamlit` re-renderings.
+
+Whenever we switch between pages via the sidebar, a simple `if-else` statement ensure that the proper page is loaded: 
+
+````python
+if st.session_state.page == "emotion":
+    st.subheader("Sentiment Analysis")
+    if "emotion_model" not in st.session_state:
+        st.write("Loading model...")
+    else:
+        // ...
+    
+elif st.session_state.page == "patent":
+    st.subheader("USPTO Patent Evaluation")
+    // ...
+````
+
+#### **Sentiment Analysis**
+
+Sentiment Analysis is relatively simple. It uses the `ModelImplementation` class detailed above to switch between four pre-existing HuggingFace models for the sentiment analysis:
+
+- [cardiffnlp/twitter-roberta-base-sentiment](https://huggingface.co/cardiffnlp/twitter-roberta-base-sentiment)
+- [finiteautomata/beto-sentiment-analysis](https://huggingface.co/finiteautomata/beto-sentiment-analysis)
+- [bhadresh-savani/distilbert-base-uncased-emotion](https://huggingface.co/bhadresh-savani/distilbert-base-uncased-emotion)
+- [siebert/sentiment-roberta-large-english](https://huggingface.co/siebert/sentiment-roberta-large-english)
+
+A method called `ParseEmotionOutput()` is used to process labels outputted by the [cardiffnlp/twitter-roberta-base-sentiment](https://huggingface.co/cardiffnlp/twitter-roberta-base-sentiment) model in particular.
+
+Upon loading, if a model hasn't been instantiated yet, the page will create a new model with a pre-set model name and cache it for later use:
+
+````python
+def emotion_model_change():
+    st.session_state.emotion_model = ModelImplementation(
+        st.session_state.emotion_model_name,
+        AutoModelForSequenceClassification,
+        st.session_state.emotion_model_name,
+        AutoTokenizer,
+        pipeline,
+        ParseEmotionOutput,
+        classifier_args={ "task" : "sentiment-analysis" },
+        placeholders=["@AmericanAir just landed - 3hours Late Flight - and now we need to wait TWENTY MORE MINUTES for a gate! I have patience but none for incompetence."]
+    )
+
+if "emotion_model_name" not in st.session_state:
+    st.session_state.emotion_model_name = "cardiffnlp/twitter-roberta-base-sentiment"
+    emotion_model_change()
+````
+
+The method `emotion_model_change()` can be called to switch between different models, based on the session-saved `emotion_model_name` value. By default, the [cardiffnlp/twitter-roberta-base-sentiment](https://huggingface.co/cardiffnlp/twitter-roberta-base-sentiment) model is used. To switch between models, we use a `Streamlit` `selectbox` module:
+
+````python
+model_option = st.selectbox(
+    "What sentiment analysis model do you want to use? NOTE: Lag may occur when loading a new model!",
+    emotion_model_names,
+    on_change=emotion_model_change,
+    key="emotion_model_name"
+)
+
+form = st.form(key='sentiment-analysis-form')
+text_input = form.text_area(
+    "Enter some text for sentiment analysis! If you just want to test it out without entering anything, just press the \"Submit\" button and the model will look at the placeholder.", 
+    placeholder=st.session_state.emotion_model.placeholders[0]
+)
+submit = form.form_submit_button('Submit')
+````
+
+When the page loads, a placeholder from the current model is printed inside a `Streamlit` `text_area` module. When the user clicks on the `form_submit_button` button, the app will use whatever model is currently cached to generate output predictions. If no user input was provided, the placeholder value is passed as the input instead.
+
+````python
+if submit:
+    if text_input is None or len(text_input.strip()) == 0:
+        to_eval = st.session_state.emotion_model.placeholders[0]
+    else:
+        to_eval = text_input.strip()
+    st.write("You entered:")
+    st.markdown("> {}".format(to_eval))
+    st.write("Using the NLP model:")
+    st.markdown("> {}".format(st.session_state.emotion_model_name))
+    label, score = st.session_state.emotion_model.predict(to_eval)  
+    st.markdown("#### Result:")
+    st.markdown("**{}**: {}".format(label,score))
+````
+
+#### **USPTO Patent Acceptance Prediction**
+
+The back-end for the USPTO Patent Acceptance Prediction is similar to that of **Sentiment Analysis**, but with some major differences.
+
+Instead of switching between for pre-trained HuggingFace models, we use two fine-tuned models. These models are available online:
+
+- [rk2546/uspto-patents-abstracts](https://huggingface.co/rk2546/uspto-patents-abstracts)
+- [rk2546/uspto-patents-claims](https://huggingface.co/rk2546/uspto-patents-claims)
+
+and are built off of a pre-existing model named [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased). They are fine-tuned off of the USPTO dataset. The tokenizer used to parse text uses the same [distilbert-base-uncased](https://huggingface.co/distilbert-base-uncased) model but is left unmodified.
+
+````python
+if "patent_data" not in st.session_state:
+    f = open('./data/val.json')
+    valData = json.load(f)
+    f.close()
+
+    patent_data = {}
+    for num, label, abstract, claim in zip(valData["patent_numbers"],valData["labels"], valData["abstracts"], valData["claims"]):
+        patent_data[num] = {"patent_number":num,"label":label,"abstract":abstract,"claim":claim}
+
+    st.session_state.patent_data = patent_data
+    st.session_state.patent_num = list(patent_data.keys())[0]
+    st.session_state.weight = 0.5
+    st.session_state.patent_abstract_model = ModelImplementation(
+        'rk2546/uspto-patents-abstracts',
+        DistilBertForSequenceClassification,
+        'distilbert-base-uncased',
+        DistilBertTokenizerFast,
+        TextClassificationPipeline,
+        ParsePatentOutput,
+        classifier_args={"return_all_scores":True},
+    )
+    print("Patent abstracts model initialized")
+    st.session_state.patent_claim_model = ModelImplementation(
+        'rk2546/uspto-patents-claims',
+        DistilBertForSequenceClassification,
+        'distilbert-base-uncased',
+        DistilBertTokenizerFast,
+        TextClassificationPipeline,
+        ParsePatentOutput,
+        classifier_args={"return_all_scores":True},
+    )
+    print("Patent claims model initialized")
+````
+
+In addition, instead of the user selecting a particular model to use, instead the interface allows for two separate inputs - one for an Abstract draft and another for a Claims draft. These fields can be pre-populated using an `st.selectbox` that, when selected, pre-fills the two inputs with examples from `val.json`:
+
+````python
+patent_select_list = list(st.session_state.patent_data.keys())
+patent_index_option = st.selectbox(
+    "Want to pre-populate with an existing patent? Select the index number of  below.",
+    patent_select_list,
+    key="patent_num",
+)
+
+//...
+
+with st.form(key='patent-form'):
+    col1, col2 = st.columns(2)
+    with col1:
+        abstract_input = st.text_area(
+            "Enter the abstract of the patent below", 
+            placeholder=st.session_state.patent_data[st.session_state.patent_num]["abstract"],
+            height=400
+        )
+    with col2:
+        claim_input = st.text_area(
+            "Enter the claims of the patent below", 
+            placeholder=st.session_state.patent_data[st.session_state.patent_num]["claim"],
+            height=400
+        )
+    weight_val = st.slider(
+        "How much do the abstract and claims weight when aggregating a total softmax score?",
+        min_value=-1.0,
+        max_value=1.0,
+        value=0.5,
+    )
+    submit = st.form_submit_button('Submit')
+````
+
+One unique addition is a slider `st.slider` that lets the user tune how much the two models' softmax outputs are combined. This way, some customizability is allowed - if the user wants only an Abstract to be used, then the weight can be adjusted to bias the Abstract model's output completely.
+
+### **Training the USPTO Patent Acceptance Predictor**
+
+While the Sentiment Analysis function does not need to be trained, the model for the patent prediction needs to be fine-tuned from an existing model. The scripts that do so are available in both `src/train.py` and `src/patent_train.ipynb`, with the latter being a more user-readable option. **NOTE:** the `src/train.py` implementation is a tad outdated. Please rely on `src/patent_train.ipynb` if you want to try for yourself.
+
+The general idea was to divide the training into steps:
+
+1. Parse only the necessary data out from the raw USPTO data,
+2. Separate the data into training and validation data
+3. Fine-tune an existing HuggingFace model into two separate models, each of which separately perform predictions onto abstracts and claims respectively.
+4. Publish the two models onto HuggingFace to make the models easily accessible.
+
+The only data columns parsed from the original USPTO data were:
+- "patent_number" <- Unique identifier
+- "abstract" <- Input #1
+- "claims" <- Input #2
+- "decision" <- Output Labels
+
+We also filter rows such that the values in the "decision" column only consist of "ACCEPTED" or "REJECTED"; these values are then coded into `1` and `0` respectively for easier data processing. Beyond splitting the data into training and validation data, we also split each into two sub-datasets - one consisting of Abstracts, and another for Claims. No pre-processing was performed further.
+
+During fine-tuning, we use a custom class that will be used by HuggingFace's various transformers:
+
+````python
+class USPTODataset(Dataset):
+    def __init__(self, encodings, labels):
+        self.encodings = encodings
+        self.labels = labels
+    def __getitem__(self, idx):
+        item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}
+        item['labels'] = torch.tensor(self.labels[idx])
+        return item
+    def __len__(self):
+        return len(self.labels)
+````
+
+The training data is fitted into an instance of this class, and that instance is fed into the training algorithm. First, all input strings are tokenized. The tokenized data is then fed into instances of `USPTODataset`.
+
+````python
+train_abstracts_encodings = tokenizer(trainData["abstracts"], truncation=True, padding=True)
+train_claims_encodings = tokenizer(trainData["claims"], truncation=True, padding=True)
+
+train_abstracts_dataset = USPTODataset(train_abstracts_encodings, trainData["labels"])
+train_claims_dataset = USPTODataset(train_claims_encodings, trainData["labels"])
+````
+
+When initializing the model that we'll fine-tune, we also add an additional step to run the training on the GPU rather than the CPU. This is a time optimization measure.
+
+````python
+device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')
+model = DistilBertForSequenceClassification.from_pretrained(model_name)
+model.to(device)
+model.train()
+````
+
+Finally, we load the datasets into wrapper classes that are used during the training. We then train for `10` epochs.
+
+````python
+train_abstracts_loader = DataLoader(train_abstracts_dataset, batch_size=32, shuffle=True)
+train_claims_loader = DataLoader(train_claims_dataset, batch_size=32, shuffle=True)
+optim = AdamW(model.parameters(), lr=5e-5)
+
+def Train(loader, save_path, num_train_epochs=2):
+  batch_num = len(loader)
+  for epoch in range(num_train_epochs):
+    print(f'\t- Training epoch {epoch+1}/{num_train_epochs}')
+    batch_count = 0
+    for batch in loader:
+      print(f'{batch_count}|{batch_num} - {round((batch_count/batch_num)*100)}%', end="")
+      #print('\t\t- optim zero grad')
+      optim.zero_grad()
+      #print('\t\t- input_ids')
+      input_ids = batch['input_ids'].to(device)
+      #print('\t\t- attention_mask')
+      attention_mask = batch['attention_mask'].to(device)
+      #print('\t\t- labels0')
+      labels = batch['labels'].to(device)
+      #print('\t\t- outputs')
+      outputs = model(input_ids, attention_mask=attention_mask, labels=labels)
+            
+      #print('\t\t- loss')
+      loss = outputs[0]
+      #print('\t\t- backwards')
+      loss.backward()
+      #print('\t\t- step')
+      optim.step()
+
+      batch_count += 1
+      print("\r", end="")
+    
+    model.save_pretrained(save_path, from_pt=True) 
+    print(f'Saved model in {save_path}!\n')
+
+print("=== TRAINING ABSTRACTS ===")
+Train(train_abstracts_loader,upsto_abstracts_model_path, num_train_epochs=10)
+print("----")
+print("=== TRAINING CLAIMS ===")
+Train(train_claims_loader,upsto_claims_model_path, num_train_epochs=10)
+````
+
+---
+
 ## Milestone 3
 
 Click the link to access the HuggingFace Streamlit application: