add all application files

LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

README.md

@@ -1,13 +1,54 @@
 ---
-title: Exploring Intelligent Writing Assistance
-emoji: 🏃
-colorFrom: gray
-colorTo: yellow
+title: Exploring Intelligent Writing Assistance with Text Style Transfer
+emoji: :twisted_rightwards_arrows:
+colorFrom: blue
+colorTo: green
 sdk: streamlit
 sdk_version: 1.10.0
-app_file: app.py
-pinned: false
+app_file: apps/app.py
+models: ["sentence-transformers/all-MiniLM-L6-v2", "cffl/bert-base-styleclassification-subjective-neutral", "cffl/bart-base-styletransfer-subjective-to-neutral", "cointegrated/roberta-base-formality", "prithivida/informal_to_formal_styletransfer"]
+pinned: true
 license: apache-2.0
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# Exploring Intelligent Writing Assistance
+
+A demonstration of how the NLP task of _text style transfer_ can be applied to enhance the human writing experience using [HuggingFace Transformers](https://huggingface.co/) and [Streamlit](https://streamlit.io/).
+
+![](/static/images/app_screenshot.png)
+
+> This repo accompanies Cloudera Fast Forward Labs' [blog series](https://blog.fastforwardlabs.com/2022/03/22/an-introduction-to-text-style-transfer.html) in which we explore the task of automatically neutralizing subjectivity bias in free text.
+
+The goal of this application is to demonstrate how the NLP task of text style transfer can be applied to enhance the human writing experience. In this sense, we intend to peel back the curtains on how an intelligent writing assistant might function — walking through the logical steps needed to automatically re-style a piece of text (from informal-to-formal **or** subjective-to-neutral) while building up confidence in the model output.
+
+Through the application, we emphasize the imperative for a human-in-the-loop user experience when designing natural language generation systems. We believe text style transfer has the potential to empower writers to better express themselves, but not by blindly generating text. Rather, generative models, in conjunction with interpretability methods, should be combined to help writers understand the nuances of linguistic style and suggest stylistic edits that may improve their writing.
+
+## Project Structure
+
+```
+.
+├── LICENSE
+├── README.md
+├── apps
+│   ├── app.py
+│   ├── app_utils.py
+│   ├── data_utils.py
+│   └── visualization_utils.py
+├── requirements.txt
+├── scripts
+│   ├── download_models.py
+│   ├── install_dependencies.py
+│   └── launch_app.py
+├── setup.py
+├── src
+│   ├── __init__.py
+│   ├── content_preservation.py
+│   ├── style_classification.py
+│   ├── style_transfer.py
+│   └── transformer_interpretability.py
+├── static
+│   └── images
+└── tests
+    ├── __init__.py
+    └── test_model_classes.py
+```

apps/app.py

@@ -0,0 +1,386 @@
+# ###########################################################################
+#
+#  CLOUDERA APPLIED MACHINE LEARNING PROTOTYPE (AMP)
+#  (C) Cloudera, Inc. 2022
+#  All rights reserved.
+#
+#  Applicable Open Source License: Apache 2.0
+#
+#  NOTE: Cloudera open source products are modular software products
+#  made up of hundreds of individual components, each of which was
+#  individually copyrighted.  Each Cloudera open source product is a
+#  collective work under U.S. Copyright Law. Your license to use the
+#  collective work is as provided in your written agreement with
+#  Cloudera.  Used apart from the collective work, this file is
+#  licensed for your use pursuant to the open source license
+#  identified above.
+#
+#  This code is provided to you pursuant a written agreement with
+#  (i) Cloudera, Inc. or (ii) a third-party authorized to distribute
+#  this code. If you do not have a written agreement with Cloudera nor
+#  with an authorized and properly licensed third party, you do not
+#  have any rights to access nor to use this code.
+#
+#  Absent a written agreement with Cloudera, Inc. (“Cloudera”) to the
+#  contrary, A) CLOUDERA PROVIDES THIS CODE TO YOU WITHOUT WARRANTIES OF ANY
+#  KIND; (B) CLOUDERA DISCLAIMS ANY AND ALL EXPRESS AND IMPLIED
+#  WARRANTIES WITH RESPECT TO THIS CODE, INCLUDING BUT NOT LIMITED TO
+#  IMPLIED WARRANTIES OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY AND
+#  FITNESS FOR A PARTICULAR PURPOSE; (C) CLOUDERA IS NOT LIABLE TO YOU,
+#  AND WILL NOT DEFEND, INDEMNIFY, NOR HOLD YOU HARMLESS FOR ANY CLAIMS
+#  ARISING FROM OR RELATED TO THE CODE; AND (D)WITH RESPECT TO YOUR EXERCISE
+#  OF ANY RIGHTS GRANTED TO YOU FOR THE CODE, CLOUDERA IS NOT LIABLE FOR ANY
+#  DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, PUNITIVE OR
+#  CONSEQUENTIAL DAMAGES INCLUDING, BUT NOT LIMITED TO, DAMAGES
+#  RELATED TO LOST REVENUE, LOST PROFITS, LOSS OF INCOME, LOSS OF
+#  BUSINESS ADVANTAGE OR UNAVAILABILITY, OR LOSS OR CORRUPTION OF
+#  DATA.
+#
+# ###########################################################################
+
+import pandas as pd
+from PIL import Image
+import streamlit as st
+import streamlit.components.v1 as components
+
+from apps.data_utils import (
+    DATA_PACKET,
+    format_classification_results,
+)
+from apps.app_utils import (
+    DisableableButton,
+    reset_page_progress_state,
+    get_cached_style_intensity_classifier,
+    get_cached_word_attributions,
+    get_sti_metric,
+    get_cps_metric,
+    generate_style_transfer,
+)
+from apps.visualization_utils import build_altair_classification_plot
+
+# SESSION STATE UTILS
+if "page_progress" not in st.session_state:
+    st.session_state.page_progress = 1
+
+if "st_result" not in st.session_state:
+    st.session_state.st_result = False
+
+
+# PAGE CONFIG
+ffl_favicon = Image.open("static/images/cldr-favicon.ico")
+st.set_page_config(
+    page_title="CFFL: Text Style Transfer",
+    page_icon=ffl_favicon,
+    layout="centered",
+    initial_sidebar_state="expanded",
+)
+
+# SIDEBAR
+ffl_logo = Image.open("static/images/ffllogo2@1x.png")
+st.sidebar.image(ffl_logo)
+
+st.sidebar.write(
+    "This prototype accompanies our [Text Style Transfer](https://blog.fastforwardlabs.com/2022/03/22/an-introduction-to-text-style-transfer.html)\
+     blog series in which we explore the task of automatically neutralizing subjectivity bias in free text."
+)
+
+st.sidebar.markdown("## Select a style attribute")
+style_attribute = st.sidebar.selectbox(
+    "What style would you like to transfer between?", options=DATA_PACKET.keys()
+)
+STYLE_ATTRIBUTE_DATA = DATA_PACKET[style_attribute]
+
+st.sidebar.markdown("## Start over")
+st.sidebar.caption(
+    "This application is intended to be run sequentially from top to bottom. If you wish to alter selections after \
+    advancing through the app, push the button below to start fresh from the beginning."
+)
+st.sidebar.button("Restart from beginning", on_click=reset_page_progress_state)
+
+# MAIN CONTENT
+st.markdown("# Exploring Intelligent Writing Assistance")
+
+st.write(
+    """
+    The goal of this application is to demonstrate how the NLP task of _text style transfer_ can be applied to enhance the human writing experience. 
+    In this sense, we intend to peel back the curtains on how an intelligent writing assistant might function — walking through the logical steps needed to 
+    automatically re-style a piece of text while building up confidence in the model output. 
+
+    We emphasize the imperative for a human-in-the-loop user experience when designing natural language generation systems. We believe text style 
+    transfer has the potential to empower writers to better express themselves, but not by blindly generating text. Rather, generative models, in conjunction with 
+    interpretability methods, should be combined to help writers understand the nuances of linguistic style and suggest stylistic edits that _may_ improve their writing.
+
+    Select a style attribute from the sidebar and enter some text below to get started!
+    """
+)
+
+## 1. INPUT EXAMPLE
+if st.session_state.page_progress > 0:
+    st.write("### 1. Input some text")
+
+    with st.container():
+
+        col1_1, col1_2 = st.columns([1, 3])
+        with col1_1:
+            input_type = st.radio(
+                "Type your own or choose from a preset example",
+                ("Choose preset", "Enter text"),
+                horizontal=False,
+            )
+        with col1_2:
+            if input_type == "Enter text":
+                text_sample = st.text_input(
+                    f"Enter some text to modify style from {style_attribute}",
+                    help="You can also select one of our preset examples by toggling the radio button to the left.",
+                )
+            else:
+                option = st.selectbox(
+                    f"Select a preset example to modify style from {style_attribute}",
+                    [
+                        f"Example {i+1}"
+                        for i in range(len(STYLE_ATTRIBUTE_DATA.examples))
+                    ],
+                    help="You can also enter your own text by toggling the radio button to the left.",
+                )
+
+                idx_key = int(option.split(" ")[-1]) - 1
+                text_sample = STYLE_ATTRIBUTE_DATA.examples[idx_key]
+
+        st.text_area(
+            "Preview Text",
+            value=text_sample,
+            placeholder="Enter some text above or toggle to choose a preset!",
+            disabled=True,
+        )
+
+    if text_sample != "":
+        db1 = DisableableButton(1, "Let's go!")
+        db1.create_enabled_button()
+
+## 2. CLASSIFY INPUT
+if st.session_state.page_progress > 1:
+    db1.disable()
+
+    st.write("### 2. Detect style")
+    st.write(
+        f"""
+            Before we can transfer style, we need to ensure the input text isn't already of the target style! To do so,
+            we classify the sample text with a model that has been fine-tuned to differentiate between
+            {STYLE_ATTRIBUTE_DATA.attribute_AND_string} tones. 
+            
+            In a product setting, you could imagine this style detection process running continuously inside your favorite word processor as you write, 
+            prompting you for action when it detects language that is at odds with your desired tone of voice.
+            """
+    )
+
+    with st.spinner("Detecting style, hang tight!"):
+
+        sic = get_cached_style_intensity_classifier(style_data=STYLE_ATTRIBUTE_DATA)
+        cls_result = sic.score(text_sample)
+
+    cls_result_df = pd.DataFrame(
+        cls_result[0]["distribution"],
+        columns=["Score"],
+        index=[v for k, v in sic.pipeline.model.config.id2label.items()],
+    )
+
+    with st.container():
+
+        format_cls_result = format_classification_results(
+            id2label=sic.pipeline.model.config.id2label, cls_result=cls_result
+        )
+        st.markdown("##### Distribution Between Style Classes")
+        chart = build_altair_classification_plot(format_cls_result)
+        st.altair_chart(chart.interactive(), use_container_width=True)
+
+        st.markdown(
+            f"""
+            - **:hugging_face: Model:** [{STYLE_ATTRIBUTE_DATA.cls_model_path}]({STYLE_ATTRIBUTE_DATA.build_model_url("cls")})
+            - **Input Text:** *"{text_sample}"*
+            - **Classification Result:** \t {cls_result[0]["label"].capitalize()} ({round(cls_result[0]["score"]*100, 2)}%)
+            """
+        )
+        st.write(" ")
+
+    if cls_result[0]["label"].lower() != STYLE_ATTRIBUTE_DATA.target_attribute:
+        st.info(
+            f"""
+            **Looks like we have room for improvement!**
+            
+            The distribution of classification scores suggests that the input text is more {STYLE_ATTRIBUTE_DATA.attribute_THAN_string}. Therefore,
+            an automated style transfer may improve our intended tone of voice."""
+        )
+        db2 = DisableableButton(2, "Let's see why")
+        db2.create_enabled_button()
+    else:
+        st.success(
+            f"""**No work to be done!** \n\n\n The distribution of classification scores suggests that the input text is less \
+            {STYLE_ATTRIBUTE_DATA.attribute_THAN_string}. Therefore, there's no need to transfer style. \
+            Enter a different text prompt or select one of the preset examples to re-run the analysis with."""
+        )
+
+## 3. Here's why
+if st.session_state.page_progress > 2:
+    db2.disable()
+    st.write("### 3. Interpret the classification result")
+    st.write(
+        f"""
+        Interpreting our model's output is a crucial practice that helps build trust and justify taking real-world action from the
+        model predictions. In this case, we apply a popular model interpretability technique called [Integrated Gradients](https://arxiv.org/pdf/1703.01365.pdf) 
+        to the Transformer-based classifier to explain the model's prediction in terms of its features."""
+    )
+
+    with st.spinner("Interpreting the prediction, hang tight!"):
+        word_attributions_visual = get_cached_word_attributions(
+            text_sample=text_sample, style_data=STYLE_ATTRIBUTE_DATA
+        )
+        components.html(html=word_attributions_visual, height=200, scrolling=True)
+
+    st.write(
+        f"""
+        The visual above displays word attributions using the [Transformers Interpret](https://github.com/cdpierse/transformers-interpret) library. 
+        Positive attribution values (green) indicate tokens that contribute positively towards the 
+        predicted class ({STYLE_ATTRIBUTE_DATA.source_attribute}), while negative values (red) indicate tokens that contribute negatively towards the predicted class.
+        
+        Visualizing word attributions is a helpful way to build intuition about what makes the input text _{STYLE_ATTRIBUTE_DATA.source_attribute}_!"""
+    )
+    db3 = DisableableButton(3, "Next")
+    db3.create_enabled_button()
+
+
+## 4. SUGGEST GENERATED EDIT
+if st.session_state.page_progress > 3:
+    db3.disable()
+
+    st.write("### 4. Generate a suggestion")
+    st.write(
+        f"Now that we've verified the input text is in fact *{STYLE_ATTRIBUTE_DATA.source_attribute}* and understand why that's the case, we can utilize a \
+            text style transfer model to generate a suggested replacement that retains the same semantic meaning, but achieves the *{STYLE_ATTRIBUTE_DATA.target_attribute}* target style.\
+            \n\n Expand the accordian below to toggle generation parameters, then click the button to transfer style!"
+    )
+
+    with st.expander("Toggle generation parameters"):
+
+        # st.markdown("##### Text generation parameters")
+        st.write("**max_gen_length**")
+        max_gen_length = st.slider(
+            "Whats the maximum generation length desired?", 1, 250, 200, 10
+        )
+        st.write("**num_beams**")
+        num_beams = st.slider(
+            "How many beams to use for beam-search decoding?", 1, 8, 4
+        )
+        st.write("**temperature**")
+        temperature = st.slider(
+            "What sensitivity value to model next token probabilities?",
+            0.0,
+            1.0,
+            1.0,
+        )
+
+    st.markdown(
+        f"""
+        **:hugging_face: Model:** [{STYLE_ATTRIBUTE_DATA.seq2seq_model_path}]({STYLE_ATTRIBUTE_DATA.build_model_url("seq2seq")})
+        """
+    )
+
+    col4_1, col4_2, col4_3 = st.columns([1, 5, 4])
+    with col4_2:
+        st.markdown(
+            f"""
+            - **Max Generation Length:** {max_gen_length}
+            - **Number of Beams:** {num_beams}
+            - **Temperature:** {temperature}
+            """
+        )
+    with col4_3:
+        with st.container():
+            st.write("")
+            st.button(
+                "Generate style transfer",
+                key="generate_text",
+                on_click=generate_style_transfer,
+                kwargs={
+                    "text_sample": text_sample,
+                    "style_data": STYLE_ATTRIBUTE_DATA,
+                    "max_gen_length": max_gen_length,
+                    "num_beams": num_beams,
+                    "temperature": temperature,
+                },
+            )
+
+    if st.session_state.st_result:
+        st.warning(
+            f"""**{STYLE_ATTRIBUTE_DATA.source_attribute.capitalize()} Input:** "{text_sample}" """
+        )
+        st.info(
+            f"""
+            **{STYLE_ATTRIBUTE_DATA.target_attribute.capitalize()} Suggestion:** "{st.session_state.st_result[0]}" """
+        )
+        db4 = DisableableButton(4, "Next")
+        db4.create_enabled_button()
+
+## 5. EVALUATE THE SUGGESTION
+if st.session_state.page_progress > 4:
+    db4.disable()
+    st.write("### 5. Evaluate the suggestion")
+    st.markdown(
+        """
+        Blindly prompting a writer with style suggestions without first checking quality would make for a noisy, error-prone product
+        with a poor user experience. Ultimately, we only want to suggest high quality edits. But what makes for a suggestion-worthy edit?
+
+        A comprehensive quality evaluation for text style transfer output should consider three criteria:
+        1. **Style strength** - To what degree does the generated text achieve the target style? 
+        2. **Content preservation** - To what degree does the generated text retain the semantic meaning of the source text?
+        3. **Fluency** - To what degree does the generated text appear as if it were produced naturally by a human?
+
+        Below, we apply automated evaluation metrics - _Style Transfer Intensity (STI)_ & _Content Preservation Score (CPS)_ - to
+        measure the first two of these criteria by comparing the input text to the generated suggestion.
+        """
+    )
+
+    with st.spinner("Evaluating text style transfer, hang tight!"):
+
+        sti = get_sti_metric(
+            input_text=text_sample,
+            output_text=st.session_state.st_result[0],
+            style_data=STYLE_ATTRIBUTE_DATA,
+        )
+        cps = get_cps_metric(
+            input_text=text_sample,
+            output_text=st.session_state.st_result[0],
+            style_data=STYLE_ATTRIBUTE_DATA,
+        )
+
+    st.markdown(
+        """<hr style="height:2px;border:none;color:#333;background-color:#333;" /> """,
+        unsafe_allow_html=True,
+    )
+
+    col5_1, col5_2, col5_3 = st.columns([3, 1, 3])
+
+    with col5_1:
+        st.metric(
+            label="Style Transfer Intensity (STI)",
+            value=f"{round(sti[0]*100,2)}%",
+        )
+        st.caption(
+            f"""
+                STI compares the style class distributions (determined by the [style classifier]({STYLE_ATTRIBUTE_DATA.build_model_url("cls")}))
+                between the input text and generated suggestion using Earth Mover's Distance. Therefore, STI can be thought of as the percentage of the possible target style distribution
+                that was captured during the transfer.
+                """
+        )
+
+    with col5_3:
+        st.metric(
+            label="Content Preservation Score (CPS)",
+            value=f"{round(cps[0]*100,2)}%",
+        )
+        st.caption(
+            f"""
+                CPS compares the latent space embeddings (determined by [SentenceBERT]({STYLE_ATTRIBUTE_DATA.build_model_url("sbert")}))
+                between the input text and generated suggestion using cosine similarity. Therefore, CPS can be thought of as the percentage of content that was perserved
+                during the style transfer.
+                """
+        )

apps/app_utils.py

@@ -0,0 +1,262 @@
+# ###########################################################################
+#
+#  CLOUDERA APPLIED MACHINE LEARNING PROTOTYPE (AMP)
+#  (C) Cloudera, Inc. 2022
+#  All rights reserved.
+#
+#  Applicable Open Source License: Apache 2.0
+#
+#  NOTE: Cloudera open source products are modular software products
+#  made up of hundreds of individual components, each of which was
+#  individually copyrighted.  Each Cloudera open source product is a
+#  collective work under U.S. Copyright Law. Your license to use the
+#  collective work is as provided in your written agreement with
+#  Cloudera.  Used apart from the collective work, this file is
+#  licensed for your use pursuant to the open source license
+#  identified above.
+#
+#  This code is provided to you pursuant a written agreement with
+#  (i) Cloudera, Inc. or (ii) a third-party authorized to distribute
+#  this code. If you do not have a written agreement with Cloudera nor
+#  with an authorized and properly licensed third party, you do not
+#  have any rights to access nor to use this code.
+#
+#  Absent a written agreement with Cloudera, Inc. (“Cloudera”) to the
+#  contrary, A) CLOUDERA PROVIDES THIS CODE TO YOU WITHOUT WARRANTIES OF ANY
+#  KIND; (B) CLOUDERA DISCLAIMS ANY AND ALL EXPRESS AND IMPLIED
+#  WARRANTIES WITH RESPECT TO THIS CODE, INCLUDING BUT NOT LIMITED TO
+#  IMPLIED WARRANTIES OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY AND
+#  FITNESS FOR A PARTICULAR PURPOSE; (C) CLOUDERA IS NOT LIABLE TO YOU,
+#  AND WILL NOT DEFEND, INDEMNIFY, NOR HOLD YOU HARMLESS FOR ANY CLAIMS
+#  ARISING FROM OR RELATED TO THE CODE; AND (D)WITH RESPECT TO YOUR EXERCISE
+#  OF ANY RIGHTS GRANTED TO YOU FOR THE CODE, CLOUDERA IS NOT LIABLE FOR ANY
+#  DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, PUNITIVE OR
+#  CONSEQUENTIAL DAMAGES INCLUDING, BUT NOT LIMITED TO, DAMAGES
+#  RELATED TO LOST REVENUE, LOST PROFITS, LOSS OF INCOME, LOSS OF
+#  BUSINESS ADVANTAGE OR UNAVAILABILITY, OR LOSS OR CORRUPTION OF
+#  DATA.
+#
+# ###########################################################################
+
+from typing import List
+
+import tokenizers
+import streamlit as st
+
+from src.style_transfer import StyleTransfer
+from src.style_classification import StyleIntensityClassifier
+from src.content_preservation import ContentPreservationScorer
+from src.transformer_interpretability import InterpretTransformer
+from apps.data_utils import StyleAttributeData, string_to_list_string
+
+# CALLBACKS
+def increment_page_progress():
+    st.session_state.page_progress += 1
+
+
+def reset_page_progress_state():
+    del st.session_state.st_result
+    st.session_state.page_progress = 1
+
+
+# UTILITY CLASSES
+class DisableableButton:
+    """
+    Utility class for creating "disable-able" buttons upon click.
+
+    We initialize an empty container, then update that container with buttons
+    upon calling `create_enabled_button` and `disable` methods where clicking
+    is enabled and then disabled, respectively.
+
+    """
+
+    def __init__(self, button_number, button_text):
+        self.button_number = button_number
+        self.button_text = button_text
+
+    def _init_placeholder_container(self):
+        self.ph = st.empty()
+
+    def create_enabled_button(self):
+        self._init_placeholder_container()
+        self.ph.button(
+            self.button_text,
+            on_click=increment_page_progress,
+            key=f"ph{self.button_number}_before",
+            disabled=False,
+        )
+
+    def disable(self):
+        self.ph.button(
+            self.button_text, key=f"ph{self.button_number}_after", disabled=True
+        )
+
+
+# CACHED FUNCTIONS
+@st.cache(
+    hash_funcs={tokenizers.Tokenizer: lambda _: None},
+    allow_output_mutation=True,
+    show_spinner=False,
+)
+def get_cached_style_intensity_classifier(
+    style_data: StyleAttributeData,
+) -> StyleIntensityClassifier:
+    """
+    Return a cached style classifier.
+
+    This function overwrites the existing model's config values for
+    `id2label` and `label2id`.
+
+    Args:
+        style_data (StyleAttributeData)
+
+    Returns:
+        StyleIntensityClassifier
+    """
+    sic = StyleIntensityClassifier(style_data.cls_model_path)
+
+    # create or overwrite id-label lookup in model config
+    sic.pipeline.model.config.__dict__["id2label"] = {
+        i: a
+        for i, a in enumerate(
+            [
+                style_data.source_attribute.capitalize(),
+                style_data.target_attribute.capitalize(),
+            ]
+        )
+    }
+    sic.pipeline.model.config.__dict__["label2id"] = {
+        v: k for k, v in sic.pipeline.model.config.__dict__["id2label"].items()
+    }
+
+    return sic
+
+
+@st.cache(
+    hash_funcs={tokenizers.Tokenizer: lambda _: None},
+    allow_output_mutation=True,
+    show_spinner=False,
+)
+def get_cached_word_attributions(
+    text_sample: str, style_data: StyleAttributeData
+) -> str:
+    """
+    Calculated word attributions and return HTML visual.
+
+     This function overwrites the existing model's config values for
+    `id2label` and `label2id`.
+
+    Args:
+        text_sample (str)
+        style_data (StyleAttributeData)
+
+    Returns:
+        str
+    """
+    it = InterpretTransformer(cls_model_identifier=style_data.cls_model_path)
+
+    # create or overwrite id-label lookup in model config
+    it.explainer.id2label = {
+        i: a
+        for i, a in enumerate(
+            [
+                style_data.source_attribute.capitalize(),
+                style_data.target_attribute.capitalize(),
+            ]
+        )
+    }
+    it.explainer.label2id = {v: k for k, v in it.explainer.id2label.items()}
+    return it.visualize_feature_attribution_scores(text_sample).data
+
+
+@st.cache(
+    hash_funcs={tokenizers.Tokenizer: lambda _: None},
+    allow_output_mutation=True,
+    show_spinner=False,
+)
+def get_sti_metric(
+    input_text: str, output_text: str, style_data: StyleAttributeData
+) -> List[float]:
+    """
+    Calculate Style Transfer Intensity (STI)
+
+    Args:
+        input_text (str)
+        output_text (str)
+        style_data (StyleAttributeData)
+
+    Returns:
+        List[float]
+    """
+    sti = StyleIntensityClassifier(
+        model_identifier=style_data.cls_model_path,
+    )
+    return sti.calculate_transfer_intensity_fraction(
+        string_to_list_string(input_text), string_to_list_string(output_text)
+    )
+
+
+@st.cache(
+    hash_funcs={tokenizers.Tokenizer: lambda _: None},
+    allow_output_mutation=True,
+    show_spinner=False,
+)
+def get_cps_metric(
+    input_text: str, output_text: str, style_data: StyleAttributeData
+) -> List[float]:
+    """
+    Calculate Content Preservation Score (CPS)
+
+    Args:
+        input_text (str)
+        output_text (str)
+        style_data (StyleAttributeData)
+
+    Returns:
+        List[float]
+    """
+    cps = ContentPreservationScorer(
+        cls_model_identifier=style_data.cls_model_path,
+        sbert_model_identifier=style_data.sbert_model_path,
+    )
+    return cps.calculate_content_preservation_score(
+        string_to_list_string(input_text),
+        string_to_list_string(output_text),
+        mask_type="none",
+    )
+
+
+def generate_style_transfer(
+    text_sample: str,
+    style_data: StyleAttributeData,
+    max_gen_length: int,
+    num_beams: int,
+    temperature: int,
+):
+    """
+    Run inference on seq2seq model and persist result to
+    `session_state` varaible.
+
+    Args:
+        text_sample (str): _description_
+        style_data (StyleAttributeData): _description_
+        max_gen_length (int): _description_
+        num_beams (int): _description_
+        temperature (int): _description_
+    """
+    with st.spinner("Transferring style, hang tight!"):
+
+        generate_kwargs = {
+            "max_gen_length": max_gen_length,
+            "num_beams": num_beams,
+            "temperature": temperature,
+        }
+
+        st_class = StyleTransfer(
+            model_identifier=style_data.seq2seq_model_path,
+            **generate_kwargs,
+        )
+
+        st_result = st_class.transfer(text_sample)
+
+    st.session_state.st_result = st_result

apps/data_utils.py

@@ -0,0 +1,150 @@
+# ###########################################################################
+#
+#  CLOUDERA APPLIED MACHINE LEARNING PROTOTYPE (AMP)
+#  (C) Cloudera, Inc. 2022
+#  All rights reserved.
+#
+#  Applicable Open Source License: Apache 2.0
+#
+#  NOTE: Cloudera open source products are modular software products
+#  made up of hundreds of individual components, each of which was
+#  individually copyrighted.  Each Cloudera open source product is a
+#  collective work under U.S. Copyright Law. Your license to use the
+#  collective work is as provided in your written agreement with
+#  Cloudera.  Used apart from the collective work, this file is
+#  licensed for your use pursuant to the open source license
+#  identified above.
+#
+#  This code is provided to you pursuant a written agreement with
+#  (i) Cloudera, Inc. or (ii) a third-party authorized to distribute
+#  this code. If you do not have a written agreement with Cloudera nor
+#  with an authorized and properly licensed third party, you do not
+#  have any rights to access nor to use this code.
+#
+#  Absent a written agreement with Cloudera, Inc. (“Cloudera”) to the
+#  contrary, A) CLOUDERA PROVIDES THIS CODE TO YOU WITHOUT WARRANTIES OF ANY
+#  KIND; (B) CLOUDERA DISCLAIMS ANY AND ALL EXPRESS AND IMPLIED
+#  WARRANTIES WITH RESPECT TO THIS CODE, INCLUDING BUT NOT LIMITED TO
+#  IMPLIED WARRANTIES OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY AND
+#  FITNESS FOR A PARTICULAR PURPOSE; (C) CLOUDERA IS NOT LIABLE TO YOU,
+#  AND WILL NOT DEFEND, INDEMNIFY, NOR HOLD YOU HARMLESS FOR ANY CLAIMS
+#  ARISING FROM OR RELATED TO THE CODE; AND (D)WITH RESPECT TO YOUR EXERCISE
+#  OF ANY RIGHTS GRANTED TO YOU FOR THE CODE, CLOUDERA IS NOT LIABLE FOR ANY
+#  DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, PUNITIVE OR
+#  CONSEQUENTIAL DAMAGES INCLUDING, BUT NOT LIMITED TO, DAMAGES
+#  RELATED TO LOST REVENUE, LOST PROFITS, LOSS OF INCOME, LOSS OF
+#  BUSINESS ADVANTAGE OR UNAVAILABILITY, OR LOSS OR CORRUPTION OF
+#  DATA.
+#
+# ###########################################################################
+
+import os
+from typing import List
+from collections import defaultdict
+from dataclasses import dataclass
+
+import numpy as np
+
+
+@dataclass
+class StyleAttributeData:
+    source_attribute: str
+    target_attribute: str
+    examples: List[str]
+    cls_model_path: str
+    seq2seq_model_path: str
+    sbert_model_path: str = "sentence-transformers/all-MiniLM-L6-v2"
+    hf_base_url: str = "https://huggingface.co/"
+
+    def __post_init__(self):
+        self._make_attribute_selection_string()
+        self._make_attribute_AND_string()
+        self._make_attribute_THAN_string()
+
+    def _make_attribute_selection_string(self):
+        self.attribute_selecting_string = (
+            f"{self.source_attribute}-{self.target_attribute}"
+        )
+
+    def _make_attribute_AND_string(self):
+        self.attribute_AND_string = (
+            f"**{self.source_attribute}** and **{self.target_attribute}**"
+        )
+
+    def _make_attribute_THAN_string(self):
+        self.attribute_THAN_string = (
+            f"**{self.source_attribute}** than **{self.target_attribute}**"
+        )
+
+    def build_model_url(self, model_type: str):
+        """
+        Build a complete HuggingFace url for the given `model_type`.
+
+        Args:
+            model_type (str): "cls", "seq2seq", "sbert"
+        """
+        attr_name = f"{model_type}_model_path"
+        return os.path.join(self.hf_base_url, getattr(self, attr_name))
+
+
+# instantiate data classes & collect all data class instances
+DATA_PACKET = {
+    "subjective-to-neutral": StyleAttributeData(
+        source_attribute="subjective",
+        target_attribute="neutral",
+        examples=[
+            "another strikingly elegant four-door design for the bentley s3 continental came from james.",
+            "the band plays an engaging and contagious rhythm known as brega pop and calypso.",
+            "chemical abstracts service (cas), a prominent division of the american chemical society, is the world's leading source of chemical information.",
+            "the final fight scene is with the martial arts great, master ninja sho kosugi.",
+        ],
+        cls_model_path="cffl/bert-base-styleclassification-subjective-neutral",
+        seq2seq_model_path="cffl/bart-base-styletransfer-subjective-to-neutral",
+    ),
+    "informal-to-formal": StyleAttributeData(
+        source_attribute="informal",
+        target_attribute="formal",
+        examples=[
+            "that was funny LOL",
+            "btw - ur avatar looks familiar",
+            "i loooooooooooooooooooooooove going to the movies.",
+            "haha, thatd be dope",
+        ],
+        cls_model_path="cointegrated/roberta-base-formality",
+        seq2seq_model_path="prithivida/informal_to_formal_styletransfer",
+    ),
+}
+
+
+def format_classification_results(id2label: dict, cls_result):
+    """
+    Formats classification output to be plotted using Altair.
+
+    Args:
+        id2label (dict): Transformer model's label dictionary
+        cls_result (List): Classification pipeline output
+    """
+
+    labels = [v for k, v in id2label.items()]
+
+    format_cls_result = []
+
+    for i in range(len(labels)):
+        temp = defaultdict()
+        temp["type"] = labels[i].capitalize()
+        temp["value"] = round(cls_result[0]["distribution"][i], 4)
+
+        if i == 0:
+            temp["percentage_start"] = 0
+            temp["percentage_end"] = temp["value"]
+        else:
+            temp["percentage_start"] = 1 - temp["value"]
+            temp["percentage_end"] = 1
+
+        format_cls_result.append(temp)
+
+    return format_cls_result
+
+
+def string_to_list_string(text: str):
+    return np.expand_dims(np.array(text), axis=0).tolist()

apps/visualization_utils.py

@@ -0,0 +1,159 @@
+# ###########################################################################
+#
+#  CLOUDERA APPLIED MACHINE LEARNING PROTOTYPE (AMP)
+#  (C) Cloudera, Inc. 2022
+#  All rights reserved.
+#
+#  Applicable Open Source License: Apache 2.0
+#
+#  NOTE: Cloudera open source products are modular software products
+#  made up of hundreds of individual components, each of which was
+#  individually copyrighted.  Each Cloudera open source product is a
+#  collective work under U.S. Copyright Law. Your license to use the
+#  collective work is as provided in your written agreement with
+#  Cloudera.  Used apart from the collective work, this file is
+#  licensed for your use pursuant to the open source license
+#  identified above.
+#
+#  This code is provided to you pursuant a written agreement with
+#  (i) Cloudera, Inc. or (ii) a third-party authorized to distribute
+#  this code. If you do not have a written agreement with Cloudera nor
+#  with an authorized and properly licensed third party, you do not
+#  have any rights to access nor to use this code.
+#
+#  Absent a written agreement with Cloudera, Inc. (“Cloudera”) to the
+#  contrary, A) CLOUDERA PROVIDES THIS CODE TO YOU WITHOUT WARRANTIES OF ANY
+#  KIND; (B) CLOUDERA DISCLAIMS ANY AND ALL EXPRESS AND IMPLIED
+#  WARRANTIES WITH RESPECT TO THIS CODE, INCLUDING BUT NOT LIMITED TO
+#  IMPLIED WARRANTIES OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY AND
+#  FITNESS FOR A PARTICULAR PURPOSE; (C) CLOUDERA IS NOT LIABLE TO YOU,
+#  AND WILL NOT DEFEND, INDEMNIFY, NOR HOLD YOU HARMLESS FOR ANY CLAIMS
+#  ARISING FROM OR RELATED TO THE CODE; AND (D)WITH RESPECT TO YOUR EXERCISE
+#  OF ANY RIGHTS GRANTED TO YOU FOR THE CODE, CLOUDERA IS NOT LIABLE FOR ANY
+#  DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, PUNITIVE OR
+#  CONSEQUENTIAL DAMAGES INCLUDING, BUT NOT LIMITED TO, DAMAGES
+#  RELATED TO LOST REVENUE, LOST PROFITS, LOSS OF INCOME, LOSS OF
+#  BUSINESS ADVANTAGE OR UNAVAILABILITY, OR LOSS OR CORRUPTION OF
+#  DATA.
+#
+# ###########################################################################
+
+from typing import Iterable
+
+import altair as alt
+from captum.attr._utils.visualization import (
+    VisualizationDataRecord,
+    format_word_importances,
+    _get_color,
+)
+
+try:
+    from IPython.display import display, HTML
+
+    HAS_IPYTHON = True
+except ImportError:
+    HAS_IPYTHON = False
+
+def format_classname(classname):
+    return f'<td>{classname}</td>'
+
+def visualize_text(
+    datarecords: Iterable[VisualizationDataRecord], legend: bool = True
+) -> "HTML":  # In quotes because this type doesn't exist in standalone mode
+    assert HAS_IPYTHON, (
+        "IPython must be available to visualize text. "
+        "Please run 'pip install ipython'."
+    )
+
+    dom = []
+    dom.append(
+        '<head><link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/bootstrap@4.0.0/dist/css/bootstrap.min.css" integrity="sha384-Gn5384xqQ1aoWXA+058RXPxPg6fy4IWvTNh0E263XmFcJlSAwiGgFAW/dAiS6JXm" crossorigin="anonymous"></head>'
+    )
+    dom.append("""<table width:100; class="table">""")
+    rows = [
+        "<thead>"
+        "<tr>"
+        "<th scope='col'><span class='text-nowrap'>Predicted Label</span></th>"
+        "<th scope='col'><span class='text-nowrap'>Attribution Score</span></th>"
+        "<th scope='col'><span class='text-nowrap'>Feature Importance</span></th>"
+        "</tr>"
+        "</thead>"
+    ]
+    for datarecord in datarecords:
+        rows.append(
+            "".join(
+                [
+                    "<tbody>",
+                    "<tr>",
+                    format_classname(
+                        f"{datarecord.pred_class.capitalize()}"
+                    ),
+                    format_classname(f"{round(datarecord.attr_score.item(), 2)}"),
+                    format_word_importances(
+                        datarecord.raw_input_ids, datarecord.word_attributions
+                    ),
+                    "<tr>",
+                    "</tbody>",
+                ]
+            )
+        )
+
+    dom.append("".join(rows))
+    dom.append("</table>")
+
+    if legend:
+        dom.append("<div class='row'>")
+        dom.append("<div class='col-6'>")
+        dom.append("<b>Legend: </b>")
+
+        for value, label in zip([-1, 0, 1], ["Negative", "Neutral", "Positive"]):
+            dom.append(
+                '<span style="display: inline-block; width: 10px; height: 10px; \
+                border: 1px solid; background-color: \
+                {value}"></span> {label}  '.format(
+                    value=_get_color(value), label=label
+                )
+            )
+        dom.append("</div>")
+        dom.append("<div class='col-6'></div>")
+
+        dom.append("</div>")
+
+    html = HTML("".join(dom))
+    display(html)
+
+    return html
+
+
+def build_altair_classification_plot(format_cls_result):
+    """
+    Builds Altair bar chart for classification results.
+
+    Args:
+        format_cls_result (List): Output from `format_classification_results()`
+    """
+    source = alt.pd.DataFrame(format_cls_result)
+
+    color_scale = alt.Scale(
+        domain=[record["type"] for record in format_cls_result],
+        range=["#00A3AF", "#F96702"],
+    )
+
+    c = (
+        alt.Chart(source)
+        .mark_bar(size=50)
+        .encode(
+            x=alt.X(
+                "percentage_start:Q", axis=alt.Axis(title="Style Distribution (%)")
+            ),
+            x2=alt.X2("percentage_end:Q"),
+            color=alt.Color(
+                "type:N",
+                legend=alt.Legend(title="Attribute"),
+                scale=color_scale,
+            ),
+        )
+        .properties(height=150)
+    )
+
+    return c

requirements.txt

@@ -0,0 +1,10 @@
+torch==1.12.0
+transformers==4.20.1
+pyemd==0.5.1
+streamlit==1.10.0
+attrs==21.4.0
+jinja2==3.1.2
+jsonschema==4.7.2
+pyrsistent==0.18.1
+transformers-interpret==0.7.2
+-e .

scripts/download_models.py

@@ -0,0 +1,70 @@
+# ###########################################################################
+#
+#  CLOUDERA APPLIED MACHINE LEARNING PROTOTYPE (AMP)
+#  (C) Cloudera, Inc. 2022
+#  All rights reserved.
+#
+#  Applicable Open Source License: Apache 2.0
+#
+#  NOTE: Cloudera open source products are modular software products
+#  made up of hundreds of individual components, each of which was
+#  individually copyrighted.  Each Cloudera open source product is a
+#  collective work under U.S. Copyright Law. Your license to use the
+#  collective work is as provided in your written agreement with
+#  Cloudera.  Used apart from the collective work, this file is
+#  licensed for your use pursuant to the open source license
+#  identified above.
+#
+#  This code is provided to you pursuant a written agreement with
+#  (i) Cloudera, Inc. or (ii) a third-party authorized to distribute
+#  this code. If you do not have a written agreement with Cloudera nor
+#  with an authorized and properly licensed third party, you do not
+#  have any rights to access nor to use this code.
+#
+#  Absent a written agreement with Cloudera, Inc. (“Cloudera”) to the
+#  contrary, A) CLOUDERA PROVIDES THIS CODE TO YOU WITHOUT WARRANTIES OF ANY
+#  KIND; (B) CLOUDERA DISCLAIMS ANY AND ALL EXPRESS AND IMPLIED
+#  WARRANTIES WITH RESPECT TO THIS CODE, INCLUDING BUT NOT LIMITED TO
+#  IMPLIED WARRANTIES OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY AND
+#  FITNESS FOR A PARTICULAR PURPOSE; (C) CLOUDERA IS NOT LIABLE TO YOU,
+#  AND WILL NOT DEFEND, INDEMNIFY, NOR HOLD YOU HARMLESS FOR ANY CLAIMS
+#  ARISING FROM OR RELATED TO THE CODE; AND (D)WITH RESPECT TO YOUR EXERCISE
+#  OF ANY RIGHTS GRANTED TO YOU FOR THE CODE, CLOUDERA IS NOT LIABLE FOR ANY
+#  DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, PUNITIVE OR
+#  CONSEQUENTIAL DAMAGES INCLUDING, BUT NOT LIMITED TO, DAMAGES
+#  RELATED TO LOST REVENUE, LOST PROFITS, LOSS OF INCOME, LOSS OF
+#  BUSINESS ADVANTAGE OR UNAVAILABILITY, OR LOSS OR CORRUPTION OF
+#  DATA.
+#
+# ###########################################################################
+
+from apps.data_utils import DATA_PACKET
+from src.style_transfer import StyleTransfer
+from src.style_classification import StyleIntensityClassifier
+from src.content_preservation import ContentPreservationScorer
+
+
+def load_and_cache_HF_models(style_data_packet):
+    """
+    This utility function is used to download and cache models needed for all style
+    attributes in `apps.data_utils.DATA_PACKET`
+
+    Args:
+        style_data_packet (dict)
+    """
+
+    for style_data in style_data_packet.keys():
+        try:
+            st = StyleTransfer(model_identifier=style_data.seq2seq_model_path)
+            sic = StyleIntensityClassifier(style_data.cls_model_path)
+            cps = ContentPreservationScorer(
+                cls_model_identifier=style_data.cls_model_path,
+                sbert_model_identifier=style_data.sbert_model_path,
+            )
+
+            del st, sic, cps
+        except Exception as e:
+            print(e)
+
+if __name__=="__main__":
+    load_and_cache_HF_models(DATA_PACKET)

scripts/install_dependencies.py

@@ -0,0 +1,41 @@
+# ###########################################################################
+#
+#  CLOUDERA APPLIED MACHINE LEARNING PROTOTYPE (AMP)
+#  (C) Cloudera, Inc. 2022
+#  All rights reserved.
+#
+#  Applicable Open Source License: Apache 2.0
+#
+#  NOTE: Cloudera open source products are modular software products
+#  made up of hundreds of individual components, each of which was
+#  individually copyrighted.  Each Cloudera open source product is a
+#  collective work under U.S. Copyright Law. Your license to use the
+#  collective work is as provided in your written agreement with
+#  Cloudera.  Used apart from the collective work, this file is
+#  licensed for your use pursuant to the open source license
+#  identified above.
+#
+#  This code is provided to you pursuant a written agreement with
+#  (i) Cloudera, Inc. or (ii) a third-party authorized to distribute
+#  this code. If you do not have a written agreement with Cloudera nor
+#  with an authorized and properly licensed third party, you do not
+#  have any rights to access nor to use this code.
+#
+#  Absent a written agreement with Cloudera, Inc. (“Cloudera”) to the
+#  contrary, A) CLOUDERA PROVIDES THIS CODE TO YOU WITHOUT WARRANTIES OF ANY
+#  KIND; (B) CLOUDERA DISCLAIMS ANY AND ALL EXPRESS AND IMPLIED
+#  WARRANTIES WITH RESPECT TO THIS CODE, INCLUDING BUT NOT LIMITED TO
+#  IMPLIED WARRANTIES OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY AND
+#  FITNESS FOR A PARTICULAR PURPOSE; (C) CLOUDERA IS NOT LIABLE TO YOU,
+#  AND WILL NOT DEFEND, INDEMNIFY, NOR HOLD YOU HARMLESS FOR ANY CLAIMS
+#  ARISING FROM OR RELATED TO THE CODE; AND (D)WITH RESPECT TO YOUR EXERCISE
+#  OF ANY RIGHTS GRANTED TO YOU FOR THE CODE, CLOUDERA IS NOT LIABLE FOR ANY
+#  DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, PUNITIVE OR
+#  CONSEQUENTIAL DAMAGES INCLUDING, BUT NOT LIMITED TO, DAMAGES
+#  RELATED TO LOST REVENUE, LOST PROFITS, LOSS OF INCOME, LOSS OF
+#  BUSINESS ADVANTAGE OR UNAVAILABILITY, OR LOSS OR CORRUPTION OF
+#  DATA.
+#
+# ###########################################################################
+
+!pip3 install -r requirements.txt && python3 scripts/download_models.py

scripts/launch_app.py

@@ -0,0 +1,41 @@
+# ###########################################################################
+#
+#  CLOUDERA APPLIED MACHINE LEARNING PROTOTYPE (AMP)
+#  (C) Cloudera, Inc. 2022
+#  All rights reserved.
+#
+#  Applicable Open Source License: Apache 2.0
+#
+#  NOTE: Cloudera open source products are modular software products
+#  made up of hundreds of individual components, each of which was
+#  individually copyrighted.  Each Cloudera open source product is a
+#  collective work under U.S. Copyright Law. Your license to use the
+#  collective work is as provided in your written agreement with
+#  Cloudera.  Used apart from the collective work, this file is
+#  licensed for your use pursuant to the open source license
+#  identified above.
+#
+#  This code is provided to you pursuant a written agreement with
+#  (i) Cloudera, Inc. or (ii) a third-party authorized to distribute
+#  this code. If you do not have a written agreement with Cloudera nor
+#  with an authorized and properly licensed third party, you do not
+#  have any rights to access nor to use this code.
+#
+#  Absent a written agreement with Cloudera, Inc. (“Cloudera”) to the
+#  contrary, A) CLOUDERA PROVIDES THIS CODE TO YOU WITHOUT WARRANTIES OF ANY
+#  KIND; (B) CLOUDERA DISCLAIMS ANY AND ALL EXPRESS AND IMPLIED
+#  WARRANTIES WITH RESPECT TO THIS CODE, INCLUDING BUT NOT LIMITED TO
+#  IMPLIED WARRANTIES OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY AND
+#  FITNESS FOR A PARTICULAR PURPOSE; (C) CLOUDERA IS NOT LIABLE TO YOU,
+#  AND WILL NOT DEFEND, INDEMNIFY, NOR HOLD YOU HARMLESS FOR ANY CLAIMS
+#  ARISING FROM OR RELATED TO THE CODE; AND (D)WITH RESPECT TO YOUR EXERCISE
+#  OF ANY RIGHTS GRANTED TO YOU FOR THE CODE, CLOUDERA IS NOT LIABLE FOR ANY
+#  DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, PUNITIVE OR
+#  CONSEQUENTIAL DAMAGES INCLUDING, BUT NOT LIMITED TO, DAMAGES
+#  RELATED TO LOST REVENUE, LOST PROFITS, LOSS OF INCOME, LOSS OF
+#  BUSINESS ADVANTAGE OR UNAVAILABILITY, OR LOSS OR CORRUPTION OF
+#  DATA.
+#
+# ###########################################################################
+
+!streamlit run apps/app.py --server.port $CDSW_APP_PORT --server.address 127.0.0.1

setup.py

@@ -0,0 +1,50 @@
+# ###########################################################################
+#
+#  CLOUDERA APPLIED MACHINE LEARNING PROTOTYPE (AMP)
+#  (C) Cloudera, Inc. 2022
+#  All rights reserved.
+#
+#  Applicable Open Source License: Apache 2.0
+#
+#  NOTE: Cloudera open source products are modular software products
+#  made up of hundreds of individual components, each of which was
+#  individually copyrighted.  Each Cloudera open source product is a
+#  collective work under U.S. Copyright Law. Your license to use the
+#  collective work is as provided in your written agreement with
+#  Cloudera.  Used apart from the collective work, this file is
+#  licensed for your use pursuant to the open source license
+#  identified above.
+#
+#  This code is provided to you pursuant a written agreement with
+#  (i) Cloudera, Inc. or (ii) a third-party authorized to distribute
+#  this code. If you do not have a written agreement with Cloudera nor
+#  with an authorized and properly licensed third party, you do not
+#  have any rights to access nor to use this code.
+#
+#  Absent a written agreement with Cloudera, Inc. (“Cloudera”) to the
+#  contrary, A) CLOUDERA PROVIDES THIS CODE TO YOU WITHOUT WARRANTIES OF ANY
+#  KIND; (B) CLOUDERA DISCLAIMS ANY AND ALL EXPRESS AND IMPLIED
+#  WARRANTIES WITH RESPECT TO THIS CODE, INCLUDING BUT NOT LIMITED TO
+#  IMPLIED WARRANTIES OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY AND
+#  FITNESS FOR A PARTICULAR PURPOSE; (C) CLOUDERA IS NOT LIABLE TO YOU,
+#  AND WILL NOT DEFEND, INDEMNIFY, NOR HOLD YOU HARMLESS FOR ANY CLAIMS
+#  ARISING FROM OR RELATED TO THE CODE; AND (D)WITH RESPECT TO YOUR EXERCISE
+#  OF ANY RIGHTS GRANTED TO YOU FOR THE CODE, CLOUDERA IS NOT LIABLE FOR ANY
+#  DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, PUNITIVE OR
+#  CONSEQUENTIAL DAMAGES INCLUDING, BUT NOT LIMITED TO, DAMAGES
+#  RELATED TO LOST REVENUE, LOST PROFITS, LOSS OF INCOME, LOSS OF
+#  BUSINESS ADVANTAGE OR UNAVAILABILITY, OR LOSS OR CORRUPTION OF
+#  DATA.
+#
+# ###########################################################################
+
+from setuptools import setup, find_packages
+
+setup(
+    name="text-style-transfer",
+    version="0.1.0",
+    description="A library and demo application for the NLP task of text style transfer.",
+    author="Andrew Reed",
+    author_email="andrew.reed.r@gmail.com",
+    packages=find_packages(),
+)

src/content_preservation.py

@@ -0,0 +1,366 @@
+# ###########################################################################
+#
+#  CLOUDERA APPLIED MACHINE LEARNING PROTOTYPE (AMP)
+#  (C) Cloudera, Inc. 2022
+#  All rights reserved.
+#
+#  Applicable Open Source License: Apache 2.0
+#
+#  NOTE: Cloudera open source products are modular software products
+#  made up of hundreds of individual components, each of which was
+#  individually copyrighted.  Each Cloudera open source product is a
+#  collective work under U.S. Copyright Law. Your license to use the
+#  collective work is as provided in your written agreement with
+#  Cloudera.  Used apart from the collective work, this file is
+#  licensed for your use pursuant to the open source license
+#  identified above.
+#
+#  This code is provided to you pursuant a written agreement with
+#  (i) Cloudera, Inc. or (ii) a third-party authorized to distribute
+#  this code. If you do not have a written agreement with Cloudera nor
+#  with an authorized and properly licensed third party, you do not
+#  have any rights to access nor to use this code.
+#
+#  Absent a written agreement with Cloudera, Inc. (“Cloudera”) to the
+#  contrary, A) CLOUDERA PROVIDES THIS CODE TO YOU WITHOUT WARRANTIES OF ANY
+#  KIND; (B) CLOUDERA DISCLAIMS ANY AND ALL EXPRESS AND IMPLIED
+#  WARRANTIES WITH RESPECT TO THIS CODE, INCLUDING BUT NOT LIMITED TO
+#  IMPLIED WARRANTIES OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY AND
+#  FITNESS FOR A PARTICULAR PURPOSE; (C) CLOUDERA IS NOT LIABLE TO YOU,
+#  AND WILL NOT DEFEND, INDEMNIFY, NOR HOLD YOU HARMLESS FOR ANY CLAIMS
+#  ARISING FROM OR RELATED TO THE CODE; AND (D)WITH RESPECT TO YOUR EXERCISE
+#  OF ANY RIGHTS GRANTED TO YOU FOR THE CODE, CLOUDERA IS NOT LIABLE FOR ANY
+#  DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, PUNITIVE OR
+#  CONSEQUENTIAL DAMAGES INCLUDING, BUT NOT LIMITED TO, DAMAGES
+#  RELATED TO LOST REVENUE, LOST PROFITS, LOSS OF INCOME, LOSS OF
+#  BUSINESS ADVANTAGE OR UNAVAILABILITY, OR LOSS OR CORRUPTION OF
+#  DATA.
+#
+# ###########################################################################
+
+from typing import List
+
+import torch
+import pandas as pd
+from transformers import (
+    AutoTokenizer,
+    AutoModel,
+    AutoModelForSequenceClassification,
+)
+
+
+class ContentPreservationScorer:
+    """
+    Utility for calculating Content Preservation Score between
+    two pieces of text (i.e. input and output of TST model).
+
+    This custom evaluation metric aims to quantify content preservation by
+    first modifying text to remove all style-related tokens leaving just
+    content related tokens behind. Style tokens are determind on a
+    sentence-by-sentence basis by extracting out salient token attributions
+    from a trained Style Classifier (BERT) so contextual information is
+    perserved in the attribution scores. Style tokens are then masked/removed
+    from the text. We pass the style-less sentences through a pre-trained,
+    but not fine-tuned SentenceBert model to compute sentence embeddings.
+    Cosine similarity on the embeddings produces a score that should represent
+    content preservation.
+
+    PSUEDO-CODE: (higher score is better preservation)
+    1. mask out style tokens for input and output text (1str)
+    2. get SBERT embedddings for each (multi)
+    3. calculate cosine similarity (multi pairs)
+
+    Attributes:
+        cls_model_identifier (str)
+        sbert_model_identifier (str)
+
+    """
+
+    def __init__(self, cls_model_identifier: str, sbert_model_identifier: str):
+
+        self.cls_model_identifier = cls_model_identifier
+        self.sbert_model_identifier = sbert_model_identifier
+        self.device = (
+            torch.cuda.current_device() if torch.cuda.is_available() else "cpu"
+        )
+
+        self._initialize_hf_artifacts()
+
+    def _initialize_hf_artifacts(self):
+        """
+        Initialize a HuggingFace artifacts (tokenizer and model) according
+        to the provided identifiers for both SBert and the classification model.
+        Then initialize the word attribution explainer with the HF model+tokenizer.
+
+        """
+
+        # sbert
+        self.sbert_tokenizer = AutoTokenizer.from_pretrained(
+            self.sbert_model_identifier
+        )
+        self.sbert_model = AutoModel.from_pretrained(self.sbert_model_identifier)
+
+        # classifer
+        self.cls_tokenizer = AutoTokenizer.from_pretrained(self.cls_model_identifier)
+        self.cls_model = AutoModelForSequenceClassification.from_pretrained(
+            self.cls_model_identifier
+        )
+        self.cls_model.to(self.device)
+
+    def compute_sentence_embeddings(self, input_text: List[str]) -> torch.Tensor:
+        """
+        Compute sentence embeddings for each sentence provided a list of text strings.
+
+        Args:
+            input_text (List[str]) - list of input sentences to encode
+
+        Returns:
+            sentence_embeddings (torch.Tensor)
+
+        """
+        # tokenize sentences
+        encoded_input = self.sbert_tokenizer(
+            input_text,
+            padding=True,
+            truncation=True,
+            max_length=256,
+            return_tensors="pt",
+        )
+
+        # to device
+        self.sbert_model.eval()
+        self.sbert_model.to(self.device)
+        encoded_input = {k: v.to(self.device) for k, v in encoded_input.items()}
+
+        # compute token embeddings
+        with torch.no_grad():
+            model_output = self.sbert_model(**encoded_input)
+
+        return (
+            self.mean_pooling(model_output, encoded_input["attention_mask"])
+            .detach()
+            .cpu()
+        )
+
+    def calculate_content_preservation_score(
+        self,
+        input_text: List[str],
+        output_text: List[str],
+        threshold: float = 0.3,
+        mask_type: str = "pad",
+        return_all: bool = False,
+    ) -> List[float]:
+        """
+        Calcualates the content preservation score (CPS) between two pieces of text.
+
+        Args:
+            input_text (list) - list of input texts with indicies corresponding
+                to counterpart in output_text
+            ouptput_text (list) - list of output texts with indicies corresponding
+                to counterpart in input_text
+            return_all (bool) - If true, return dict containing intermediate
+                text with style masking applied, along with scores
+            mask_type (str) - "pad", "remove", or "none"
+
+        Returns:
+            A list of floats with corresponding content preservation scores.
+
+        PSUEDO-CODE: (higher score is better preservation)
+            1. mask out style tokens for input and output text (1str)
+            2. get SBERT embedddings for each (multi)
+            3. calculate cosine similarity (multi pairs)
+        """
+        if len(input_text) != len(output_text):
+            raise ValueError(
+                "input_text and output_text must be of same length with corresponding items"
+            )
+
+        if mask_type != "none":
+            # Mask out style tokens
+            masked_input_text = [
+                self.mask_style_tokens(text, mask_type=mask_type, threshold=threshold)
+                for text in input_text
+            ]
+            masked_output_text = [
+                self.mask_style_tokens(text, mask_type=mask_type, threshold=threshold)
+                for text in output_text
+            ]
+
+            # Compute SBert embeddings
+            input_embeddings = self.compute_sentence_embeddings(masked_input_text)
+            output_embeddings = self.compute_sentence_embeddings(masked_output_text)
+        else:
+            # Compute SBert embeddings on unmasked text
+            input_embeddings = self.compute_sentence_embeddings(input_text)
+            output_embeddings = self.compute_sentence_embeddings(output_text)
+
+        # Calculate cosine similarity
+        scores = self.cosine_similarity(input_embeddings, output_embeddings)
+
+        if return_all:
+            output = {
+                "scores": scores,
+                "masked_input_text": masked_input_text
+                if mask_type != "none"
+                else input_text,
+                "masked_output_text": masked_output_text
+                if mask_type != "none"
+                else output_text,
+            }
+            return output
+        else:
+            return scores
+
+    def calculate_feature_attribution_scores(
+        self, text: str, class_index: int = 0, as_norm: bool = False
+    ) -> List[tuple]:
+        """
+        Calcualte feature attributions using integrated gradients by passing
+        a string of text as input.
+
+        Args:
+            text (str) - text to get attributions for
+            class_index (int) - Optional output index to provide attributions for
+
+        """
+        attributions = self.explainer(text, index=class_index)
+
+        if as_norm:
+            return self.format_feature_attribution_scores(attributions)
+
+        return attributions
+
+    def mask_style_tokens(
+        self,
+        text: str,
+        threshold: float = 0.3,
+        mask_type: str = "pad",
+        class_index: int = 0,
+    ) -> str:
+        """
+        Utility function to mask out style tokens from a given string of text.
+
+        Style tokens are determined by first calculating feature importances (via
+        word attributions from trained StyleClassifer) for each token in the input sentence.
+        We then normalize the absolute values of attributions scores to see how much each token
+        contributes as a percentage overall style classification and rank those in descending order.
+
+        We then select the top N tokens that account for the cumulative _threshold_ amount (%) of
+        total styleattribution. By using cumulative percentages, N is not a fixed number and we
+        ultimately take however many tokens are needed to account for _threshold_ % of the overall
+        style.
+
+        We can optionally return a string with these style tokens padded out or completely removed
+        by toggling _mask_type_ between "pad" and "remove".
+
+        Args:
+            text (str)
+            threshold (float) - percentage of style attribution as cutoff for masking selection.
+            mask_type (str) - "pad" or "remove", indicates how to handle style tokens
+            class_index (str)
+
+        Returns:
+            text (str)
+
+        """
+
+        # get attributions and format as sorted dataframe
+        attributions = self.calculate_feature_attribution_scores(
+            text, class_index=class_index, as_norm=False
+        )
+        attributions_df = self.format_feature_attribution_scores(attributions)
+
+        # select tokens to mask
+        token_idxs_to_mask = []
+
+        # If the first token accounts for more than the set
+        # threshold, take just that token to mask. Otherwise,
+        # take all tokens up to the threshold
+        if attributions_df.iloc[0]["cumulative"] > threshold:
+            token_idxs_to_mask.append(attributions_df.index[0])
+        else:
+            token_idxs_to_mask.extend(
+                attributions_df[
+                    attributions_df["cumulative"] <= threshold
+                ].index.to_list()
+            )
+
+        # Build text sequence with tokens masked out
+        mask_map = {"pad": "[PAD]", "remove": ""}
+        toks = [token for token, score in attributions]
+        for idx in token_idxs_to_mask:
+            toks[idx] = mask_map[mask_type]
+
+        if mask_type == "remove":
+            toks = [token for token in toks if token != ""]
+
+        # Decode that sequence
+        masked_text = self.explainer.tokenizer.decode(
+            self.explainer.tokenizer.convert_tokens_to_ids(toks),
+            skip_special_tokens=False,
+        )
+
+        # Remove special characters other than [PAD]
+        for special_token in self.explainer.tokenizer.all_special_tokens:
+            if special_token != "[PAD]":
+                masked_text = masked_text.replace(special_token, "")
+
+        return masked_text.strip()
+
+    @staticmethod
+    def format_feature_attribution_scores(attributions: List[tuple]) -> pd.DataFrame:
+        """
+        Utility for formatting attribution scores for style token mask selection
+
+        Sorts a given List[tuple] where tuples represent (token, score) by the
+        normalized absolute value of each token score.
+
+        """
+
+        df = pd.DataFrame(attributions, columns=["token", "score"])
+        df["abs_norm"] = df["score"].abs() / df["score"].abs().sum()
+        df = df.sort_values(by="abs_norm", ascending=False)
+        df["cumulative"] = df["abs_norm"].cumsum()
+        return df
+
+    @staticmethod
+    def cosine_similarity(tensor1: torch.Tensor, tensor2: torch.Tensor) -> List[float]:
+        """
+        Calculate cosine similarity on pairs of embedddings.
+
+        Can handle 1D Tensor for single pair or 2D Tensors with corresponding indicies
+        for matrix operation on multiple pairs.
+
+        """
+
+        assert tensor1.shape == tensor2.shape
+
+        # ensure 2D tensor
+        if tensor1.ndim == 1:
+            tensor1 = tensor1.unsqueeze(0)
+            tensor2 = tensor2.unsqueeze(0)
+
+        cos_sim = torch.nn.CosineSimilarity(dim=1, eps=1e-6)
+        return [round(val, 4) for val in cos_sim(tensor1, tensor2).tolist()]
+
+    @staticmethod
+    def mean_pooling(model_output, attention_mask):
+        """
+        Peform mean pooling over token embeddings to create sentence embedding. Here we take
+        the attention mask into account for correct averaging on active token positions.
+
+        CODE BORROWED FROM:
+            https://www.sbert.net/examples/applications/computing-embeddings/README.html#sentence-embeddings-with-transformers
+
+        """
+
+        token_embeddings = model_output[
+            0
+        ]  # First element of model_output contains all token embeddings
+        input_mask_expanded = (
+            attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
+        )
+        sum_embeddings = torch.sum(token_embeddings * input_mask_expanded, 1)
+        sum_mask = torch.clamp(input_mask_expanded.sum(1), min=1e-9)
+
+        return sum_embeddings / sum_mask

src/style_classification.py

@@ -0,0 +1,245 @@
+# ###########################################################################
+#
+#  CLOUDERA APPLIED MACHINE LEARNING PROTOTYPE (AMP)
+#  (C) Cloudera, Inc. 2022
+#  All rights reserved.
+#
+#  Applicable Open Source License: Apache 2.0
+#
+#  NOTE: Cloudera open source products are modular software products
+#  made up of hundreds of individual components, each of which was
+#  individually copyrighted.  Each Cloudera open source product is a
+#  collective work under U.S. Copyright Law. Your license to use the
+#  collective work is as provided in your written agreement with
+#  Cloudera.  Used apart from the collective work, this file is
+#  licensed for your use pursuant to the open source license
+#  identified above.
+#
+#  This code is provided to you pursuant a written agreement with
+#  (i) Cloudera, Inc. or (ii) a third-party authorized to distribute
+#  this code. If you do not have a written agreement with Cloudera nor
+#  with an authorized and properly licensed third party, you do not
+#  have any rights to access nor to use this code.
+#
+#  Absent a written agreement with Cloudera, Inc. (“Cloudera”) to the
+#  contrary, A) CLOUDERA PROVIDES THIS CODE TO YOU WITHOUT WARRANTIES OF ANY
+#  KIND; (B) CLOUDERA DISCLAIMS ANY AND ALL EXPRESS AND IMPLIED
+#  WARRANTIES WITH RESPECT TO THIS CODE, INCLUDING BUT NOT LIMITED TO
+#  IMPLIED WARRANTIES OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY AND
+#  FITNESS FOR A PARTICULAR PURPOSE; (C) CLOUDERA IS NOT LIABLE TO YOU,
+#  AND WILL NOT DEFEND, INDEMNIFY, NOR HOLD YOU HARMLESS FOR ANY CLAIMS
+#  ARISING FROM OR RELATED TO THE CODE; AND (D)WITH RESPECT TO YOUR EXERCISE
+#  OF ANY RIGHTS GRANTED TO YOU FOR THE CODE, CLOUDERA IS NOT LIABLE FOR ANY
+#  DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, PUNITIVE OR
+#  CONSEQUENTIAL DAMAGES INCLUDING, BUT NOT LIMITED TO, DAMAGES
+#  RELATED TO LOST REVENUE, LOST PROFITS, LOSS OF INCOME, LOSS OF
+#  BUSINESS ADVANTAGE OR UNAVAILABILITY, OR LOSS OR CORRUPTION OF
+#  DATA.
+#
+# ###########################################################################
+
+from typing import List, Union
+
+import torch
+import numpy as np
+from pyemd import emd
+from transformers import pipeline
+
+
+class StyleIntensityClassifier:
+    """
+    Utility for classifying style and calculating Style Transfer Intensity between
+    two pieces of text (i.e. input and output of TST model).
+
+    This custom evaluation metric aims to quantify the magnitude of transferred
+    style between two texts. To accomplish this, we pass input and output texts
+    through a trained style classifier to produce two distributions. We then
+    utilize Earth Movers Distance (EMD) to calculate the minimum "cost"/"work"
+    required to turn the input distribution into the output distribution. This
+    metric allows us to capture a more nuanced, per-example measure of style
+    transfer when compared to simply aggregating binary classifications over
+    records in a dataset.
+
+    Attributes:
+        model_identifier (str)
+
+    """
+
+    def __init__(self, model_identifier: str):
+        self.model_identifier = model_identifier
+        self.device = torch.cuda.current_device() if torch.cuda.is_available() else -1
+        self._build_pipeline()
+
+    def _build_pipeline(self):
+
+        self.pipeline = pipeline(
+            task="text-classification",
+            model=self.model_identifier,
+            device=self.device,
+            return_all_scores=True,
+        )
+
+    def score(self, input_text: Union[str, List[str]]):
+        """
+        Classify a given input text using the model initialized by the class.
+
+        Args:
+            input_text (`str` or `List[str]`) - Input text for classification
+
+        Returns:
+            classification (dict) - a dictionary containing the label, score, and
+                distribution between classes
+
+        """
+        if isinstance(input_text, str):
+            tmp = list()
+            tmp.append(input_text)
+            input_text = tmp
+
+        result = self.pipeline(input_text)
+        distributions = np.array(
+            [[label["score"] for label in item] for item in result]
+        )
+        return [
+            {
+                "label": self.pipeline.model.config.id2label[scores.argmax()],
+                "score": round(scores.max(), 4),
+                "distribution": scores.tolist(),
+            }
+            for scores in distributions
+        ]
+
+    def calculate_transfer_intensity(
+        self, input_text: List[str], output_text: List[str], target_class_idx: int = 1
+    ) -> List[float]:
+        """
+        Calcualates the style transfer intensity (STI) between two pieces of text.
+
+        Args:
+            input_text (list) - list of input texts with indicies corresponding
+                to counterpart in output_text
+            ouptput_text (list) - list of output texts with indicies corresponding
+                to counterpart in input_text
+            target_class_idx (int) - index of the target style class used for directional
+                score correction
+
+        Returns:
+            A list of floats with corresponding style transfer intensity scores.
+
+        """
+
+        if len(input_text) != len(output_text):
+            raise ValueError(
+                "input_text and output_text must be of same length with corresponding items"
+            )
+
+        input_dist = [item["distribution"] for item in self.score(input_text)]
+        output_dist = [item["distribution"] for item in self.score(output_text)]
+
+        return [
+            self.calculate_emd(input_dist[i], output_dist[i], target_class_idx)
+            for i in range(len(input_dist))
+        ]
+
+    def calculate_transfer_intensity_fraction(
+        self, input_text: List[str], output_text: List[str], target_class_idx: int = 1
+    ) -> List[float]:
+        """
+        Calcualates the style transfer intensity (STI) _fraction_ between two pieces of text.
+        See `calcualte_sti_fraction()` for details.
+
+        Args:
+            input_text (list) - list of input texts with indicies corresponding
+                to counterpart in output_text
+            ouptput_text (list) - list of output texts with indicies corresponding
+                to counterpart in input_text
+            target_class_idx (int) - index of the target style class used for directional
+                score correction
+
+        Returns:
+            A list of floats with corresponding style transfer intensity scores.
+
+        """
+
+        if len(input_text) != len(output_text):
+            raise ValueError(
+                "input_text and output_text must be of same length with corresponding items"
+            )
+
+        input_dist = [item["distribution"] for item in self.score(input_text)]
+        output_dist = [item["distribution"] for item in self.score(output_text)]
+
+        return [
+            self.calculate_sti_fraction(
+                input_dist[i],
+                output_dist[i],
+                ideal_dist=[0.0, 1.0],
+                target_class_idx=target_class_idx,
+            )
+            for i in range(len(input_dist))
+        ]
+
+    def calculate_sti_fraction(
+        self, input_dist, output_dist, ideal_dist=[0.0, 1.0], target_class_idx=1
+    ):
+        """
+        Calculate the direction-corrected style transfer intensity fraction between
+        two style distributions of equal length.
+
+        If output_dist moves closer towards target style class, the metric represents the percentage of
+        the possible _target_ style distribution that was captured during the transfer. If output_dist
+        moves further from the target style class, the metric represents the percentage of the possible
+        _source_ style distribution that was captured.
+
+        Args:
+            input_dist (list) - probabilities assigned to the style classes
+                from the input text to style transfer model
+            output_dist (list) - probabilities assigned to the style classes
+                from the outut text of the style transfer model
+            ideal_dist (list, optional): The maximum possibly distribution. Defaults to [0.0, 1.0].
+            target_class_idx (int, optional)
+
+        Returns:
+            sti_fraction (float)
+        """
+
+        sti = self.calculate_emd(input_dist, output_dist, target_class_idx)
+
+        if sti > 0:
+            potential = self.calculate_emd(input_dist, ideal_dist, target_class_idx)
+        else:
+            potential = self.calculate_emd(
+                input_dist, ideal_dist[::-1], target_class_idx
+            )
+
+        return sti / potential
+
+    @staticmethod
+    def calculate_emd(input_dist, output_dist, target_class_idx):
+        """
+        Calculate the direction-corrected Earth Mover's Distance (aka Wasserstein distance)
+        between two distributions of equal length. Here we penalize the EMD score if
+        the output text style moved further away from the target style.
+
+        Reference: https://github.com/passeul/style-transfer-model-evaluation/blob/master/code/style_transfer_intensity.py
+
+        Args:
+            input_dist (list) - probabilities assigned to the style classes
+                from the input text to style transfer model
+            output_dist (list) - probabilities assigned to the style classes
+                from the outut text of the style transfer model
+
+        Returns:
+            emd (float) - Earth Movers Distance between the two distributions
+
+        """
+
+        N = len(input_dist)
+        distance_matrix = np.ones((N, N))
+        dist = emd(np.array(input_dist), np.array(output_dist), distance_matrix)
+
+        transfer_direction_correction = (
+            1 if output_dist[target_class_idx] >= input_dist[target_class_idx] else -1
+        )
+
+        return round(dist * transfer_direction_correction, 4)

src/style_transfer.py

@@ -0,0 +1,94 @@
+# ###########################################################################
+#
+#  CLOUDERA APPLIED MACHINE LEARNING PROTOTYPE (AMP)
+#  (C) Cloudera, Inc. 2022
+#  All rights reserved.
+#
+#  Applicable Open Source License: Apache 2.0
+#
+#  NOTE: Cloudera open source products are modular software products
+#  made up of hundreds of individual components, each of which was
+#  individually copyrighted.  Each Cloudera open source product is a
+#  collective work under U.S. Copyright Law. Your license to use the
+#  collective work is as provided in your written agreement with
+#  Cloudera.  Used apart from the collective work, this file is
+#  licensed for your use pursuant to the open source license
+#  identified above.
+#
+#  This code is provided to you pursuant a written agreement with
+#  (i) Cloudera, Inc. or (ii) a third-party authorized to distribute
+#  this code. If you do not have a written agreement with Cloudera nor
+#  with an authorized and properly licensed third party, you do not
+#  have any rights to access nor to use this code.
+#
+#  Absent a written agreement with Cloudera, Inc. (“Cloudera”) to the
+#  contrary, A) CLOUDERA PROVIDES THIS CODE TO YOU WITHOUT WARRANTIES OF ANY
+#  KIND; (B) CLOUDERA DISCLAIMS ANY AND ALL EXPRESS AND IMPLIED
+#  WARRANTIES WITH RESPECT TO THIS CODE, INCLUDING BUT NOT LIMITED TO
+#  IMPLIED WARRANTIES OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY AND
+#  FITNESS FOR A PARTICULAR PURPOSE; (C) CLOUDERA IS NOT LIABLE TO YOU,
+#  AND WILL NOT DEFEND, INDEMNIFY, NOR HOLD YOU HARMLESS FOR ANY CLAIMS
+#  ARISING FROM OR RELATED TO THE CODE; AND (D)WITH RESPECT TO YOUR EXERCISE
+#  OF ANY RIGHTS GRANTED TO YOU FOR THE CODE, CLOUDERA IS NOT LIABLE FOR ANY
+#  DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, PUNITIVE OR
+#  CONSEQUENTIAL DAMAGES INCLUDING, BUT NOT LIMITED TO, DAMAGES
+#  RELATED TO LOST REVENUE, LOST PROFITS, LOSS OF INCOME, LOSS OF
+#  BUSINESS ADVANTAGE OR UNAVAILABILITY, OR LOSS OR CORRUPTION OF
+#  DATA.
+#
+# ###########################################################################
+
+from typing import List, Union
+
+import torch
+from transformers import pipeline
+
+
+class StyleTransfer:
+    """
+    Model wrapper for a Text2TextGeneration pipeline used to transfer a style attribute on a given piece of text.
+
+    Attributes:
+        model_identifier (str) - Path to the model that will be used by the pipeline to make predictions
+        max_gen_length (int) - Upper limit on number of tokens the model can generate as output
+
+    """
+
+    def __init__(
+        self,
+        model_identifier: str,
+        max_gen_length: int = 200,
+        num_beams=4,
+        temperature=1,
+    ):
+        self.model_identifier = model_identifier
+        self.max_gen_length = max_gen_length
+        self.num_beams = num_beams
+        self.temperature = temperature
+        self.device = torch.cuda.current_device() if torch.cuda.is_available() else -1
+        self._build_pipeline()
+
+    def _build_pipeline(self):
+
+        self.pipeline = pipeline(
+            task="text2text-generation",
+            model=self.model_identifier,
+            device=self.device,
+            max_length=self.max_gen_length,
+            num_beams=self.num_beams,
+            temperature=self.temperature,
+        )
+
+    def transfer(self, input_text: Union[str, List[str]]) -> List[str]:
+        """
+        Transfer the style attribute on a given piece of text using the
+        initialized `model_identifier`.
+
+        Args:
+            input_text (`str` or `List[str]`) - Input text for style transfer
+
+        Returns:
+            generated_text (`List[str]`) - The generated text outputs
+
+        """
+        return [item["generated_text"] for item in self.pipeline(input_text)]

src/transformer_interpretability.py

@@ -0,0 +1,148 @@
+# ###########################################################################
+#
+#  CLOUDERA APPLIED MACHINE LEARNING PROTOTYPE (AMP)
+#  (C) Cloudera, Inc. 2022
+#  All rights reserved.
+#
+#  Applicable Open Source License: Apache 2.0
+#
+#  NOTE: Cloudera open source products are modular software products
+#  made up of hundreds of individual components, each of which was
+#  individually copyrighted.  Each Cloudera open source product is a
+#  collective work under U.S. Copyright Law. Your license to use the
+#  collective work is as provided in your written agreement with
+#  Cloudera.  Used apart from the collective work, this file is
+#  licensed for your use pursuant to the open source license
+#  identified above.
+#
+#  This code is provided to you pursuant a written agreement with
+#  (i) Cloudera, Inc. or (ii) a third-party authorized to distribute
+#  this code. If you do not have a written agreement with Cloudera nor
+#  with an authorized and properly licensed third party, you do not
+#  have any rights to access nor to use this code.
+#
+#  Absent a written agreement with Cloudera, Inc. (“Cloudera”) to the
+#  contrary, A) CLOUDERA PROVIDES THIS CODE TO YOU WITHOUT WARRANTIES OF ANY
+#  KIND; (B) CLOUDERA DISCLAIMS ANY AND ALL EXPRESS AND IMPLIED
+#  WARRANTIES WITH RESPECT TO THIS CODE, INCLUDING BUT NOT LIMITED TO
+#  IMPLIED WARRANTIES OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY AND
+#  FITNESS FOR A PARTICULAR PURPOSE; (C) CLOUDERA IS NOT LIABLE TO YOU,
+#  AND WILL NOT DEFEND, INDEMNIFY, NOR HOLD YOU HARMLESS FOR ANY CLAIMS
+#  ARISING FROM OR RELATED TO THE CODE; AND (D)WITH RESPECT TO YOUR EXERCISE
+#  OF ANY RIGHTS GRANTED TO YOU FOR THE CODE, CLOUDERA IS NOT LIABLE FOR ANY
+#  DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, PUNITIVE OR
+#  CONSEQUENTIAL DAMAGES INCLUDING, BUT NOT LIMITED TO, DAMAGES
+#  RELATED TO LOST REVENUE, LOST PROFITS, LOSS OF INCOME, LOSS OF
+#  BUSINESS ADVANTAGE OR UNAVAILABILITY, OR LOSS OR CORRUPTION OF
+#  DATA.
+#
+# ###########################################################################
+
+import torch
+from transformers_interpret import SequenceClassificationExplainer
+from transformers import (
+    AutoTokenizer,
+    AutoModelForSequenceClassification,
+)
+
+from apps.visualization_utils import visualize_text
+
+class CustomSequenceClassificationExplainer(SequenceClassificationExplainer):
+    """
+    Subclassing to replace `visualize()` method with custom styling.
+
+    Namely, removing a few columns, styling fonts, and re-arrangning legend position.
+    """
+
+    def visualize(self, html_filepath: str = None, true_class: str = None):
+        """
+        Visualizes word attributions. If in a notebook table will be displayed inline.
+        Otherwise pass a valid path to `html_filepath` and the visualization will be saved
+        as a html file.
+        If the true class is known for the text that can be passed to `true_class`
+        """
+        tokens = [token.replace("Ġ", "") for token in self.decode(self.input_ids)]
+        attr_class = self.id2label[self.selected_index]
+
+        if self._single_node_output:
+            if true_class is None:
+                true_class = round(float(self.pred_probs))
+            predicted_class = round(float(self.pred_probs))
+            attr_class = round(float(self.pred_probs))
+        else:
+            if true_class is None:
+                true_class = self.selected_index
+            predicted_class = self.predicted_class_name
+
+        score_viz = self.attributions.visualize_attributions(  # type: ignore
+            self.pred_probs,
+            predicted_class,
+            true_class,
+            attr_class,
+            tokens,
+        )
+
+        # NOTE: here is the overwritten function
+        html = visualize_text([score_viz])
+
+        if html_filepath:
+            if not html_filepath.endswith(".html"):
+                html_filepath = html_filepath + ".html"
+            with open(html_filepath, "w") as html_file:
+                html_file.write(html.data)
+
+        return html
+
+
+class InterpretTransformer:
+    """
+    Utility for visualizing word attribution scores from Transformer models.
+
+    This class utilizes the [Transformers Interpret](https://github.com/cdpierse/transformers-interpret)
+    libary to calculate word attributions using a techinique called Integrated Gradients.
+
+    Attributes:
+        cls_model_identifier (str)
+
+    """
+
+    def __init__(self, cls_model_identifier: str):
+
+        self.cls_model_identifier = cls_model_identifier
+        self.device = (
+            torch.cuda.current_device() if torch.cuda.is_available() else "cpu"
+        )
+
+        self._initialize_hf_artifacts()
+
+    def _initialize_hf_artifacts(self):
+        """
+        Initialize a HuggingFace artifacts (tokenizer and model) according
+        to the provided identifiers for both SBert and the classification model.
+        Then initialize the word attribution explainer with the HF model+tokenizer.
+
+        """
+
+        # classifer
+        self.cls_tokenizer = AutoTokenizer.from_pretrained(self.cls_model_identifier)
+        self.cls_model = AutoModelForSequenceClassification.from_pretrained(
+            self.cls_model_identifier
+        )
+        self.cls_model.to(self.device)
+
+        # transformers interpret
+        self.explainer = CustomSequenceClassificationExplainer(
+            self.cls_model, self.cls_tokenizer
+        )
+
+    def visualize_feature_attribution_scores(self, text: str, class_index: int = 0):
+        """
+        Calculates and visualizes feature attributions using integrated gradients.
+
+        Args:
+            text (str) - text to get attributions for
+            class_index (int) - Optional output index to provide attributions for
+
+        """
+        self.explainer(text, index=class_index)
+        return self.explainer.visualize()

tests/test_model_classes.py

@@ -0,0 +1,164 @@
+# ###########################################################################
+#
+#  CLOUDERA APPLIED MACHINE LEARNING PROTOTYPE (AMP)
+#  (C) Cloudera, Inc. 2022
+#  All rights reserved.
+#
+#  Applicable Open Source License: Apache 2.0
+#
+#  NOTE: Cloudera open source products are modular software products
+#  made up of hundreds of individual components, each of which was
+#  individually copyrighted.  Each Cloudera open source product is a
+#  collective work under U.S. Copyright Law. Your license to use the
+#  collective work is as provided in your written agreement with
+#  Cloudera.  Used apart from the collective work, this file is
+#  licensed for your use pursuant to the open source license
+#  identified above.
+#
+#  This code is provided to you pursuant a written agreement with
+#  (i) Cloudera, Inc. or (ii) a third-party authorized to distribute
+#  this code. If you do not have a written agreement with Cloudera nor
+#  with an authorized and properly licensed third party, you do not
+#  have any rights to access nor to use this code.
+#
+#  Absent a written agreement with Cloudera, Inc. (“Cloudera”) to the
+#  contrary, A) CLOUDERA PROVIDES THIS CODE TO YOU WITHOUT WARRANTIES OF ANY
+#  KIND; (B) CLOUDERA DISCLAIMS ANY AND ALL EXPRESS AND IMPLIED
+#  WARRANTIES WITH RESPECT TO THIS CODE, INCLUDING BUT NOT LIMITED TO
+#  IMPLIED WARRANTIES OF TITLE, NON-INFRINGEMENT, MERCHANTABILITY AND
+#  FITNESS FOR A PARTICULAR PURPOSE; (C) CLOUDERA IS NOT LIABLE TO YOU,
+#  AND WILL NOT DEFEND, INDEMNIFY, NOR HOLD YOU HARMLESS FOR ANY CLAIMS
+#  ARISING FROM OR RELATED TO THE CODE; AND (D)WITH RESPECT TO YOUR EXERCISE
+#  OF ANY RIGHTS GRANTED TO YOU FOR THE CODE, CLOUDERA IS NOT LIABLE FOR ANY
+#  DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, PUNITIVE OR
+#  CONSEQUENTIAL DAMAGES INCLUDING, BUT NOT LIMITED TO, DAMAGES
+#  RELATED TO LOST REVENUE, LOST PROFITS, LOSS OF INCOME, LOSS OF
+#  BUSINESS ADVANTAGE OR UNAVAILABILITY, OR LOSS OR CORRUPTION OF
+#  DATA.
+#
+# ###########################################################################
+
+import pytest
+import transformers
+
+from src.style_transfer import StyleTransfer
+from src.style_classification import StyleIntensityClassifier
+from src.content_preservation import ContentPreservationScorer
+from src.transformer_interpretability import InterpretTransformer
+
+
+@pytest.fixture
+def subjectivity_example_data():
+    examples = [
+        """there is an iconic roadhouse, named "spud's roadhouse", which sells fuel and general shop items , has great meals and has accommodation.""",
+        "chemical abstracts service (cas), a prominent division of the american chemical society, is the world's leading source of chemical information.",
+        "the most serious scandal was the iran-contra affair.",
+        "another strikingly elegant four-door saloon for the s3 continental came from james young.",
+        "other ambassadors also sent their messages of condolence following her passing.",
+    ]
+
+    ground_truth = [
+        'there is a roadhouse, named "spud\'s roadhouse", which sells fuel and general shop items and has accommodation.',
+        "chemical abstracts service (cas), a division of the american chemical society, is a source of chemical information.",
+        "one controversy was the iran-contra affair.",
+        "another four-door saloon for the s3 continental came from james young.",
+        "other ambassadors also sent their messages of condolence following her death.",
+    ]
+
+    return {"examples": examples, "ground_truth": ground_truth}
+
+
+@pytest.fixture
+def subjectivity_styletransfer():
+    MODEL_PATH = "cffl/bart-base-styletransfer-subjective-to-neutral"
+    return StyleTransfer(model_identifier=MODEL_PATH, max_gen_length=200)
+
+
+@pytest.fixture
+def subjectivity_styleintensityclassifier():
+    CLS_MODEL_PATH = "cffl/bert-base-styleclassification-subjective-neutral"
+    return StyleIntensityClassifier(model_identifier=CLS_MODEL_PATH)
+
+
+@pytest.fixture
+def subjectivity_contentpreservationscorer():
+    CLS_MODEL_PATH = "cffl/bert-base-styleclassification-subjective-neutral"
+    SBERT_MODEL_PATH = "sentence-transformers/all-MiniLM-L6-v2"
+    return ContentPreservationScorer(
+        cls_model_identifier=CLS_MODEL_PATH, sbert_model_identifier=SBERT_MODEL_PATH
+    )
+
+
+@pytest.fixture
+def subjectivity_interprettransformer():
+    CLS_MODEL_PATH = "cffl/bert-base-styleclassification-subjective-neutral"
+    return InterpretTransformer(cls_model_identifier=CLS_MODEL_PATH)
+
+
+# test class initialization
+def test_StyleTransfer_init(subjectivity_styletransfer):
+    assert isinstance(
+        subjectivity_styletransfer.pipeline,
+        transformers.pipelines.text2text_generation.Text2TextGenerationPipeline,
+    )
+
+
+def test_StyleIntensityClassifier_init(subjectivity_styleintensityclassifier):
+    assert isinstance(
+        subjectivity_styleintensityclassifier.pipeline,
+        transformers.pipelines.text_classification.TextClassificationPipeline,
+    )
+
+
+def test_ContentPreservationScorer_init(subjectivity_contentpreservationscorer):
+    assert isinstance(
+        subjectivity_contentpreservationscorer.cls_model,
+        transformers.models.bert.modeling_bert.BertForSequenceClassification,
+    )
+    assert isinstance(
+        subjectivity_contentpreservationscorer.sbert_model,
+        transformers.models.bert.modeling_bert.BertModel,
+    )
+
+
+def test_InterpretTransformer_init(subjectivity_interprettransformer):
+    assert isinstance(
+        subjectivity_interprettransformer.cls_model,
+        transformers.models.bert.modeling_bert.BertForSequenceClassification,
+    )
+
+
+# test class functionality
+def test_StyleTransfer_transfer(subjectivity_styletransfer, subjectivity_example_data):
+    assert subjectivity_example_data[
+        "ground_truth"
+    ] == subjectivity_styletransfer.transfer(subjectivity_example_data["examples"])
+
+
+def test_StyleIntensityClassifier_calculate_transfer_intensity_fraction(
+    subjectivity_styleintensityclassifier, subjectivity_example_data
+):
+    sti_frac = (
+        subjectivity_styleintensityclassifier.calculate_transfer_intensity_fraction(
+            input_text=subjectivity_example_data["examples"],
+            output_text=subjectivity_example_data["ground_truth"],
+        )
+    )
+    assert sti_frac == [
+        0.9891820847234861,
+        0.9808499743983614,
+        0.8070009460737938,
+        0.9913705583756346,
+        0.9611679711017459,
+    ]
+
+
+def test_ContentPreservationScorer_calculate_content_preservation_score(
+    subjectivity_contentpreservationscorer, subjectivity_example_data
+):
+    cps = subjectivity_contentpreservationscorer.calculate_content_preservation_score(
+        input_text=subjectivity_example_data["examples"],
+        output_text=subjectivity_example_data["ground_truth"],
+        mask_type="none",
+    )
+    assert cps == [0.9369, 0.9856, 0.7328, 0.9718, 0.9709]