Duplicate from ja-818/speech_and_text_emotion_recognition

Co-authored-by: Julian Arango <ja-818@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1 @@
+*.wav filter=lfs diff=lfs merge=lfs -text

.github/workflows/check_size.yml

@@ -0,0 +1,16 @@
+name: Check file size
+on:               # or directly `on: [push]` to run the action on every push on any branch
+  pull_request:
+    branches: [main]
+
+  # to run this workflow manually from the Actions tab
+  workflow_dispatch:
+
+jobs:
+  sync-to-hub:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check large files
+        uses: ActionsDesk/lfs-warning@v2.0
+        with:
+          filesizelimit: 10485760 # this is 10MB so we can sync to HF Spaces

.github/workflows/sync_hf.yml

@@ -0,0 +1,20 @@
+name: Sync to Hugging Face hub
+on:
+  push:
+    branches: [main]
+
+  # to run this workflow manually from the Actions tab
+  workflow_dispatch:
+
+jobs:
+  sync-to-hub:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+        with:
+          fetch-depth: 0
+          lfs: true
+      - name: Push to hub
+        env:
+          HF_TOKEN: ${{ secrets.HF_TOKEN }}
+        run: git push https://ja-818:$HF_TOKEN@huggingface.co/spaces/ja-818/speech_and_text_emotion_recognition main

.gitignore

@@ -0,0 +1,129 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+pip-wheel-metadata/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+.python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/

LICENSE

@@ -0,0 +1,121 @@
+Creative Commons Legal Code
+
+CC0 1.0 Universal
+
+    CREATIVE COMMONS CORPORATION IS NOT A LAW FIRM AND DOES NOT PROVIDE
+    LEGAL SERVICES. DISTRIBUTION OF THIS DOCUMENT DOES NOT CREATE AN
+    ATTORNEY-CLIENT RELATIONSHIP. CREATIVE COMMONS PROVIDES THIS
+    INFORMATION ON AN "AS-IS" BASIS. CREATIVE COMMONS MAKES NO WARRANTIES
+    REGARDING THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS
+    PROVIDED HEREUNDER, AND DISCLAIMS LIABILITY FOR DAMAGES RESULTING FROM
+    THE USE OF THIS DOCUMENT OR THE INFORMATION OR WORKS PROVIDED
+    HEREUNDER.
+
+Statement of Purpose
+
+The laws of most jurisdictions throughout the world automatically confer
+exclusive Copyright and Related Rights (defined below) upon the creator
+and subsequent owner(s) (each and all, an "owner") of an original work of
+authorship and/or a database (each, a "Work").
+
+Certain owners wish to permanently relinquish those rights to a Work for
+the purpose of contributing to a commons of creative, cultural and
+scientific works ("Commons") that the public can reliably and without fear
+of later claims of infringement build upon, modify, incorporate in other
+works, reuse and redistribute as freely as possible in any form whatsoever
+and for any purposes, including without limitation commercial purposes.
+These owners may contribute to the Commons to promote the ideal of a free
+culture and the further production of creative, cultural and scientific
+works, or to gain reputation or greater distribution for their Work in
+part through the use and efforts of others.
+
+For these and/or other purposes and motivations, and without any
+expectation of additional consideration or compensation, the person
+associating CC0 with a Work (the "Affirmer"), to the extent that he or she
+is an owner of Copyright and Related Rights in the Work, voluntarily
+elects to apply CC0 to the Work and publicly distribute the Work under its
+terms, with knowledge of his or her Copyright and Related Rights in the
+Work and the meaning and intended legal effect of CC0 on those rights.
+
+1. Copyright and Related Rights. A Work made available under CC0 may be
+protected by copyright and related or neighboring rights ("Copyright and
+Related Rights"). Copyright and Related Rights include, but are not
+limited to, the following:
+
+  i. the right to reproduce, adapt, distribute, perform, display,
+     communicate, and translate a Work;
+ ii. moral rights retained by the original author(s) and/or performer(s);
+iii. publicity and privacy rights pertaining to a person's image or
+     likeness depicted in a Work;
+ iv. rights protecting against unfair competition in regards to a Work,
+     subject to the limitations in paragraph 4(a), below;
+  v. rights protecting the extraction, dissemination, use and reuse of data
+     in a Work;
+ vi. database rights (such as those arising under Directive 96/9/EC of the
+     European Parliament and of the Council of 11 March 1996 on the legal
+     protection of databases, and under any national implementation
+     thereof, including any amended or successor version of such
+     directive); and
+vii. other similar, equivalent or corresponding rights throughout the
+     world based on applicable law or treaty, and any national
+     implementations thereof.
+
+2. Waiver. To the greatest extent permitted by, but not in contravention
+of, applicable law, Affirmer hereby overtly, fully, permanently,
+irrevocably and unconditionally waives, abandons, and surrenders all of
+Affirmer's Copyright and Related Rights and associated claims and causes
+of action, whether now known or unknown (including existing as well as
+future claims and causes of action), in the Work (i) in all territories
+worldwide, (ii) for the maximum duration provided by applicable law or
+treaty (including future time extensions), (iii) in any current or future
+medium and for any number of copies, and (iv) for any purpose whatsoever,
+including without limitation commercial, advertising or promotional
+purposes (the "Waiver"). Affirmer makes the Waiver for the benefit of each
+member of the public at large and to the detriment of Affirmer's heirs and
+successors, fully intending that such Waiver shall not be subject to
+revocation, rescission, cancellation, termination, or any other legal or
+equitable action to disrupt the quiet enjoyment of the Work by the public
+as contemplated by Affirmer's express Statement of Purpose.
+
+3. Public License Fallback. Should any part of the Waiver for any reason
+be judged legally invalid or ineffective under applicable law, then the
+Waiver shall be preserved to the maximum extent permitted taking into
+account Affirmer's express Statement of Purpose. In addition, to the
+extent the Waiver is so judged Affirmer hereby grants to each affected
+person a royalty-free, non transferable, non sublicensable, non exclusive,
+irrevocable and unconditional license to exercise Affirmer's Copyright and
+Related Rights in the Work (i) in all territories worldwide, (ii) for the
+maximum duration provided by applicable law or treaty (including future
+time extensions), (iii) in any current or future medium and for any number
+of copies, and (iv) for any purpose whatsoever, including without
+limitation commercial, advertising or promotional purposes (the
+"License"). The License shall be deemed effective as of the date CC0 was
+applied by Affirmer to the Work. Should any part of the License for any
+reason be judged legally invalid or ineffective under applicable law, such
+partial invalidity or ineffectiveness shall not invalidate the remainder
+of the License, and in such case Affirmer hereby affirms that he or she
+will not (i) exercise any of his or her remaining Copyright and Related
+Rights in the Work or (ii) assert any associated claims and causes of
+action with respect to the Work, in either case contrary to Affirmer's
+express Statement of Purpose.
+
+4. Limitations and Disclaimers.
+
+ a. No trademark or patent rights held by Affirmer are waived, abandoned,
+    surrendered, licensed or otherwise affected by this document.
+ b. Affirmer offers the Work as-is and makes no representations or
+    warranties of any kind concerning the Work, express, implied,
+    statutory or otherwise, including without limitation warranties of
+    title, merchantability, fitness for a particular purpose, non
+    infringement, or the absence of latent or other defects, accuracy, or
+    the present or absence of errors, whether or not discoverable, all to
+    the greatest extent permissible under applicable law.
+ c. Affirmer disclaims responsibility for clearing rights of other persons
+    that may apply to the Work or any use thereof, including without
+    limitation any person's Copyright and Related Rights in the Work.
+    Further, Affirmer disclaims responsibility for obtaining any necessary
+    consents, permissions or other rights required for any use of the
+    Work.
+ d. Affirmer understands and acknowledges that Creative Commons is not a
+    party to this document and has no duty or obligation with respect to
+    this CC0 or use of the Work.

README.md

@@ -0,0 +1,11 @@
+---
+title: Speech and Text Emotion Recognition
+emoji: 🦧
+colorFrom: black
+colorTo: black
+sdk: gradio
+sdk_version: 3.21.0
+app_file: app.py
+pinned: false
+duplicated_from: ja-818/speech_and_text_emotion_recognition
+---

app.py

@@ -0,0 +1,52 @@
+import os
+import gradio as gr
+from models import infere_speech_emotion, infere_text_emotion, infere_voice2text
+
+# Create a Gradio app object
+with gr.Blocks() as demo:
+    gr.Markdown(
+        ''' 
+        # Speech and Text Emotion Recognition
+        ## Determining someone's emotions can be challenging based solely on their tone or words
+        ### This app uses both to provide a more accurate analysis of emotional expression in a single audio recording
+        '''
+        )
+    
+    # Upload audio input and output fields
+    with gr.Tab("Upload audio"):
+        with gr.Row():
+            upload_input = gr.Audio(label="Audio file", source="upload", type="filepath")
+            with gr.Column():
+                upload_output_1 = gr.Textbox(label="Text from the audio")
+                upload_output_2 = gr.Textbox(label="Speech emotion")
+                upload_output_3 = gr.Textbox(label="Text emotion")
+        btn0 = gr.Button("Analyze audio")
+        # Examples to be used as input 
+        gr.Examples(
+            [ 
+                os.path.join(os.path.dirname(__file__), "audio/a_good_dream.wav"),
+                os.path.join(os.path.dirname(__file__), "audio/hype_in_ai.wav"),
+            ],
+            upload_input,
+            label="Examples in which speech and words express different emotions:"        
+        )
+    # Input-output logic based on button click
+    btn0.click(fn=infere_voice2text, inputs=upload_input, outputs=upload_output_1)
+    btn0.click(fn=infere_speech_emotion, inputs=upload_input, outputs=upload_output_2)
+    upload_output_1.change(fn=infere_text_emotion, inputs=upload_output_1, outputs=upload_output_3)
+    
+    # Record audio input and output fields
+    with gr.Tab("Record audio"):   
+        with gr.Row():
+            record_input = gr.Audio(label="Audio recording", source="microphone", type="filepath")
+            with gr.Column():
+                record_output_1 = gr.Textbox(label="Text from the audio")
+                record_output_2 = gr.Textbox(label="Speech emotion")
+                record_output_3 = gr.Textbox(label="Text emotion")
+        btn1 = gr.Button("Analyze audio")   
+    # Input-output logic based on button click
+    btn1.click(fn=infere_voice2text, inputs=record_input, outputs=record_output_1)
+    btn1.click(fn=infere_speech_emotion, inputs=record_input, outputs=record_output_2)
+    record_output_1.change(fn=infere_text_emotion, inputs=record_output_1, outputs=record_output_3)
+
+demo.launch()

audio/a_good_dream.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:970765169ddf95678a118d45c8a77e290a4ab18be5f3f6a313670cbc54423b9d
+size 858676

audio/hype_in_ai.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:26fef5c2e4ed3b8b7640e28a998cdac1f1f9ade7573098a35fbea7bbd33bef4d
+size 1329444

models.py

@@ -0,0 +1,26 @@
+# Import the necessary libraries
+from transformers import pipeline
+
+# Initialize the text classification model with a pre-trained model
+model_text_emotion = pipeline("text-classification", model="j-hartmann/emotion-english-distilroberta-base")
+
+# Initialize the audio classification model with a pre-trained SER model
+model_speech_emotion = pipeline("audio-classification", model="aherzberg/ser_model_fixed_label")
+
+# Initialize the automatic speech recognition model with a pre-trained model that is capable of converting speech to text
+model_voice2text = pipeline("automatic-speech-recognition", model="openai/whisper-tiny.en")
+
+# A function that uses the initialized text classification model to predict the emotion of a given text input
+def infere_text_emotion(text):
+    return model_text_emotion(text)[0]["label"].capitalize()
+
+# A function that uses the initialized audio classification model to predict the emotion of a given speech input
+def infere_speech_emotion(text):
+    # Dict that maps the speech model emotions with the text's ones
+    emotions_dict = {"angry": "Anger", "disgust": "Disgust", "fear": "Fear", "happy": "Joy", "neutral": "Neutral", "sad": "Sadness"}
+    inference = model_speech_emotion(text)[0]["label"]
+    return emotions_dict[inference]
+
+# A function that uses the initialized automatic speech recognition model to convert speech (as an audio file) to text
+def infere_voice2text(audio_file):
+    return model_voice2text(audio_file)["text"]

requirements.txt

@@ -0,0 +1,3 @@
+gradio
+transformers
+torch