Create README.md

README.md

@@ -0,0 +1,93 @@
+---
+license: apache-2.0
+---
+
+
+
+# Model
+
+## TL;DR
+
+CLAP is to audio what CLIP is to image. This is an improved CLAP checkpoint, specifically trained on music and speech.
+
+## Description
+
+CLAP (Contrastive Language-Audio Pretraining) is a neural network trained on a variety of (audio, text) pairs. It can be instructed in to predict the most relevant text snippet, given an audio, without directly optimizing for the task. The CLAP model uses a SWINTransformer to get audio features from a log-Mel spectrogram input, and a RoBERTa model to get text features. Both the text and audio features are then projected to a latent space with identical dimension. The dot product between the projected audio and text features is then used as a similar score.
+
+
+# Usage
+
+You can use this model for zero shot audio classification or extracting audio and/or textual features.
+
+# Uses
+
+## Perform zero-shot audio classification
+
+### Using `pipeline`
+
+```python
+from datasets import load_dataset
+from transformers import pipeline
+
+dataset = load_dataset("ashraq/esc50")
+audio = dataset["train"]["audio"][-1]["array"]
+
+audio_classifier = pipeline(task="zero-shot-audio-classification", model="ylacombe/larger_clap_music_and_speech")
+output = audio_classifier(audio, candidate_labels=["Sound of a dog", "Sound of vaccum cleaner"])
+print(output)
+>>> [{"score": 0.999, "label": "Sound of a dog"}, {"score": 0.001, "label": "Sound of vaccum cleaner"}]
+```
+
+## Run the model:
+
+You can also get the audio and text embeddings using `ClapModel`
+
+### Run the model on CPU:
+
+```python
+from datasets import load_dataset
+from transformers import ClapModel, ClapProcessor
+
+librispeech_dummy = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
+audio_sample = librispeech_dummy[0]
+
+model = ClapModel.from_pretrained("ylacombe/larger_clap_music_and_speech")
+processor = ClapProcessor.from_pretrained("ylacombe/larger_clap_music_and_speech")
+
+inputs = processor(audios=audio_sample["audio"]["array"], return_tensors="pt")
+audio_embed = model.get_audio_features(**inputs)
+```
+
+### Run the model on GPU:
+
+```python
+from datasets import load_dataset
+from transformers import ClapModel, ClapProcessor
+
+librispeech_dummy = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
+audio_sample = librispeech_dummy[0]
+
+model = ClapModel.from_pretrained("ylacombe/larger_clap_music_and_speech").to(0)
+processor = ClapProcessor.from_pretrained("ylacombe/larger_clap_music_and_speech")
+
+inputs = processor(audios=audio_sample["audio"]["array"], return_tensors="pt").to(0)
+audio_embed = model.get_audio_features(**inputs)
+```
+
+
+# Citation
+
+If you are using this model for your work, please consider citing the original paper:
+```
+@misc{https://doi.org/10.48550/arxiv.2211.06687,
+  doi = {10.48550/ARXIV.2211.06687},
+  url = {https://arxiv.org/abs/2211.06687},
+  author = {Wu, Yusong and Chen, Ke and Zhang, Tianyu and Hui, Yuchen and Berg-Kirkpatrick, Taylor and Dubnov, Shlomo},
+  keywords = {Sound (cs.SD), Audio and Speech Processing (eess.AS), FOS: Computer and information sciences, FOS: Computer and information sciences, FOS: Electrical engineering, electronic engineering, information engineering, FOS: Electrical engineering, electronic engineering, information engineering},
+  title = {Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation},
+  publisher = {arXiv},
+  year = {2022},
+  copyright = {Creative Commons Attribution 4.0 International}
+}
+```
+