fusestudio 0.1.0
Convert slide and script contents to video
Requires Python: >=3.8,<3.9

## Steps
# Project name: **img2vid**

## Project Structure

## Description

### Objective: 

The aim of this project is to generate videos from slides and scripts. 

Curriculum engineers will create slides and write scripts in Google Slides. Using those slides this project will create a video of slide show with a voiceover generated using a text-to-speech model.

## Sample

![Sample GIF: Domains of NLP](docs/static/sample_gif.gif)

## Installation

1. Run `curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -` to install poetry.
2. Run `source $HOME/.poetry/env` to make poetry accessible across the system.

### Setup env
Poetry uses `pyproject.toml` and `poetry.lock` files. Both these files are available in the repo.

1. Python 3.8 is used for this project. It needs to be available in the system before setting up venv or existing Python 3.8 environment can be used.
2. Run `poetry config --local virtualenvs.in-project true` to enable in project venv.
3. Run `poetry env use /full/path/to/python` or `poetry env use python3.8 ` if you have python executable in your `PATH`. Details can be found [here](https://python-poetry.org/docs/managing-environments/). It will create the environment.
4. Environment details can be seen by running `poetry env info`.
5. `poetry add module_name` can be used to add new modules and `poetry install` to install the added modules.


## Error Handling


## Working of the Project
![General Working](docs/static/general_working.jpg)

The above diagram shows the general working mechanism of the project.

The process into creating a video from slides and script is divided into the following steps:

1. Conversion: pptx/dop to pdf
    - The downloaded presentation slide is converted into images in two sub processes:
    - PPTX/ODP to PDF
        - We will be converting the presentation slide into pdf using the libreoffice in Linux. Initially, Libreoffice unpacks the presentation slides **including animations.** If there are two animation in a slide then there will be two corresponding page in the pdf.
        eg:

        <img src="docs/static/animation1.jpg" alt="Animation_1" width="200"/>
        <br>
        <img src="docs/static/animation2.jpg" alt="Animation_2" width="200"/>
        <br>
        <img src="docs/static/animation3.jpg" alt="Animation_3" width="200"/>
        <br>
    - Libreoffice used [Expand Animations](https://github.com/monperrus/ExpandAnimations/releases/download/0.12/ExpandAnimations-0.12.oxt) to unpacks the animations.

<details><summary><b>Installing Libre Office and Adding Addons/Extensions</b></summary>

1.  Add the LibreOffice PPA Repository
    `sudo add-apt-repository ppa:libreoffice/ppa`

2. Update Your System
    `sudo apt update`

3. Upgrade or Install LibreOffice
    `sudo apt install libreoffice`

4. Installing addons/extensions: ExpandAnimations
    - Download the extension: ExpandAnimations and save it anywhere on your computer.
        - [Expand Animations](https://github.com/monperrus/ExpandAnimations/releases/download/0.12/ExpandAnimations-0.12.oxt)
    - In LibreOffice, select Tools → Extension Manager from the menu bar. 
    - In the Extension Manager dialog, click Add.
    - A file browser window opens. In Add Extension(s) dialog, you can find the extension's files in your system folders. The extension's files have OXT file extension.
    - Find and select the extension you want to install and click Open. 
</details>


2. Conversion: pdf to images
    - The converted pdf is now converted into images using [pdf2image](https://pypi.org/project/pdf2image/) module.

3. Generation: audio
    - The raw script download is processed to extract the sentences using regex pattern filtering as shown in [Process Script](src/generate_script.py)
    - We will now be generating audio using [Mozilla TTS](https://github.com/mozilla/TTS) or [TTS](https://pypi.org/project/TTS/)

    e.g: Generating audio using provided sentence 
    ```
    tts --text "Bill got in the habit of asking himself “Is that thought true?” and if he wasn’t absolutely certain it was, he just let it go." \
    --config_path config.json \
    --model_path tts_model.pth.tar \
    --out_path folder/output.wav \
    --vocoder_path vocoder_model.pth.tar \
    --vocoder_config_path config_vocoder.json
    ```
    - The project is using the specific checkpoints and models as listed in [tts_config](config/tts_config.py)

4. Generation: video
    - The length of the audio files with combination to FPS is used to determine the length of the video
    - Using the video duration, the images of slides are slideshowed using [Moviepy](https://pypi.org/project/moviepy/) and each individual videos are combined using [ffmpeg](https://www.ffmpeg.org/)

This is the overall working of the project from presentation slide to video generation.



# Resources

## Papers

### Text-to-Spectrogram
- [Tacotron](https://arxiv.org/abs/1703.10135)
- [Tacotron2](https://arxiv.org/abs/1712.05884)
- [Glow TTS](https://arxiv.org/abs/2005.11129)

### Vocoders
- [MelGAN](https://arxiv.org/abs/1910.06711)
- [MultiBandMelGAN](https://arxiv.org/abs/2005.05106)


## Collab Notebooks
- [English - LJSpeech + Universal FullBand-MelGAN + GlowTTS](https://colab.research.google.com/drive/1NC4eQJFvVEqD8L4Rd8CVK25_Z-ypaBHD?usp=sharing)
- [English - LJSpeech - Tacotron2 with Dynamic Convolutional Attention and Universal Fullband-MelGAN](https://colab.research.google.com/drive/1uV2CD1hWx5FGUrcIQjdZfI717AMj8-kP?usp=sharing)
- [Spanish - Tacotron2 DDC with Universal FullBand-MelGAN](https://colab.research.google.com/drive/1u_16ZzHjKYFn1HNVuA4Qf_i2MMFB9olY?usp=sharing)


## License 

Fusemachines