Upload 18 files

.gitignore

@@ -0,0 +1,15 @@
+venv
+__pycache__
+.venv
+output/
+*.mkv
+*.wav
+*.mp3
+*.mp4
+*.webm
+pretrained_models
+tmp
+dist
+build
+*.spec
+audio_cache

README.md

@@ -1,13 +1,144 @@
----
-title: Dubbing
-emoji: 🐠
-colorFrom: yellow
-colorTo: indigo
-sdk: gradio
-sdk_version: 4.8.0
-app_file: app.py
-pinned: false
-license: apache-2.0
----
-
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# Weeablind
+
+A program to dub multi-lingual media and anime using modern AI speech synthesis, diarization, language identification, and voice cloning.
+
+## Why
+
+Many shows, movies, news segments, interviews, and videos will never receive proper dubs to other languages, and dubbing something from scratch can be an enormous undertaking. This presents a common accessibility hurdle for people with blindness, dyslexia, learning disabilities, or simply folks that don't enjoy reading subtitles. This program aims to create a pleasant alternative for folks facing these struggles.
+
+This software is a product of war. My sister turned me onto my now-favorite comedy anime "The Disastrous Life of Saiki K." but Netflix never ordered a dub for the 2nd season. I'm blind and cannot and will not ever be able to read subtitles, but I MUST know how the story progresses! Netflix has forced my hand and I will bring AI-dubbed anime to the blind!
+
+## How
+
+This project relies on some rudimentary slapping together of some state of the art technologies. It uses numerous audio processing libraries and techniques to analyze and synthesize speech that tries to stay in-line with the source video file. It primarily relies on ffmpeg and pydub for audio and video editing, Coqui TTS for speech synthesis, speechbrain for language identification, and pyannote.audio for speaker diarization.
+
+You have the option of dubbing every subtitle in the video, setting the s tart and end times, dubbing only foreign-language content, or full-blown multi-speaker dubbing with speaking rate and volume matching.
+
+## When?
+
+This project is currently what some might call in alpha. The major, core functionality is in place, and it's possible to use by cloning the repo, but it's only starting to be ready for a first release. There are numerous optimizations, UX, and refactoring that need to be done before a first release candidate. Stay tuned for regular updates, and feel free to extend a hand with contributions, testing, or suggestions if this is something you're interested in.
+
+## The Name
+
+I had the idea to call the software Weeablind as a portmanteaux of Weeaboo (someone a little too obsessed with anime), and blind. I might change it to something else in the future like Blindtaku, DubHub, or something similar and more catchy because the software can be used for far more than just anime.
+
+## Setup
+
+There are currently no prebuilt-binaries to download, this is something I am looking into, but many of these dependencies are not easy to bundle with something like PyInstaller
+
+The program works best on Linux, but will also run on Windows.
+
+### System Prerequisits
+You will need to install [FFmpeg](https://ffmpeg.org/download.html) on your system and make sure it's callable from terminal or in your system PATH
+
+For using Coqui TTS, you will also need Espeak-NG which you can get from your package manager on Linux or [here](https://github.com/espeak-ng/espeak-ng/releases) on Windows
+
+On Windows, pip requires MSVC Build Tools to build Coqui. You can install it here:
+https://visualstudio.microsoft.com/visual-cpp-build-tools/
+
+Coqui TTS and Pyannote diarization will also both perform better if you have CUDA set up on your system to use your GPU. This should work out of the box on Linux but getting it set up on Windows takes some doing. This [blog post](https://saturncloud.io/blog/how-to-run-mozilla-ttscoqui-tts-training-with-cuda-on-a-windows-system/) should walk you through the process. If you can't get it working, don't fret, you can still use them on your CPU.
+
+The latest version of Python works on Linux, but Spleeter only works on 3.10 and Pyannote can be finicky with that too. 3.10 seems to work the best on on Windows. You can get it from the Microsoft Store.
+
+### Setup from Source
+To use the project, you'll need to clone the repository and install the dependencies in a virtual enviormonet.
+
+```
+git clone https://github.com/FlorianEagox/weeablind.git
+cd weeablind
+python3.10 -m venv venv
+# Windows
+.\venv\Scripts\activate
+# Linux
+source ./venv/bin/activate
+```
+This project has a lot of dependencies, and pip can struggle with conflicts, so it's best to install from the lock file like this:
+```
+pip install -r requirements-win-310.txt --no-deps
+```
+You can try from the regular requirements file, but it can take a heck of a long time and requires some rejiggering sometimes. 
+
+Installing the dependencies can take a hot minute and uses a lot of space (~8 GB).
+
+If you don't need certain features for instance, language filtering, you can omit speechbrain from the readme. 
+
+once this is completed, you can run the program with
+
+```
+python weeablind.py
+```
+
+## Usage
+Start by either selecting a video from your computer or pasting a link to a YT video and pressing enter. It should download the video and lot the subs and audio.
+
+### Loading a video
+Once a video is loaded, you can preview the subtitles that will be dubbed. If the wrong language is loaded, or the wrong audio stream, switch to the streams tab and select the correct ones.
+
+### Cropping
+You can specify a start and end time if you only need to dub a section of the video, for example to skip the opening theme and credits of a show. Use timecode syntax like 2:17 and press enter. 
+
+### Configuring Voices
+By default, a "Sample" voice should be initialized. You can play around with different configurations and test the voice before dubbing with the "Sample Voice" button in the "Configure Voices" tab. When you have parameters you're happy with, clicking "Update Voices" will re-asign it to that slot. If you choose the SYSTEM tts engine, the program will use Windows' SAPI5 Narrorator or Linux espeak voices by default. This is extremely fast but sounds very robotic. Selecting Coqui gives you a TON of options to play around with, but you will be prompted to download often very heavy TTS models. VCTK/VITS is my favorite model to dub with as it's very quick, even on CPU, and there are hundreds of speakers to choose from. It is loaded by default. If you have ran diarization, you can select different voices from the listbox and change their properties as well.
+
+### Language Filtering
+In the subtitles tab, you filter the subtitles to exclude lines spoken in your selected language so only the foreign language gets dubbed. This is useful for multi-lingual videos, but not videos all in one language.
+
+### Diarization
+Running diarization will attempt to assign the correct speaker to all the subtitles and generate random voices for the total number of speakers detected. In the futre, you'll be able to specify the diarization pipeline and number of speakers if you know ahead of time. Diarization is only useful for videos with multiple speakers and the accuracy can very massively.
+
+### Background Isolation
+In the "Streams" tab, you can run vocal isolation which will attempt to remove the vocals from your source video track but retain the background. If you're using a multi-lingual video and running language filtering as well, you'll need to run that first to keep the english (or whatever source language's vocals).
+
+### Dubbing
+Once you've configured things how you like, you can press the big, JUICY run dubbing button. This can take a while to run. Once completed, you should have something like "MyVideo-dubbed.mkv" in the `output` directory. This is your finished video!
+
+## Things to do
+- A better filtering system for language detection. Maybe inclusive and exclusive or confidence threshhold
+- Find some less copyrighted multi-lingual / non-english content to display demos publicly
+- de-anglicanization it so the user can select their target language instead of just English
+- FIX PYDUB'S STUPID ARRAY DISTORTION so we don't have to perform 5 IO operations per dub!!!
+- ~~run a vocal isolation / remover on the source audio to remove / mitigate the original speakers?~~
+- ~~A proper setup guide for all platforms~~
+- remove or fix the broken espeak implementation to be cross-platform
+- ~~Uninitialized, singletons for heavy models upon startup (e.g. only intialize pyannote/speechbrain pipelines when needed)~~
+- Abstraction for singletons of Coqui voices using the same model to reduce memory footprint
+- ~~GUI tab to list and select audio / subtitle streams w/ FFMPEG~~
+- ~~Move the tabs into their own classes~~
+- ~~Add labels and screen reader landmarks to all the controls~~
+- ~~Single speaker or multi speaker control switch~~
+- ~~Download YouTube video with Closed Captions~~
+- ~~GUI to select start and end time for dubbing~~
+- Throw up a Flask server on my website so you can try it with minimal features.
+- Use OCR to generate subtitles for videos that don't have sub streams
+- Use OCR for non-text based subtitles
+- Make a cool logo?
+- Learn how to package python programs as binaries to make releases
+- ~~Remove the copyrighted content from this repo (sorry not sorry TV Tokyo)~~
+- Save and import config files for later
+- ~~Support for all subtitle formats~~
+- Maybe slap in an ASR library for videos without subtitles?
+- Maybe support for magnet URLs or the arrLib to pirate media (who knows???)
+
+### Diarization
+- Filter subtitles by the selected voice from the listbox
+- Select from multiple diarization models / pipelines
+- Optimize audio trakcs for diarizaiton by isolating lines speech based on subtitle timings
+- Investigate Diart?
+
+### TTS
+
+- ~~Rework the speed control to use PyDub to speed up audio.~~
+- ~~match the volume of the speaker to TTS~~
+- Checkbox to remove sequential subtitle entries and entries that are tiny, e.g. "nom" "nom" "nom" "nom"
+- investigate voice conversion?
+- Build an asynchronous queue of operations to perform
+- Started - Asynchronous GUI for Coqui model downloads
+- Add support for MyCroft Mimic 3
+- Add Support for PiperTTS
+
+### Cloning
+- Create a cloning mode to select subtitles and export them to a dataset or wav compilation for Coqui XTTS
+- Use diaries and subtitles to isolate and build training datasets
+- Build a tool to streamline the manual creation of datasets
+
+###### (oh god that's literally so many things, the scope of this has gotten so big how will this ever become a thing)

Voice.py

@@ -0,0 +1,159 @@
+from enum import Enum, auto
+import abc
+import os
+import threading
+from time import sleep
+from TTS.api import TTS
+from TTS.utils import manage
+import pyttsx3
+from espeakng import ESpeakNG
+import numpy as np
+from torch.cuda import is_available
+import time
+
+class Voice(abc.ABC):
+	class VoiceType(Enum):
+		ESPEAK = "ESpeak"
+		COQUI = "Coqui TTS"
+		SYSTEM = "System Voices"
+
+	def __new__(cls, voice_type, init_args=[], name="Unnamed"):
+		if cls is Voice:
+			if voice_type == cls.VoiceType.ESPEAK:
+				return super().__new__(ESpeakVoice)
+			elif voice_type == cls.VoiceType.COQUI:
+				return super().__new__(CoquiVoice)
+			elif voice_type == cls.VoiceType.SYSTEM:
+				return super().__new__(SystemVoice)
+		else:
+			return super().__new__(cls)
+
+	def __init__(self, voice_type, init_args=[], name="Unnamed"):
+		self.voice_type = voice_type
+		self.name = name
+		self.voice_option = None
+
+	@abc.abstractmethod
+	def speak(self, text, file_name):
+		pass
+
+	def set_speed(self, speed):
+		pass
+
+	@abc.abstractmethod
+	def set_voice_params(self, voice=None, pitch=None):
+		pass
+
+	@abc.abstractmethod
+	def list_voice_options(self):
+		pass
+
+	def calibrate_rate(self):
+		output_path = './output/calibration.wav'
+		calibration_phrase_long = "In the early morning light, a vibrant scene unfolds as the quick brown fox jumps gracefully over the lazy dog. The fox's russet fur glistens in the sun, and its swift movements captivate onlookers. With a leap of agility, it soars through the air, showcasing its remarkable prowess. Meanwhile, the dog, relaxed and unperturbed, watches with half-closed eyes, acknowledging the fox's spirited display. The surrounding nature seems to hold its breath, enchanted by this charming spectacle. The gentle rustling of leaves and the distant chirping of birds provide a soothing soundtrack to this magical moment. The two animals, one lively and the other laid-back, showcase the beautiful harmony of nature, an ageless dance that continues to mesmerize all who witness it."
+		calibration_phrase_chair = "A chair is a piece of furniture with a raised surface used to sit on, commonly for use by one person. Chairs are most often supported by four legs and have a back; however, a chair can have three legs or could have a different shape. A chair without a back or arm rests is a stool, or when raised up, a bar stool."
+		calibration_phrase = "Hello? Testing, testing. Is.. is this thing on? Ah! Hello Gordon! I'm... assuming that's your real name... You wouldn't lie to us. Would you? Well... You finally did it! You survived the resonance cascade! You brought us all to hell and back, alive! You made it to the ultimate birthday bash at the end of the world! You beat the video game! And... now I imagine you'll... shut it down. Move on with your life. Onwards and upwards, ay Gordon? I don't.. know... how much longer I have to send this to you so I'll try to keep it brief. Not my specialty. Perhaps this is presumptuous of me but... Must this really be the end of our time together? Perhaps you could take the science team's data, transfer us somewhere else, hmm? Now... it doesn't have to be Super Punch-Out for the Super Nintendo Entertainment System. Maybe a USB drive, or a spare floppy disk. You could take us with you! We could see the world! We could... I'm getting a little ahead of myself, surely. Welp! The option's always there! You changed our lives, Gordon. I'd like to think it was for the better. And I don't know what's going to happen to us once you exit the game for good. But I know we'll never forget you. I hope you won't forget us. Well... This is where I get off. Goodbye Gordon!"
+		self.speak(calibration_phrase, output_path)
+
+	def get_wpm(words, duration):
+		return (len(words.split(' ')) / duration * 60)
+
+class ESpeakVoice(Voice):
+	def __init__(self, init_args=[], name="Unnamed"):
+		super().__init__(Voice.VoiceType.ESPEAK, init_args, name)
+		self.set_voice_params(init_args)
+
+	def speak(self, text, file_name):
+		self.voice.synth_wav(text, file_name)
+
+	def set_speed(self, speed):
+		# current_speaker.set_speed(60*int((len(text.split(' ')) / (sub.end.total_seconds() - sub.start.total_seconds()))))
+		self.voice.speed = speed
+
+	def set_voice_params(self, voice=None, pitch=None):
+		if voice:
+			self.voice.voice = voice
+		if pitch:
+			self.voice.pitch = pitch
+
+	def list_voice_options(self):
+		# Optionally, you can return available voice options for ESpeak here
+		pass
+
+class CoquiVoice(Voice):
+	def __init__(self, init_args=None, name="Coqui Voice"):
+		super().__init__(Voice.VoiceType.COQUI, init_args, name)
+		self.voice = TTS().to('cuda' if is_available() else 'cpu')
+		self.langs = ["All Languages"] + list({lang.split("/")[1] for lang in self.voice.list_models()})
+		self.langs.sort()
+		self.selected_lang = 'en'
+		self.is_multispeaker = False
+		self.speaker = None
+		self.speaker_wav = None
+
+	def speak(self, text, file_path=None):
+		if file_path:
+			return self.voice.tts_to_file(
+				text,
+				file_path=file_path,
+				speaker=self.speaker,
+				language= 'en' if self.voice.is_multi_lingual else None,
+				speaker_wav=self.speaker_wav
+			)
+		else:
+			return np.array(self.voice.tts(
+				text,
+				speaker=self.speaker,
+				language= 'en' if self.voice.is_multi_lingual else None
+			))
+
+	def set_voice_params(self, voice=None, speaker=None, speaker_wav=None, progress=None):
+		if voice and voice != self.voice_option:
+			if progress:
+				progress(0, "downloading")
+				download_thread = threading.Thread(target=self.voice.load_tts_model_by_name, args=(voice,))
+				download_thread.start()
+				while(download_thread.is_alive()):
+					# I'll remove this check if they accept my PR c:
+					bar = manage.tqdm_progress if hasattr(manage, "tqdm_progress") else None
+					if bar:
+						progress_value = int(100*(bar.n / bar.total))
+						progress(progress_value, "downloading")
+					time.sleep(0.25)  # Adjust the interval as needed
+				progress(-1, "done!")
+			else:
+				self.voice.load_tts_model_by_name(voice)
+			self.voice_option = self.voice.model_name
+		self.is_multispeaker = self.voice.is_multi_speaker
+		self.speaker = speaker
+
+	def list_voice_options(self):
+		return self.voice.list_models()
+
+	def is_model_downloaded(self, model_name):
+		return os.path.exists(os.path.join(self.voice.manager.output_prefix, self.voice.manager._set_model_item(model_name)[1]))
+
+	def list_speakers(self):
+		return self.voice.speakers if self.voice.is_multi_speaker else []
+
+class SystemVoice(Voice):
+	def __init__(self, init_args=[], name="Unnamed"):
+		super().__init__(Voice.VoiceType.SYSTEM, init_args, name)
+		self.voice = pyttsx3.init()
+		self.voice_option = self.voice.getProperty('voice')
+
+	def speak(self, text, file_name):
+		self.voice.save_to_file(text, file_name)
+		self.voice.runAndWait()
+		return file_name
+
+	def set_speed(self, speed):
+		self.voice.setProperty('rate', speed)
+
+	def set_voice_params(self, voice=None, pitch=None):
+		if voice:
+			self.voice.setProperty('voice', voice)
+			self.voice_option = self.voice.getProperty('voice')
+
+	def list_voice_options(self):
+		return [voice.name for voice in self.voice.getProperty('voices')]

app_state.py

@@ -0,0 +1,7 @@
+from Voice import Voice
+
+video = None
+speakers = [Voice(Voice.VoiceType.COQUI, name="Sample")]
+speakers[0].set_voice_params('tts_models/en/vctk/vits', 'p326') # p340
+current_speaker = speakers[0]
+sample_speaker = current_speaker

diarize.py

@@ -0,0 +1,75 @@
+# This file contains all functions related to diarizing a video including optimization and processing a speech diary (rttm file)
+# These functions use a functional approach as I didn't wanted to group them and not bloat the video class with such specific functions
+# Perhaps going forward I should abstract diary entries as their own objects similar to dub_line, but I haven't decidded yet as diaries might be useful for voice cloning as well
+
+import app_state
+import utils
+from Voice import Voice
+from pyannote.audio import Pipeline
+import torchaudio.transforms as T
+import torchaudio
+import random
+
+pipeline = None
+
+# Read RTTM files generated by Pyannote into an array containing the speaker, start, and end of their speech in the audio
+def load_diary(file):
+	diary = []
+	with open(file, 'r', encoding='utf-8') as diary_file:
+		for line in diary_file.read().strip().split('\n'):
+			line_values = line.split(' ')
+			diary.append([line_values[7], float(line_values[3]), float(line_values[4])])
+	total_speakers = len(set(line[0] for line in diary))
+	app_state.speakers = initialize_speakers(total_speakers)
+	return diary
+
+# Time Shift the speech diary to be in line with the start time
+def update_diary_timing(diary, start_time):
+	return [[int(line[0].split('_')[1]), line[1] + start_time, line[2]] for line in diary]
+
+def initialize_speakers(speaker_count):
+	speakers = []
+	speaker_options = app_state.sample_speaker.list_speakers()
+	for i in range(speaker_count):
+		speakers.append(Voice(Voice.VoiceType.COQUI, f"Voice {i}"))
+		speakers[i].set_voice_params('tts_models/en/vctk/vits', random.choice(speaker_options))
+	return speakers
+
+def find_nearest_speaker(diary, sub):
+	return diary[
+		utils.find_nearest(
+			[diary_entry[1] for diary_entry in diary],
+			sub.start
+		)
+	][0]
+
+
+
+def optimize_audio_diarization(video):
+	crop = video.crop_audio(True)
+	waveform, sample_rate = torchaudio.load(crop)
+	# Apply noise reduction
+	noise_reduce = T.Vad(sample_rate=sample_rate)
+	clean_waveform = noise_reduce(waveform)
+	
+	# Normalize audio
+	normalize = T.Resample(orig_freq=sample_rate, new_freq=sample_rate)
+	normalized_waveform = normalize(clean_waveform)
+
+	return normalized_waveform, sample_rate
+
+def run_diarization(video):
+	global pipeline # Probably should move this to app state?
+	if not pipeline:
+		pipeline = Pipeline.from_pretrained("pyannote/speaker-diarization-3.0", use_auth_token="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")
+		import torch
+		pipeline.to(torch.device("cuda"))
+	output = utils.get_output_path(video.file, ".rttm")
+	optimized, sample_rate = optimize_audio_diarization(video)
+	diarization = pipeline({"waveform": optimized, "sample_rate": sample_rate})
+	with open(output, "w") as rttm:
+		diarization.write_rttm(rttm)
+	diary = load_diary(output)
+	diary = update_diary_timing(diary, video.start_time)
+	for sub in video.subs_adjusted:
+		sub.voice = find_nearest_speaker(diary, sub)

dub_line.py

@@ -0,0 +1,135 @@
+from dataclasses import dataclass
+from Voice import Voice
+import ffmpeg
+import utils
+import app_state
+import srt
+from re import compile, sub as substitute
+from pydub import AudioSegment
+from audiotsm import wsola
+from audiotsm.io.wav import WavReader, WavWriter
+from audiotsm.io.array import ArrayReader, ArrayWriter
+from speechbrain.pretrained import EncoderClassifier
+import numpy as np
+from language_detection import detect_language
+remove_xml = compile(r'<[^>]+>|\{[^}]+\}')
+language_identifier_model = None # EncoderClassifier.from_hparams(source="speechbrain/lang-id-voxlingua107-ecapa", savedir="tmp")
+
+@dataclass
+class DubbedLine:
+	start: float
+	end: float
+	text: str
+	index: int
+	voice: int = 0
+	language: str = ""
+
+	# This is highly inefficient as it writes and reads the same file many times
+	def dub_line_file(self, match_volume=True, output=False):
+		output_path = utils.get_output_path(str(self.index), '.wav', path='files')
+		tts_audio = app_state.speakers[self.voice].speak(self.text, output_path)
+		rate_adjusted = self.match_rate(tts_audio, self.end-self.start)
+		segment = AudioSegment.from_wav(rate_adjusted)
+		if match_volume:
+			segment = self.match_volume(app_state.video.get_snippet(self.start, self.end), segment)
+		if output:
+			segment.export(output_path, format='wav')
+		return segment
+
+	# This should ideally be a much more efficient way to dub.
+	# All functions should pass around numpy arrays rather than reading and writting files. For some reason though, it gives distroted results
+	def dub_line_ram(self, output=True):
+		output_path = utils.get_output_path(str(self.index), '.wav', path='files')
+		tts_audio = app_state.speakers[self.voice].speak(self.text)
+		rate_adjusted = self.match_rate_ram(tts_audio, self.end-self.start)
+		data = rate_adjusted / np.max(np.abs(rate_adjusted))
+		# This causes some kind of wacky audio distrotion we NEED to fix ;C
+		audio_as_int = (data * (2**15)).astype(np.int16).tobytes()
+		segment = AudioSegment(
+			audio_as_int,
+			frame_rate=22050,
+			sample_width=2,
+			channels=1
+		)
+		if output:
+			segment.export(output_path, format='wav')
+		return segment
+
+	def match_rate(self, target_path, source_duration, destination_path=None, clamp_min=0, clamp_max=4):
+		if destination_path == None:
+			destination_path = target_path.split('.')[0] + '-timeshift.wav'
+		duration = float(ffmpeg.probe(target_path)["format"]["duration"])
+		rate = duration*1/source_duration
+		rate = np.clip(rate, clamp_min, clamp_max)
+		with WavReader(target_path) as reader:
+			with WavWriter(destination_path, reader.channels, reader.samplerate) as writer:
+				tsm = wsola(reader.channels, speed=rate)
+				tsm.run(reader, writer)
+		return destination_path
+
+	def match_rate_ram(self, target, source_duration, outpath=None, clamp_min=0.8, clamp_max=2.5):
+		num_samples = len(target)
+		target = target.reshape(1, num_samples)
+		duration = num_samples / 22050
+		rate = duration*1/source_duration
+		rate = np.clip(rate, clamp_min, clamp_max)
+		reader = ArrayReader(target)
+		tsm = wsola(reader.channels, speed=rate)
+		if not outpath:
+			rate_adjusted = ArrayWriter(channels=1)
+			tsm.run(reader, rate_adjusted)
+			return rate_adjusted.data
+		else:
+			rate_adjusted = WavWriter(outpath, 1, 22050)
+			tsm.run(reader, rate_adjusted)
+			rate_adjusted.close()
+			return outpath
+
+	def match_volume(self, source_snippet, target):
+		# ratio = source_snippet.rms / (target.rms | 1)
+		ratio = source_snippet.dBFS - target.dBFS
+		# adjusted_audio = target.apply_gain(ratio)
+		adjusted_audio = target + ratio
+		return adjusted_audio
+		# adjusted_audio.export(output_path, format="wav")
+
+	def get_language(self, source_snippet):
+		if not self.language:
+			self.language = detect_language(source_snippet)
+		return self.language
+
+
+def filter_junk(subs, minimum_duration=0.1, remove_repeats=True):
+	filtered = []
+	previous = ""
+	for sub in subs:
+		if (sub.end - sub.start) > minimum_duration:
+			if sub.text != previous:
+				filtered.append(sub)
+		previous = sub.text
+	return filtered
+
+# This function is designed to handle two cases
+#	1 We just have a path to an srt that we want to import
+#	2 You have a file containing subs, but not srt (a video file, a vtt, whatever)
+# 		In this case, we must extract or convert the subs to srt, and then read it in (export then import)
+def load_subs(import_path="", extract_subs_path=False, filter=True):
+	if extract_subs_path: # For importing an external subtitles file
+		(
+			ffmpeg
+			.input(extract_subs_path)
+			.output(import_path)
+			.global_args('-loglevel', 'error')
+			.run(overwrite_output=True)
+		)
+	with open(import_path, "r", encoding="utf-8") as f:
+		original_subs = list(srt.parse(f.read()))
+		return filter_junk([
+			DubbedLine(
+				sub.start.total_seconds(),
+				sub.end.total_seconds(),
+				substitute(remove_xml, '', sub.content),
+				sub.index
+			)
+			for sub in original_subs
+		])

language_detection.py

@@ -0,0 +1,13 @@
+# This is used to detect the spoken language in an audio file
+# I wanted to abstract it to it's own file, just like vocal isolation & diarization
+from speechbrain.pretrained import EncoderClassifier
+
+language_identifier_model = None
+
+def detect_language(file):
+	global language_identifier_model
+	if not language_identifier_model:
+		language_identifier_model = EncoderClassifier.from_hparams(source="speechbrain/lang-id-voxlingua107-ecapa", savedir="tmp") #, run_opts={"device":"cuda"})
+	signal = language_identifier_model.load_audio(file)
+	prediction = language_identifier_model.classify_batch(signal)
+	return prediction[3][0].split(' ')[1]

loading subs pseudocode

@@ -0,0 +1,20 @@
+user loads video
+	video is file:
+		file has subs?
+			load the first subs
+			display all subs
+			user selects new subs:
+				load subs with given stream index
+	video is YT link:
+		download all subs (if any)
+		subs?
+			display the subs
+			user selects subs (vtt)
+				convert the subs to srt
+				load subs
+	there are no subs!?!?!:
+		This is the spooky zone
+		offer to upload a subtitle file?
+
+		offer to attempt video OCR???
+		attempt ASR + Translation? This would be fucking insane don't do this please don't add this feature this is literally impossible, right???

requirements-linux310.txt

@@ -0,0 +1,237 @@
+absl-py==2.0.0
+accelerate==0.24.1
+aiohttp==3.8.6
+aiosignal==1.3.1
+alembic==1.12.1
+antlr4-python3-runtime==4.9.3
+anyascii==0.3.2
+anyio==3.7.1
+appdirs==1.4.4
+asteroid-filterbanks==0.4.0
+astunparse==1.6.3
+async-timeout==4.0.3
+attrs==23.1.0
+audioread==3.0.1
+audiotsm==0.1.2
+Babel==2.13.1
+bangla==0.0.2
+blinker==1.7.0
+bnnumerizer==0.0.2
+bnunicodenormalizer==0.1.6
+Brotli==1.1.0
+cachetools==5.3.2
+certifi==2023.7.22
+cffi==1.16.0
+charset-normalizer==3.3.2
+clean-fid==0.1.35
+click==7.1.2
+clip-anytorch==2.5.2
+colorama==0.4.6
+coloredlogs==15.0.1
+colorlog==6.7.0
+contourpy==1.1.1
+coqpit==0.0.17
+cycler==0.12.1
+Cython==0.29.30
+dateparser==1.1.8
+decorator==5.1.1
+docker-pycreds==0.4.0
+docopt==0.6.2
+einops==0.6.1
+encodec==0.1.1
+exceptiongroup==1.1.3
+ffmpeg-python==0.2.0
+filelock==3.13.1
+Flask==2.3.3
+flatbuffers==1.12
+fonttools==4.43.1
+frozenlist==1.4.0
+fsspec==2023.6.0
+ftfy==6.1.1
+future==0.18.3
+g2pkk==0.1.2
+gast==0.4.0
+gitdb==4.0.11
+GitPython==3.1.40
+google-auth==2.23.4
+google-auth-oauthlib==0.4.6
+google-pasta==0.2.0
+greenlet==3.0.1
+grpcio==1.59.2
+gruut==2.2.3
+gruut-ipa==0.13.0
+gruut-lang-de==2.0.0
+gruut-lang-en==2.0.0
+gruut-lang-es==2.0.0
+gruut-lang-fr==2.0.2
+h11==0.12.0
+h2==4.1.0
+h5py==3.10.0
+hpack==4.0.0
+httpcore==0.13.7
+httpx==0.19.0
+huggingface-hub==0.18.0
+humanfriendly==10.0
+hyperframe==6.0.1
+HyperPyYAML==1.2.2
+idna==3.4
+imageio==2.31.6
+inflect==5.6.2
+itsdangerous==2.1.2
+jamo==0.4.1
+jieba==0.42.1
+Jinja2==3.1.2
+joblib==1.3.2
+jsonlines==1.2.0
+jsonmerge==1.9.2
+jsonschema==4.19.2
+jsonschema-specifications==2023.7.1
+julius==0.2.7
+k-diffusion==0.0.16
+keras==2.9.0
+Keras-Preprocessing==1.1.2
+kiwisolver==1.4.5
+kornia==0.7.0
+lazy_loader==0.3
+libclang==16.0.6
+librosa==0.10.0
+lightning==2.1.0
+lightning-utilities==0.9.0
+llvmlite==0.40.1
+Mako==1.2.4
+Markdown==3.5.1
+markdown-it-py==3.0.0
+MarkupSafe==2.1.3
+matplotlib==3.7.3
+mdurl==0.1.2
+mpmath==1.3.0
+msgpack==1.0.7
+multidict==6.0.4
+mutagen==1.47.0
+networkx==2.8.8
+nltk==3.8.1
+norbert==0.2.1
+num2words==0.5.13
+numba==0.57.0
+numpy==1.22.0
+nvidia-cublas-cu12==12.1.3.1
+nvidia-cuda-cupti-cu12==12.1.105
+nvidia-cuda-nvrtc-cu12==12.1.105
+nvidia-cuda-runtime-cu12==12.1.105
+nvidia-cudnn-cu12==8.9.2.26
+nvidia-cufft-cu12==11.0.2.54
+nvidia-curand-cu12==10.3.2.106
+nvidia-cusolver-cu12==11.4.5.107
+nvidia-cusparse-cu12==12.1.0.106
+nvidia-nccl-cu12==2.18.1
+nvidia-nvjitlink-cu12==12.3.52
+nvidia-nvtx-cu12==12.1.105
+oauthlib==3.2.2
+omegaconf==2.3.0
+onnxruntime-gpu==1.16.1
+opt-einsum==3.3.0
+optuna==3.4.0
+packaging==23.1
+pandas==1.5.3
+pathtools==0.1.2
+Pillow==10.0.1
+platformdirs==3.11.0
+pooch==1.8.0
+primePy==1.3
+protobuf==3.20.3
+psutil==5.9.6
+py-espeak-ng==0.1.8
+pyannote.audio==3.0.1
+pyannote.core==5.0.0
+pyannote.database==5.0.1
+pyannote.metrics==3.2.1
+pyannote.pipeline==3.0.1
+pyasn1==0.5.0
+pyasn1-modules==0.3.0
+pycparser==2.21
+pycryptodomex==3.19.0
+pydub==0.25.1
+Pygments==2.16.1
+pynndescent==0.5.10
+pyparsing==3.1.1
+pypinyin==0.49.0
+pysbd==0.3.4
+python-crfsuite==0.9.9
+python-dateutil==2.8.2
+pytorch-lightning==2.1.0
+pytorch-metric-learning==2.3.0
+pyttsx3==2.90
+pytz==2023.3.post1
+PyYAML==6.0.1
+referencing==0.30.2
+regex==2023.10.3
+requests==2.31.0
+requests-oauthlib==1.3.1
+resize-right==0.0.2
+rfc3986==1.5.0
+rich==13.6.0
+rpds-py==0.10.6
+rsa==4.9
+ruamel.yaml==0.18.4
+ruamel.yaml.clib==0.2.8
+safetensors==0.4.0
+scikit-image==0.22.0
+scikit-learn==1.3.0
+scipy==1.11.3
+semver==3.0.2
+sentencepiece==0.1.99
+sentry-sdk==1.34.0
+setproctitle==1.3.3
+shellingham==1.5.4
+six==1.16.0
+smmap==5.0.1
+sniffio==1.3.0
+sortedcontainers==2.4.0
+soundfile==0.12.1
+soxr==0.3.7
+speechbrain==0.5.15
+spleeter==2.4.0
+SQLAlchemy==2.0.23
+srt==3.5.3
+sympy==1.12
+tabulate==0.9.0
+tbb==2021.10.0
+tensorboard==2.9.1
+tensorboard-data-server==0.6.1
+tensorboard-plugin-wit==1.8.1
+tensorboardX==2.6.2.2
+tensorflow==2.9.3
+tensorflow-estimator==2.9.0
+tensorflow-io-gcs-filesystem==0.34.0
+termcolor==2.3.0
+threadpoolctl==3.2.0
+tifffile==2023.9.26
+tokenizers==0.13.3
+torch==2.1.0
+torch-audiomentations==0.11.0
+torch-pitch-shift==1.2.4
+torchaudio==2.1.0
+torchdiffeq==0.2.3
+torchmetrics==1.2.0
+torchsde==0.2.6
+torchvision==0.16.0
+tqdm==4.64.1
+trainer==0.0.31
+trampoline==0.1.2
+transformers==4.33.3
+triton==2.1.0
+TTS==0.19.1
+typer==0.3.2
+typing_extensions==4.8.0
+tzlocal==5.2
+umap-learn==0.5.4
+Unidecode==1.3.7
+urllib3==2.0.7
+wandb==0.15.12
+wcwidth==0.2.9
+websockets==12.0
+Werkzeug==3.0.1
+wrapt==1.15.0
+wxPython==4.2.1
+yarl==1.9.2
+yt-dlp==2023.10.13

requirements.txt

@@ -0,0 +1,14 @@
+tts # <-- Coqui TTS engine
+pyannote.audio
+ffmpeg-python
+srt
+py-espeak-ng
+pydub
+# pyAudio # <--- Needed on Windows, breaks on Linux
+-f https://extras.wxpython.org/wxPython4/extras/linux/gtk3/ubuntu-22.04
+wxpython 
+pyttsx3 # <-- System TTS engine
+yt-dlp # <-- Downloading YT vids
+audiotsm # <-- Audio timestretching
+speechbrain # <-- Audio Language Identification
+spleeter # <-- Vocal / Background isolation

synth.py

@@ -0,0 +1,33 @@
+# Formerly the prototypical file, synth. Now it's just a graveyard of functions that may never return?
+from pydub import AudioSegment
+
+
+import concurrent.futures
+from utils import get_output_path
+
+
+# This function was intended to run with multiprocessing, but Coqui won't play nice with that.
+def dub_task(sub, i):
+	print(f"{i}/{len(subs_adjusted)}")
+	try:
+		return dub_line_ram(sub)
+		# empty_audio = empty_audio.overlay(line, sub.start*1000) 
+	except Exception as e:
+		print(e)
+		with open(f"output/errors/{i}-rip.txt", 'w') as f:
+			f.write(e)
+		# total_errors += 1
+
+# This may be used for multithreading?
+def combine_segments():
+	empty_audio = AudioSegment.silent(total_duration * 1000, frame_rate=22050)
+	total_errors = 0
+	for sub in subs_adjusted:
+		print(f"{sub.index}/{len(subs_adjusted)}")
+		try:
+			segment = AudioSegment.from_file(f'output/files/{sub.index}.wav')
+			empty_audio = empty_audio.overlay(segment, sub.start*1000)
+		except:
+			total_errors += 1
+	empty_audio.export('new.wav')
+	print(total_errors)

test.py

@@ -0,0 +1,12 @@
+# This file is just a quick script for whatever I'm testing at the time, it's not really important 
+
+# testing XTTS / VC models
+
+from TTS.api import TTS
+tts = TTS("tts_models/multilingual/multi-dataset/xtts_v2").to('cuda')
+
+# generate speech by cloning a voice using default settings
+tts.tts_to_file(text="Welcome to DougDoug, where we solve problems that no one has",
+                file_path="/media/tessa/SATA SSD1/AI MODELS/cloning/output/doug.wav",
+                speaker_wav="/media/tessa/SATA SSD1/AI MODELS/cloning/doug.wav",
+                language="en")

utils.py

@@ -0,0 +1,53 @@
+import os.path
+import app_state
+import numpy as np
+from pydub.playback import play
+from pydub import AudioSegment
+from torch.cuda import is_available
+
+APP_NAME = "WeeaBlind"
+test_video_name = "./output/download.webm"
+default_sample_path = "./output/sample.wav"
+test_start_time = 94
+test_end_time =  1324
+gpu_detected = is_available()
+
+def create_output_dir():
+	path = './output/files'
+	if not os.path.exists(path):
+		os.makedirs(path)
+
+def get_output_path(input, suffix, prefix='', path=''):
+	filename = os.path.basename(input)
+	filename_without_extension = os.path.splitext(filename)[0]
+	return os.path.join(os.path.dirname(os.path.abspath(__file__)), 'output', path, f"{prefix}{filename_without_extension}{suffix}")
+
+def timecode_to_seconds(timecode):
+	parts = list(map(float, timecode.split(':')))
+	seconds = parts[-1]
+	if len(parts) > 1:
+		seconds += parts[-2] * 60
+	if len(parts) > 2:
+		seconds += parts[-3] * 3600
+	return seconds
+
+def seconds_to_timecode(seconds):
+	hours = int(seconds // 3600)
+	minutes = int((seconds % 3600) // 60)
+	seconds = seconds % 60
+	timecode = ""
+	if hours:
+		timecode += f"{hours}:"
+	if minutes:
+		timecode += f"{minutes}:" 
+	timecode = f"{timecode}{seconds:05.2f}"
+	return timecode
+
+# Finds the closest element in an arry to the given value
+def find_nearest(array, value):
+	return (np.abs(np.asarray(array) - value)).argmin()
+
+def sampleVoice(text, output=default_sample_path):
+	play(AudioSegment.from_file(app_state.sample_speaker.speak(text, output)))
+
+snippet_export_path = get_output_path("video_snippet", "wav")

video.py

@@ -0,0 +1,219 @@
+"""
+The Video class represents a reference to a video from either a file or web link. This class should implement the ncessary info to dub a video.
+"""
+
+from io import StringIO
+import time
+import ffmpeg
+from yt_dlp import YoutubeDL
+import utils
+from pydub import AudioSegment
+from dub_line import load_subs
+import json
+import numpy as np
+import librosa
+import soundfile as sf
+
+class Video:
+	def __init__(self, video_URL, loading_progress_hook=print):
+		self.start_time = self.end_time = 0
+		self.downloaded = False
+		self.subs = self.subs_adjusted = self.subs_removed = []
+		self.background_track = self.vocal_track = None
+		self.speech_diary = self.speech_diary_adjusted = None
+		self.load_video(video_URL, loading_progress_hook)
+
+
+	# This is responsible for loading the app's audio and subtitles from a video file or YT link
+	def load_video(self, video_path, progress_hook=print):
+		sub_path = ""
+		if video_path.startswith("http"):
+			self.downloaded = True
+			try:
+				video_path, sub_path, self.yt_sub_streams = self.download_video(video_path, progress_hook)
+			except: return
+			progress_hook({"status":"complete"})
+		else:
+			self.downloaded = False
+		self.file = video_path
+		if not (self.downloaded and not sub_path):
+			try:
+				self.subs = self.subs_adjusted = load_subs(utils.get_output_path(self.file, '.srt'), sub_path or video_path)
+			except:
+				progress_hook({"status": "subless"})
+		self.audio = AudioSegment.from_file(video_path)
+		self.duration = float(ffmpeg.probe(video_path)["format"]["duration"])
+		if self.subs:
+			self.update_time(0, self.duration)
+
+	def download_video(self, link, progress_hook=print):
+		options = {
+			'outtmpl': 'output/%(id)s.%(ext)s',
+			'writesubtitles': True,
+			"subtitleslangs": ["all"],
+			"progress_hooks": (progress_hook,)
+		}
+		try:
+			with YoutubeDL(options) as ydl:
+				info = ydl.extract_info(link)
+				return ydl.prepare_filename(info), list(info["subtitles"].values())[0][-1]["filepath"] if info["subtitles"] else None, info["subtitles"]
+		except Exception as e:
+			print('AHHH\n',e,'\nAHHHHHH')
+			progress_hook({"status": "error", "error": e})
+			raise e
+
+
+	def update_time(self, start, end):
+		self.start_time = start
+		self.end_time = end
+		# clamp the subs to the crop time specified
+		start_line = utils.find_nearest([sub.start for sub in self.subs], start)
+		end_line = utils.find_nearest([sub.start for sub in self.subs], end)
+		self.subs_adjusted = self.subs[start_line:end_line]
+		if self.speech_diary:
+			self.update_diary_timing()
+
+	def list_streams(self):
+		probe = ffmpeg.probe(self.file)["streams"]
+		if self.downloaded:
+			subs = [{"name": stream[-1]['name'], "stream": stream[-1]['filepath']} for stream in self.yt_sub_streams.values()]
+		else:
+			subs = [{"name": stream['tags'].get('language', 'unknown'), "stream": stream['index']} for stream in probe if stream["codec_type"] == "subtitle"]
+		return {
+			"audio": [stream for stream in probe if stream["codec_type"] == "audio"],
+			"subs": subs
+		}
+
+	def get_snippet(self, start, end):
+		return self.audio[start*1000:end*1000]
+	
+	# Crops the video's audio segment to reduce memory size
+	def crop_audio(self, isolated_vocals):
+		# ffmpeg -i .\saiki.mkv -vn -ss 84 -to 1325 crop.wav
+		source_file = self.vocal_track if isolated_vocals and self.vocal_track else self.file
+		output = utils.get_output_path(source_file, "-crop.wav")
+		(
+			ffmpeg
+			.input(self.file, ss=self.start_time, to=self.end_time)
+			.output(output)
+			.global_args('-loglevel', 'error')
+			.global_args('-vn')
+			.run(overwrite_output=True)
+		)
+		return output
+
+	def filter_multilingual_subtiles(self, progress_hook=print, exclusion="English"):
+		multi_lingual_subs = []
+		removed_subs = []
+		# Speechbrain is being a lil bitch about this path on Windows all of the sudden
+		snippet_path = "video_snippet.wav" # utils.get_output_path('video_snippet', '.wav')
+		for i, sub in enumerate(self.subs_adjusted):
+			self.get_snippet(sub.start, sub.end).export(snippet_path, format="wav")
+			if sub.get_language(snippet_path) != exclusion:
+				multi_lingual_subs.append(sub)
+			else:
+				removed_subs.append(sub)
+			progress_hook(i, f"{i}/{len(self.subs_adjusted)}: {sub.text}")
+		self.subs_adjusted = multi_lingual_subs
+		self.subs_removed = removed_subs
+		progress_hook(-1, "done")
+
+	# This funxion is is used to only get the snippets of the audio that appear in subs_adjusted after language filtration or cropping, irregardless of the vocal splitting.
+	# This should be called AFTER filter multilingual and BEFORE vocal isolation. Not useful yet
+	# OKAY THERE HAS TO BE A FASTER WAY TO DO THIS X_X
+
+	# def isolate_subs(self):
+	# 	base = AudioSegment.silent(duration=self.duration*1000, frame_rate=self.audio.frame_rate, channels=self.audio.channels, frame_width=self.audio.frame_width)
+	# 	samples = np.array(base.get_array_of_samples())
+	# 	frame_rate = base.frame_rate
+		
+	# 	for sub in self.subs_adjusted:
+	# 		copy = np.array(self.get_snippet(sub.start, sub.end).get_array_of_samples())
+	# 		start_sample = int(sub.start * frame_rate)
+	# 		end_sample = int(sub.end * frame_rate)
+			
+	# 		# Ensure that the copy array has the same length as the region to replace
+	# 		copy = copy[:end_sample - start_sample]  # Trim if necessary
+			
+	# 		samples[start_sample:end_sample] = copy
+
+	# 	return AudioSegment(
+	# 		samples.tobytes(),
+	# 		frame_rate=frame_rate,
+	# 		sample_width=base.sample_width,  # Adjust sample_width as needed (2 bytes for int16)
+	# 		channels=base.channels
+	# 	)
+
+	def isolate_subs(self, subs):
+		empty_audio = AudioSegment.silent(self.duration * 1000, frame_rate=self.audio.frame_rate)
+		empty_audio = self.audio
+		first_sub = subs[0]
+		empty_audio = empty_audio[0:first_sub.start].silent((first_sub.end-first_sub.start)*1000)
+		for i, sub in enumerate(subs[:-1]):
+			print(sub.text)
+			empty_audio = empty_audio[sub.end:subs[i+1].start].silent((subs[i+1].start-sub.end)*1000, frame_rate=empty_audio.frame_rate, channels=empty_audio.channels, sample_width=empty_audio.sample_width, frame_width=empty_audio.frame_width)
+
+		return empty_audio
+
+	def run_dubbing(self, progress_hook=None):
+		total_errors = 0
+		operation_start_time = time.process_time()
+		empty_audio = AudioSegment.silent(self.duration * 1000, frame_rate=22050)
+		status = ""
+		# with concurrent.futures.ThreadPoolExecutor(max_workers=100) as pool:
+		# 	tasks = [pool.submit(dub_task, sub, i) for i, sub in enumerate(subs_adjusted)]		
+		# 	for future in concurrent.futures.as_completed(tasks):
+		# 		pass
+		for i, sub in enumerate(self.subs_adjusted):
+			status = f"{i}/{len(self.subs_adjusted)}"
+			progress_hook(i, f"{status}: {sub.text}")
+			try:
+				line = sub.dub_line_file(False)
+				empty_audio = empty_audio.overlay(line, sub.start*1000)
+			except Exception as e:
+				print(e)
+				total_errors += 1
+		self.dub_track = empty_audio.export(utils.get_output_path(self.file, '-dubtrack.wav'), format="wav").name
+		progress_hook(i+1, "Mixing New Audio")
+		self.mix_av(mixing_ratio=1)
+		progress_hook(-1)
+		print(f"TOTAL TIME TAKEN: {time.process_time() - operation_start_time}")
+		# print(total_errors)
+
+	# This runs an ffmpeg command to combine the audio, video, and subtitles with a specific ratio of how loud to make the dubtrack
+	def mix_av(self, mixing_ratio=1, dubtrack=None, output_path=None):
+		# i hate python, plz let me use self in func def
+		if not dubtrack: dubtrack = self.dub_track
+		if not output_path: output_path = utils.get_output_path(self.file, '-dubbed.mkv')
+
+		input_video = ffmpeg.input(self.file)
+		input_audio = input_video.audio
+		if self.background_track:
+			input_audio = ffmpeg.input(self.background_track)
+		input_dub = ffmpeg.input(dubtrack).audio
+
+		mixed_audio = ffmpeg.filter([input_audio, input_dub], 'amix', duration='first', weights=f"1 {mixing_ratio}")
+
+		output = (
+			# input_video['s']
+			ffmpeg.output(input_video['v'], mixed_audio, output_path, vcodec="copy", acodec="aac")
+			.global_args('-loglevel', 'error')
+			.global_args('-shortest')
+		)
+		ffmpeg.run(output, overwrite_output=True)
+
+	# Change the subs to either a file or a different stream from the video file
+	def change_subs(self, stream_index=-1):
+		if self.downloaded:
+			sub_path = list(self.yt_sub_streams.values())[stream_index][-1]['filepath']
+			self.subs = self.subs_adjusted = load_subs(utils.get_output_path(sub_path, '.srt'), sub_path)
+		else:
+			# ffmpeg -i output.mkv -map 0:s:1 frick.srt
+			sub_path = utils.get_output_path(self.file, '.srt')
+			ffmpeg.input(self.file).output(sub_path, map=f"0:s:{stream_index}").run(overwrite_output=True)
+			self.subs = self.subs_adjusted = load_subs(sub_path)
+
+	def change_audio(self, stream_index=-1):
+		audio_path = utils.get_output_path(self.file, f"-${stream_index}.wav")
+		ffmpeg.input(self.file).output(audio_path, map=f"0:a:{stream_index}").run(overwrite_output=True)
+		self.audio = AudioSegment.from_file(audio_path)

vocal_isolation.py

@@ -0,0 +1,47 @@
+from spleeter.separator import Separator
+from spleeter.audio import adapter
+from pydub import AudioSegment
+import numpy as np
+import utils
+
+separator = None # Separator('spleeter:2stems')
+# I don't have any clue on how to make this work yet, just ignore for now. Ideally we'd never have to serialize the audio to wav and then rea read it but alas, bad implementations of PCM will be the death of me
+def seperate_ram(video):
+	audio_loader = adapter.AudioAdapter.default()
+	sample_rate = 44100
+	audio = video.audio
+	# arr = np.array(audio.get_array_of_samples(), dtype=np.float32).reshape((-1, audio.channels)) / (
+	#         1 << (8 * audio.sample_width - 1)), audio.frame_rate
+	arr = np.array(audio.get_array_of_samples())
+	audio, _ = audio_loader.load_waveform(arr)
+	# waveform, _ = audio_loader.load('/path/to/audio/file', sample_rate=sample_rate)
+
+	print("base audio\n", base_audio, "\n")
+	# Perform the separation :
+	# prediction = separator.separate(audio)
+
+def seperate_file(video, isolate_subs=True):
+	global separator
+	if not separator:
+		separator = Separator('spleeter:2stems')
+	source_audio_path = utils.get_output_path(video.file, '-audio.wav')
+	isolated_path = utils.get_output_path(video.file, '-isolate.wav')
+	separator.separate_to_file(
+		(video.audio).export(source_audio_path, format="wav").name,
+		'./output/',
+		filename_format='{filename}-{instrument}.{codec}'
+	)
+	# separator.separate_to_file(
+	# 	video.isolate_subs().export(source_audio_path, format="wav").name,
+	# 	'./output/',
+	# 	filename_format='{filename}-{instrument}.{codec}'
+	# )
+	background_track = utils.get_output_path(source_audio_path, '-accompaniment.wav')
+	# If we removed primary langauge subs from a multilingual video, we'll need to add them back to the background.
+	if video.subs_removed:
+		background = AudioSegment.from_file(background_track)
+		for sub in video.subs_removed:
+			background = background.overlay(video.get_snippet(sub.start, sub.end), int(sub.start*1000))
+		background.export(background_track, format="wav")
+	video.background_track = background_track
+	video.vocal_track = utils.get_output_path(isolated_path, '-vocals.wav')

weeablind.py

@@ -0,0 +1,163 @@
+import wx
+import wx.adv
+from Voice import Voice
+from pydub import AudioSegment
+from pydub.playback import play
+from tabs.ConfigureVoiceTab import ConfigureVoiceTab
+from tabs.SubtitlesTab import SubtitlesTab
+from tabs.ListStreams import ListStreamsTab
+import threading
+import utils
+from video import Video
+import app_state
+from video import Video
+import json
+
+class GUI(wx.Panel):
+	def __init__(self, parent):
+		super().__init__(parent)
+
+		# Labels
+		lbl_title = wx.StaticText(self, label="WeeaBlind")
+		lbl_GPU = wx.StaticText(self, label=f"GPU Detected? {utils.gpu_detected}")
+		lbl_GPU.SetForegroundColour((0, 255, 0) if utils.gpu_detected else (255, 0, 0))
+		lbl_main_file = wx.StaticText(self, label="Choose a video file or link to a YouTube video:")
+		lbl_start_time = wx.StaticText(self, label="Start Time:")
+		lbl_end_time = wx.StaticText(self, label="End Time:")
+
+		# Controls
+		btn_choose_file = wx.Button(self, label="Choose File")
+		btn_choose_file.Bind(wx.EVT_BUTTON, self.open_file)
+
+		self.txt_main_file = wx.TextCtrl(self, style=wx.TE_PROCESS_ENTER, value=utils.test_video_name)
+		self.txt_main_file.Bind(wx.EVT_TEXT_ENTER, lambda event: self.load_video(self.txt_main_file.Value))
+
+		self.txt_start = wx.TextCtrl(self, style=wx.TE_PROCESS_ENTER, value=utils.seconds_to_timecode(0))
+		self.txt_end = wx.TextCtrl(self, style=wx.TE_PROCESS_ENTER, value=utils.seconds_to_timecode(0))
+		self.txt_start.Bind(wx.EVT_TEXT_ENTER, self.change_crop_time)
+		self.txt_end.Bind(wx.EVT_TEXT_ENTER, self.change_crop_time)
+
+		self.chk_match_volume = wx.CheckBox(self, label="Match Speaker Volume")
+		self.chk_match_volume.SetValue(True)
+
+		self.lb_voices = wx.ListBox(self, choices=[speaker.name for speaker in app_state.speakers])
+		self.lb_voices.Bind(wx.EVT_LISTBOX, self.on_voice_change)
+		self.lb_voices.Select(0)
+
+		tab_control = wx.Notebook(self)
+		self.tab_voice_config = ConfigureVoiceTab(tab_control, self)
+		tab_control.AddPage(self.tab_voice_config, "Configure Voices")
+		self.tab_subtitles = SubtitlesTab(tab_control, self)
+		tab_control.AddPage(self.tab_subtitles, "Subtitles")
+		self.streams_tab = ListStreamsTab(tab_control, self)
+		tab_control.AddPage(self.streams_tab, "Video Streams")
+		btn_run_dub = wx.Button(self, label="Run Dubbing!")
+		btn_run_dub.Bind(wx.EVT_BUTTON, self.run_dub)
+		sizer = wx.GridBagSizer(vgap=5, hgap=5)
+
+		sizer.Add(lbl_title, pos=(0, 0), span=(1, 2), flag=wx.CENTER | wx.ALL, border=5)
+		sizer.Add(lbl_GPU, pos=(0, 3), span=(1, 1), flag=wx.CENTER | wx.ALL, border=5)
+		sizer.Add(lbl_main_file, pos=(2, 0), span=(1, 2), flag=wx.LEFT | wx.TOP, border=5)
+		sizer.Add(self.txt_main_file, pos=(3, 0), span=(1, 2), flag=wx.EXPAND | wx.LEFT | wx.RIGHT | wx.BOTTOM, border=5)
+		sizer.Add(btn_choose_file, pos=(3, 2), span=(1, 1), flag=wx.ALIGN_RIGHT | wx.RIGHT | wx.BOTTOM, border=5)
+		sizer.Add(lbl_start_time, pos=(4, 0), flag=wx.LEFT | wx.TOP, border=5)
+		sizer.Add(self.txt_start, pos=(4, 1), flag= wx.TOP | wx.RIGHT, border=5)
+		sizer.Add(lbl_end_time, pos=(5, 0), flag=wx.LEFT | wx.TOP, border=5)
+		sizer.Add(self.txt_end, pos=(5, 1), flag= wx.TOP | wx.RIGHT, border=5)
+		sizer.Add(self.chk_match_volume, pos=(6, 0), span=(1, 2), flag=wx.LEFT | wx.TOP, border=5)
+		sizer.Add(self.lb_voices, pos=(7, 0), span=(1, 1), flag=wx.EXPAND | wx.LEFT | wx.TOP, border=5)
+		sizer.Add(tab_control, pos=(7, 1), span=(1, 3), flag=wx.EXPAND | wx.ALL, border=5)
+		sizer.Add(btn_run_dub, pos=(9, 2), span=(1, 1), flag=wx.ALIGN_RIGHT | wx.RIGHT | wx.BOTTOM, border=5)
+		sizer.AddGrowableCol(1)
+		self.tab_voice_config.update_voice_fields(None)
+
+		self.SetSizerAndFit(sizer)
+
+	def open_file(self, evenet):
+		dlg = wx.FileDialog(
+			frame, message="Choose a file",
+			wildcard="*.*",
+			style=wx.FD_OPEN | wx.FD_CHANGE_DIR
+		)
+		if dlg.ShowModal() == wx.ID_OK:
+			self.load_video(dlg.GetPath())
+		dlg.Destroy()
+
+	def load_video(self, video_path):
+		def update_ui():
+			self.txt_main_file.Value = app_state.video.file
+			self.txt_start.SetValue(utils.seconds_to_timecode(app_state.video.start_time))
+			self.txt_end.SetValue(utils.seconds_to_timecode(app_state.video.end_time))
+			self.tab_subtitles.create_entries()
+
+		def initialize_video(progress=True):
+			app_state.video = Video(video_path, update_progress if progress else print)
+			wx.CallAfter(update_ui)
+			wx.CallAfter(self.streams_tab.populate_streams, app_state.video.list_streams())
+
+		if video_path.startswith("http"):
+			dialog = wx.ProgressDialog("Downloading Video", "Download starting", 100, self)
+
+			def update_progress(progress=None):
+				status = progress['status'] if progress else "waiting"
+				total = progress.get("fragment_count", progress.get("total_bytes", 0))
+				if status == "downloading" and total:
+					completed = progress.get("fragment_index", progress.get("downloaded_bytes", 1))
+					percent_complete = int(100 * (completed / total))
+					wx.CallAfter(dialog.Update, percent_complete, f"{status}: {percent_complete}% \n {progress['info_dict'].get('fulltitle', '')}")
+				elif status == "complete":
+					if dialog:
+						wx.CallAfter(dialog.Destroy)
+				elif status == "error":
+					wx.CallAfter(wx.MessageBox,
+						f"Failed to download video with the following Error:\n {str(progress['error'])}",
+						"Error",
+						wx.ICON_ERROR
+					)
+					update_progress({"status": "complete"})
+
+			threading.Thread(target=initialize_video).start()
+		else:
+			initialize_video(False)
+
+	def change_crop_time(self, event):
+		app_state.video.update_time(
+			utils.timecode_to_seconds(self.txt_start.Value),
+			utils.timecode_to_seconds(self.txt_end.Value)
+		)
+		self.tab_subtitles.create_entries()
+
+	def update_voices_list(self):
+		self.lb_voices.Set([speaker.name for speaker in app_state.speakers])
+		self.lb_voices.Select(self.lb_voices.Strings.index(app_state.current_speaker.name))
+
+	def on_voice_change(self, event):
+		app_state.current_speaker = app_state.speakers[self.lb_voices.GetSelection()]
+		app_state.sample_speaker = app_state.current_speaker
+		self.tab_voice_config.update_voice_fields(event)
+
+	def run_dub(self, event):
+		progress_dialog = wx.ProgressDialog(
+			"Dubbing Progress",
+			"Starting...",
+			maximum=len(app_state.video.subs_adjusted) + 1,  # +1 for combining phase
+			parent=self,
+			style=wx.PD_APP_MODAL | wx.PD_AUTO_HIDE
+		)
+		dub_thread = None
+		def update_progress(i, text=""):
+			if i == -1:
+				return wx.CallAfter(progress_dialog.Destroy)
+			wx.CallAfter(progress_dialog.Update, i, text)
+
+		dub_thread = threading.Thread(target=app_state.video.run_dubbing, args=(update_progress,))
+		dub_thread.start()
+
+if __name__ == '__main__':
+	utils.create_output_dir()
+	app = wx.App(False)
+	frame = wx.Frame(None, wx.ID_ANY, utils.APP_NAME, size=(800, 800))
+	frame.Center()
+	gui = GUI(frame)
+	frame.Show()
+	app.MainLoop()