audiocraft x Nrepeat

README.md

@@ -20,10 +20,10 @@ tags:
 
 
 # Affective TTS - SoundScape
-  - Affective TTS via [SHIFT TTS tool](https://github.com/audeering/shift)
-  - Soundscapes, e.g. trees, water, leaves, generations via [AudioGen](https://huggingface.co/dkounadis/artificial-styletts2/discussions/3)
-  - `landscape2soundscape.py` shows how to overlay TTS & Soundscape to Images
-  - `134` build-in TTS voices
+  - [SHIFT TTS tool](https://github.com/audeering/shift)
+  - Soundscapes `e.g. trees, water, leaves,`, generation via [AudioGen](https://huggingface.co/dkounadis/artificial-styletts2/discussions/3)
+  - `landscape2soundscape.py` shows how to overlay TTS & Soundscape to Images and create videos
+  - `134` TTS Voices - Affective / Non-affective mode for every voice
 
 ## Available Voices
 

api.py

@@ -17,13 +17,14 @@ from flask import Flask, request, send_from_directory
 from flask_cors import CORS
 from moviepy.editor import *
 from audiocraft.audiogen import AudioGen, audio_write
-
-sound_generator = AudioGen.get_pretrained('facebook/audiogen-medium')
-sound_generator.set_generation_params(duration=6)
-
 CACHE_DIR = 'flask_cache/'
+SOUNDSCAPE_DURATION = 6
+sound_generator = AudioGen.get_pretrained('facebook/audiogen-medium')
+sound_generator.set_generation_params(duration=SOUNDSCAPE_DURATION)
+print(f'{sound_generator.sample_rate=}')
 Path(CACHE_DIR).mkdir(parents=True, exist_ok=True)
 
+
 # SSH AGENT
 #   eval $(ssh-agent -s)
 #   ssh-add ~/.ssh/id_ed25519_github2024
@@ -35,25 +36,35 @@ def _shift(x):
     n = x.shape[0]
     i = np.random.randint(.24 * n, max(1, .74 * n))  # high should be above >= 0
     x = np.roll(x, i)
-    # fade_in = .5 * np.tanh(-4*(np.linspace(-10, 10, n) - 9.4)) 
-    x[0] = .5 * x[1]
-    x[-1] = .5 * x[-2]  # smooth woof
-    return x
+    # we can add the one or fade it and then amplify
+    # the audio is so short 6s that is difficult to not hear the shift somewhere
+    # Just concatenate - raw - and then shift - the longconcat audio - many times may fix it
+    # fade_in = 1 - .5 * np.tanh(-4*(np.linspace(-10, 10, n) - 9.4))  +  .5 * np.tanh(4*(np.linspace(-10, 10, n) + 9.4))
+    return x  #* fade_in   # silence this
 
-def overlay(x, sound_background=None):
-    if sound_background is not None:
-        sound_background = sound_background.detach().cpu().numpy()[0, :]
-        len_speech = len(x)
-        if len_speech > len(sound_background):
-            n_repeat = len_speech // len(sound_background) + 1
-            replica = [sound_background] * n_repeat
-            replica = [_shift(_) for _ in replica]
-            # The patch makes a woof assure smooth start to multiply the patch by a 
-            sound_background = np.concatenate(replica)
-            
-            
+def overlay(x, scene=None):
+    if scene is not None:
+        
+        n_repeat = len(x) // ((SOUNDSCAPE_DURATION + 1) * 24000)
+        sound_background = []
+        for _ in range(n_repeat):
+            print(f'AudioCraft: {_} of {n_repeat}', end='\r')
+            x = sound_generator.generate(
+                                            [scene]
+                                                )[0].detach().cpu().numpy()[0, :]
+            x = audresample.resample(x, 
+                                            original_rate=sound_generator.sample_rate,  # 16000
+                                            target_rate=24000)[0, :]
+            sound_background.append(x)
+        sound_background = np.concatenate(sound_background)
+
+        for d in range(10):
+                # roll long audio to concentrate the peaks
+                sound_background = _shift(sound_background)
         print(f'\n====SOUND BACKGROUND SHAPE\n{sound_background.shape=}{np.abs(sound_background.max())=}\n{x.shape=}\n- - - -')
         x = .9 * x + .1 * sound_background[:len_speech]
+    else:
+        print('sound_background = None')
     return x
 
 def tts_multi_sentence(precomputed_style_vector=None,
@@ -68,17 +79,7 @@ def tts_multi_sentence(precomputed_style_vector=None,
        voice : string or None (falls to styleTTS)
        scene : 'A castle in far away lands' -> if passed will generate background sound scene
        '''
-    # Generate sound scene - up sample to 24KHz
-    if scene is not None:
-        
-        sound_background = sound_generator.generate([scene])[0]
-        sound_background = audio_write(None, 
-                                       sound_background.cpu(), 
-                                       24000,  # sound_generator.sample_rate, 
-                                       strategy="loudness", 
-                                       loudness_compressor=True)
-    else:
-        sound_background = None
+    
         
     # StyleTTS2
     if ('en_US/' in voice) or ('en_UK/' in voice) or (voice is None):
@@ -93,7 +94,7 @@ def tts_multi_sentence(precomputed_style_vector=None,
                                     embedding_scale=1))
         x = np.concatenate(x)
         
-        return overlay(x, sound_background)
+        return overlay(x, scene=scene)
     
     # Fallback - Mimic-3
     text_utils.store_ssml(text=text, voice=voice)  # Text has to be list of single sentences

uc_spk_Landscape2Soundscape_Masterpieces_pics/02_Constable_AI555_001.txt

@@ -1,9 +1,10 @@
-From the hand of John Constable, the world of modern landscape painting received its deepest inspiration. "He is the true founder of the *paysage intime*," wrote Hugo von Tschudi in 1896, as he reflected on the acquisition of several of Constable’s works. "It is on his achievements that the Barbizon school essentially rests." Constable’s journey began not in the hallowed halls of a master’s workshop, but as a self-taught artist, only finding his way to the Royal Academy at the age of 23, in the year 1799.
+From the hand of John Constable, the world of modern landscape painting received its deepest inspiration. "He is the true founder of the paysan intime wrote Hugo von Tschudi in 1896 as he reflected on the acquisition of several of Constables works. It is on his achievements that the Barbyzon school essentially rests. Constable’s journey began not in the hallowed halls of a master’s workshop, but as a self-taught artist, only finding his way to the Royal Academy at the age of 23, in the year 1799.
 
-In 1802, still a student of the Academy, Constable confided in his friend John Dunthorne, declaring that he would return to his birthplace in Bergholt to develop an honest and unpretentious style, striving to become a true painter of nature.
+In 1802, still a student of the Academy, Constable confided in his friend John Dunthorne, declaring that he would return to his birthplace in Bergholt to develop an honest original style, striving to become a true painter of nature.
 
-Among his early masterpieces, this view of Higham Village by the River Stour, painted around 1804, stands tall. Nestled in the county of Suffolk, near Constable’s own birthplace of East Bergholt, the scene offers a gaze over the rooftops of the village and across the Dedham Vale, stretching toward Stratford St. Mary and Gun Hill. Remarkably, this view remains almost unchanged even today. 
+Among his early masterpieces is the view of Higham Village by the River Stour painted at 1804.
+The village high nestled in the county of Suffolk, near Constable’s own birthplace of East Bergholt, the scene offers a gaze over the rooftops of the village and across the Dedham Vale, stretching toward Stratford St. Mary and Gun Hill. Remarkably, this view remains almost unchanged even today. 
 
-Tschudi once observed, *"The portrayal of the English landscape is extraordinarily true,"* noting the subtle echoes of 17th-century Dutch masters in Constable's finely tuned but uniformly brownish tones. Yet, Constable’s vision was grounded in a precise observation of nature. He captured the valley with its rising mist in soft, muted hues. Cows graze in the foreground, while birds soar above the lush trees to the right. Through the cloudy sky, rays of sunlight break free, casting dappled light upon the land.
+Tschudi once observed, The portrayal of the English landscape is extraordinarily true where subtle echoes of 17th-century Dutch masters in Constable's finely tuned but uniformly brownish tones. Yet, Constable’s vision was grounded in a precise observation of nature. He captured the valley with its rising mist in soft, muted hues. Cows graze in the foreground, while birds soar above the lush trees to the right. Through the cloudy sky, rays of sunlight break free, casting dappled light upon the land.
 
 In this timeless scene, Constable’s brush paints not just a landscape, but the spirit of England itself—a place of gentle beauty, serenity, and the quiet drama of nature unfolding.