generate sp embedding

CLAUDE.md

@@ -1,96 +0,0 @@
-# CLAUDE.md
-
-This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
-
-## Project Overview
-
-KaniTTS-2 is a Gradio-based web application for text-to-speech generation using the KaniTTS model family from PyPI. It's designed to run on HuggingFace Spaces with GPU acceleration via the `@spaces.GPU` decorator.
-
-## Running the Application
-
-```bash
-python app.py
-```
-
-The app launches on `0.0.0.0:7860` with a Gradio interface.
-
-## Architecture
-
-### Initialization Flow
-
-The application follows a strict initialization sequence that must be maintained:
-
-1. **Dependency Setup** ([app.py:1-3](app.py#L1-L3)): `create_env.setup_dependencies()` runs first to install the dev version of transformers from GitHub. This uses a `/tmp/deps_installed` marker to prevent repeated installations.
-
-2. **Configuration Loading** ([app.py:12-13](app.py#L12-L13)): Loads `model_config.yaml` which defines multiple model checkpoints (e.g., "test-135000", "test-130000") with their HuggingFace paths and parameters.
-
-3. **Examples Loading** ([app.py:15-17](app.py#L15-L17)): Loads `examples.yaml` via the `Examples` adapter class, which transforms YAML examples into Gradio-compatible list-of-lists format.
-
-4. **Model Initialization** ([app.py:19-20](app.py#L19-L20)): `InitModels` loads all models upfront so the UI can switch between them without latency. Each model is a `KaniTTS` instance initialized directly with its config using unpacking (`**config`).
-
-### Key Components
-
-**InitModels** ([util.py:24-57](util.py#L24-L57))
-- Lazy initializer that constructs a map of `model_name -> KaniTTS`
-- Loads all models immediately in `__call__` for zero-switching latency
-- Each `KaniTTS` instance is initialized by unpacking its config directly: `KaniTTS(**config)`
-- No longer requires `NemoAudioPlayer` or HuggingFace token as these are handled internally by the PyPI package
-
-**Examples** ([util.py:59-97](util.py#L59-L97))
-- Adapter converting YAML examples to Gradio `gr.Examples` rows
-- Order must match UI inputs: `[text, model_dropdown, temp, top_p, rp]`
-- Centralizes format and defaults so UI input order changes only require updates here and in [app.py](app.py)
-
-**generate_speech_gpu** ([app.py:22-55](app.py#L22-L55))
-- Decorated with `@spaces.GPU` for HuggingFace Spaces GPU allocation
-- Calls the model directly: `audio, _ = model(text, temperature=t, top_p=top_p, repetition_penalty=rp)`
-- Returns only the audio tuple: `(sample_rate, audio)` - the text output is ignored
-
-### Configuration Files
-
-**model_config.yaml**
-- Defines multiple model checkpoints under the `models` key
-- Each model specifies: `model_name` (HF repo path), `device_map`, `use_bematts`, `audio_step`, `use_learnable_rope`
-- All config parameters are passed directly to `KaniTTS` constructor via unpacking
-
-**examples.yaml**
-- List of example prompts with their generation parameters
-- Each example requires: `text`, `model`, and optionally `temperature`, `top_p`, `repetition_penalty`
-- Missing parameters fall back to defaults in [util.py:90-92](util.py#L90-L92): temperature=1.0, top_p=0.95, repetition_penalty=1.1
-
-### Dependencies
-
-- **kani-tts==1.0.1**: Core TTS library from PyPI providing `KaniTTS` class with all inference logic
-- **gradio>=4.0.0**: UI framework
-- **transformers**: Installed from GitHub main branch via [create_env.py](create_env.py) for latest features
-
-### Environment Variables
-
-- `OMP_NUM_THREADS=4`: Set in [create_env.py:6](create_env.py#L6) to limit OpenMP threading
-
-## Important Implementation Notes
-
-### Model Inference
-- The KaniTTS model is called directly as a callable: `audio, text = model(text, temperature=..., top_p=..., repetition_penalty=...)`
-- Returns tuple of `(audio, text)` but only audio is used in the UI
-- No `max_tokens` parameter - the model handles sequence length internally
-- Sample rate is hardcoded to 22050 Hz ([app.py:48](app.py#L48))
-
-### Example Caching
-- Examples use `cache_examples=True` ([app.py:131](app.py#L131)) to pre-generate audio, speeding up demo interactions
-- If you modify generation logic, cached examples may need regeneration
-
-### GPU Allocation
-- The `@spaces.GPU` decorator is critical for HuggingFace Spaces deployment
-- Without it, the app runs on CPU which is significantly slower
-- Device selection falls back gracefully: `"cuda" if torch.cuda.is_available() else "cpu"` ([app.py:35](app.py#L35))
-
-### Input Order Dependency
-- The order of inputs in `gr.Examples` ([app.py:128](app.py#L128)) must exactly match the order in `Examples.__call__` ([util.py:94](util.py#L94))
-- Current order: `[text, model_dropdown, temp, top_p, rp]`
-- Changing this requires updates in both locations
-
-### Removed Features
-- **Speaker selection**: No longer supported in the new API
-- **Time reporting**: Generation timing is not tracked
-- **Max tokens slider**: Sequence length is handled automatically by the model

app.py

@@ -4,10 +4,9 @@ setup_dependencies()
 
 import spaces
 import gradio as gr
-from util import InitModels, load_config, Examples
+from util import InitModels, load_config, Examples, SpeakerManager
 import numpy as np
 import torch
-import json
 
 config = load_config("./model_config.yaml")
 models_configs = config.models
@@ -19,13 +18,30 @@ examples = examples_maker()
 init_models = InitModels(models_configs)
 models = init_models()
 
-# Load speaker map
-with open("./speakers/speaker_map.json", "r") as f:
-    speaker_map = json.load(f)
+# Initialize speaker manager
+speaker_manager = SpeakerManager()
 
 
 @spaces.GPU
-def generate_speech_gpu(text, model_choice, speaker_choice, t, top_p, rp):
+def generate_embedding_gpu(audio_data):
+    """
+    Generate speaker embedding from audio on GPU
+    """
+    try:
+        if audio_data is None:
+            return "No audio provided"
+
+        embedding = speaker_manager.generate_embedding(audio_data)
+        print("Embedding generated successfully!")
+        return speaker_manager.get_status()
+
+    except Exception as e:
+        print(f"Error generating embedding: {str(e)}")
+        return f"Error: {str(e)}"
+
+
+@spaces.GPU
+def generate_speech_gpu(text, model_choice, mode, speaker_choice, t, top_p, rp):
     """
     Generate speech from text using the selected model on GPU
     """
@@ -42,8 +58,8 @@ def generate_speech_gpu(text, model_choice, speaker_choice, t, top_p, rp):
 
         selected_model = models[model_choice]
 
-        # Get speaker embedding path
-        speaker_emb = speaker_map.get(speaker_choice) if speaker_choice else None
+        # Get speaker embedding based on mode
+        speaker_emb = speaker_manager.get_speaker_emb(mode, speaker_choice)
 
         print(f"Generating speech with {model_choice}...")
         audio, _ = selected_model(
@@ -76,12 +92,39 @@ with gr.Blocks(title="😻 KaniTTS - Text to Speech", theme=gr.themes.Ocean()) a
                 label="Selected Model"
             )
 
+            # Speaker mode selector
+            speaker_mode = gr.Radio(
+                choices=["select", "generate"],
+                value="select",
+                label="Speaker Mode"
+            )
+
+            # Speaker selection (visible in "select" mode)
             speaker_dropdown = gr.Dropdown(
-                choices=list(speaker_map.keys()),
-                value=list(speaker_map.keys())[0],
-                label="Speaker"
+                choices=speaker_manager.get_speaker_names(),
+                value=speaker_manager.get_speaker_names()[0] if speaker_manager.get_speaker_names() else None,
+                label="Speaker",
+                visible=True
             )
 
+            # Audio upload and embedding generation (visible in "generate" mode)
+            with gr.Group(visible=False) as embedding_group:
+                audio_input = gr.Audio(
+                    label="Upload or Record Audio (16kHz recommended)",
+                    type="numpy",
+                    sources=["upload", "microphone"]
+                )
+
+                with gr.Row():
+                    run_embedding_btn = gr.Button("Run Embedding", variant="secondary")
+                    clean_embedding_btn = gr.Button("Clean", variant="stop")
+
+                embedding_status = gr.Textbox(
+                    label="Embedding Status",
+                    value="No embedding generated",
+                    interactive=False
+                )
+
             text_input = gr.Textbox(
                 label="Text",
                 placeholder="Enter your text ...",
@@ -112,10 +155,36 @@ with gr.Blocks(title="😻 KaniTTS - Text to Speech", theme=gr.themes.Ocean()) a
                 type="numpy"
             )
     
+    # Toggle visibility based on speaker mode
+    def toggle_speaker_mode(mode):
+        if mode == "select":
+            return gr.update(visible=True), gr.update(visible=False)
+        else:  # generate
+            return gr.update(visible=False), gr.update(visible=True)
+
+    speaker_mode.change(
+        fn=toggle_speaker_mode,
+        inputs=[speaker_mode],
+        outputs=[speaker_dropdown, embedding_group]
+    )
+
+    # Embedding generation events
+    run_embedding_btn.click(
+        fn=generate_embedding_gpu,
+        inputs=[audio_input],
+        outputs=[embedding_status]
+    )
+
+    clean_embedding_btn.click(
+        fn=speaker_manager.clean,
+        inputs=[],
+        outputs=[embedding_status]
+    )
+
     # GPU generation event
     generate_btn.click(
         fn=generate_speech_gpu,
-        inputs=[text_input, model_dropdown, speaker_dropdown, temp, top_p, rp],
+        inputs=[text_input, model_dropdown, speaker_mode, speaker_dropdown, temp, top_p, rp],
         outputs=[audio_output]
     )
 
@@ -125,7 +194,7 @@ with gr.Blocks(title="😻 KaniTTS - Text to Speech", theme=gr.themes.Ocean()) a
 
         gr.Examples(
             examples=examples,
-            inputs=[text_input, model_dropdown, speaker_dropdown, temp, top_p, rp],
+            inputs=[text_input, model_dropdown, speaker_mode, speaker_dropdown, temp, top_p, rp],
             fn=generate_speech_gpu,
             outputs=[audio_output],
             cache_examples=True,

examples.yaml

@@ -16,7 +16,7 @@ examples:
     repetition_penalty: 1.1
 
   - text: >-
-      Have some wine, the March Hare said in an encouraging tone. Alice looked all round the table, but there was nothing on it but tea. I don't see any wine, she remarked. There isn't any, said the March Hare. Then it wasn't very civil of you to offer it, said Alice angrily. It wasn't very civil of you to sit down without being invited, said the March Hare. I didn't know it was YOUR table, said Alice; it's laid for a great many more than three. Your hair wants cutting, said the Hatter.  He had been looking at Alice for some time with great curiosity, and this was his first speech. You should learn not to make personal remarks, Alice said with some severity; it's very rude.
+      Have some wine, the March Hare said in an encouraging tone. Alice looked all round the table, but there was nothing on it but tea. I don't see any wine, she remarked. There isn't any, said the March Hare. Then it wasn't very civil of you to offer it, said Alice angrily. It wasn't very civil of you to sit down without being invited, said the March Hare. I didn't know it was YOUR table, said Alice; it's laid for a great many more than three. Your hair wants cutting, said the Hatter.
     model: "Exp-1"
     speaker: "Andrew (en)"
     temperature: 1

make_speaker.py

@@ -1,934 +0,0 @@
-import torch
-
-
-sp1 = [
-  -0.056959059089422226,
-  -0.07431771606206894,
-  -0.0007744207978248596,
-  0.14234879612922668,
-  -0.07791736721992493,
-  0.044718775898218155,
-  -0.0025952402502298355,
-  -0.07470317929983139,
-  -0.07941192388534546,
-  -0.06825936585664749,
-  -0.07658017426729202,
-  0.10334887355566025,
-  0.1488780379295349,
-  0.05303303524851799,
-  0.03563545644283295,
-  0.006154931616038084,
-  -0.019980307668447495,
-  0.006242025177925825,
-  -0.07650502771139145,
-  0.011423020623624325,
-  -0.0828247144818306,
-  0.09208501130342484,
-  0.11241830140352249,
-  0.18787875771522522,
-  -0.05311708152294159,
-  0.00862691830843687,
-  -0.07777128368616104,
-  0.21713057160377502,
-  0.06223568692803383,
-  -0.07252423465251923,
-  -0.08889792859554291,
-  -0.07656295597553253,
-  -0.005845929961651564,
-  0.037021245807409286,
-  0.0019470887491479516,
-  0.03835201635956764,
-  0.09467420727014542,
-  -0.09536144882440567,
-  0.1606171429157257,
-  0.04815187677741051,
-  -0.0461551658809185,
-  -0.10729880630970001,
-  -0.0836249515414238,
-  -0.014936363324522972,
-  -0.07798093557357788,
-  -0.04525156319141388,
-  -0.02704687789082527,
-  0.031187091022729874,
-  -0.08094345778226852,
-  0.03038698248565197,
-  -0.07573038339614868,
-  -0.04615750536322594,
-  0.03339000046253204,
-  0.012611067853868008,
-  -0.040278010070323944,
-  -0.05864224582910538,
-  -0.07855084538459778,
-  -0.01116874534636736,
-  -0.07860837131738663,
-  0.022938184440135956,
-  -0.0490887314081192,
-  0.012859633192420006,
-  0.05778036639094353,
-  -0.07337072491645813,
-  0.033186428248882294,
-  0.038420047610998154,
-  -0.051903847604990005,
-  0.18860727548599243,
-  -0.08550824970006943,
-  0.09781505167484283,
-  -0.050179820507764816,
-  -0.09151201695203781,
-  0.06224966049194336,
-  -0.07560431957244873,
-  0.03041389212012291,
-  -0.08223845064640045,
-  -0.07560642808675766,
-  -0.07574097812175751,
-  -0.07378879189491272,
-  0.09276977181434631,
-  -0.0721750557422638,
-  -0.019931843504309654,
-  -0.016255822032690048,
-  -0.08486099541187286,
-  0.015946706756949425,
-  0.219291090965271,
-  -0.003677315777167678,
-  0.03579730913043022,
-  -0.06582938879728317,
-  -0.07598833739757538,
-  -0.0017783945659175515,
-  -0.07621932774782181,
-  0.14186978340148926,
-  -0.08100845664739609,
-  -0.016534771770238876,
-  -0.039146170020103455,
-  -0.08936012536287308,
-  -0.0884183868765831,
-  0.0818035677075386,
-  -0.08333006501197815,
-  -0.08697599172592163,
-  0.08900929987430573,
-  0.1540442854166031,
-  0.06929890811443329,
-  0.023458357900381088,
-  0.11819791793823242,
-  0.05528293922543526,
-  -0.07724623382091522,
-  -0.08985307067632675,
-  -0.07698521018028259,
-  -0.08518560975790024,
-  -0.03854784741997719,
-  0.05612029507756233,
-  0.19019906222820282,
-  -0.0532655343413353,
-  0.10331130772829056,
-  0.08962190896272659,
-  0.07186102122068405,
-  -0.08073300868272781,
-  -0.03013928048312664,
-  -0.017707977443933487,
-  0.061603646725416183,
-  0.08824963122606277,
-  -0.040111809968948364,
-  -0.04101881384849548,
-  -0.023703360930085182,
-  0.16742625832557678,
-  -0.08780555427074432
-]
-
-sp2 = [
-  -0.050281789153814316,
-  0.023217787966132164,
-  0.043346893042325974,
-  -0.043094731867313385,
-  -0.04279769957065582,
-  0.019955242052674294,
-  -0.07015533745288849,
-  -0.0764705017209053,
-  0.052317824214696884,
-  -0.08275781571865082,
-  0.1702217310667038,
-  -0.0029036293271929026,
-  0.16897068917751312,
-  -0.033592864871025085,
-  0.17897063493728638,
-  -0.07369279116392136,
-  -0.08292099088430405,
-  -0.07833810150623322,
-  -0.07124188542366028,
-  0.09324893355369568,
-  -0.08365192264318466,
-  -0.039345383644104004,
-  0.13911734521389008,
-  -0.021355967968702316,
-  -0.06709616631269455,
-  -0.0733819454908371,
-  0.00957417581230402,
-  -0.07639335095882416,
-  -0.042981404811143875,
-  -0.062470581382513046,
-  -0.08963305503129959,
-  0.052217237651348114,
-  0.009040125645697117,
-  -0.07530245929956436,
-  0.07278499752283096,
-  -0.09641268104314804,
-  0.11474557220935822,
-  -0.09885910898447037,
-  0.061111241579055786,
-  -0.08349886536598206,
-  -0.03521011769771576,
-  -0.1121467873454094,
-  -0.06835179030895233,
-  -0.04818527400493622,
-  -0.08024188131093979,
-  0.09246063232421875,
-  -0.0786546915769577,
-  -0.0678088366985321,
-  0.12170662730932236,
-  0.006078201346099377,
-  -0.07792742550373077,
-  -0.023563995957374573,
-  -0.07990031689405441,
-  -0.08500851690769196,
-  0.017435764893889427,
-  -0.008875912986695766,
-  -0.01564612425863743,
-  0.12621375918388367,
-  -0.05536468327045441,
-  -0.08361390978097916,
-  0.15475153923034668,
-  -0.0753680020570755,
-  0.025031298398971558,
-  -0.07504384219646454,
-  0.11817727982997894,
-  0.22630754113197327,
-  0.1351470947265625,
-  0.14117325842380524,
-  -0.07466709613800049,
-  -0.017258374020457268,
-  -0.058375779539346695,
-  -0.09002719074487686,
-  0.1529882848262787,
-  0.054032210260629654,
-  -0.07923610508441925,
-  0.04768155887722969,
-  0.013129397295415401,
-  -0.07510846108198166,
-  -0.07405361533164978,
-  -0.06735733896493912,
-  0.07549432665109634,
-  -0.07524596899747849,
-  0.023765340447425842,
-  -0.04690846800804138,
-  -0.0456087701022625,
-  0.013412008062005043,
-  -0.07029068470001221,
-  -0.09484000504016876,
-  0.008935618214309216,
-  0.14847226440906525,
-  0.10177381336688995,
-  -0.0767863541841507,
-  0.20601123571395874,
-  -0.07636284828186035,
-  -0.07587302476167679,
-  -0.0172903873026371,
-  0.031998757272958755,
-  -0.024688469246029854,
-  -0.08434971421957016,
-  -0.08759179711341858,
-  0.02792895957827568,
-  0.0656527578830719,
-  0.05782775953412056,
-  0.09333281219005585,
-  -0.06884845346212387,
-  -0.07939416170120239,
-  0.15371105074882507,
-  -0.026411522179841995,
-  0.07004162669181824,
-  -0.07803111523389816,
-  -0.0015616194577887654,
-  0.10434804111719131,
-  -0.008118604309856892,
-  0.06711513549089432,
-  -0.035488326102495193,
-  -0.066922128200531,
-  -0.06753159314393997,
-  -0.08739693462848663,
-  -0.06943987309932709,
-  -0.0800175592303276,
-  -0.07267877459526062,
-  -0.0846465677022934,
-  0.05650201067328453,
-  -0.0858917236328125,
-  0.11707165837287903,
-  -0.07632513344287872,
-  -0.07149224728345871,
-  0.0010398230515420437
-]
-
-sp3 = [
-  0.044863346964120865,
-  -0.08795837312936783,
-  0.00623549846932292,
-  -0.041037190705537796,
-  0.15892581641674042,
-  0.06204115226864815,
-  -0.07757825404405594,
-  0.12236713618040085,
-  0.004264748655259609,
-  -0.09987714886665344,
-  0.05126308649778366,
-  0.037995338439941406,
-  -0.08919689059257507,
-  0.04392874240875244,
-  -0.09021709114313126,
-  -0.0738280862569809,
-  0.02076968550682068,
-  0.1070074662566185,
-  0.049832168966531754,
-  0.17484578490257263,
-  -0.013567082583904266,
-  0.10016804933547974,
-  -0.08356668800115585,
-  0.0636911690235138,
-  -0.009811697527766228,
-  -0.07636874169111252,
-  -0.06126904860138893,
-  -0.053117018193006516,
-  0.01112096942961216,
-  0.011951063759624958,
-  0.11347390711307526,
-  0.07987657189369202,
-  0.09502555429935455,
-  0.045208342373371124,
-  -0.047236792743206024,
-  -0.07695288211107254,
-  -0.00397157296538353,
-  0.2076280117034912,
-  -0.08805114775896072,
-  -0.06516429036855698,
-  0.0346195288002491,
-  -0.007755806669592857,
-  0.12718522548675537,
-  -0.04961549863219261,
-  -0.08501579612493515,
-  -0.08136767894029617,
-  -0.015555771067738533,
-  -0.0896003246307373,
-  0.008332539349794388,
-  -0.08981787413358688,
-  0.12871037423610687,
-  -0.08342160284519196,
-  -0.06250006705522537,
-  -0.0906406119465828,
-  0.0739695206284523,
-  -0.07614704221487045,
-  0.0028914636932313442,
-  -0.06738509982824326,
-  -0.0908239558339119,
-  -0.004341233056038618,
-  -0.08561462163925171,
-  -0.06059376150369644,
-  -0.08309012651443481,
-  0.02291426807641983,
-  -0.08814181387424469,
-  -0.07243850082159042,
-  -0.10541810095310211,
-  -0.006142698228359222,
-  0.03829303756356239,
-  -0.08143717795610428,
-  -0.052729811519384384,
-  -0.08436775207519531,
-  -0.0825762078166008,
-  -0.023270217701792717,
-  -0.09676963090896606,
-  0.18764223158359528,
-  -0.0035655321553349495,
-  0.060881830751895905,
-  0.019288599491119385,
-  -0.0646059513092041,
-  0.2043074071407318,
-  -0.052328258752822876,
-  -0.08275464177131653,
-  0.006811313796788454,
-  -0.09141694009304047,
-  -0.0643858015537262,
-  -0.04983310401439667,
-  0.022007863968610764,
-  0.0523625947535038,
-  0.12141022831201553,
-  -0.09637351334095001,
-  0.09305799007415771,
-  -0.08631274104118347,
-  -0.08203761279582977,
-  0.01890491507947445,
-  -0.09548700600862503,
-  0.06787174195051193,
-  0.0185020100325346,
-  0.13169577717781067,
-  -0.07152362912893295,
-  0.08089947700500488,
-  0.0007502693333663046,
-  -0.04203244671225548,
-  -0.09299170970916748,
-  -0.06988460570573807,
-  -0.07600069046020508,
-  0.016998400911688805,
-  0.11065729707479477,
-  0.1770820915699005,
-  -0.07554581016302109,
-  0.14596983790397644,
-  -0.06950408965349197,
-  -0.08761636167764664,
-  0.05795743688941002,
-  -0.09482874721288681,
-  -0.07928589731454849,
-  -0.07619829475879669,
-  0.026845542713999748,
-  -0.08625771850347519,
-  -0.08191405981779099,
-  -0.033759284764528275,
-  -0.09005032479763031,
-  -0.09133398532867432,
-  0.13270556926727295,
-  -0.03323248401284218,
-  -0.06472556293010712,
-  -0.07661902159452438,
-  -0.0922679528594017
-]
-
-
-sp4 = [
-  0.014975552447140217,
-  0.1201590821146965,
-  -0.04833684489130974,
-  -0.042657576501369476,
-  -0.07680147886276245,
-  0.008444481529295444,
-  0.04672827199101448,
-  0.0655755028128624,
-  0.11940808594226837,
-  -0.07818085700273514,
-  -0.07619981467723846,
-  -0.034715279936790466,
-  -0.08239906281232834,
-  0.02117394655942917,
-  0.026922911405563354,
-  -0.02619064413011074,
-  -0.08279731124639511,
-  -0.08409284800291061,
-  -0.07563990354537964,
-  -0.07746758311986923,
-  0.017148327082395554,
-  0.07626708596944809,
-  0.07004616409540176,
-  0.03208186477422714,
-  -0.03150057792663574,
-  0.11792197823524475,
-  -0.07686424255371094,
-  0.004818132147192955,
-  -0.09032411873340607,
-  -0.08447712659835815,
-  0.020521923899650574,
-  -0.058463796973228455,
-  -0.07045496255159378,
-  0.0969952866435051,
-  -0.0420924574136734,
-  -0.02713640220463276,
-  -0.08808104693889618,
-  0.0425461009144783,
-  0.04754403978586197,
-  -0.0019016547594219446,
-  -0.04388159513473511,
-  0.025083905085921288,
-  -0.08270448446273804,
-  -0.0770757868885994,
-  0.09254211187362671,
-  0.10774955153465271,
-  -0.06317340582609177,
-  -0.07892616838216782,
-  -0.08005274832248688,
-  0.04106053337454796,
-  0.0981193482875824,
-  0.032208967953920364,
-  0.032562531530857086,
-  -0.040666233748197556,
-  -0.09163665026426315,
-  0.14852704107761383,
-  -0.07776651531457901,
-  0.09260601550340652,
-  -0.08714006841182709,
-  0.02349991351366043,
-  0.09777960926294327,
-  0.012837720103561878,
-  -0.0681685283780098,
-  0.09760434180498123,
-  0.11893817782402039,
-  -0.06687002629041672,
-  0.10464373975992203,
-  -0.07898904383182526,
-  0.03718216344714165,
-  -0.001790654263459146,
-  0.1210879310965538,
-  0.09112358093261719,
-  0.07040399312973022,
-  -0.021451229229569435,
-  -0.016307493671774864,
-  0.11010289192199707,
-  0.19557251036167145,
-  -0.06409434229135513,
-  -0.07242149859666824,
-  0.0815720334649086,
-  -0.0742398202419281,
-  0.15758396685123444,
-  0.0034991512075066566,
-  -0.08164040744304657,
-  0.09242386370897293,
-  -0.07533959299325943,
-  0.12539155781269073,
-  0.034026630222797394,
-  -0.07615572959184647,
-  -0.07524855434894562,
-  -0.0046832324005663395,
-  0.02044426091015339,
-  -0.07627419382333755,
-  0.0906405821442604,
-  0.15664952993392944,
-  0.014610418118536472,
-  -0.06705746054649353,
-  0.12825961410999298,
-  0.06035701185464859,
-  0.17567360401153564,
-  -0.0796094685792923,
-  -0.07080274820327759,
-  -0.08287626504898071,
-  -0.06109911575913429,
-  -0.022632954642176628,
-  -0.07906034588813782,
-  -0.0585855208337307,
-  -0.07637544721364975,
-  -0.08893950283527374,
-  0.08905342221260071,
-  0.06906967610120773,
-  0.07786651700735092,
-  -0.03081165812909603,
-  -0.10391682386398315,
-  -0.009698718786239624,
-  0.09618605673313141,
-  0.1353370100259781,
-  -0.009754122234880924,
-  -0.0798230841755867,
-  0.14158795773983002,
-  0.07175704091787338,
-  0.009440293535590172,
-  -0.08403725177049637,
-  -0.010469981469213963,
-  -0.09179911017417908,
-  0.012903331778943539,
-  0.03833264485001564,
-  0.1914415955543518
-]
-
-sp5 = [
-  -0.07472408562898636,
-  0.2077360600233078,
-  -0.057473067194223404,
-  -0.07846137136220932,
-  0.0486067496240139,
-  -0.09742959588766098,
-  0.012818996794521809,
-  -0.04314102977514267,
-  -0.07742120325565338,
-  -0.012238459661602974,
-  0.16626425087451935,
-  -0.06014818325638771,
-  0.030748408287763596,
-  -0.07885394245386124,
-  0.08880891650915146,
-  0.04357529431581497,
-  -0.08909814804792404,
-  -0.08589012920856476,
-  0.017018966376781464,
-  -0.06919971853494644,
-  -0.0647362470626831,
-  -0.06408999860286713,
-  -0.06747046113014221,
-  -0.08739134669303894,
-  -0.06523431837558746,
-  -0.07173540443181992,
-  -0.07459395378828049,
-  -0.07758451998233795,
-  -0.09113436937332153,
-  0.02833479642868042,
-  0.010935867205262184,
-  -0.04095727205276489,
-  -0.008447790518403053,
-  -0.015916690230369568,
-  -0.06783552467823029,
-  0.16540202498435974,
-  0.08896030485630035,
-  -0.022869249805808067,
-  -0.08067937195301056,
-  -0.07047552615404129,
-  -0.0791078507900238,
-  -0.11707916855812073,
-  -0.08358949422836304,
-  -0.08547090739011765,
-  -0.0013615615898743272,
-  -0.08036443591117859,
-  -0.04231594502925873,
-  -0.0837881937623024,
-  -0.02109750732779503,
-  0.06521718949079514,
-  0.03358446806669235,
-  0.1431589126586914,
-  -0.027682557702064514,
-  0.03217494115233421,
-  -0.06871805340051651,
-  -0.06577359139919281,
-  0.10372865945100784,
-  -0.05901371315121651,
-  0.0447809062898159,
-  -0.07911419868469238,
-  0.2223190814256668,
-  0.033684857189655304,
-  -0.01980946958065033,
-  -0.009113002568483353,
-  0.061996620148420334,
-  -0.06760461628437042,
-  0.0793633908033371,
-  0.09014952927827835,
-  0.06929260492324829,
-  -0.01617756299674511,
-  0.07633700966835022,
-  0.016587991267442703,
-  -0.0069177825935184956,
-  -0.019375547766685486,
-  0.026005519554018974,
-  -0.0949050784111023,
-  0.13627009093761444,
-  0.0007139204535633326,
-  0.016387799754738808,
-  -0.07186658680438995,
-  -0.009011861868202686,
-  0.022241994738578796,
-  0.06350655108690262,
-  0.10589846968650818,
-  -0.09742557257413864,
-  -0.06443683803081512,
-  0.004297737963497639,
-  -0.09466835856437683,
-  0.04342241585254669,
-  0.08106304705142975,
-  0.12201986461877823,
-  0.04398150369524956,
-  -0.009540509432554245,
-  0.06304764747619629,
-  -0.08143053948879242,
-  0.1555684208869934,
-  -0.06324099004268646,
-  0.04688173905014992,
-  -0.08392588794231415,
-  0.19633686542510986,
-  0.08215086907148361,
-  -0.07445380836725235,
-  0.05648314580321312,
-  -0.07521350681781769,
-  -0.08106789737939835,
-  -0.0769851952791214,
-  -0.09946925938129425,
-  0.07573969662189484,
-  -0.008974740281701088,
-  -0.0393686518073082,
-  -0.0017469975864514709,
-  -0.08926048129796982,
-  -0.04622442275285721,
-  -0.09214682877063751,
-  0.005119341425597668,
-  0.10839592665433884,
-  -0.050333209335803986,
-  -0.09676054120063782,
-  0.047501906752586365,
-  -0.06837188452482224,
-  0.1552308201789856,
-  0.02244103141129017,
-  -0.08457397669553757,
-  -0.05681031569838524,
-  0.015423906035721302,
-  -0.08144135773181915,
-  -0.06842615455389023,
-  -0.03671957924962044
-]
-
-
-sp6 = [
-  -0.004930280148983002,
-  0.08562461286783218,
-  -0.0764380544424057,
-  0.12423505634069443,
-  0.06380770355463028,
-  -0.0833282396197319,
-  -0.0704629123210907,
-  -0.02359308861196041,
-  -0.11950968950986862,
-  -0.05383031815290451,
-  0.13100019097328186,
-  -0.04609416425228119,
-  0.11273174732923508,
-  -0.0883885845541954,
-  -0.016859225928783417,
-  0.03442854806780815,
-  -0.1029595360159874,
-  -0.07559429109096527,
-  0.12149295210838318,
-  0.014217427931725979,
-  -0.06553817540407181,
-  0.06483729183673859,
-  -0.0832972452044487,
-  -0.0895114466547966,
-  -0.07193689048290253,
-  -0.0770263597369194,
-  0.00022856975556351244,
-  0.011081762611865997,
-  -0.06022211164236069,
-  -0.07772520184516907,
-  -0.04475982487201691,
-  0.15953606367111206,
-  -0.08595475554466248,
-  0.07281705737113953,
-  -0.043590039014816284,
-  0.024075673893094063,
-  -0.03666220232844353,
-  0.05976352468132973,
-  -0.00404775096103549,
-  -0.09037335962057114,
-  -0.025358861312270164,
-  -0.08681948482990265,
-  -0.08855558931827545,
-  -0.0653468519449234,
-  0.05669263005256653,
-  -0.07210364937782288,
-  -0.05875953659415245,
-  -0.09035740792751312,
-  0.028884833678603172,
-  -0.07896765321493149,
-  0.02423865534365177,
-  0.13390737771987915,
-  -0.013620391488075256,
-  -0.02224227413535118,
-  0.015568692237138748,
-  0.11594505608081818,
-  0.11687905341386795,
-  -0.018582934513688087,
-  -0.09030970931053162,
-  0.005574168637394905,
-  -0.04361603781580925,
-  -0.03851548582315445,
-  -0.08371025323867798,
-  -0.03500206768512726,
-  -0.02459687739610672,
-  -0.07222030311822891,
-  0.08121480792760849,
-  0.03040383756160736,
-  -0.03100021556019783,
-  -0.07651577889919281,
-  0.03292612358927727,
-  0.026235831901431084,
-  -0.08650624006986618,
-  -0.01090069767087698,
-  0.03657924383878708,
-  -0.08384150266647339,
-  0.064586341381073,
-  0.09112914651632309,
-  0.07503412663936615,
-  -0.07587313652038574,
-  0.12387727946043015,
-  -0.020256295800209045,
-  0.040244076400995255,
-  -0.06343528628349304,
-  -0.04044422134757042,
-  0.02972465194761753,
-  0.04687220975756645,
-  -0.008625105954706669,
-  0.1008363887667656,
-  0.010900158435106277,
-  -0.09016355872154236,
-  0.04838952794671059,
-  -0.04105637967586517,
-  0.013639458455145359,
-  -0.07048259675502777,
-  -0.08235885202884674,
-  -0.08882340788841248,
-  -0.09107483923435211,
-  0.03668944537639618,
-  0.0019158608047291636,
-  -0.022357558831572533,
-  0.1465294361114502,
-  0.07629132270812988,
-  -0.06898661702871323,
-  -0.08680594712495804,
-  -0.08412013947963715,
-  -0.002412576461210847,
-  0.1506911963224411,
-  -0.009846117347478867,
-  -0.0802307203412056,
-  0.020010171458125114,
-  -0.06089577078819275,
-  -0.08643854409456253,
-  -0.1062886118888855,
-  -0.08187893778085709,
-  0.0315963439643383,
-  -0.07040993869304657,
-  -0.060491085052490234,
-  -0.030400332063436508,
-  0.13255496323108673,
-  -0.06075480952858925,
-  -0.08898318558931351,
-  -0.08366542309522629,
-  -0.006396912969648838,
-  0.09778483957052231,
-  0.05774940177798271,
-  -0.053698617964982986,
-  -0.07357174903154373
-]
-
-sp7 = [
-  0.09096527099609375,
-  0.12818661332130432,
-  0.05300923064351082,
-  -0.0620405450463295,
-  -0.07956791669130325,
-  -0.09087102115154266,
-  0.021690314635634422,
-  0.0144176771864295,
-  0.0019766734912991524,
-  0.007868418470025063,
-  -0.07859831303358078,
-  -0.060859933495521545,
-  -0.0775991678237915,
-  0.01997179351747036,
-  0.15887515246868134,
-  0.07313112914562225,
-  -0.09271703660488129,
-  -0.08671814203262329,
-  -0.07813264429569244,
-  -0.07720690965652466,
-  -0.07696140557527542,
-  0.18065419793128967,
-  0.10781082510948181,
-  -0.022600548341870308,
-  0.06215902790427208,
-  -0.02438243292272091,
-  -0.07941883057355881,
-  0.07533253729343414,
-  -0.09328891336917877,
-  -0.05382213741540909,
-  -0.06141037493944168,
-  -0.07790078967809677,
-  0.1648797243833542,
-  0.0423060841858387,
-  -0.07286743074655533,
-  0.04643263295292854,
-  -0.06554959714412689,
-  -0.032457154244184494,
-  0.09949684888124466,
-  0.025956032797694206,
-  0.03757084533572197,
-  0.025580935180187225,
-  -0.08542721718549728,
-  -0.002371697686612606,
-  -0.018300838768482208,
-  -0.09333072602748871,
-  0.09380193054676056,
-  -0.08460453152656555,
-  -0.08267955482006073,
-  0.011247556656599045,
-  0.06678486615419388,
-  0.06353003531694412,
-  -0.05372804403305054,
-  -0.04275272786617279,
-  -0.09475936740636826,
-  -0.02697761356830597,
-  -0.08031945675611496,
-  0.09705562144517899,
-  0.0034417700953781605,
-  0.026172643527388573,
-  0.15678609907627106,
-  0.15548571944236755,
-  0.08869600296020508,
-  0.0034438855946063995,
-  0.04129115864634514,
-  -0.06807658076286316,
-  -0.048452526330947876,
-  0.00348582467995584,
-  -0.005682673305273056,
-  0.05030336230993271,
-  0.024773158133029938,
-  0.08784151077270508,
-  0.08264995366334915,
-  0.07744842767715454,
-  0.006688673049211502,
-  0.11568285524845123,
-  -0.0006149086402729154,
-  -0.07731792330741882,
-  -0.07520095258951187,
-  0.050927311182022095,
-  -0.07624947279691696,
-  -0.017263416200876236,
-  -0.07306291908025742,
-  -0.04022892937064171,
-  -0.043212465941905975,
-  0.025481276214122772,
-  0.009810780175030231,
-  -0.03424441069364548,
-  -0.07623236626386642,
-  -0.07723627239465714,
-  0.14873334765434265,
-  -0.07702631503343582,
-  0.013342263177037239,
-  -0.07819023728370667,
-  -0.06493690609931946,
-  0.007031683810055256,
-  -0.11437463015317917,
-  -0.04450619965791702,
-  -0.0739501342177391,
-  0.03370510786771774,
-  0.006866330746561289,
-  0.04877154901623726,
-  0.09061554819345474,
-  -0.06745804101228714,
-  0.03955249860882759,
-  -0.08013571798801422,
-  -0.050169579684734344,
-  -0.07888628542423248,
-  -0.09185811877250671,
-  -0.008135518059134483,
-  -0.032329343259334564,
-  0.05627875775098801,
-  -0.034713007509708405,
-  0.03831075131893158,
-  0.051426082849502563,
-  0.12307145446538925,
-  0.13895918428897858,
-  -0.055210549384355545,
-  -0.0824555903673172,
-  -0.046466924250125885,
-  0.138591930270195,
-  -0.06990546733140945,
-  -0.07223182171583176,
-  -0.026878757402300835,
-  0.12467526644468307,
-  0.06654545664787292,
-  0.04314073547720909,
-  0.0024285614490509033
-]
-
-
-torch.save(torch.tensor(sp1), './speakers/speaker_1.pt') # Kore (en)
-torch.save(torch.tensor(sp2), './speakers/speaker_2.pt') # Puck (en)
-torch.save(torch.tensor(sp3), './speakers/speaker_3.pt') # Andrew (en)
-
-torch.save(torch.tensor(sp4), './speakers/speaker_4.pt') # Aisulu (ky)
-torch.save(torch.tensor(sp5), './speakers/speaker_5.pt') # Baike (ky)
-
-
-torch.save(torch.tensor(sp6), './speakers/speaker_6.pt') # Ash (es)
-torch.save(torch.tensor(sp7), './speakers/speaker_7.pt') # Nova (es)
-

util.py

@@ -2,6 +2,8 @@ from kani_tts import KaniTTS
 from kani_tts import SpeakerEmbedder
 
 import os
+import json
+import torch
 from omegaconf import OmegaConf
 
 
@@ -62,6 +64,124 @@ class InitModels:
         print("All models loaded!")
         return models
 
+class SpeakerManager:
+    """
+    Manages speaker embeddings for the TTS application.
+
+    Supports two modes:
+    1. Select speaker: Load pre-saved speaker embeddings from speaker_map.json
+    2. Generate embedding: Generate speaker embedding from uploaded audio using SpeakerEmbedder
+
+    Parameters
+    ----------
+    speaker_map_path : str
+        Path to speaker_map.json file
+
+    Methods
+    -------
+    get_speaker_emb(mode, speaker_name=None) -> str | torch.Tensor | None
+        Returns speaker embedding based on mode:
+        - "select": Returns path to .pt file from speaker_map
+        - "generate": Returns cached generated embedding tensor or None
+
+    generate_embedding(audio_data, sample_rate) -> torch.Tensor
+        Generates speaker embedding from audio using SpeakerEmbedder.
+        Expects audio at 16kHz. Caches the result internally.
+
+    clean()
+        Clears cached generated embedding.
+
+    get_speaker_names() -> list[str]
+        Returns list of available speaker names from speaker_map.json.
+    """
+
+    def __init__(self, speaker_map_path: str = "./speakers/speaker_map.json"):
+        self.speaker_map_path = speaker_map_path
+        self.speaker_map = self._load_speaker_map()
+        self.cached_embedding = None
+        self.embedder = None
+
+    def _load_speaker_map(self):
+        """Load speaker map from JSON file."""
+        if not os.path.exists(self.speaker_map_path):
+            return {}
+        with open(self.speaker_map_path, "r") as f:
+            return json.load(f)
+
+    def get_speaker_names(self):
+        """Get list of available speaker names."""
+        return list(self.speaker_map.keys())
+
+    def get_speaker_emb(self, mode: str, speaker_name: str = None):
+        """
+        Get speaker embedding based on mode.
+
+        Parameters
+        ----------
+        mode : str
+            Either "select" or "generate"
+        speaker_name : str, optional
+            Name of speaker from speaker_map (only used in "select" mode)
+
+        Returns
+        -------
+        str | torch.Tensor | None
+            Path to .pt file (select mode) or embedding tensor (generate mode)
+        """
+        if mode == "select":
+            if speaker_name and speaker_name in self.speaker_map:
+                return self.speaker_map[speaker_name]
+            return None
+        elif mode == "generate":
+            return self.cached_embedding
+        return None
+
+    def generate_embedding(self, audio_data, sample_rate: int = 16000):
+        """
+        Generate speaker embedding from audio data.
+
+        Parameters
+        ----------
+        audio_data : tuple | np.ndarray
+            Either (sample_rate, audio_array) tuple from Gradio or numpy array
+        sample_rate : int
+            Sample rate of the audio (default: 16000)
+
+        Returns
+        -------
+        torch.Tensor
+            Generated speaker embedding [1, 128]
+        """
+        # Initialize embedder lazily
+        if self.embedder is None:
+            self.embedder = SpeakerEmbedder()
+
+        # Handle Gradio audio format (sr, audio) tuple
+        if isinstance(audio_data, tuple):
+            sample_rate, audio_array = audio_data
+        else:
+            audio_array = audio_data
+
+        # Generate embedding
+        embedding = self.embedder.embed_audio(audio_array, sample_rate=sample_rate)
+
+        # Cache the result
+        self.cached_embedding = embedding
+
+        return embedding
+
+    def clean(self):
+        """Clear cached generated embedding."""
+        self.cached_embedding = None
+        return "Embedding cleared"
+
+    def get_status(self):
+        """Get current status of generated embedding."""
+        if self.cached_embedding is not None:
+            return "✅ Embedding ready"
+        return "No embedding generated"
+
+
 class Examples:
 
     """
@@ -77,7 +197,7 @@ class Examples:
     --------
     - Produces a list-of-lists whose order must match the `inputs` order
         used when constructing `gr.Examples` in `app.py`.
-    - Current order: `[text, model_dropdown, speaker_dropdown, temp, top_p, rp]`.
+    - Current order: `[text, model_dropdown, speaker_mode, speaker_dropdown, temp, top_p, rp]`.
 
     Why this exists
     ---------------
@@ -93,12 +213,13 @@ class Examples:
         for e in self.exam_cfg.examples:
             text                = e.get("text")
             model               = e.get("model")
+            speaker_mode        = e.get("speaker_mode", "select")  # Default to "select" mode
             speaker             = e.get("speaker", "Kore (en)")
             temperature         = e.get("temperature", 1.0)
             top_p               = e.get("top_p", 0.95)
             repetition_penalty  = e.get("repetition_penalty", 1.1)
-            # Order must match gr.Examples inputs: [text, model_dropdown, speaker_dropdown, temp, top_p, rp]
-            rows.append([text, model, speaker, temperature, top_p, repetition_penalty])
+            # Order must match gr.Examples inputs: [text, model_dropdown, speaker_mode, speaker_dropdown, temp, top_p, rp]
+            rows.append([text, model, speaker_mode, speaker, temperature, top_p, repetition_penalty])
 
         return rows