raincandy-u/Llama-3-Aplite-Instruct-4x8B-MoE · did this have degenerate routing?

did this have degenerate routing?

by Kquant03 - opened Apr 22

Apr 22

I purposely gave Azathoth degenerate parameters through making every prompt exactly the same. Wanted to see how badly it would affect it. Didn't hurt it too bad but I'm wondering if your prompts being somewhat similar gave it any degenerate routing.

raincandy-u

Owner Apr 23

My merge config😿

base_model: Meta-Llama-3-8B-Instruct
experts:
  - source_model: Meta-Llama-3-8B-Instruct
    positive_prompts:
    - "explain"
    - "chat"
    - "assistant"
  - source_model: Llama3-8B-OpenHermes-DPO
    positive_prompts:
    - "python"
    - "math"
    - "solve"
    - "code"
  - source_model: Llama-3-SLERP-8B
    positive_prompts:
    - "chat"
    - "assistant"
    - "AI"
  - source_model: hf-llama3-8b-orpo-v0.0
    positive_prompts:
    - "think"
    - "chat"
    - "code"
    - "roleplay"
gate_mode: hidden
dtype: float16

Kquant03

Apr 23

Yeah but like...when you ran the merge, in your terminal it will tell you which layers (if any) are degenerate

raincandy-u

Owner Apr 23

Well... It seems like I did not save it sorry...🤕

raincandy-u changed discussion status to closed Apr 23

Kquant03

Apr 24

it's okay lol I was just wondering

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment