Keep Responding in the wrong language despite the prompt template instructing to reply in a specific language

#132
by tdecae - opened

Hi, the title says it all, we keep getting answers back in the wrong language (English when it should be Spanish). Could this be due to the fact that some of the documents that were embedded and stored were in English as well as Spanish? and if so, do we need a separate pipeline (one for english, with only english docs, and another for spanish with only spanish docs..). We could run langdetect to divert incoming messages but this seems like a lot of bother. Is there another way to fix this?
Thanks

I have the same problem with French in DSPY, I rewrote the dspy.Anyscale this way but with no very good results:
import os
import random
import re
import shutil
import subprocess
from typing import Literal

from dsp.modules.adapter import TurboAdapter, DavinciAdapter, LlamaAdapter

import backoff
import requests

class Anyscale(dspy.HFModel):
def init(self, model, **kwargs):
super().init(model=model, is_client=True)
self.session = requests.Session()
self.api_base = os.getenv("ANYSCALE_API_BASE")
self.token = os.getenv("ANYSCALE_API_KEY")
self.model = model
self.kwargs = {
"temperature": 0.0,
"n": 1,
**kwargs,
}

def _generate(self, prompt, use_chat_api=True, **kwargs):
    url = f"{self.api_base}/completions"
    
    kwargs = {**self.kwargs, **kwargs}

    temperature = kwargs.get("temperature")
    max_tokens = kwargs.get("max_tokens", 150) 
    system_prompt = kwargs.get("system_prompt","Vous êtes un robot de discussion générale. Vous vous exprimez uniquement en français. Vos réponses sont concises, elles ne dépassent pas 50 mots, mais restent informatives.")

    if use_chat_api:
        url = f"{self.api_base}/chat/completions"
        #print(f"**** {url} {system_prompt}")
        messages = [
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": prompt},
        ]
        body = {
            "model": self.model,
            "messages": messages,
            "temperature": temperature,
            "max_tokens": max_tokens,
        }
    else:
        body = {
            "model": self.model,
            "prompt": f"[INST]{system_prompt}[/INST]{prompt}",
            "temperature": temperature,
            "max_tokens": max_tokens,
        }

    headers = {"Authorization": f"Bearer {self.token}"}

    try:
        completions = []
        for i in range(kwargs.get('n', 1)):
            with self.session.post(url, headers=headers, json=body) as resp:
                resp_json = resp.json()
                if use_chat_api:
                    completions.extend([resp_json.get('choices', [])[0].get('message', {}).get('content', "")])
                else:
                    completions.extend([resp_json.get('choices', [])[0].get('text', "")])
        response = {"prompt": prompt, "choices": [{"text": c} for c in completions]}
        return response
    except Exception as e:
        print(f"Failed to parse JSON response: {e}")
        raise Exception("Received invalid JSON response from server")

me too problem with French. On some other project I worked on, it transpired the embedding was causing the problem as opposed the LLM itself.

My bad, I mixed DSPy and more specific problem of Mistral, but the given class answers the question and the results are usually correct as long as I don't use the DSPy compiler, for example:

model = Anyscale(model='mistralai/Mixtral-8x7B-Instruct-v0.1',max_tokens=1024)
model( "Qui sont Adam et Ève ?") # How are Adam and Eve

[" Adam et Ève sont les premiers êtres humains mentionnés dans la Genèse, le premier livre de la Bible. Ils sont considérés comme les ancêtres de toute l'humanité."] # Adam and Eve are etc.
But when I use this instance inside a DSPy compiler results come in English.

Using the OpenAI interface, it works too :

import openai
client = openai.OpenAI(
base_url = "https://api.endpoints.anyscale.com/v1",
api_key = "esecret_*******"
)

Note: not all arguments are currently supported and will be ignored by the backend.

chat_completion = client.chat.completions.create(
model="mistralai/Mixtral-8x7B-Instruct-v0.1",
messages=[{"role": "system", "content": "Vous êtes un assistant efficace, vos réponses sont courtes et précises"},
{"role": "user", "content": "Qui était d'Alembert ?"}],
temperature=0.7
)
print(chat_completion.choices[0].message.content)

I get the good answer :

Jean le Rond d'Alembert (1717-1783) était un philosophe, mathématicien et encyclopédiste français, célèbre pour ses contributions en mécanique et en équations différentielles. Il a travaillé en étroite collaboration avec Denis Diderot sur l'Encyclopédie, servant de co-rédacteur en chef et rédigeant de nombreux articles.

Mistral AI_ org

Hi there, if you ever have issues to get a model adhere to a language you want, try playing around with prefixes, basically appending something to the final string before its sent for completion.

The idea is something like this: [INST] Answer always in spanish to the user.\n\nHi there! [/INST] Aquí está la respuesta en español:

This might be able to help make it more consistent and adhere to the language better. <3

Thanks will try

Sign up or log in to comment