About Ask2Democracy project

This model was trained during the 2023 Somos NLP Hackathon and it's part of the ask2democracy project. Our focus during the hackathon was on enhancing Retretrieval Augmented Generation (RAG) capabilities in spanish, using an open source model adapted for public debate discussions. This generative model is intended to be integrated with the retrieval system exposed in the project demo (currently integrated with OpenAI), in order to generate conversational source based answers. However, we encountered performance limitations due to the model's large size, which caused issues when running it on limited hardware. Specifically, we observed an inference time of approximately 70 seconds when using a GPU.

To address this issue, we are currently working on optimizing ways to integrate the model into the AskDemocracy space demo. Remaining work is required in order to improve the model's performance. Further updates are expected to be integrated in the AskDemocracy space demo.

Developed by:

🇨🇴 Jorge Henao
🇨🇴 David Torres

What's baizemocracy-lora-7B-cfqa-conv model?

This model is an open-source chat model fine-tuned with LoRA inspired by Baize project. It was trained with the Baize datasets and the ask2democracy-cfqa-salud-pension dataset, wich contains almost 4k instructions to answers questions based on a context relevant to citizen concerns and public debate in spanish.

Two model variations was trained during the Hackathon Somos NLP 2023:

A conversational style focused model: focused in a more conversational way of asking questions, dee Pre-proccessing dataset section.
A generative context focused model: This model variation is more focused on source based augmented retrieval generation Baizemocracy-RAGfocused.

Testing is a work in progress, we decide to share both model variations with community in order to invovle more people experimenting what it works better and find other possible use cases.

Training Parameters

Base Model: LLaMA-7B
Training Epoch: 1
Batch Size: 16
Maximum Input Length: 512
Learning Rate: 2e-4
LoRA Rank: 8
Updated Modules: All Linears

Training Dataset

Ask2Democracy-cfqa-salud-pension (3,806)
Standford Alpaca (51,942)
Quora Dialogs (54,456):
StackOverflow Dialogs (57,046)
Alpacaca chat Dialogs
Medical chat Dialogs

How to use it

import time
import torch
from peft import PeftModel, PeftConfig
from transformers import AutoModelForCausalLM, AutoTokenizer

peft_model_id = "hackathon-somos-nlp-2023/baizemocracy-lora-7B-cfqa-conv"
config = PeftConfig.from_pretrained(peft_model_id)
base_model = AutoModelForCausalLM.from_pretrained("decapoda-research/llama-7b-hf", return_dict=True, load_in_8bit=True, device_map='auto')
tokenizer = AutoTokenizer.from_pretrained(peft_model_id)

# Load the Lora model
tuned_model = PeftModel.from_pretrained(base_model, peft_model_id)

def generate(text):
  stt = time.time()
  print("hackathon-somos-nlp-2023/baizemocracy-lora-7B-cfqa response:")
  inputs = tokenizer(text, return_tensors="pt")
  input_ids = inputs["input_ids"].cuda()
  with torch.cuda.amp.autocast():
      tuned_model.eval()

      generation_output = tuned_model.generate(
          input_ids=input_ids[:,1:-1],
          generation_config=generation_config,
          return_dict_in_generate=True,
          output_scores=True,
          max_new_tokens=256
      )
      for s in generation_output.sequences:
          output = tokenizer.decode(s)
          print(output)
  ent = time.time()
  elapsed_time = round(ent - stt, 2)
  print (f"{elapsed_time} seconds")

Example outputs

baizemocracy-lora-7B-cfqa-conv model:

#Text taken from Mexican political reform from https://www.gob.mx/cms/uploads/attachment/file/3080/EXPLICACION_AMPLIADA_REFORMA_POLITICA_ELECTORAL.pdf
text = """
The conversation between human and AI assistant.Given the context answer the Human question.
Context:'Ratificación del Plan Nacional de Desarrollo y de la Estrategia Nacional de
Seguridad Pública
Se adiciona como facultad de la Cámara de Diputados la aprobación del Plan Nacional de Desarrollo, con lo que la pluralidad de intereses y las visiones expresadas por las distintas fuerzas
políticas que componen la Cámara de Diputados quedarán plasmadas en la ruta que el Ejecutivo
Federal traza para sus acciones durante cada sexenio.
De igual manera, el Senado de la República ratificará la Estrategia Nacional de Seguridad Pública. Toda vez que la función principal del Estado es garantizar la seguridad, es indispensable que
dicha estrategia sea aprobada por un órgano representativo de la voluntad popular como es el
caso del Senado.
El papel que desempeñarán las Cámaras del Congreso de la Unión en el contexto de la Reforma
Política-Electoral permite aumentar el nivel de corresponsabilidad entre los Poderes de la Unión,
al mismo tiempo que preserva la capacidad del Estado mexicano para responder oportunamente ante las amenazas al orden público y para poner en marcha acciones de trascendencia nacional.'
[|Human|] ¿cual será la nueva facultad de la cámara?"""
generate(text)
output:
[|AI|] La nueva facultad de la Cámara de Diputados será la aprobación del Plan Nacional de Desarrollo.

About dataset pre-processing

Ask2Democracy-cfqa-salud-pension dataset was pre-processed in a conversational style in two variations like this:


def format_instruction_without_context(example):
  example["topic"] = example['input']
  input = "La conversación entre un humano y un asistente de IA."
  input += "\n[|Human|] "+example['input']
  input += "\n[|AI|] "+example["output"]
  if len(example["topics"])>0:
    topics = ", ".join(example["topics"])
    input += "\n[|Human|] "+"¿En cuáles tópicos clasificarías su respuesta?"
    input += "\n[|AI|] "+f"Aquí una lista de tópicos: {topics}."
    example["topic"] += f" ({topics})"
  example["input"] = input
  return example`

def format_instruction_with_context(example):
  example["topic"] = example['input']
  context = example['instruction'].replace("Given the context please answer the question. Context:","")
  context = ' '.join(context.strip().split())[1:-3]
  input = "La conversación entre un humano y un asistente de IA."
  input += "\n[|Human|] "+example['input']+f"\nPara responder la pregunta, usa el siguiente contexto:\n{context}"
  input += "\n[|AI|] "+example["output"]
  if len(example["topics"])>0:
    topics = ", ".join(example["topics"])
    input += "\n[|Human|] "+"¿En cuáles tópicos clasificarías su respuesta?"
    input += "\n[|AI|] "+f"Aquí una lista de tópicos: {topics}."
    example["topic"] += f" ({topics})"
  example["input"] = input
  return example

More details can be found in the Ask2Democracy project GitHub

somosnlp-hackathon-2023
/

baizemocracy-lora-7B-cfqa-conv

About Ask2Democracy project