language:
- en
license: apache-2.0
library_name: transformers
tags:
- code
- finance
- qa
- assistant
- reasoning
metrics:
- code_eval
model-index:
- name: Moe-2x7b-QA-Code
results:
- task:
type: text-generation
name: Text Generation
dataset:
name: AI2 Reasoning Challenge (25-Shot)
type: ai2_arc
config: ARC-Challenge
split: test
args:
num_few_shot: 25
metrics:
- type: acc_norm
value: 65.19
name: normalized accuracy
source:
url: >-
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=nextai-team/Moe-2x7b-QA-Code
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: HellaSwag (10-Shot)
type: hellaswag
split: validation
args:
num_few_shot: 10
metrics:
- type: acc_norm
value: 85.36
name: normalized accuracy
source:
url: >-
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=nextai-team/Moe-2x7b-QA-Code
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MMLU (5-Shot)
type: cais/mmlu
config: all
split: test
args:
num_few_shot: 5
metrics:
- type: acc
value: 61.71
name: accuracy
source:
url: >-
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=nextai-team/Moe-2x7b-QA-Code
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: TruthfulQA (0-shot)
type: truthful_qa
config: multiple_choice
split: validation
args:
num_few_shot: 0
metrics:
- type: mc2
value: 65.23
source:
url: >-
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=nextai-team/Moe-2x7b-QA-Code
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: Winogrande (5-shot)
type: winogrande
config: winogrande_xl
split: validation
args:
num_few_shot: 5
metrics:
- type: acc
value: 77.35
name: accuracy
source:
url: >-
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=nextai-team/Moe-2x7b-QA-Code
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: GSM8k (5-shot)
type: gsm8k
config: main
split: test
args:
num_few_shot: 5
metrics:
- type: acc
value: 49.66
name: accuracy
source:
url: >-
https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=nextai-team/Moe-2x7b-QA-Code
name: Open LLM Leaderboard
Model Description
Moe-2x7b-QA-Code is a state-of-the-art language model specialized in Question Answering (QA) and code-related queries. Leveraging the Mixture of Experts (MoE) architecture, this model has been trained on a diverse dataset encompassing technical documentation, forums, and code repositories to provide accurate and context-aware responses to both technical and general questions.
How to Use
from transformers import AutoTokenizer
import transformers
import torch
model = "nextai-team/Moe-2x7b-QA-Code"
tokenizer = AutoTokenizer.from_pretrained(model)
pipeline = transformers.pipeline(
"text-generation",
model=model,
device_map="auto",
model_kwargs={"torch_dtype": torch.float16},
)
def generate_resposne(query):
messages = [{"role": "user", "content": query}]
prompt = pipeline.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
return outputs[0]['generated_text']
response = generate_resposne("How to learn coding .Please provide a step by step procedure")
print(response)
Intended Use
This model is intended for developers, data scientists, and researchers seeking to integrate sophisticated natural language understanding and code generation functionalities into their applications. Ideal use cases include but are not limited to:
Automated coding assistance Technical support bots Educational tools for learning programming Enhancing code review processes
Model Architecture employs a Mixture of Experts (MoE) architecture, which allows it to efficiently manage its vast number of parameters for specialized tasks. This architecture facilitates the model's ability to discern subtle nuances in programming languages and natural language queries, leading to more accurate code generation and question answering performance.
Performance demonstrates significant improvements in accuracy and relevance over its predecessor, particularly in complex coding scenarios and detailed technical queries. Benchmarks and performance metrics can be provided upon request.
Limitations and Bias
This model, like any other, has its limitations. It may exhibit biases inherent in the training data or struggle with questions outside its training scope. Users should critically assess the model's outputs, especially for sensitive or critical applications.
Training Data
The Moe-2x7b-QA-Code model was trained on a curated dataset comprising technical documentation, Stack Overflow posts, GitHub repositories, and other code-related content. This extensive training set ensures the model's proficiency in understanding and generating code-related content alongside general language understanding.
Training Procedure
The model was trained using a Mixture of Experts (MoE) approach, allowing it to dynamically leverage different subsets of parameters for different types of input data. This method enhances the model's capacity and efficiency, enabling it to excel in a wide range of QA and coding tasks.
Model Architecture
Moe-2x7b-QA-Code employs an advanced MoE architecture with 2x7 billion parameters, optimized for high performance in QA and coding tasks. This architecture enables the model to efficiently process and generate accurate responses to complex queries.
Contact Https://nextai.co.in
Open LLM Leaderboard Evaluation Results
Detailed results can be found here
| Metric | Value |
|---|---|
| Avg. | 67.42 |
| AI2 Reasoning Challenge (25-Shot) | 65.19 |
| HellaSwag (10-Shot) | 85.36 |
| MMLU (5-Shot) | 61.71 |
| TruthfulQA (0-shot) | 65.23 |
| Winogrande (5-shot) | 77.35 |
| GSM8k (5-shot) | 49.66 |