michaelfeil

/

ct2fast-starcoder

like 12

I see something weird in my results and I would like anyone can help me. I tried to use ctsfast to replace huggingface llm in Pandas AI.
I passed the following prompts but the result is nothing about coding but only suggesting 2 names

You are provided with the following pandas DataFrames:

{'Name': ['Alice', 'Bob', 'Charlie', 'David', 'Eve'], 'Age': [25, 30, 22, 35, 28], 'City': ['New York', 'San Francisco', 'Los Angeles', 'Chicago', 'Miami']}

This is the initial python code to be updated:

# TODO import all the dependencies required
import pandas as pd

def analyze_data(dfs: list[pd.DataFrame]) -> dict:
    """
    Analyze the data
    1. Prepare: Preprocessing and cleaning data if necessary
    2. Process: Manipulating data for analysis (grouping, filtering, aggregating, etc.)
    3. Analyze: Conducting the actual analysis (if the user asks to plot a chart save it to an image in /temp_chart.png and do not show the chart.)
    4. Output: return a dictionary of:
    - type (possible values "text", "number", "dataframe", "plot")
    - value (can be a string, a dataframe or the path of the plot, NOT a dictionary)
    Example output: { "type": "text", "value": "The average loan amount is $15,000." }
    """

Using the provided dataframes (dfs), update the python code based on the last question in the conversation.

Updated code:

My result:

"\nYou are provided with the following pandas DataFrames:\n\n{'Name': ['Alice', 'Bob', 'Charlie', 'David', 'Eve'], 'Age': [25, 30, 22, 35, 28], 'City': ['New York', 'San Francisco', 'Los Angeles', 'Chicago', 'Miami']}\nWrite Python code to do the following query using pandas\n\nWho are two oldest people?\n\n\n\nDavid, Eve\n\n\n\nWhat is the name of the person who has the most number of friends?\n\n\n\nDavid\n\n\n\nWhat is the name of the person with the highest age?\n\n\n\nDavid\n\n\n\nWhat is the name of the person who has the most number of friends and the highest age?\n\n\n\nDavid\n\n\n\nWhat is the name of the person who has the most number of friends and the highest age?\n\n\n\nDavid\n\n\n\nWhat is the name of the person who has the most number of friends and the highest age?\n\n\n\nDavid\n\n\n\nWhat is the name of the person who has the most number of friends and the highest age?\n\n\n\nDavid\n\n\n\nWhat is the name of the person who has the most number of friends and the highest age?\n\n\n\nDavid\n\n\n\nWhat is the name of the person who has the most number of friends and the highest age?\n\n\n\nDavid\n\n\n\nWhat is the name of the person who has the most number of friends and the highest age?\n\n\n\nDavid\n\n\n\nWhat is the name of the person who has the most number of friends and the highest age?\n\n\n\nDavid\n\n\n\nWhat is the name of the person who has the most number of friends and the highest age?\n\n\n\nDavid\n\n\n\nWhat is the name of the person who has the most number of friends and the highest age?\n\n\n\nDavid\n\n\n\nWhat is the name of the person who has the most number of friends and the highest age?\n\n\n\nDavid\n\n\n\nWhat is the name of the person who has the most number of friends and the highest age?\n\n\n\nDavid\n\n\n\nWhat is the name of the person who has the most number of friends and the highest age?\n\n\n\nDavid\n\n\n\nWhat is the name of the person who has the most number of friends and the highest age?\n\n\n\nDavid\n\n\n\nWhat is the name of the person who has the most number of friends and the highest age?\n\n\n\n\nDavid\n\n\n\nWhat is the name of the person who has the most number of friends and the highest age?\n\n\n\n\nDavid\n\n\n\nWhat is the name of the person who has the most number of friends and the highest age?\n\n\n\n\nDavid\n\n\n\nWhat is the name of the person who has the most number of friends and the highest age?\n\n\n\nDavid\n\n\n\nWhat is the name of the person who has the most number of friends and the highest age?\n\n\n\nDavid\n\n\n\nWhat is the name of the person who has the most number of friends and the highest age?\n\n\n\nDavid\n\n\n\nWhat is the name of the person who has the most number of friends and the highest age?\n\n\n<"]

@Bilibili Thanks!
For context: Usually the quantization decreases the relevant metrics. A viable test to perform is to validate if quantization impacts the performance.

I would say, that GPTQ, at time of their paper, had some issues to perform a good quantization, and int8-quantization is really underwhelming, while int4 performance works as expected with gptq.
In general, int8 should not significantly impact performance. In some cases this may also lead to a non-significant increase in performance, i assume since human-eval is such a small dataset. For me the result of the quantization sound viable.

@thanhnew2001 I noticed you are querying the starcoder model with instructions, while starcoder was just trained on documents.
I would resort with such a request to a model like StarChat-Beta, which is instruction tuned, potentially also derivatives from llama-2chat.