Interesting but needs more work?

#2
by pranjal-codefire - opened

Seems to work for generating a lot of queries, however, fails for following and many more similar scenarios

CREATE TABLE employee (
ID int(10) NOT NULL AUTO_INCREMENT,
name varchar(255) DEFAULT '',
tel1 varchar(255) DEFAULT NULL,
tel2 varchar(255) DEFAULT NULL,
pass varchar(255) DEFAULT '',
company varchar(256) DEFAULT NULL,
employee_code int(11) NOT NULL DEFAULT '0',
joining_date date DEFAULT NULL,
designation varchar(255) DEFAULT NULL,
) ENGINE=MyISAM AUTO_INCREMENT=211 DEFAULT CHARSET=latin1

And ran the following prompt
"for mysql database give me the number of employees joined in each designation on monthly basis for last 1 year "

Got the following response

SELECT employee.designation,
count(employee.id) AS number_of_employees
FROM employee
WHERE employee.joining_date >= (CURRENT_DATE - INTERVAL '1 year')
GROUP BY employee.designation
ORDER BY number_of_employees DESC;

Where as ChatGPT gives following, which seems more appropriate for the prompt

SELECT
DATE_FORMAT(joining_date, '%Y-%m') AS month_year,
designation,
COUNT(*) AS num_joined
FROM employee
WHERE joining_date >= DATE_SUB(CURDATE(), INTERVAL 1 YEAR)
GROUP BY month_year, designation
ORDER BY month_year, num_joined DESC;

gpt-3.5-turbo-0613 gives me

SELECT MONTH(joining_date) AS month, designation, COUNT(*) AS num_employees
FROM employee
WHERE joining_date >= DATE_SUB(CURDATE(), INTERVAL 1 YEAR)
GROUP BY MONTH(joining_date), designation

Defog.ai org

Thanks for trying out the model, and for the feedback!

Great point – our model is not instruction fine tuned yet (though we have an instruction fine-tuned update coming soon). While the model currently works well for generating queries, it often fails for questions that humans ask in a vague (to an LLM) way. ChatGPT and gpt-3.5-turbo are both instruction fine-tuned, and handle these slightly "human-language" questions better

In this case, asking How employees joined in each month and each designation over the last year? does give the correct response. We should get to parity or better results with gpt-3.5-turbo for these kinds of questions within the next month (mostly by asking a lot of contractors for their data labeling preferences).

Thanks again for trying this out – please keep the feedback coming!

Sign up or log in to comment