Does Daily Software Engineering Work Need Reasoning Models?
•
5
Yi Cui
onekq
AI & ML interests
Benchmark, Code Generation,
Articles
Organizations
Posts
4
Post
2479
Here is my latest study on OpenAI🍓o1🍓.
A Case Study of Web App Coding with OpenAI Reasoning Models (2409.13773)
I wrote an easy-to-read blogpost to explain finding.
https://huggingface.co/blog/onekq/daily-software-engineering-work-reasoning-models
INSTRUCTION FOLLOWING is the key.
100% instruction following + Reasoning = new SOTA
But if the model misses or misunderstands one instruction, it can perform far worse than non-reasoning models.
A Case Study of Web App Coding with OpenAI Reasoning Models (2409.13773)
I wrote an easy-to-read blogpost to explain finding.
https://huggingface.co/blog/onekq/daily-software-engineering-work-reasoning-models
INSTRUCTION FOLLOWING is the key.
100% instruction following + Reasoning = new SOTA
But if the model misses or misunderstands one instruction, it can perform far worse than non-reasoning models.
Post
402
Announce 🎉 WebApp1K-Duo 🎉
onekq-ai/WebApp1K-Duo-React
This is to keep up the challenge after OpenAI o1 models saturated the WebApp1K benchmark. The new benchmark brings SOTA to 67%. Let the hill climbing commence!
onekq-ai/WebApp1K-models-leaderboard
PS: I will publish more findings soon.
onekq-ai/WebApp1K-Duo-React
This is to keep up the challenge after OpenAI o1 models saturated the WebApp1K benchmark. The new benchmark brings SOTA to 67%. Let the hill climbing commence!
onekq-ai/WebApp1K-models-leaderboard
PS: I will publish more findings soon.
models
None public yet
datasets
None public yet