Post
1045
After the announcements yesterday, I got a chance to try the new gemini-1.5-flash model from
@goog1e
, it is almost as good as gpt-4o on the StaticAnalaysisEval (
patched-codes/static-analysis-eval) It is also a bit faster than gpt-4o and much cheaper.
I did run into a recitation flag with an example in the dataset where the api refused to fix the vulnerability and flagged the input as using copyrighted content. This is something you cannot unset even with the safety filters and seems to be an existing bug https://issuetracker.google.com/issues/331677495
But overall you get gpt-4o level performance for 7% the price, we are thinking of making it default in patchwork - https://github.com/patched-codes/patchwork You can use the
I did run into a recitation flag with an example in the dataset where the api refused to fix the vulnerability and flagged the input as using copyrighted content. This is something you cannot unset even with the safety filters and seems to be an existing bug https://issuetracker.google.com/issues/331677495
But overall you get gpt-4o level performance for 7% the price, we are thinking of making it default in patchwork - https://github.com/patched-codes/patchwork You can use the
google_api_key and model options to choose gemini-1.5-flash-latest to run it with patchwork.
