OpenAI GPT-3 vs Specialized Models [Benchmark] - Should AI companies be really worried ?
Tags: gpt3, ai, api, googlecloud
Published: 2023-01-06 14:37:41



OpenAI’s Objective is to create Artificial General Intelligence - “highly autonomous systems that outperform humans at most economically valuable work” . 

They are planning on combining text, image and speech models.

In this article, we focus on one of their actual milestones, **GPT-3** . Can it really achieve state of the art performance on any language task (compared to specialized models)?

**OpenAI’s GPT3** : Open AI is challenging these companies (especially NLP models) with their GPT3 model. 

Let’s ask ChatGPT (GPT-3 optimized for dialogue) what GPT-3 is ?


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/y4l0k16qw0ysvc69yt1f.png)



**Specialized AI Models** : There are a lot of AI companies training specialized models for specific tasks and providing access to them through APIs. Among witch big tech companies (Google, Amazon, Microsoft and IBM) as well as smaller companies focused on specific tasks (DeepL(translation) , Deepgram (speech) , Clarifia (vision) ) 


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ahu87b9zlc3vzqsthbs8.png)



Large Language Models like GPT-3 should be able to compete with specialized models on a lot of Natural Language Processing (NLP) tasks without fine tuning. That's referred to as **Zero Shot Learning** . Let's verify that!

## Benchmark
In order to verify that, we chose four tasks : **Keywords Extraction, Sentiment Analysis, language detection** and **translation** to benchmark GPT3 against other state of the art proprietary models from different companies.

We've done that using a single API : [Eden AI](https://edenai.co?referral=open-ai-vs-the-world). Code snippets will be available for each task so that you can reproduce the predictions yourself on your own data.

There is an Open Sourced version of EdenAI  that you can [⭐ find on github ⭐ ](https://github.com/edenai/edenai-apis) as a python module ! 

Please refer there for more information on the different APIs used in the benchmark.

### Language Detection: 
Language detection is simply the task of returning in what language a text is written.


**1/ Dataset :** We used an interesting dataset [from huggingface](https://huggingface.co/datasets/papluca/language-identification) with 20 languages : arabic (ar), bulgarian (bg), german (de), modern greek (el), english (en), spanish (es), french (fr), hindi (hi), italian (it), japanese (ja), dutch (nl), polish (pl), portuguese (pt), russian (ru), swahili (sw), thai (th), turkish (tr), urdu (ur), vietnamese (vi), and chinese (zh)


**2/ Evaluation**
We compare Open AI to Google GCP, Amazon AWS, and IBM.
We took a few hundreds of examples and evaluated the models performance on them with the **Accuracy** metric.

**3/ Results** 
The results are shown below with OpenAI ranking 3rd out of the four AI providers we chose.
 
1. Amazon 96%
2. Google 95%
3. OpenAI 89%
4. IBM 87%


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kpsqrhjo9ffyddr45t1e.png)



### Sentiment analysis :
This tasks is about understanding the sentiment of the writer when writing a specific piece of text. It can be _Positive_ _Negative_ or _Neutral_ .



**1/ Dataset:** Most of the datasets we've found did not include a “neutral” sentiment except for the [Twitter Sentiment Analysis dataset](https://www.kaggle.com/datasets/jp797498e/twitter-entity-sentiment-analysis?resource=download 
) from kaggle. 

**2/ Evaluation:**
We compare Open AI to Google GCP’s, Amazon AWS’s and IBM’s APIs. using the **Accuracy** metric


**3/ Results :**
As shown below, once again, OpenAI gets the 3rd place.

1. Amazon 76%
2. Google 66%
3. Openai 61%
4. IBM 56%



![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/64ime0gcfuyslbnsxvky.png)



### Keyword extraction :
Keyword or Keyphrase Extraction is about being able to extract the words or phrases that most represent a given text.



**1/ Dataset**: we selected our datasets from the public github repository [AutomaticKeyphraseExtraction](https://github.com/snkim/AutomaticKeyphraseExtraction ) 
Most of the datasets listed there were too long for the 4k token limit of OpenAI so we had to go with the Hulth2003 abstracts dataset.
Since the different providers are trained to return keywords and keyphrases present in the original text, we did some cleaning to remove all keywords that were not present in the abstracts. We ended up with 470 abstracts.

**2/ Evaluation**
We compare Open AI to Microsoft, Amazon and IBM.
And we measured their performances using the **average precision** metric.


**3/ Results** : This time, OpenAI's GPT-3 was ranked last.

1/ Microsoft 0.6513312046679187
3/ IBM 0.6022276518997
2/ Amazon 0.4954784007523
4/ OpenAI 0.2598775421


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rcaucnm175wa9kkiyoz0.png)


### Translation :
Automatically translate a text from a language A to a language B.


**1/ Dataset :** we chose a dataset from the Language Technology Research Group at the University of Helsinki’s [Tatoeba Translation Challenge ](https://github.com/Helsinki-NLP/Tatoeba-Challenge ).
We took 100 of examples from different latin languages pairs : deu-fra, eng-fra, fra -ita, deu-spa , deu-swe which constitutes a 500 example test dataset.


**2/ Evaluation**
We compare Open AI to DeepL, ModernMT, NeuralSpace, Amazon AWS and Google GCP.
A [lot of metrics](https://en.wikipedia.org/wiki/Evaluation_of_machine_translation#Automatic_evaluation) exist for automatic machine translation evaluation. We chose  [COMET by Unbabel](https://github.com/Unbabel/COMET) (wmt21-comet-da) which is based on a machine learning model trained to get state-of-the-art levels of correlation with human judgements. (read more on [their paper](https://aclanthology.org/2020.emnlp-main.213/) ) .
 

**3/ Results:**
The scores are not interpretable but are used to rank machine translation models. And here again, OpenAI ranks last in this task.

1. DeepL : 0.19001633345126925 
2. ModernMT : 0.17788391513374424 
3. Amazon : 0.16483921567053203 
4. Neuralspace : 0.163133354485786 
5. Google : 0.16280640903935437 
6. OpenAI : 0.15934198508564865


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6zt3y11v9sq1jhrrtoft.png)



## Conclusion

OpenAI's GPT-3's results are quite impressive. We’re getting closer to a kind of general (zero shot) nlp models multitasking without fine tuning. 

**But in practice**, for the moment, if you need an API for one of the tasks we presented in the benchmark, **OpenAI's GPT3 wouldn't be the best choice** .
In addition to the fact of it not being the best performant model, the **4k input tokens limit** can be problematic for reasonably long texts. 

We still do need to closely watch the new models OpenAI is working on. As [Sam Altman talked about in an interview](https://www.youtube.com/watch?v=WHoWGNQRXb0), they are And implementing a continuous learning approach which would make their model constantly improving by feeding on the internet.

They are also planing on unifying their models to deal with multiple input types which resulting in a single model capable of analyzing any type of data.

## Code Snippets : 

**Language Detection :**

```python
import json
import requests

headers = {"Authorization": "Bearer 🔑 Your_API_Key"}

url ="https://api.edenai.run/v2/translation/language_detection"
payload={"providers": "google,amazon,openai,ibm", 'text': "this is a test"}

response = requests.post(url, json=payload, headers=headers)

result = json.loads(response.text)
print(result['google'])
print(result['amazon'])
print(result['ibm'])
print(result['openai'])

```

**Sentiment Analysis :**


```python
import json
import requests

headers = {"Authorization": "Bearer 🔑 Your_API_Key"}

url ="https://api.edenai.run/v2/text/sentiment_analysis"
payload={"providers": "google,amazon,ibm,openai", 'language': "en", 'text': "this is a test"}

response = requests.post(url, json=payload, headers=headers)

result = json.loads(response.text)
print(result['google'])
print(result['amazon'])
# ...

```

**Keyword Extraction:**


```python
import json
import requests

headers = {"Authorization": "Bearer 🔑 Your_API_Key"}

url ="https://api.edenai.run/v2/text/keyword_extraction"
payload={"providers": "microsoft,amazon,ibm,openai", "language": "en", "text": "this is a test of Eden AI"}

response = requests.post(url, json=payload, headers=headers)

result = json.loads(response.text)
print(result['microsoft'])
print(result['amazon'])
#...

```

**Translation :**

```python
import json
import requests

headers = {"Authorization": "Bearer 🔑 Your_API_Key"}

url ="https://api.edenai.run/v2/translation/automatic_translation"
payload={ "providers": "deepl,modernmt,neuralspace,amazon,google,openai", 
'source_language':"en", 
'target_language':"fr", 
'text': "this is a test"}

response = requests.post(url, json=payload, headers=headers)

result = json.loads(response.text)
print(result['deepl']['text'])
print(result['google']['text'])
# ...

```