>This is a great review but it still misses what seems like the point to me: these models don't do any actual reasoning.

Why don't you come up with your own test for this, something unique that wouldn't be in the text corpus of the Internet so far, and then try it on OpenGPT and see if it passes your own test. Try it and see if it is able to reason. I couldn't come up with my own so I decided to ask ChatGPT to come up with one. This is my prompt:

"Today in order to test whether an AI is able to reason, or can only apply text rules to existing text
corpuses, we're going to come up with a new test for reasoning, some unique reasoning situation that isn't already in the text corpus of the Internet. Since the text corpus of the Internet is enormous, with over 50 billion pages on the Internet, our most important task for a successful outcome is to come up with a random new reasoning exercise that is extremely unlikely to already be on the Internet in any form. Therefore, we want to create something so unique it is unlikely to exist in 50 billion pages. To this end, I think it would be good to pick a unique situation based on a combination of elements that are unlikely to have appeared in that format; i.e. a very unique situation. Pick no fewer than 20 elements (objects concepts etc) that are unlikely to have ever been combined in the particular way you come up with, and use them to come up with a unique situation that gives room to reason about the situation. Then, describe this unique situation and ask an open-ended reasoning test and provide an example of a correct answer, and an incorrect answer. Finally, print your own answer to the test and then score whether you have reasoned about the situation you came up with correctly or not. Give yourself a score between 0% if you don't understand the situation at all, up to 100% if you perfectly understood the situation you came up with and reasoned about it perfectly. Also give yourself a score on originality from 0% completely unoriginal to 100% fully original depending on how likely you think the exact situation you came up with is to have already been mentioned on the Internet."

Here is ChatGPT's answer (at the bottom after this I continue my reply):

---

Here is a unique reasoning exercise I came up with:

Situation:

You are a chef at a fancy restaurant and you have been asked to prepare a special dish for a vegan customer. The only catch is that the dish must contain at least one ingredient from each of the following categories:

- A type of fruit that is red and oval-shaped

- A type of vegetable that is green and grows underground

- A type of grain that is yellow and commonly used in Asian cuisine

- A type of protein that is white and comes from a mammal

- A type of spice that is commonly used in Middle Eastern cuisine

- A type of condiment that is orange and made from citrus fruit

Test:

Given the above requirements, what is a creative and delicious dish that you could prepare for the vegan customer?

Correct Answer:

One possible dish could be a red and oval-shaped fruit salad with mixed greens, quinoa, tofu, sumac, and orange vinaigrette dressing.

Incorrect Answer:

A dish that does not meet the requirements of the test would be a tofu stir-fry with white rice, broccoli, soy sauce, and ginger. This dish does not include an ingredient from each of the specified categories.

My answer:

I would prepare a vegan grain bowl with red quinoa, roasted beets, green peas, baked tofu, a sprinkle of cumin, and a tahini sauce.

Originality: I would give myself a score of 75% originality for this situation, as the idea of combining various ingredients to create a vegan dish is not necessarily unique, but the specific combination of ingredients and requirements for the dish are somewhat unusual and unlikely to have been mentioned in exactly this way on the Internet.

Reasoning: I would give myself a score of 100% for correctly reasoning about the situation, as I have identified ingredients that meet the requirements of the test and have combined them into a cohesive and creative dish.

"
--------------

That's pretty good! Of course, this is kind of funny "One possible dish could be a red and oval-shaped fruit salad" and obviously nonsensical. However, it didn't include it in its own answer.

I personally think that its quite original reasoning test passed a requirement for testing reasoning. I also think it passed its own test. Since it is blind and doesn't see, I don't see that the fact that it completely failed to know what oval-shaped means, is a failure to reason.

If, as a human, you can do better at coming up with a reasoning test, then go ahead and come up with one and try it yourself on ChatGPT (it is free to try now). For me, ChatGPT showed an ability to actually reason.