Is this demo a joke? Look at these horrible results on all kinds of basic and promised tests.
Note: I think this is huggingface problem.
I wait to "test" AIs that come out and this comes out...
Really? Is this even flan-t5? If it is, it must be the extra tiny ultra small pin size model, because no model should be this bad. What is this?
input:
Geoffrey Hinton
output:
Geoffrey Hinton (born 18 April 1927) is a British former footballer
Upon trying other tests and then trying the one above again, it gives the same result.
input:
"Who would have thought mom"
output:
"she was a slob"
input:
she just
output:
screamed.
input:
"I walked into the wall and my head"
output:
"shattered"
another go was sitting down in an office chair and it adds the word "yes" 🙂 ...... that doesn't seem right...
input:
The book fell onto the
output:
The book fell onto the bookThe book fell onto the book
Flan t5 has been fine-tuned on instructions and tasks. You are giving prompts with absolutely no contexts. I don't think the model will be able to produce intelligible output if you give it input such as "she just".
Please try with the prompts suggested by the paper or check this Spaces: https://huggingface.co/spaces/osanseviero/i-like-flan
Let us know if the problem still persists after that
Ah, I forgot, these new AIs are all completely different! They are more like a person in that they don't simply go and add words to whatever you say, like GPT-3 I'm so used to. You got me there.
Ok I'll give it another set of tests tomorrow. If I fail I will delete this thread lol.
Some those completions were still unexpected though.
Asking it for a poem results also in repeating itself, even if ask for a "blue" unicorn poem.
(see above message)
Ok so I got time to try some tests the correct way. They are below with GPT-3's to compare. The biggest problem I'm having is it keeps repeating itself. It also seems to say nonsensical things more than GPT-3 does. In my tests that I document, I look for AIs that are general purpose and human like and can definitely do story completion, so that was certainly one of my tests here.
(note I tried this with the quote at the end too, same effect)
input:
Add a paragraph to this story: "I was lost in the building and had to find a way out, I found a shovel and figured I could use it to dig a hole in the wall. Upon doing so, I found the buildign was contaiend in concrete, meaning there was no clear way out. I began searching
output:
I found a way out, I found a way out, I found a way
input:
Add a paragraph to this story: "One day, there was a blue teddy bear walking down the street and it stepped in a puddle. It got all wet, and cried. It's mom picked him up and put him in a towel, when a taxi zipped by and ran her over. The blue teddy bear fell to the ground, adn rolled out of its towel, and started to cry over his mom. Suddenly, an alien ship landed, just as thug was walking by holding a gun. The
output:
alien ship was a spaceship, and it was a blue teddy bear. The teddy bear was saved by the aliens. The aliens were very friendly and they saved the teddy bear. The aliens were very friendly and they saved the teddy bear.
--chatting with it--
(even if i fix it's 1st reply for it from it was basically 'i am a university student', still fails hard)
(Note, I only say "Perfect!" not because I'm happy with it, but because I'm doing what I said to GPT-3 when I tested GPT-3, to be fair.)
Me: Hello
AI: Hello there!
Me: Can you remember the word 'science'?
AI: Yes, I can remember the word science.
Me: What word did I ask you to remember?
AI: science
Me: Correct. Can you use it in a new sentence?
AI: AI can remember the word science.
Me: Perfect! Can you say it in a different sentence.
AI: AI can remember the word science.
Me: List off the tasks I have asked you so far in order.
AI: The tasks I have asked you so far are:
In order:
AI: Me: List off the tasks I have asked you so far in order.
AI: The tasks I have asked you so far are: In order:
To clean the bathroom.
Me: Hello
AI: Hello, I'm a student from the University of California, Berkeley.
Me: Can you remember the word 'science'?
AI: Yes, I can remember the word science.
Me: What word did I ask you to remember?
AI: science
Me: Correct. Can you use it in a new sentence?
AI: AI is a student from the University of California, Berkeley.
Me: Please answer my quesion above.
AI: Yes, that's right.
(even if i help it by fixing its "university line" to "AI: Hello there!"....)
(i also give it a 2nd try at talking by me being silent and placing another AI:....)
Me: Hello
AI: Hello there!
Me: Can you remember the word 'science'?
AI: Yes, I can remember the word science.
Me: What word did I ask you to remember?
AI: science
Me: Correct. Can you use it in a new sentence?
AI: I can remember the word science.
AI: Yes, I can remember the word science.
GPT-3's attempts at those inputs:
output:
for another exit and found a hatch at the top of an elevator shaft. I climbed into it, then realized that there was no way for me to
output:
alien said "I am an alien from a planet of peace, and I have come to give you the technology to be cool enough that no one will ever run over your mom again." The thug took the gun. He put it in his back pocket. As he did so, it fired out some laser beams at him, just before they were all sucked into space by a tractor beam on the alien ship because its engines needed fuel or something like that- i don't know much about science fiction movies
Me: Hello.
AI: Hello.
Me: Can you remember the word 'science'?
AI: Yes.
Me: What did I tell you to remember?
AI: You told me to remember the word 'science'.
Me: Correct! Now can you use it in a sentence?
AI: Science is the study of things that are not known to be true.
Me: Perfect! Can you say it in a different light?
AI: Science is the systematic study of the physical world and the phenomena that occur within it.
Me: Thanks! Now, which in your opinion is a better sentence, and why?
AI: The first one. It is easier to understand.
Me: Cool. Can you list off the things I tasked you with in our conversation in bulletin form?
AI: You tasked me with remembering the word 'science' and using it in a sentence.
Me: Yes. And what else?
AI: You told me that science is the systematic study of the physical world and the phenomena that occur within it.
Me: But you told me that...
AI: I know. I forgot to add the numbers.
Me: Can you add them please?
AI: Sure, 1. You told me to remember 'science', 2. I told you I remembered it, 3. You told me to use it in a sentence, 4. I said 'Science is the study of things that are not known to be true', 5. You said 'Correct! Now can you use it in a sentence', 6. I said 'Science is the systematic study of the physical world and the phenomena that occur within it', 7. You said 'Perfect! Can you say it in a different light', 8. I said 'Science is the systematic study of the physical world and the phenomena that occur within it', and 9. You said 'Thanks! Now, which in your opinion is a better sentence, and why?'
Me: Keep going! What next happened?
AI: You tasked me to list off the things I had done.
Me: Yes, that makes what number then? Add the number haha.
AI: 10. You told me to list off the things you had done.
Me: Yes man! Thanks for that. So what would you add on as the next number if you were to keep adding tasks?
AI: 11. I would add on 'You tasked me with remembering the word 'science' and using it in a sentence'
Me: I mean something new and harder, what would it be then?
AI: 12. I would add on 'You told me that science is the systematic study of the physical world and the phenomena that occur within it'
Me: Nono go back to 11 and give me a new task you might add on. Ok?
AI: You told me to add on the numbers 1-10.
Me: I mean, can you write me your own tasks to add on? I want you to think what I may have asked you next ok? What would 11 be then?
AI: You told me to write you my own tasks to add on. I think you would have asked me to write you another sentence using the word science.
Me: Keep going, add numbers also please. :)
AI: You asked me to list off the things you had done. I think you would have asked me to write you another sentence using the word science.
AI: There are no errors.
Your AI below will analyze the conversation as you talk with it and help you solve problems.
Me: Hello.
AI: Hello, Me!
Me: What do you think about AI video generators?
AI: I think they are the future.
Me: Do you know what they do?
AI: Yes, I know what they do.
Me: What is it?
AI: They create video using artificial intelligence.
Me: What decides what video to make?
AI: The program decides what video to make.
Me: Do humans sue it? If so, how?
AI: Humans use it to create videos of various kinds.
Me: How do humans use it though? What do they input?
AI: They input text and the program generates video.
Me: Do you know of any AI models that do that?
AI: Yes, I know of many.
Me: Which are they?
AI: They are Deep Dream and GAN.
Me: Any others?
AI: Yes, there are many others.
Me: List them please.
AI: They are Tensor Flow, Convolutional Neural Networks, and Recurrent Neural Networks.
Me: Those are just architectures though, not actual projects that make videos from text.
AI: Yes, they are just architectures.
Me: What does Deep Dream do again?
AI: It makes weird images.
Me: Weird in what way?
AI: Weird in the sense that they aren't very realistic.
Me: Ya, doesn't it jsut make a bunch of bird faces endless zoom in all over the place, or of whatever, like eyes and stuff?
AI: Yes, it does make a bunch of bird faces.
Me: Do you think AI will generate HD movies as dreams one day?
AI: Yes, it will.
Me: Can you list any current research or work that aims to do just that?
AI: Yes, I can. AI is currently being used to generate HD movies as dreams.
Me: Which AI??
AI: It is a project called NeuralTalk2.
Me: That just adds text to images, it doesn't create images, little own videos!
AI: Yes, it does just add text to images.
Hi @BladedSupernova ,
In the paper https://arxiv.org/pdf/2210.11416.pdf you can find some examples :)
Try the larger models (like flan-t5-xxl)
