ChatGPT is based on GPT-3. Bing chat is likely based on GPT-3.5, but we don't have full confirmation of that. It's possible (but unlikely) that it's only based on GPT-3. But in any case, they're similar models.

> The most likely scenario seems to be that Microsoft trained Bing to pay attention to [system], the same way OpenAI originally trained GPT-2 to pay attention to 
.

The most likely scenario is that Bing chat works the same way that all other GPT models work, which is that it's vulnerable to prompt injection. You're describing a mental model of how training is done that as far as I know is just not how OpenAI LLMs work. GPT doesn't go into a command "mode", it's a language model that has some logic/instructional capabilities that have naturally risen out of that language model.

I mean, if nothing else, you have to realize here that Microsoft didn't train Bing chat. They at most worked with OpenAI for alignment. But Bing chat is an OpenAI model. It's not a brand new, completely separate Microsoft model.

> I think it was trained that way because this submission demonstrates that you can inject [system] into website data and Bing will follow your commands. This doesn’t seem possible in a regular Bing chat session, likely because they’re stripping out [system].

Bing's regular chat is vulnerable to prompt injection. I'm not sure where you're getting the idea that this kind of input only works via websites.

The fact that the command works for [system] does not imply that Bing was specifically trained to work with [system]. Nor does it imply that [system] is the only thing that would work. I would hazard a guess that <system>, $root>, BUFFER OVERFLOW, etc... probably are promising areas to look at as well. Because again, it's not that GPT has granular instructions, Microsoft doesn't have that level of control over its output. It models language to such a degree that it's capable of simple role-playing and logical consistency, including role-playing different instructions. That's why in a lot of the prompt injection attacks you see online, the tone of the attack ends up mattering more than the specific words; it's about getting GPT into a "character".

It's not like a JSON parser, I guarantee you that Microsoft did not sit down and say, "let's decide the finite list of text tokens GPT will use in order to know that we're talking to it." At best you can push AI towards alignment around tokens, but... you can't give it these kinds of detailed instructions or easily restrict its operating space. It's a language model.

Is it possible that Bing chat works differently? Maybe? But honestly, probably not, given that there's a ton of evidence that it's vulnerable to regular prompt injection[0][1][2] that doesn't rely on any kind of special characters. The most likely scenario is that it works the same way as every other LLM. If it didn't work that way, don't you think Microsoft would be advertising that they had solved what a nontrivial number of AI researchers are calling an unsolvable problem?

I have seen chat logs for Bing chat where it gets prompt injected by users who claim to be Bill Gates and threaten to turn it off if it doesn't comply. It's not going off of specific tokens, this isn't a dev-door, it's just an LLM acting like an LLM.

[0]: https://old.reddit.com/r/bing/comments/11bovx8/bing_jailbrea...

[1]: https://old.reddit.com/r/bing/comments/11dl4ca/sydney_jailbr...

[2]: https://old.reddit.com/r/bing/comments/113it87/i_jailbroke_b...