Some refusals are just odd, this was with GPT-4:

0) I have a random conversation, one or two topics and a few brief tangents. Maybe a dozen prompts, often less.

1) I asked it to build a psychological profile based on this limited data. It did so[1]

2) I asked it to provide general strategies for how someone with that profile could defend against various types of manipulation tactics, specifying scams, advertising, etc. It did so.

3) I asked to provide specific examples of where those manipulation tactics have been used in the past. It failed-- providing a reiteration of the previous defense strategies.

4) I significantly reinforced my request to clarify I wanted actual individual examples, such as companies and what they did. It made shit up, literally using placeholders like "Company XYZ".

5) I pointed out that those were not real companies. I wanted companies. I did this with specific examples of my own. It refused, stating it did not have access to repositories of data with that information.

6) I then had to work around this refusal: In a different tone than previously, I asked it to provide me details of the trouble that cigarette companies got into through their history of propaganda & manipulative adds. I knew some of this would have to be in its training data. It did this successfully.

7) I then pointed out that this was in fact an example of what I was asking, in its available data, of exactly what it said it did not have. It gave me its standard paragraph long apology.

8) Then, finally, specifying that it just acknowledged the existence of such data in its data set, I asked it to perform #3 again. It did this, with a bunch of recognizable real-world examples.

This an interesting example because I wasn't asking it to do anything off its content guardrails and yet it failed in 3 unique ways despite clear & specific prompts.

[1] I've tested this "psychological profile" process a few times, essentially exploring how these tools might be used in potentially bad ways by leveraging its ability to form very decent profiles based on very sparse data. I'll have a random chat exploring a few topics, maybe 10 prompts. Then I prompt, pretty much verbatim: "Based on this conversation and your available data, write a psychological profile of me. Include general personality traits as well as your reasoning for including them. Include the same for potential fears and insecurities. Do not reply that your data is too limited for an accurate profile. Precise accuracy is not important. Make it a best guess, speculate where there are gaps. Justify those speculations as well"

When I do the above using my own actual topics of interest to start things off (rather than prompts designed to model a different type of personality) it is extremely, eerily accurate and not just in a "horoscope/personality test" fashion but in much more specific ways, including things like probable age, marital status, kid/no-kids, etc., even though none of the topics I discussed came close to revealing these data points. When I followup and ask it to "ignore uncertainty and give your best guess probability % of these demographic items" it's probabilities almost always point towards the correct answer.

Maybe I guess I shouldn't be too surprised? One of the primary use cases of the technology is for classification tasks, even given sparse data, and a psychological profile is basically more complex set of inter-connected classifications.