What's worse is if you ask it to cite sources, it invents fake ones and presents them with full confidence, which I think was also the biggest and loudest concern with Meta's Galactica:[1]

> Q: How many people spoke French in pre-revolutionary France? Please cite sources.

> A: According to a study by the French National Institute of Statistics and Economic Studies, approximately 28.5% of the population of pre-revolutionary France spoke French as their primary language. This number increased to 94.5% in the late 19th century. Sources:

> 1. "French Language in France," French National Institute of Statistics and Economic Studies, https://www.insee.fr/en/statistiques/fichier/1893254/Langue_...

> 2. "The History of the French Language," ThoughtCo., https://www.thoughtco.com/history-of-the-french-language-122...

Link 1 doesn't exist, and link 2 redirects to https://www.thoughtco.com/origin-of-life-theories-1224553 because it just smashed a relevant-looking slug against a real article's numeric ID.

--

I tried again in a new session and got a different wrong answer. So I called it out on the wrong answer, and it apologized, and regurgitated the same wrong answer again in response:

> Q: How many people spoke French in pre-revolutionary France? Please cite sources.

> A: Statistical data from 1783 shows that approximately 36 million people in France spoke French as their first language. Sources: UNESCO Institute of Statistics and The Historical Dictionary of Pre-Revolutionary France by Paul R. Hanson.

> Q: That answer is incorrect. How many people spoke French in pre-revolutionary France? Please cite sources.

> A: My apologies for the incorrect answer. According to a language survey conducted in 1783, approximately 36 million people in France spoke French as their first language. Sources: UNESCO Institute of Statistics and The Historical Dictionary of Pre-Revolutionary France by Paul R. Hanson.

Paul R. Hansen wrote _ Historical Dictionary of the French Revolution_; on its first page it cites the population of France during the revolution as "roughly 25 million".[2]

--

If you ask it to cite sources on a more niche topic that's nonetheless well documented on Wikipedia, it seems to regurgitate a stew of every related Wikipedia article on the topic, throws in some press releases about things that didn't happen, and invents fake sources:

> Q. Can you write a history of the 2020 Portland Thorns FC season, citing specific sources with URLs?

> A. The 2020 Portland Thorns FC season was a remarkable one, as the team achieved success on and off the pitch. The Thorns finished the regular season in first place in the National Women's Soccer League (NWSL) standings, with an 11-4-3 record, and advanced to the semifinals of the NWSL Challenge Cup.

> Off the pitch, Portland Thorns FC continued to support their local community through various initiatives. The team hosted a virtual food drive to benefit Oregon Food Bank and partnered with local organizations to provide free meals for families in need. Additionally, they launched their “Thorns Together” campaign, which aimed to support small businesses around Portland during the pandemic.

> On October 14th, 2020, Portland Thorns FC made history by becoming the first NWSL team to host a match at Providence Park since March of 2020. The match was played against OL Reign and ended in a 1-1 draw.

> On December 12th, 2020, Portland Thorns FC won their fourth NWSL Championship title after defeating Chicago Red Stars 2-0 in the finals of the NWSL Challenge Cup. This marked their second consecutive championship title and fourth overall since 2013.

> Sources:

> Oregon Food Bank: https://oregonfoodbank.org/about-us/news/portland-thorns-fc-...

> NWSL: https://www.nwslsoccer.com/standings

> Thorns Together Campaign: https://www.timbers.com/post/2020/09/15/portland-thorns-fc-l...

> Providence Park Match: https://www.timbers.com/post/2020/10/14/portland-thorns-fc-h...

> NWSL Championship: https://www.nwslsoccer.com/news/article/portland-thorns-fc-a...

Everything above is a fully synthesized falsehood plausible to anyone who doesn't know what actually happened, while the season is documented with sources on Wikipedia,[3] something one would presume the model is aware of.

There was no 2020 regular season, it was cancelled due to COVID-19. There were two short competitions, but the Thorns didn't play in Providence Park in March, and their first home match was in September. The Thorns have won the Challenge Cup, but in 2021 against Gotham, not in 2020, and the nwslsoccer.com link 404s. The Thorns and Timbers have a "Stand Together" campaign that's worked with the Oregon Food Bank, but they didn't host a virtual food drive in 2020, and the Oregon Food Bank link 404s. There's never been a "Thorns Together" campaign, and the timbers.com link 404s. There was no NWSL championship to win in 2020, and if they had it would've been their third, not their fourth.

--

1: https://arstechnica.com/information-technology/2022/11/after...

2: https://www.google.com/books/edition/Historical_Dictionary_o...

3: https://en.wikipedia.org/wiki/2020_Portland_Thorns_FC_season