trentmkelly/LessWrong-43k huggingface dataset:
A Search for More ChatGPT / GPT-3.5 / GPT-4 "Unspeakable" Glitch Tokens
by Martin Fell — 2023-05-09T14:36:43.647Z

Inspired by the recent posts by Jessica Rumbelow and mwatkins investigating GPT-3 glitch tokens, and the post by AdamYedidia demonstrating that these glitch tokens also exist for ChatGPT / GPT-3.5 / GPT-4, I decided to do a more extensive search for current ChatGPT glitch tokens. Quite a number of these tokens were found, which I've divided into four categories.

If you're interested in playing around with them, I'd suggest the "polysemantic" tokens listed below, which display the most variable behaviour. " ForCanBeConverted" is a good starter. Other good ones include " YYSTACK" and " JSBracketAccess". The "unspeakable" tokens can also elicit some unusual responses.

Also just to mention that although the search was conducted using the free research preview of ChatGPT, additional testing confirms that these tokens also produce unusual behaviour on gpt-3.5-turbo on the OpenAI Playground, and on Microsoft's Bing AI.

Method

GPT-3.5 and GPT-4 use a different tokenizer than GPT-3, called cl100k_base. This tokenizer has a vocabulary of just over 100,000 tokens, in contrast to the tokenizer used by GPT-3, r50k_base, with a vocabulary of just over 50,000 tokens. Additionally, while the weights for the r50k_base tokenizer are publicly available, this is not the case for cl100k_base. This is significant because the method used by Jessica Rumbelow and mwatkins to identify potential anomalous tokens required access to the weights, and as such the same technique cannot be used at present on cl100k_base.

Since this required a manual search, I decided early on to restrict the search to a smaller range of tokens, at least for this initial search. A comment by Slimepriestess on AdamYedidia's post noted that the " petertodd" token was about 3/4 of the way through the list of tokens on the 50k model, and given how interesting that token is, I decided to restrict the search to tokens 70,000 – 97,999 in the hope of finding an interesting selection within that range.

To get the full list of tokens for cl100k_base, I downloaded the "cl100k_base.tiktoken.txt" from https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken found on the tiktoken GitHub repository, and wrote a short python script to generate prompts consisting of lists of tokens. To prevent the possibility of the tokenizer selecting the wrong token in the prompt, each token was preceded by and succeeded b...
How to Interpret Prediction Market Prices as Probabilities
by SimonM — 2023-05-09T14:12:27.394Z

Or "how to account for fees"

Over at the EA Forum, bob wrote an article "Don't Interpret Prediction Market Prices as Probabilities".

In his post, he makes 3[1] cases against prediction market prices generating "fair" prices.

Opportunity cost - an incorrect price may be insufficiently incorrect for traders to put right because the reward isn't as appealing as alternative trades
Individual Risk-aversion - an incorrect price may be insufficiently incorrect for traders to put right because the risk of loss doesn't compensate for the potential gain
Correlation with the real world - an incorrect price may be insufficiently incorrect for traders to put right because in the world where they are wrong, it is especially unpleasant for them

I believe that all three of these are reasonable complaints. I also believe only one of them causes prediction market prices to be materially "wrong" after accounting for some simple adjustments.

Specifically, I think that the market outcomes correlating with the real world does lead to non-fair prices, and I discussed this at length here.

For the other 2 cases, I do not think the effect on prediction markets enough to stop treating prices as ~probabilities. What it means is we have to be more careful about how we interpret prices.

Risk Aversion

In general[2], fair bets are unappealing to most people. A certain $50 is more appealing to me that a 50-50 shot at $100. This means that the market price of a coin flip might deviate from 50% if no-one is willing to risk $ to put it right. However, as long as the stakes are small enough, it doesn't need to deviate far before someone is willing to start to put the price correctly. In general, the stakes available in prediction markets tend to be small. In my experience, the largest mispricings have generally come where large quantities of money all appear in the same market and there isn't enough liquidity from "smart" money to take the other side. (eg Mayweather-McGregor). This happens infrequently, especially in prediction markets where the $ involved is generally quite small. The volumes involved in prediction markets are generally small enough relative to the wealth of smart money, that I think it is reasonably easy to ignore this issue in the short term.

Opportunity Cost

When placing a bet on a prediction markets there are a number of costs associated with that trade:

Time to price the market
Fe

...
Stampy's AI Safety Info - New Distillations #2 [April 2023]
by markov — 2023-05-09T13:31:48.625Z
Link: https://aisafety.info/?state=8QZH_8222_7626_8EL6_7749_8XBK_8XV7_8PYW_89ZU_7729_6412_6920_8G1G_7580_8H0O_7632_7772_6350_8C7T_8HIA_8IZE_7612_8EL9_89ZQ_8PYV_

Hey! This is another update from the distillers at the AI Safety Info website (and its more playful clone Stampy).

Here are a couple of the answers that we wrote up over the last month (April 2023). As always let us know if there are any questions that you guys have that you would like to see answered.

The list below redirects to individual links, and the collective URL above renders all of the answers in the list on one page at once.

What is a subagent?

How could a superintelligent AI use the internet to take over the physical world?

Are there any AI alignment projects which governments could usefully put a very large amount of resources into?

What is deceptive alignment?

How does Redwood Research do adversarial training?

How does DeepMind do adversarial training?

What is outer alignment?

What is inner alignment?

What are ‘true names’ in the context of AI alignment?

Will there be a discontinuity in AI capabilities?

Isn't the real concern technological unemployment?

What can we expect the motivations of a superintelligent machine to be?

What is shard theory?

What are "pivotal acts"?

What is perverse instantiation?

Wouldn't a superintelligence be slowed down by the need to do experiments in the physical world?

Is the UN concerned about existential risk from AI?

What are some of the leading AI capabilities organizations?

What is "whole brain emulation"?

What are the "no free lunch" theorems?

What is feature visualization?

What are some AI governance exercises and projects I can try?

What is "metaphilosophy" and how does it relate to AI safety?

What is AI alignment?

Are there any detailed example stories of what unaligned AGI would look like?

What is a shoggoth?

Crossposted to the Effective Altruism Forum: https://forum.effectivealtruism.org/posts/FyfBJJJAZAdpfE9MX/stampy-s-ai-safety-info-new-distillations-2
Quote quiz answer
by jasoncrawford — 2023-05-09T13:27:34.750Z
Link: https://rootsofprogress.org/quote-quiz-answer

Here’s the answer to the recent quote quiz:

The author was Ted Kaczynski, aka the Unabomber. The quote was taken from his manifesto, “Industrial Society and Its Future.” Here’s a slightly longer, and unaltered, quote:

First let us postulate that the computer scientists succeed in developing intelligent machines that can do all things better than human beings can do them. In that case presumably all work will be done by vast, highly organized systems of machines and no human effort will be necessary. Either of two cases might occur. The machines might be permitted to make all of their own decisions without human oversight, or else human control over the machines might be retained. If the machines are permitted to make all their own decisions, we can’t make any conjectures as to the results, because it is impossible to guess how such machines might behave. We only point out that the fate of the human race would be at the mercy of the machines. It might be argued that the human race would never be foolish enough to hand over all power to the machines. But we are suggesting neither that the human race would voluntarily turn power over to the machines nor that the machines would willfully seize power. What we do suggest is that the human race might easily permit itself to drift into a position of such dependence on the machines that it would have no practical choice but to accept all of the machines’ decisions. As society and the problems that face it become more and more complex and as machines become more and more intelligent, people will let machines make more and more of their decisions for them, simply because machine-made decisions will bring better results than man-made ones. Eventually a stage may be reached at which the decisions necessary to keep the system running will be so complex that human beings will be incapable of making them intelligently. At that stage the machines will be in effective control. People won’t be able to just turn the machines off, because they will be so dependent on them that turning them off would amount to suicide.

All I did was replace the word “machines” with “AI”.

My point here is not to try to discredit this argument by associating it with a terrorist: I think we should evaluate ideas on their merits, apart from who held or espoused them. Rather, I’m interested in intellectual history, in the genealogy of ideas. I think it’s ...
Does reversible computation let you compute the complexity class PSPACE as efficiently as normal computers compute the complexity class P?
by Noosphere89 — 2023-05-09T13:18:09.025Z

Specifically, I am asking whether reversible computers let you implement PSPACE-complete algorithms to solve PSPACE-complete problems, and in particular, do so efficiently, ideally as efficient as normal computers compute the complexity class P.

I'm interested in this question because I've seen some sources saying that reversible computation can implement PSPACE algorithms while conventional computers can only implement algorithms in the complexity class P.

The sources I have are these:

https://www.lesswrong.com/posts/2BJBZh7Rvxr6GaLQs/negentropy-overrated#bHr5gobPhh5KLvxbA

and this Chegg source, which claims that reversible Turing Machines with a polynomial bound on space is equal to PSPACE.

https://www.chegg.com/homework-help/reversible-pspace-pspace-shown-problem-quantified-satisfiabi-chapter-3-problem-9p-solution-9781107002173-exc

I'd like any answer to this question, but ideally an answer would either show that reversible computation is able to implement PSPACE-complete algorithms as efficiently as normal computers implement algorithms in the P complexity class, or show that reversible computation can't do this, and show what complexity class can reversible computation efficiently implement algorithms.
EconTalk podcast: "Eliezer Yudkowsky on the Dangers of AI"
by TekhneMakre — 2023-05-09T11:14:27.173Z
Link: https://www.econtalk.org/eliezer-yudkowsky-on-the-dangers-of-ai/

Youtube link: https://www.youtube.com/watch?v=fZlZQCTqIEo

From the description: "Eliezer Yudkowsky insists that once artificial intelligence becomes smarter than people, everyone on earth will die. Listen as Yudkokwsky speaks with EconTalk's Russ Roberts on why we should be very, very afraid and why we're not prepared or able to manage the terrifying risks of AI."

Transcript (partial?): https://www.econtalk.org/eliezer-yudkowsky-on-the-dangers-of-ai/#audio-highlights
Most people should probably feel safe most of the time
by Kaj_Sotala — 2023-05-09T09:35:11.911Z

There is an idea that I’ve sometimes heard around rationalist and EA circles, that goes something like “you shouldn’t ever feel safe, because nobody is actually ever safe”. I think there are at least two major variations of this:

You shouldn’t ever feel safe, because something bad could happen at any time. To think otherwise is an error of rationality.
You shouldn’t ever feel safe, because AI timelines might be short and we might be about to die soon.[1] Thus, to think that you’re safe is making an error of rationality.

I’m going to argue against both of these. If you already feel like both of these are obviously wrong, you might not need the rest of this post.

Note that I only intend to dispute the intellectual argument that these are making. It’s possible to accept on an intellectual level that it would make sense to feel safe most of the time, but still not feel safe. That kind of emotional programming requires

different kinds of tools

to deal with. I’m mostly intending to say that if you feel safe, you don’t need to feel bad about that. You don’t need to make yourself feel unsafe; for most people, it’s perfectly rational to feel safe.

I do expect some of the potential readers of this post to live in a very unsafe environment - e.g. parts of current-day Ukraine, or if they live together with someone abusive - where they are actually in constant danger. For them, it may make sense to feel unsafe all the time. (If you are one of them, I genuinely hope things get better for you soon.) But these are clearly situations where something has gone badly wrong; the feeling that one has in those situations shouldn’t be something that one was actively striving for. I think that any reader who doesn’t live in an actively horrendous situation would do better to feel safe most of the time. (Short timelines don't count as a horrendous situation, for reasons that I'll get into.)

As I interpret it, the core logic in both of the “you shouldn’t ever feel safe” claims goes as follows:

To feel safe implies a belief that nothing bad is going to happen to you
But something bad can happen to you at any time, even when you don’t expect it. In the case of AI, we even have reasons to put a significant probability on this in fact happening soon.
Thus, feeling safe requires having an incorrect belief, and the rational course of action is to not

...
Summaries of top forum posts (1st to 7th May 2023)
by Zoe Williams — 2023-05-09T09:30:35.133Z

This is part of a weekly series summarizing the top posts on the EA and LW forums - you can see the full collection here. The first post includes some details on purpose and methodology. Feedback, thoughts, and corrections are welcomed.

If you'd like to receive these summaries via email, you can subscribe here.

Podcast version: Subscribe on your favorite podcast app by searching for 'EA Forum Podcast (Summaries)'. A big thanks to Coleman Snell for producing these!

Philosophy and Methodologies

Who regulates the regulators? We need to go beyond the review-and-approval paradigm

by jasoncrawford

Linkpost for

this blog post

.

Institutional Review Boards (IRBs) were put in place to review the ethics of medical trials, and initially worked well. However, after a study participant's death, they became more stringent and over-reached (eg. requiring heart attack study participants to read and sign long consent forms during a heart attack). A similar pattern occurred with the FDA, NEPA and NRC. This is due to lopsided incentives - regulators are blamed for anything that goes wrong, but neither blamed nor rewarded for how much they slow down or speed up progress. It’s also harder to remove regulations than to add them. The same pattern can be seen as corporations grow eg. Google is now very risk-averse and can require 15+ approvals for minor changes.

The author believes this is evidence the review-and-approval model is broken, and we need better ways to mitigate risk and create safety (eg. liability laws).

How much do you believe your results?

by Eric Neyman

The performance of an intervention in a trial / study is a combination of its actual effect and random noise. This means when comparing multiple interventions, you should expect the top-performing ones to be a combination of good and lucky, and therefore discount for the luck portion (eg. if it estimates 4 lives per $X, you might expect 2). The author suggests keeping this in mind when considering a study, and working hard to reduce the noise in your measurements when conducting one (eg. by increasing sample size).

A top comment by Karthik Tadepalli notes these results depend on if the true spread of intervention quality is the same order of magnitude as the spread of experimental noise. In cases of fat-tailed distributions of intervention quality, the latter becomes negligible and we don’t need to di...
Focusing on longevity research as a way to avoid the AI apocalypse
by Random Trader — 2023-05-09T04:47:18.231Z

I don't want to die.

You don't want to die.

The people who are potentially going to get us killed by pushing AI capability research don't want to die.

Our most basic goals have been aligned all this time, so it's really tragic that we are in the situation we're in right now. How did this happen?

First of all, it is a fact that people have differing opinions on how likely it is that AI will kill us all. Some think the probability is 99%+, and some others think it's something like 10%. But even the most optimistic capability researchers acknowledge that it's not 0.

If it's not 0, why are they taking the risk? Sure, there are huge economic incentives, but I think there's a deeper root cause.

Let me quote Tim Urban from Wait but Why in The AI Revolution: Our Immortality or Extinction:

"If ASI really does happen this century, and if the outcome of that is really as extreme—and permanent—as most experts think it will be, we have an enormous responsibility on our shoulders. The next million+ years of human lives are all quietly looking at us, hoping as hard as they can hope that we don’t mess this up. We have a chance to be the humans that gave all future humans the gift of life, and maybe even the gift of painless, everlasting life. Or we’ll be the people responsible for blowing it—for letting this incredibly special species, with its music and its art, its curiosity and its laughter, its endless discoveries and inventions, come to a sad and unceremonious end.

When I’m thinking about these things, the only thing I want is for us to take our time and be incredibly cautious about AI. Nothing in existence is as important as getting this right—no matter how long we need to spend in order to do so.

But thennnnnn

I think about not dying.

Not. Dying.

And then I might consider that humanity’s music and art is good, but it’s not that good, and a lot of it is actually just bad. And a lot of people’s laughter is annoying, and those millions of future people aren’t actually hoping for anything because they don’t exist. And maybe we don’t need to be over-the-top cautious, since who really wants to do that?

Cause what a massive bummer if humans figure out how to cure death right after I die."

I think this is it. People are terrified of death, and they want to get AGI and ASI as soon as possible cause they have a "nothing to lose" mindset. Aging is go...
When is Goodhart catastrophic?
by Drake Thomas — 2023-05-09T03:59:16.043Z

Thanks to Aryan Bhatt, Eric Neyman, and Vivek Hebbar for feedback.

This post gets more math-heavy over time; we convey some intuitions and overall takeaways first, and then get more detailed. Read for as long as you're getting value out of things!

TLDR

How much should you optimize for a flawed measurement? If you model optimization as selecting for high values of your goal V plus an independent error X, then the answer ends up being very sensitive to the distribution of the error X: if it’s heavy-tailed you shouldn't optimize too hard, but if it’s light-tailed you can go full speed ahead.

Related work

Why the tails come apart by Thrasymachus discusses a sort of "weak Goodhart" effect, where extremal proxy measurements won't have extremal values of your goal (even if they're still pretty good). It implicitly looks at cases similar to a normal distribution.

Scott Garrabrant's taxonomy of Goodhart's Law discusses several ways that the law can manifest. This post is about the "Regressional Goodhart" case.

Scaling Laws for Reward Model Overoptimization (Gao et al., 2022) considers very similar conditioning dynamics in real-world RLHF reward models. In their Appendix A, they show a special case of this phenomenon for light-tailed error, which we'll prove a generalization of in the next post.

Defining and Characterizing Reward Hacking (Skalse et al., 2022) shows that under certain conditions, leaving any terms out of a reward function makes it possible to increase expected proxy return while decreasing expected true return.

How much do you believe your results? by Eric Neyman tackles very similar phenomena to the ones discussed here, particularly in section IV; in this post we're interested in characterizing that sort of behavior and when it occurs. We strongly recommend reading it first if you'd like better intuitions behind some of the math presented here - though our post was written independently, it's something of a sequel to Eric's.

An Arbital page defines Goodhart's Curse and notes

The exact conditions for Goodhart's Curse applying between V and a point estimate or probability distribution over U [a proxy measure that an AI is optimizing], have not yet been written out in a convincing way.

To the extent this post adopts a reasonable frame, we think it makes progress towards this goal.

Motivation/intuition

Goodhart's Law says

When a m

...
Chilean AIS Hackathon Retrospective
by agucova — 2023-05-09T01:34:27.462Z

TL;DR

We hosted an AI Safety “Thinkathon” in Chile. We had participation from 40 students with differing skill levels and backgrounds, with groups totalling 13 submissions.
We see potential in:

Similar introductory events aiming for a broad audience
Collaborating more often with student organizations
Leveraging remote help from external mentors

We experimented with an alternative naming, having remote mentors, different problem sources, and student organization partnerships, with varying results.
We could have improved planning and communicating the difficulty of challenges.

Introduction

In February, we ran the first AI Safety Hackathon in Chile (and possibly in all of South America[1]). This post provides some details about the event, a teaser of some resulting projects and our learnings throughout.

A team working on their proposal during the second day.

Goals and overview of the event

The hackathon was meant to kick-start our nascent AI Safety Group at UC Chile, generating interest in AI Safety and encouraging people to register for our AGI Safety Fundamentals course group.

It ran between the 25th and the 28th of February, the first two days being in-person events and the other two serving as additional time for participants to work on their proposals, with some remote assistance on our part. Participants formed teams of up to four people, and could choose to assist either virtually (through Discord) or in-person (on the first two days).

We had help from Apart Research and partial funding from AI Alignment Awards.

Things we experimented with

Aiming for a broad audience, we named the event “Thinkathon” (instead of hackathon) and provided plenty of introductory material alongside the proposed problems.

We think this was the right choice, as the desired effect was reflected on the participant demographics (see below).
We could have been better at preparing participants. Some participants suggested we could have done an introductory workshop.

We incorporated the two problems from the AI Alignment Awards (Goal Misgeneralization and the Shutdown problem), alongside easier, self-contained problems aimed at students with different backgrounds (like policy or psychology).

We think most teams weren't prepared to tackle the AI Alignment Awards challenges. Most teams (77%) chose them initially regardless of their experience, getting stuck quickly.
This might have worked

...