trentmkelly/LessWrong-43k huggingface dataset:
"Membranes" is better terminology than "boundaries" alone
by Chris Lakin — 2023-05-28T22:16:21.404Z

We've been thinking about the core concept behind «boundaries» extensively, and going forward we're going to stop using the term "boundaries" (alone) but use the term "«boundaries/membranes»" instead.

Summary: We've found that whenever try to use the word "boundaries" with others in lay speech, our explainees have had trouble. We think this is because the word "boundaries" in English is overloaded with different definitions, some of which directly conflict with our intended meaning. Critch suggested "membranes" as an alternative term in his first «boundaries» post, and we suggest using that instead.

Difficulties with using the word "boundaries"

The definition of "boundaries" used in common social situations (preferences) means something different than «boundaries/membranes»

We've found that the common use of the word "boundaries" is very confusingly related to the «boundaries/membranes» concept.

For example, if you hear someone say, "Hey, you yelled at me and that crosses my boundaries!", this is very not what «boundaries/membranes» is about.

Basically, everything in this post — which describes people setting social rules about how to interact with them according to their consent or preferences — is exactly what we don't mean by the technical term. Another example: this post is not about the "boundaries" that we mean: This post talks about "boundary" rules being variable, which we'd likely call "preferences"— meanwhile we think the concept at hand is is independent of any particular agent.

Basically: "crossing (boundaries : social requirements enforced by someone)" does not imply "crossing «membrane-boundaries»".

And that's annoying because every time we explain the concept to someone using the word "boundaries", we have to qualify, "it's called boundaries, but it's not about the normal kind of boundaries, but it's also not unrelated, but please just ignore that for now" lest they get understandably confused. Ugh.

(Also, this exact conflation delayed Chipmonk's understanding of «boundaries/membranes» for the entire month of December!)

Note: Critch and we seem to agree that the concept is distinct from preferences. He says so in his Part 3b:

That's because my goal is to treat boundaries as more fundamental than preferences, rather than as merely a feature of them.

The definition of "boundaries" used to mean inert separation means something different than

...
The king token
by p.b. — 2023-05-28T19:18:16.539Z

A while back I did a short foray into interpretability research analyzing the king token of my chess move predicting transformer. At the time the result seemed simultaneously unsurprising and disappointing, and I went back to work on more substantial matters, though revisiting this little project now, I am not sure why.

The chess move predicting transformer outputs a probability distribution over actions conditioned on a state. This is different enough from large language models to be interesting in its own right: Do the activations of a token representing a piece encode affordances? Typical tactical motives (checks, skewers, double attacks…)? Strategic structures (isolanis, passed pawns, Maroczy bind…)? Specific details of a class of positions (bishop pair, damaged king position, queenless middlegame … )? Higher level abstractions (king safety, initiative, zugzwang)?

A priori this is very unclear to me. The reason I started the analysis was to find out whether typical human concepts are encoded in the transformer representations. I looked specifically at the king token to see whether king safety is a represented concept. And, in a way it actually is. When I looked at the positions that led to the largest absolute values in the principal component, they all showed the king under attack and in check by heavy pieces.

I realized that this was profoundly unsurprising because the king in these positions absolutely had to move and therefore it made sense for the activations of the king token to be large. This fact alone did not mean any subtle or interesting concepts were encoded in the principal component of the king token.

Now, eight months later I decided to dig a little deeper. All the following analyses are done on the output tokens of the final transformer block right before these are fed into the dense connected layer that generates move predictions.

Steering away from king's moves

My earlier results did imply a steering possibility and indeed, dividing the king token by 10 generally drops the probability of a king’s move in positions with an initial probability above 10% to under 1%. Dividing by 100 drops the probability even more. Setting it directly to zero didn’t change much beyond that.

However, there were three kinds of exceptions, positions where the probability of a king’s move didn’t drop as much as in most cases and where th...
Language Agents Reduce the Risk of Existential Catastrophe
by cdkg — 2023-05-28T19:10:17.729Z

This post was written by Simon Goldstein, associate professor at the Dianoia Institute of Philosophy at ACU, and Cameron Domenico Kirk-Giannini, assistant professor at Rutgers University, for submission to the Open Philanthropy AI Worldviews Contest. Both authors are currently Philosophy Fellows at the Center for AI Safety.

Abstract: Recent advances in natural language processing have given rise to a new kind of AI architecture: the language agent. By repeatedly calling an LLM to perform a variety of cognitive tasks, language agents are able to function autonomously to pursue goals specified in natural language and stored in a human-readable format. Because of their architecture, language agents exhibit behavior that is predictable according to the laws of folk psychology: they have desires and beliefs, and then make and update plans to pursue their desires given their beliefs. We argue that the rise of language agents significantly reduces the probability of an existential catastrophe due to loss of control over an AGI. This is because the probability of such an existential catastrophe is proportional to the difficulty of aligning AGI systems, and language agents significantly reduce that difficulty. In particular, language agents help to resolve three important issues related to aligning AIs: reward misspecification, goal misgeneralization, and uninterpretability.

1. Misalignment and Existential Catastrophe

There is a significant chance that artificial general intelligence will be developed in the not-so-distant future — by 2070, for example. How likely is it that the advent of AGI will lead to an existential catastrophe for humanity? Here it is worth distinguishing between two possibilities: an existential catastrophe could result from humans losing control over an AGI system (call this a misalignment catastrophe), or an existential catastrophe could result from humans using an AGI system deliberately to bring that catastrophe about (call this a malicious actor catastrophe). In what follows, we are interested in assessing the probability of a misalignment catastrophe rather than a malicious actor catastrophe.

Carlsmith (2021) helpfully structures discussion of the probability of a misalignment catastrophe around six propositions. Since we are interested in the probability of a misalignment catastrophe conditional on the develo...
Devil's Advocate: Adverse Selection Against Conscientiousness 
by lionhearted (Sebastian Marshall) — 2023-05-28T17:53:37.968Z

Partially in reply to "Request: stop advancing AI capabilities".

Consider this argument:

"[X field] is often [dangerous / bad]. If you're [pro-social / thoughtful / conscientious], please avoid [X field]."

Let's grant that these points as true. X field is indeed bad or dangerous, people who are some mix of pro-social and thoughtful and conscientious will recognize that, and they are exhorted not to join that field.

What happens next, causally?

(1) If the field has a limited supply of people with relevant background to make contributions in the field, the result will be to slow progress in the field.

(2) If the field is below a "critical mass" of contributors to it, it might fail to make progress entirely.

(3) However, if the field does already have a critical mass of people making contributions,

the end result will be adverse selection against pro-sociality, thoughtfulness, and conscientiousness in the field.

I have a profound respect, at a distance, for MIRI and Soares. I think we actually agree about the dangers, and I agree that interpretability research (for instance) is a very important under-resourced problem that would benefit from contributors.

But I also come to one different conclusion — the more thoughtful, pro-social, and conscientious people working at Meta, OpenAI, Google, etc... the more likely things go well. I think there's a critical mass already in place for continued growth of machine learning capabilities on the path to AGI, and having people who care and are thoughtful in the room participating in their development is a good thing and not a bad thing. I would respectfully advocate the opposite position of Soares, and encourage thoughtful people to consider joining the strongest teams in the field and being a voice of reason in the room while shaping development.
Reacts now enabled on 100% of posts, though still just experimenting 
by Ruby — 2023-05-28T05:36:40.953Z

Update 3

We've just released our first draft of Inline Reacts. See this comment for more information. It's currently only enabled on this post, but if it seems to be working smoothly/intuitively we may roll it out for more posts.

Update 2

I've now enabled reacts by default on all new posts. I've updated that we need to get data more quickly, and at the current rate of new posts and comments, we probably wouldn't have a sample size to do interesting comparisons anyhow.

I expect the team to focus on Reacts for the next week, then let them sit for a longer period with some tweaks, and then make a call about whether or not to keep them (and with what design).

Update

We're progressing the reacts experiments to now have reacts enabled on 50% of more posts (by default). We hope to get more data this way and figure out if the reacts should be part of the site longterm (and how to make them good).
If you do/don't like having them on your post, you can change this in the edit settings (see below).

LessWrong is experimenting with the addition of reacts to the site, as per the recent experimental Open Thread. We are now progressing to the next stage of the experiment: trying out reacts in actual discussion threads.

The dev/moderator team will be proactively looking for posts to enable to react voting on (with author permission), but also any user can enable it themselves to help us experiment:

When creating or editing a post, expand the "Options" section at the bottom and change the Voting system to Names-attached reactions

The admins will also be on the lookout for good posts to enable reacts on (with author permission).

Iterating on the react palette

We're continuing to think about what reacts should be available. Thanks to everyone who's weighed in so far.

I just spent time today and yesterday adding a range of new reacts and sorting them, with a focus on reacts that express LessWrong-style conversational moves. Hover-overs tooltips have a longer description that you can view on the live thing (reacts are enabled on this post).

We'll be tracking which ones get used, but please comment here with further feedback and requests. In other places, it'd be cool if people said things like "wish I could react with X" for various values of X.

Different palette views

Raemon made it so you can change how you view the palette. See the options at the top there:

The options are:

List
Mixed

...
My AI Alignment Research Agenda and Threat Model, right now (May 2023)
by Nicholas / Heather Kross — 2023-05-28T03:23:38.353Z
Link: https://www.thinkingmuchbetter.com/nickai/conceptual/myagenda-may2023.html

TLDR

Fairly short timelines, mildly fast takeoffs, and medium-high uncertainties. --> Looking for abstractions to help cognition-steering and value-loading. --> Grasping at / reacting to related/scary/FOMO lines of research.

Threat Model

An AI system could be built that's far smarter than any human or small group of humans. This AI system could use its intelligence to defeat any non-motivation-directing safeguards, and gain control of the world and the future of humanity and other sentient life. Based on the orthogonality thesis, this amount of power would probably not, by default, be directed towards the best interests of humanity and other sentient life. Based on the idea of instrumental convergence, such an AI would destroy everything we value in its quest to fulfill its (dumb-by-default) original goal. This AI may require new insights to build, or could arise by "scaling up" existing ML architectures. This AI may "self-improve" its architecture, or it could get smarter through prosaic "hack more cloud computing power" techniques. In either case, it could start from a position of low capabilities and end up as the most powerful entity on Earth. This all could start within as few as 1.5 years from now, and will probably happen within 10 years, barring nuclear or other catastrophe.

The simplest solution to the above problem would be "don't build superhuman AGI, at least for the near future". However, superhuman AGI is likely to be built, on purpose or by accident, by any of a handful of groups with large amounts of computational resources and talented researchers. These groups are generally not monolithic, and contain leaders and employees who disagree (internally, with other orgs, and/or with me) about the best approach to AI alignment. (See the section "The AI Landscape" below).

Imagine if any of these groups got a box, today, that said "Input your alignment solution by USB drive, push button to get a superhuman AGI that runs on that, box expires in 1 week". According to my threat model, humanity is unlikely to survive longer than 1 week in this scenario. This is despite the wildly varying (often quite good!) alignment-motivations and security-mindsets of these groups. On my view, this is (mainly) because none of these groups has an adequate pre-prepared response for the below "Two Subproblems".

The Two Subproblems

I forgot where this advice came from...
Kelly betting vs expectation maximization 
by MorgneticField — 2023-05-28T01:54:51.819Z

People talk about Kelly betting and expectation maximization as though they're alternate strategies for the same problem. Actually, they're each the best option to pick for different classes of problems. Understanding when to use Kelly betting and when to use expectation maximization is critical.

Most of the ideas for this came from Ole Peters ergodicity economics writings. Any mistakes are my own.

The parable of the casino

Alice and Bob visit a casino together. They each have $100, and they decide it'll be fun to split up, play the first game they each find, and then see who has the most money. They'll then keep doing this until their time in the casino is up in a couple days.

Alice heads left and finds a game that looks good. It's double or nothing, and there's a 60% chance of winning. That sounds good to Alice. Players buy as many tickets as they want. Each ticket resolves independently from the others at the stated odds, but all resolve at the same time. The tickets are $1 each. How many should Alice buy?

Bob heads right and finds a different game. It's a similar double or nothing game with a 60% chance of winning. He has to buy a ticket to play, but in Bob's game he's only allowed to buy one ticket. He can pay however much he wants for it, then the double or nothing is against the amount he paid for his ticket. How much should he pay for a ticket?

Alice's game is optimized by an ensemble average

Let's estimate the amount of money Alice will win, as a function how many tickets she buys. We don't know how each ticket resolves, but we can say that approximately 60% of the tickets will be winners and 40% will be losers (though we don't know which tickets will be which). This is just calculating the expected value of the bet.

If she buys x tickets, she'll make 0.6∗x∗2+0.4∗x∗0 dollars. This is a linear function that monotonically increases with x, so Alice should buy as many as she can.

Since she has $100, she can buy 100 tickets. That means she will probably come away with $120. There will be some variance here. If tickets were cheaper (say only a penny each), then she could lower her variance buy buying more tickets at the lower price.

Bob's game is optimized by a time-average

Unlike Alice's game with a result for each ticket, there's only one result to Bob's game. He either doubles his ticket price or gets nothing back.

One way people tend to ...
Project Idea: Challenge Groups for Alignment Researchers
by Adam Zerner — 2023-05-27T20:10:12.001Z

@Henrik Karlsson writes a lot about what makes people successful. One recurring theme I've found involves the people around them. Successful people almost always have a group of friends who they're constantly chatting with, bouncing ideas around with, learning from, pushing each other, supporting each other, etc. A group who is similarly talented and driven.

Consider the example of MrBeast:

MrBeast: I somehow found these other four lunatics. Three of us were college dropouts. One was a high school dropout. And one, he just like quit his job. We were all super small YouTubers.

And we basically talked every day for a thousand days in a row and did nothing but hyper-study what makes a good video. [. . .] Some days I'd get on Skype at 7 am and I’d be in the call until 10 pm and then I’d go to bed. I wake up and I do it again.

He had summoned a scene.

MrBeast: We all had like 10-20,000 subscribers when we met. And by the time we stopped talking, we all had millions of subscribers. We all hit a million subscribers within a month of each other.

There was, in other words, little luck involved. They collectively extracted the know-how needed to consistently do what they wanted to do, which was to go viral.

Or consider musicians in Sweeden. Or intellectuals who grew up in exceptional milieus.

I like to think of this as a "challenge group". It seems like a pretty huge force multiplier.

I'm not sure about the extent to which alignment researchers currently lack challenge groups, but to the extent they do, helping form them seems like a pretty impactful project. It also strikes me as something that is, and is likely to remain, moderately neglected.

Similarly, you could also help form challenge groups in other areas: aspiring alignment researchers, cryonicists, EAs more generally, rationality dojos. But to increase the likelihood that new members are matched with existing members for whom they're compatible, it's probably a good idea to start off with a somewhat narrow group.

I have more thoughts on this that I'm struggling to formulate, so I'll leave it here for now. In general I think @Henrik Karlsson has a lot of great things to say on this topic.
Introspective Bayes
by False Name — 2023-05-27T19:35:52.856Z

An attempt to demonstrate a limiting condition of an optimal Bayesian agent, or a probabilistic agent in general.

This analysis relies on the description of such an optimal Bayesian agent (hereafter OBA), in Bostrom 2014, Boxes one and ten, and associated endnotes, with supplementary research on Kolmogorov’s axioms of probability, from Kolmogorov 1933, and a definition of Kolmogorov complexity. Are these sources inaccurate, the argument relying on them, naturally, cannot be guaranteed.

Sic: It is emphasized that the optimal Bayesian agent commences with a prior probability distribution, a function assigning a probability value to each possible world, the latter defined by their being describable as the output string of a given computer program enacted by a universal Turing machine (hereafter UTM).

We first examine the nature of the possible worlds. For “universal induction”, as the OBA is to perform, it seems the agent is operating according to the Solomonoff probability of the likelihood of the world’s existence, with respect to the length of the string, itself defined as string’s Kolmogorov complexity: a shorter string, a more likely world.

Solomonoff probability of string, so world, x, M(x):=

By definition of Solomonoff probability of possible worlds, the probability is defined for all programs Turing-computable, including those non-halting. Moreover, we appeal to Kolmogorov’s probability axioms, to observe that any prior probability function of the agent, that as the prior probability function has multiple values, this implies there is more than one possible world, though, one or many, the function is defined, so that: there is a definite set, or rather collection, of possible worlds under consideration. Let the existence of this collection, be point “alfa”.

(And in general “beta” explains why Bayes’ theorem is efficacious: that it can conditionalise is , else nothing; that its work will be done is known only ex post facto, so any case of , the lack of evidence of the operation of the agent, is excluded as to form).

It is curious now to note that there is a sense in which the simplest “possible world” consists entirely of the OBA, with its sensors “detecting” only the circumstances of , so that . It is in general supposed however, that the sensor’s percepts consist of an external world, by assumption subject to its prior probability distribution, that that par...
Should Rational Animations invite viewers to read content on LessWrong?
by Writer — 2023-05-27T19:26:47.606Z

When I introduced Rational Animations, I wrote:

What I won't do is aggressively advertise LessWrong and the EA Forum. If the channel succeeds, I will organize fundraisers for EA charities. If I adapt an article for YT, I will link it in the description or just credit the author. If I use quotes from an author on LW or the EA Forum, I will probably credit them on-screen. But I will never say: "Come on LW! Plenty of cool people there!" especially if the channel becomes big. Otherwise, "plenty of cool people" becomes Reddit pretty fast.

If the channel becomes big, I will also refrain from posting direct links to LW and the EA Forum. Remind me if I ever forget. And let me know if these rules are not conservative enough.

In my most recent communications with Open Phil, we discussed the fact that a YouTube video aimed at educating on a particular topic would be more effective if viewers had an easy way to fall into an "intellectual rabbit hole" to learn more. So a potential way to increase Rational Animations' impact is to increase the number of calls to action to read the sources of our videos. LessWrong and the EA Forum are good candidates for people to go learn more. Suppose we adapt an article from the sequences. One option is to link to readthesequences.com, another is to link to the version of the article hosted on LessWrong. It seems to me that linking to the version hosted on LessWrong would dramatically increase the chance of people falling into an "intellectual rabbit hole" and learning a lot more and eventually starting to contribute.
Here are what I think are the main cons and pros of explicitly inviting people to read stuff on LessWrong:

Cons:

If lots of people join at once we may get a brief period of lower-quality posts and comments until moderation catches up.
If we can't onboard people fast enough, the quality of what's in the site will become a lot lower over time, and people producing excellent content will be driven away.

Pro:

If people read LessWrong, they will engage with important topics such as reducing existential risk from AI. Eventually, some of them might be able to contribute in important ways, such as by doing alignment research.

I'm more optimistic now about trying to invite people than I was in 2021, mainly thanks to recent adjustments in moderation policy and solutions such as the Rejected Content Section, which, in...