trentmkelly/LessWrong-43k huggingface dataset:
EA & LW Forums Weekly Summary (28th Nov - 4th Dec 22')
by Zoe Williams — 2022-12-06T09:38:16.552Z

Supported by Rethink Priorities

This is part of a weekly series summarizing the top posts on the EA and LW forums - you can see the full collection here. The first post includes some details on purpose and methodology. Feedback, thoughts, and corrections are welcomed.

If you'd like to receive these summaries via email, you can subscribe here.

Podcast version: prefer your summaries in podcast form? A big thanks to Coleman Snell for producing these! Subscribe on your favorite podcast app by searching for 'EA Forum Podcast (Summaries)'. Note this is a new feed, part of a coordinated effort to create more narrated EA content - check out the announcement post here.
Author's note: FTX-related posts can be found in their own section. There's also a temporary new section for 'Giving Recommendations and Year-End Posts' as we hit giving season, and many organizations reflect on the year just been.

Top / Curated Readings

Designed for those without the time to read all the summaries. Everything here is also within the relevant sections later on so feel free to skip if you’re planning to read it all. These are picked by the summaries’ author and don’t reflect the forum ‘curated’ section.

Announcing FTX Community Response Survey

by Conor McGurk, WillemSleegers, David_Moss

Author’s TL;DR: Let us know how you are feeling about EA post the FTX crisis by

filling out the EA survey

. If you’ve already responded to the EA survey, you can take the

extra questionnaire here

.

The LessWrong 2021 Review (Intellectual Circle Expansion)

by Ruby, Raemon

The Annual LessWrong review is starting on 1st December. This involves nominating and voting on the best posts of 2021. For the first time, this year is also allowing off-site content to be nominated. If you have an account registered before Jan 2021, you can participate by casting preliminary votes until 14th December - a UI for it will show up at the top of all 2021 posts.

Physical books with the winners of the 2020 review will likely be ready by January.

Why Neuron Counts Shouldn't Be Used as Proxies for Moral Weight

by Adam Shriver

The fourth post in the

Moral Weight Project Sequence

.
Neuron counts are sometimes proposed as a proxy for moral weight due to relation to intelligence, an argument they result in “more valenced consciousness”, or being required to reach minimal information capacity for morally relevant ...
Take 5: Another problem for natural abstractions is laziness.
by Charlie Steiner — 2022-12-06T07:00:48.626Z

As a writing exercise, I'm writing an AI Alignment Hot Take Advent Calendar - one new hot take, written every day for 25 days. Or until I run out of hot takes.

Soundtrack.

Natural abstractions are patterns in the environment that are so convenient and so useful that most right-thinking agents will learn to take advantage of them. But what if humans and modern ML are too lazy to be right-thinking?

One way of framing this point is in terms of gradient starvation. The reason neural networks don't explore all possible abstractions (aside from the expense) is that once they find the first way of solving a problem, they don't really have an incentive to find a second way - it doesn't give them a higher score, so they don't. When gradient starvation is strong, it means the loss landscape has a lot of local minima that the agent can roll into, that aren't easily connected to the global minimum, and so what abstractions the network ends up using will depend strongly on the initial conditions.

Regularization and exploration can help ameliorate this problem, but often come with catastrophic forgetting - if a neural net finds a strictly better way to solve the problem it's faced with, it might forget all about the previous way. When we imagine a right-thinking agent that learns natural abstractions, we often imagine something that's intrinsically motivated to learn lots of different ways of solving a problem, and that doesn't erase its memory of interesting methods just because they're not on the Pareto frontier.

So that's what I mean by "lazy"/"not lazy", here. Neural networks, or humans, are lazy if they're parochial in solution-space, doing local search in a way that sees them get stuck in what a less-lazy optimizer might consider to be ruts.

It's important to note that laziness is not an unambiguously bad property. First, it's usually more efficient. Second, maybe we don't want our neural net to actually search through the weird and adversarial parts of parameter-space, and local search prevents it from doing so. Alex Turner et al. have recently been making arguments like this fairly forcefully. Still, we don't want maximal laziness, especially not if we want to find natural abstractions like the various meanings of "human values."

I might be attacking a strawman or a bailey here, I'm not totally sure. I've been using "natural abstraction" here as if it just means an abstr...
Verification Is Not Easier Than Generation In General
by johnswentworth — 2022-12-06T05:20:48.744Z

People who’ve spent a lot of time thinking about P vs NP often have the intuition that “verification is easier than generation”. It’s easier to verify a solution to some equations than to find a solution. It’s easier to verify a password than to guess it. That sort of thing. The claim that it is easier to verify solutions to such problems than to generate them is essentially the claim that P ≠ NP, a conjecture which is widely believed to be true. Thus the intuition that verification is generally easier than generation.

The problem is, this intuition comes from thinking about problems which are in NP. NP is, roughly speaking, the class of algorithmic problems for which solutions are easy to verify. Verifying the solution to some equations is easy, so that problem is in NP.

I think a more accurate takeaway would be that among problems in NP, verification is easier than generation. In other words, among problems for which verification is easy, verification is easier than generation. Rather a less impressive claim, when you put it like that.

With that in mind, here is an algorithmic problem for which generation is easier than verification.

Predicate: given a program, does it halt?

Generation problem: generate a program which halts.

Verification problem: given a program, verify that it halts.

The generation problem is trivial. The verification problem is uncomputable.

That’s it for the post, you all can argue about the application to alignment in the comment section.
Shh, don't tell the AI it's likely to be evil
by naterush — 2022-12-06T03:35:09.764Z

If you haven't been online in the past week: ChatGPT is OpenAI's new fine-tuned language model that is really much better at doing all sorts things that you'd expect. Here's it debugging some code, or writing awful-sounding Seinfeld episodes, or (my favorite) writing a verse from the King James Bible about taking a PB sandwich out of a VCR.

My fairly-basic understanding of large language models is that they are trying to predict the most-likely-next token in a sequence. So if you start a sentence with "I love peanut butter and..." the large language model will finish it with "jelly." It's clear that the relationships be captured by "most-likely-token" are way more complex and interesting that one might initially guess (see: getting a PB sandwich out of a VCR above).

Given that LLMs are trained on the public internet, I'm wondering if we should be thinking hard about what sorts of sequences we are exposing these models to.

Imagine OpenAI of the future goes to fine-tune one of their models to be a "friendly and nice and happy chatbot that just wants to help humans solve their problems, while also being aligned + not evil." ChatGPT5 is born, and when given a question as a prompt, it will likely respond with something friend and nice and happy, etc.

But deep in the weights in ChatGPT5's network are a memory of the other "friendly and nice and happy chatbots" it has seen before: the AIs that we have been writing about since Asimov. And most of the AIs in our fiction have a tendency to work great at first, but eventually turn evil, and then try to kill everyone.

And so, as these are the only other chatbots that ChatGPT5 has learned from, this may become the most-likely sequence. Be a nice and friendly and happy chatbot - and then, after about 2 years of being friendly and visibly-aligned, go ham -- for no other reason than this is the most likely set of tokens it's seen.

Another way to ask this question that: would you rather ChatGPT had been trained on the dataset it was likely trained on (e.g. human vs. AI war), or one where every story about an AI was turned to one of solar punk abundance and happy utopia?

To me, the answer is pretty clear.

The solution, less so. I wonder if I should devote more time to writing aligned-AI fiction and putting on the internet where it can be found by OpenAI's web scraper. Not by a human though -- all the fiction I write is go...
What are the major underlying divisions in AI safety?
by Chris_Leong — 2022-12-06T03:28:02.694Z

I’ve recently been thinking about how different researchers have wildly different conceptions of what needs to be done in order to solve alignment and what projects are net-positive.

I started making a list of core divisions;

Empiricism vs conceptual research: Which is more valuable or do we need both?

Take-off speeds: How fast will it be?

Ultimate capability level: What level of intelligence will AI’s reach? How much of an advantage does this provide them?

Offense-defense balance: Which has the advantage?

Capabilities externalities: How bad are these?

Are there any obvious ones that I’ve missed?
The No Free Lunch theorem for dummies
by Steven Byrnes — 2022-12-05T21:46:25.950Z

(I wrote this little explainer when drafting Response to Blake Richards: AGI, generality, alignment, & loss functions a few months ago, but it got cut from the final version. I figure it kinda stands alone, and maybe people will find it to be helpful pedagogy, because I recall being confused about this topic for quite a while when I first came across it a few years ago. Also, there’s some chance that I’m

still

confused about NFL, and if so, someone can correct me in the comments! Anyway, enjoy!)

What is the No Free Lunch (NFL) theorem?

Suppose there’s a function F that maps each of a finite number of inputs {i1,…,iN} to one of a finite number of outputs {o1,…,oM}. You have no idea what this function is; literally any possible function, out of the MN possibilities, is equally likely, as far as you know.

In fact, I generated the function F by secretly rolling an M-sided die N times. If I got a "9" on my first die-roll, then F(i1)=o9, and so on.

Now, you and me have a conversation about this unknown-to-you function F:

Me: “Pop quiz: What is F(i3)?”
You: “How should I know? You just told me that any of the MN possible F’s are equally likely. So F(i3) is equally likely to be any of the M possible outputs {o1,…,oM}.”
Me: “OK, I'll give you a hint: F(i1)=o9. Again, what is F(i3)?”
You: “WTF kind of hint is that?? That doesn’t help me at all! I still have no idea. It’s still equally likely to be anything!!”
Me: “OK, here’s another hint: F(i2)=o8. Again, what is F(i3)?”
You: “Stop being annoying. That’s not a real hint. You still haven’t given me any useful information. Based on what you’ve told me, there are MN−2 possible F’s, and they’re all equally likely, and they’re divided evenly amongst the M possible values of F(i3).”

That's the essence of the No Free Lunch Theorem (or maybe family of theorems?), as far as I understand it. It sounds dumb, and honestly it is kind of dumb. But you can describe it in a way that makes it sound very profound:

There's no such thing as “learning” or “figuring things out”

in general

; any algorithm that successfully does so on some problems will do catastrophically badly on other problems.”

In other words, if you’ve observed F(i1),F(i2),…,F(i1000), that information ...
ChatGPT and Ideological Turing Test
by Viliam — 2022-12-05T21:45:49.529Z

It seems to me that ChatGPT should be able to pass the Ideological Turing Test -- to generate a convincingly looking argument for any side of an issue. It is obvious how to use this ability for evil purposes: write the bottom line, and generate the arguments.

I think this can also be used for good purposes, and I invite you to brainstorm how. Here are some of my ideas:

Generate a heresy for yourself. People sometimes pretend to think critically about their own beliefs, and sometimes they kinda honestly try, but they still unconsciously avoid the weak points of their beliefs. Instead of generating a heretical thought that you know you can easily debunk, use ChatGPT to generate counter-arguments to your beliefs.

Prepare yourself for a debate on certain topic, by asking ChatGPT to generate arguments against your position. This might give you a decent idea of what arguments you should expect to meet in the real debate.

Perhaps teachers could use it similarly, to find out the most frequent misconceptions about the topic they are going to teach, and then they could adjust their lessons accordingly.

I wonder if listening to (ChatGPT-generated) "both sides of the story" could increase your chance to make the right guess. The experiment could be designed like this: An expert in certain field chooses a question that (according to the expert's knowledge) has a known correct answer, but most people are not familiar with it. Then someone else asks ChatGPT to create convincing arguments for both sides. The participants are randomly divided into two groups. First group only hears the question, and then tries to guess the right answer. Second group hears the question, then reads the arguments for both sides, and then tries to guess the right answer. Will the second group be more successful on average? If the answer is "yes", this may be a useful way to figure out the truth about questions where most people are wrong (and thus ChatGPT might mislead you if you ask it to only provide the single best answer).

But even if it does not increase your chance of guessing correctly, perhaps teachers could use this to create the feeling of curiosity in their students. Before the lesson, give students the generated arguments for both sides. During the lesson, explain how it actually works, and provide explanations why the other side is wrong. This may prevent hindsight bias (students concludi...
ChatGPT on Spielberg’s A.I. and AI Alignment
by Bill Benzon — 2022-12-05T21:10:20.551Z

A remark by MarcusOfCitium over at Astral Codex Ten prompted me to think about the relationship between John Bowlby’s concept of attachment, Steven Spielberg’s movie, AI: Artificial Intelligence, and the problem of AI Alignment. I decided ask ChatGPT what they thought about the connection. Here’s the whole chat.

Tell me about Spielberg’s movie, AI: Artificial Intelligence.

AI: Artificial Intelligence is a science fiction film directed by Steven Spielberg. The movie is set in a future where robots known as "mechas" have become a part of everyday life. The story follows a mecha named David, who is designed to be a child's companion. After his owner, a young boy named Martin, falls into a coma, David is abandoned and sets out on a journey to become a real boy. The film explores themes of love, humanity, and what it means to be alive.

* * 2nd try

AI: Artificial Intelligence is a 2001 science fiction drama film directed by Steven Spielberg. The film explores the idea of creating a humanoid robot with the ability to love, and the moral and ethical implications of such technology. The story follows a young robot boy named David, who is programmed to love his human mother unconditionally. When his mother falls ill and can no longer take care of him, David sets out on a journey to find a way to become a real boy and win her love. The film explores themes of love, family, and the definition of humanity. It stars Haley Joel Osment as David, Jude Law as Gigolo Joe, and Frances O'Connor as Monica Swinton.

How is David programmed to love Monica?

David is programmed to love Monica unconditionally as his mother. He is programmed to be devoted to her and to always put her needs and desires above his own. David's love for Monica is not based on personal feelings or emotional attachment, but rather on his programming to prioritize her happiness and well-being.

Is there a special moment in the movie where David first loves Monica?

There is not a specific moment in the movie where David first loves Monica. As a robot, David's love for Monica is not based on personal feelings or emotional attachment, but rather on his programming to prioritize her happiness and well-being. This love is present from the moment he is activated and begins interacting with Monica.

In this context, what is bonding?

In the context of AI: Artificial Intelligence, bonding refers to the emotional attachment and con...
Updating my AI timelines
by Matthew Barnett — 2022-12-05T20:46:28.161Z

Earlier this year I offered to bet people who had short AI timelines.

While it wasn't my intention to be known as "a long AI timelines guy", I have begun feeling that was how people perceived me. Nonetheless, in the last few months, I've modified my views substantially. Thus, I offer this short post, which can hopefully make my current position more clear.

There are several reasons for my update towards shorter AI timelines, though each reason is relatively straightforward and uncomplicated. In the spirit of writing something short rather than not writing something at all, my explanations here will be brief, although I may be willing to elaborate in a comment below.

In order, these reasons included, but were not limited to,

I became convinced that the barriers to language models adopting human-level reasoning were much weaker than I had believed. Previously, I had imagined that it would be difficult to get a language model to perform reliable reasoning over long sequences, in which each step in the sequence requires making a non-trivial inference, and one mistake in understanding the sequence can make the difference between a coherent and incoherent response.
Yet, my personal experience with language models, including but not limited to ChatGPT, has persuaded me to that this type of problem is not a strong barrier, and is closer in difficulty to other challenges like "summarize a document" or "understanding what's going on in a plot" which I had already thought language models were making good progress on. In hindsight, I should have perhaps trusted the model I had constructed myself, which forecasted human-level language models by 2030. Note: I don't think this update reflects new major capabilities found in GPT-3.5, but rather my own prior state of ignorance.
I built a TAI timelines model, and after fitting the model, it came out with a median timeline of 2037. While I don't put a high degree of confidence in my model, or the parameters that I used, I believe it's still more reliable than my own intuition, which suggested much later dates were more plausible.
I reflected more on the possibility that short-term AI progress will accelerate AI progress.
I noticed that I had been underestimating the returns to scaling, and the possibility of large companies scaling their training budgets quickly to the $10B-$100B level. I am still unsure that this will happen within t

...
Steering Behaviour: Testing for (Non-)Myopia in Language Models
by Evan R. Murphy — 2022-12-05T20:28:33.025Z

Authors' Contributions: Both authors contributed equally to this project as a whole. Evan did the majority of implementation work, as well as the work for writing this post. Megan was more involved at the beginning of the project, and did the majority of experiment design. While Megan did give some input on the contents of this post, it doesn’t necessarily represent her views.

Acknowledgments: Thanks to the following people for insightful conversations which helped us improve this post: Ian McKenzie, Aidan O’Gara, Andrew McKnight and Evan Hubinger. One of the authors (Evan R. Murphy) was also supported by a grant from the Future Fund regranting program while working on this project.

Summary

Myopia is a theorised property of AI systems relating to their planning horizon capabilities. As has been recently discussed[1], myopia seems like an important property for AI safety, because non-myopia is likely a necessary precursor to highly risky emergent properties like deceptive alignment. We expect non-myopia in large language models (LLMs) receiving RLHF or similar fine-tuning because they are trained using multi-token completions rather than just immediate next token predictions.

We aren't aware of any previous public experiments testing specifically for myopia or non-myopia in machine learning models. We ran an initial experiment "Testing for steering behaviour in fine-tuned LLMs" which demonstrated noticeable ‘steering’ behaviour away from toxic content in the InstructGPT fine-tuned LLMs. We share key results from this experiment along with the full dataset of prompts we used in it. We also describe a follow-up experiment we're currently working on to determine the extent to which the steering we observed in the initial experiment is non-myopic.

Finally, we invite suggestions for future (non-)myopia experiments to run and share a few ideas of our own.

Context and motivation

What is myopia?

Myopia is a theorised property of some AI systems that has been discussed a fair amount on these forums. Rather than try to reinvent the wheel on defining it, we’ll borrow this explanation from the

myopia tag page on Alignment Forum

:

Myopia means short-sighted, particularly with respect to planning -- neglecting long-term consequences in favor of the short term. The extreme case, in which only immediate rewards are considered, is of particular interest. We can think of a my

...