trentmkelly/LessWrong-43k huggingface dataset:
What's the simplest concrete unsolved problem in AI alignment?
by agg — 2023-01-26T04:15:13.620Z

In your preferred area of AI alignment, what is the simplest concrete unsolved problem?

By "simplest", ideally the problem has been solved when any of the conditions are weakened. However, this isn't always possible, so a simpler solved version of the problem could also work (e.g., Goldbach's weak conjecture is known to be true.)

By "concrete", I mean something where given the statement of the problem and a proposed solution, a neutral third party would be able to consistently determine whether it's solved or not (e.g., not "explain [some theory] in a good way").
2022 Less Wrong Census/Survey: Request for Comments
by Screwtape — 2023-01-25T20:57:30.853Z

The first draft of the 2022 Less Wrong Census is complete, and viewable here. Please take a look at it! Please do not actually take it yet!

I have included all the standard questions such as age, country, and probability estimates on whether we live in a simulation. I have also included many of the bonus questions from previous years, and have in fact compiled a list of the questions from every previous survey I could freely find. I intend to take the Do Not Take This Survey Yet warning off of and make another post asking people to take it on Monday the 30th of January.

Yes, this is a bit later than usual. I'm comfortable counting people answering this in late January or early February as part of the previous year.

Three things I would like from you:

First, please look over this draft. Let me know if any questions are poorly phrased or pointless. Tell me if there are any dumb typos or copy and paste errors.

Second, if you work for Less Wrong or CFAR, I didn't add any of your questions. (Ben Pace in particular had a very lovely document full of possible questions, all of which I read and very few of which I used.) If you want me to add some questions, let me know and I'm happy to add entire sections for you! If you don't work for Less Wrong or CFAR, you can also suggest additions and I'll add as many as make sense to the bonus section.

Third, the data links for many previous surveys are now dead links. If you have a copy of the public data for the 2009, 2011, 2012, 2013, 2017, or 2020 Less Wrong surveys, please pass them on to me (or just publish a working link in the comments below.) As far as I know no Less Wrong Census was run in 2010, 2015, 2018, 2019, or 2021. Yes, the only Less Wrong data set I've recovered was 2014 and 2016. If I can get copies of more of them, I intend to bundle them together into one .zip file that can be backed up in lots of places.

Edit: Census post is up here.
Next steps after AGISF at UMich
by JakubK — 2023-01-25T20:57:19.460Z
Link: https://docs.google.com/document/d/1s9aPA4dKfj5Qj8ZE-1dGThnNSGyNlr3hmMDgzeMeKjw/edit#

I ran an AGISF (technical alignment track) program at the University of Michigan last semester. At the end of the program, I shared a "next steps" document full of ideas for actions to take after finishing the program, and I've refined the document into a public version that you can find here. Below are some relevant parts for people in a similar position.

Getting Feedback

I encourage you to

apply for advising at

80,000 Hours

(80k), a career advising service for people who want to find career paths that best improve the world. 80k views

risks from advanced AI

as one of the world’s

most pressing problems

. The advisors aren’t professional AI researchers, but they know a lot about AI safety and can help you think through your plans.

One more option is

career coaching

from

AI Safety Support

.

Advice and guides

For general advice on technical alignment careers, including links to other career guides, see the

Careers in alignment

document (by

Richard Ngo

of OpenAI) from Week 7.

Career opportunities

The

AGI Safety Fundamentals Opportunities Board

is an excellent resource for tracking jobs, internships, and other programs.

Getting started with academic research

The

Student ML Safety Research Stipend Opportunity

provides stipends for students doing empirical ML safety research.

Personal projects and exercises

“All skill depends on practice. So too, with memory. If you want to remember something, you need to practice remembering it, not just looking at it.

Retrieval practice—where you shut the book and try to recall what you’ve learned without looking at it—is one of the most effective studying techniques.

1

It beat fancier techniques like concept mapping in head-to-head trials.

2

Retrieval also works even if you can’t get feedback.

3

”

– Scott Young in

The 10 Essential Strategies for Deeper Learning

See

Technical AI safety exercises and projects

for a list of possible projects and exercises to try. Also consider working on a project with a friend!

Upskilling

You might be able to build skills through traditional routes like taking classes. I took

Deep Learning for Computer Vision

with Justin Johnson and it was the best EECS course I have ever taken. I highly recommend taking this class as your intro to machine learning.

You might also be able to build skills outside of school (e.g. with online courses, internships, textbooks,...
AGI will have learnt utility functions
by beren — 2023-01-25T19:42:11.139Z

This post is part of the work done at Conjecture.

Thanks to Eric Winsor, Daniel Braun, Andrea Miotti and Connor Leahy for helpful comments and feedback on the draft versions of this post.

There has been a lot of debate and discussion recently in the AI safety community about whether AGI will likely

optimize for fixed goals

or be a

wrapper mind

. The term wrapper mind is largely a restatement of the old idea of a utility maximizer, with AIXI as a canonical example. The fundamental idea is that there is an agent with some fixed utility function which it maximizes without any kind of feedback which can change its utility function.

Rather, the optimization process is assumed to be 'wrapped around' some core and unchanging utility function. The capabilities core of the agent is also totally modular and disjoint from the utility function such that arbitrary planners and utility functions can be composed so long as they have the right I/O interfaces[1]. The core 'code' of an AIXI like agent is incredibly simple and, for instance, could be implemented in this Python pseudocode [2]:

class WrapperMind():
...

def action_perception_loop():
while True:
observation = self.sensors.get_observation()
state = self.update_state(self.current_state, observation)
all_action_plans = self.generate_action_plans(state)
all_trajectories = self.world_model.generate_all_trajectories(all_action_plans, state)
optimal_plan, optimal_utility = self.evaluate_trajectories(all_trajectories)
self.execute(optimal_plan)

There's a couple of central elements to this architecture which must be included in any AIXI-like architecture. The AGI needs some sensorimotor equipment to both sense the world and execute its action plans. It needs a Bayesian filtering component to be able to update its representation of the world state given new observations and its current state. It needs a world model that can generate sets of action plans and then generate 'rollouts' which are simulations of likely futures given an action plan. Finally, it needs a utility function that can calculate ...
[RFC] Possible ways to expand on "Discovering Latent Knowledge in Language Models Without Supervision".
by gekaklam — 2023-01-25T19:03:16.218Z

Preface

We would like to thank the following people who contributed to the generation of ideas and provided feedback on this post: Alexandre Variengien, Daniel Filan, John Wentworth, Jonathan Claybrough, Jörn Stöhler, June Ku, Marius Hobbhahn, and Matt MacDermott.

We are a group of four who participate in

SERI ML Alignment Theory Scholars Program

under John Wentworth, and we are extending the paper “

Discovering Latent Knowledge in Language Models Without Supervision

" as we see it as an interesting direction where we could contribute concrete progress towards alignment.

We have a number of potential directions we could explore. The goal of this post is to get feedback on them early in order to prioritize better. This could be confirmations for the good ideas, reasons why certain ideas might be bad, references to existing similar attempts or relevant literature, potential failure modes, and generally (constructive) criticism of any kind.

Although we're all involved in all parts of the research process, the structure of our group and our main roles are as follows: Kaarel Hänni is our theorist and the main contributor of ideas; Walter Laurito and Kay Kozaronek are our experimentalists who focus mainly on deploying and running the code; Georgios Kaklamanos is our distiller and focuses on writing up and presenting the results (e.g., this post).

Brief summary of the DLK paper

In case you haven’t read the original paper, you can get a quick overview from the following

Twitter Thread from the author

(

alternative Threadreader link

). Here we’ll just mention a few key points relevant to the things we want to work on. The above figure from the paper outlines their process in a clear way.

The key insight of the paper is that coherence conditions satisfied by truth/falsehood can be leveraged to search for a model's internal representation of truth.

e.g., the question "Are cats mammals?" can be answered either "yes" or "no"
Thus, if we define as p(x+i) the probability that the statement xi is true, and p(x−i)the probability that statement xi is false, these two probabilities should sum to 1.

They introduce Contrast-Consistent Search (CCS), a method that learns a linear projection of the hidden states that is consistent across negations.

The input to CCS is a set of Yes-No questions (contrasting pairs) and access to a pre-trained m

...
Spreading messages to help with the most important century
by HoldenKarnofsky — 2023-01-25T18:20:07.322Z

Click lower right to download or find on Apple Podcasts, Spotify, Stitcher, etc.

In the most important century series, I argued that the 21st century could be the most important century ever for humanity, via the development of advanced AI systems that could dramatically speed up scientific and technological advancement, getting us more quickly than most people imagine to a deeply unfamiliar future.

In this more recent series, I’ve been trying to help answer this question: “So what? What can I do to help?”

So far, I’ve just been trying to build a picture of some of the major risks we might face (especially the risk of misaligned AI that could defeat all of humanity), what might be challenging about these risks, and why we might succeed anyway. Now I’ve finally gotten to the part where I can start laying out tangible ideas for how to help (beyond the pretty lame suggestions I gave before).

This piece is about one broad way to help: spreading messages that ought to be more widely understood.

One reason I think this topic is worth a whole piece is that practically everyone can help with spreading messages at least some, via things like talking to friends; writing explanations of your own that will appeal to particular people; and, yes, posting to Facebook and Twitter and all of that. Call it slacktivism if you want, but I’d guess it can be a big deal: many extremely important AI-related ideas are understood by vanishingly small numbers of people, and a bit more awareness could snowball. Especially because these topics often feel too “weird” for people to feel comfortable talking about them! Engaging in credible, reasonable ways could contribute to an overall background sense that it’s OK to take these ideas seriously.

And then there are a lot of potential readers who might have special opportunities to spread messages. Maybe they are professional communicators (journalists, bloggers, TV writers, novelists, TikTokers, etc.), maybe they’re non-professionals who still have sizable audiences (e.g., on Twitter), maybe they have unusual personal and professional networks, etc. Overall, the more you feel you are good at communicating with some important audience (even a small one), the more this post is for you.

That said, I’m not excited about blasting around hyper-simplified messages. As I hope this series has shown, the challenges that could lie ahead of us ...
My Model Of EA Burnout
by LoganStrohl — 2023-01-25T17:52:42.770Z

(Probably somebody else has said most of this. But I personally haven't read it, and felt like writing it down myself, so here we go.)

I think that EA [editor note: "Effective Altruism"] burnout usually results from prolonged dedication to satisfying the values you think you should have, while neglecting the values you actually have.

Setting aside for the moment what “values” are and what it means to “actually” have one, suppose that I actually value these things (among others):

True Values

Abundance
Power
Novelty
Social Harmony
Beauty
Growth
Comfort
The Wellbeing Of Others
Excitement
Personal Longevity
Accuracy

One day I learn about “global catastrophic risk”: Perhaps we’ll all die in a nuclear war, or an AI apocalypse, or a bioengineered global pandemic, and perhaps one of these things will happen quite soon.

I recognize that GCR is a direct threat to The Wellbeing Of Others and to Personal Longevity, and as I do, I get scared. I get scared in a way I have never been scared before, because I’ve never before taken seriously the possibility that everyone might die, leaving nobody to continue the species or even to remember that we ever existed—and because this new perspective on the future of humanity has caused my own personal mortality to hit me harder than the lingering perspective of my Christian upbringing ever allowed. For the first time in my life, I’m really aware that I, and everyone I will ever care about, may die.

My fear has me very focused on just two of my values: The Wellbeing Of Others and Personal Longevity. But as I read, think, and process, I realize that pretty much regardless of what my other values might be, they cannot possibly be satisfied if the entire species—or the planet, or the lightcone—is destroyed.

[This is, of course, a version of EA that’s especially focused on the far future; but I think it’s common for a very similar thing to happen when someone transitions from “soup kitchens” to “global poverty and animal welfare”. There’s an exponential increase in stakes, accompanied by a corresponding increase in the fear of lost value.]

So I reason that a new life strategy is called for.

Over time, under the influence of my “Accuracy” value as well as my “Social Harmony” value (since I’m now surrounded by people who are thinking about this stuff), I come to believe that I should value the following:

Should Values

Im

...
Thoughts on the impact of RLHF research
by paulfchristiano — 2023-01-25T17:23:16.402Z

In this post I’m going to describe my basic justification for working on RLHF in 2017-2020, which I still stand behind. I’ll discuss various arguments that RLHF research had an overall negative impact and explain why I don’t find them persuasive.

I'll also clarify that I don't think research on RLHF is automatically net positive; alignment research should address real alignment problems, and we should reject a vague association between "RLHF progress" and "alignment progress."

Background on my involvement in RLHF work

Here are some background views about alignment I held in 2015 and still hold today. I expect disagreements about RLHF will come down to disagreements about this background:

The simplest plausible strategies for alignment involve humans (maybe with the assistance of AI systems) evaluating a model’s actions based on how much we expect to like their consequences, and then training the models to produce highly-evaluated actions. (This is in contrast with, for example, trying to formally specify the human utility function, or notions of corrigibility / low-impact / etc, in some way.)
Simple versions of this approach are expected to run into difficulties, and potentially to be totally unworkable, because:

Evaluating consequences is hard.
A treacherous turn can cause trouble too quickly to detect or correct even if you are able to do so, and it’s challenging to evaluate treacherous turn probability at training time.

It’s very unclear if those issues are fatal before or after AI systems are powerful enough to completely transform human society (and in particular the state of AI alignment). Even if they are fatal, many of the approaches to resolving them still have the same basic structure of learning from expensive evaluations of actions.

In order to overcome the fundamental difficulties with RLHF, I have long been interested in techniques like iterated amplification and adversarial training. However, prior to 2017 most researchers I talked to in ML (and many researchers in alignment) thought that the basic strategy of training AI with expensive human evaluations was impractical for more boring reasons and so weren't interested in these difficulties. On top of that, we obviously weren’t able to actually implement anything more fancy than RLHF since all of these methods involve learning from expensive feedback. I worked on RLHF work to try to facilitate an...
Could AI be used to engineer a sociopolitical situation where humans can solve the problems surrounding AGI?
by hollowing — 2023-01-25T17:17:33.379Z

0

hi, i've been learning about alignment and am new to lesswrong. here's my question.
there seems to be a consensus here that AI couldn't be used to solve the problem of AI control per se. that said, is there any discussion or literature on whether a future AI might be able to generate a very impactful political strategy which, if enacted, would engineer a sociopolitical situation where humans have better prospects to solve the problems around AGI?
this question came to my mind in discussing how it seems that, in principle, there should be a way to string together words (and tone, body language, etc) to convince anyone of anything. likewise, it seems there are in principle sequences of actions which would change society/culture to any arbitrary state. however, most of these strategies are far outside the range of what a human could come up with; but a smarter AI might be able to come up with them, or in general have very intelligent ideas humans can't come up with, as Robert Miles helped illustrate to me in this video (https://youtu.be/L5pUA3LsEaw?t=359).
Progress links and tweets, 2023-01-25
by jasoncrawford — 2023-01-25T16:12:11.224Z
Link: https://rootsofprogress.org/links-and-tweets-2023-01-25

The Progress Forum

Tyler Cowen AMA is now done, read dozens of answers
Vitalik on science, his philanthropy, progress and effective altruism
On Eli Dourado’s “Heretical Thoughts on AI”

Announcements

AllSearch.ai: “Google Books on steroids” (@dwarkesh_sp)
Our World in Data is hiring a data scientist (via @MaxCRoser)

Links

Zvi on gas stoves: ruining Nice Things for “marginally better health”
A link from last week’s digest on quantum computing is probably bogus

Queries

What is the case that ~4° C of warming by 2100 will be far worse for the world?
What is the earliest technology where most users had no idea how it worked?
Why is the learning curve on corn linear and not exponential?

Quotes

The insane power potential of nanotech motors. See also the intro to Nanosystems
“Don’t send that railway through our town! … Wait, build us a branch line!”
There are no “natural” resources
The dose determines the poison

Tweets and retweets

My hypothesis about Solow’s computer productivity paradox
One blast furnace produces ~170x more iron than all of England in 1720
Stuart Buck’s grandfather hoped he would grow up to “get an indoors job”
Satellite photo of anywhere on Earth for as little as $175 ($7/km2!)
We had supersonic jets, lunar landers & nuclear reactors in the 1970s. We lost our way
A nuclear design has finally received NRC certification after 6 years
Skyscrapers used to be Art Deco and neo-Gothic, what happened? (@culturaltutor)
The Hardiman, one of the earliest exoskeleton designs
Visualisation of Probability Mass
by brook — 2023-01-25T15:09:40.974Z

This post is a tutorial for Visualisation of Probability Mass, a tool that lets you represent credences as percentages in a 10x10 grid. It's useful for turning visual intuitions into coherent percentages, or numerical intuitions into a visual representation.

At the end, I'll briefly mention other tools useful for visualising quantitative beliefs.

When can Visualisation of Probability Mass be Useful?

Visually demonstrate your expectation of how a project will turn out, or how likely it is to succeed on some metric, to show your manager
Turn a rough intuition about the harms and benefits of veganism into coherent[1] percentages
Compare your AI worries to a friends'
Guess how many people at a party are your friends vs your friends' friends vs your friends' friends' friends
Forecast the contents of your fridge by category (It's fine! I'm working, I promise! I have to get calibrated!)

Tutorial

Visualisation of Probability Mass is a tool for quickly representing images of percentages in a 10x10 grid, where each box represents 1%. You can click on any colour (on the right) and then on any tile (on the left) to change its colour.

You can add a title and description for each colour or for the graph in total, and export it as a jpeg.

Here's a worked example (numbers picked mostly at random, not intended as a true claim on this topic):

Personal Experience

Honestly, I find the visual output of Visualisation of Probability Mass more confusing than useful, though when I discussed it with others they found it quite useful. I expect that for some, most of the value is in using the visual half of the tool as an input to turn an intuition into percentages that are coherent[1], rather than as an output, unless you have <4 possible outcomes.

Tools for Related Tasks

Another tool which can be used to represent probability density intuitively is Click and Drag Probability Elicitation. However, the web demo is currently too limited to be useful, but I’m excited about its potential when fully usable. If you're building a survey (in Qualtrics, LimeSurvey, or oTree) where people can input probability density graphs, consider using it.

Tools that can be used to share quantitative beliefs as whole models include coding languages like Squiggle, Guesstimate, and Python.

Try it Yourself!

Show us your beliefs on a topic! Post it in the comments. If you can't think of an...