trentmkelly/LessWrong-43k huggingface dataset:
What is the ground reality of countries taking steps to recalibrate AI development towards Alignment first?
by  — 2023-01-29T13:26:39.705Z

You can possibly put the lid on European AI research, the biggest sign to this is civilian administration oversight over the internet which is incredibly tight compared to European Union's non authoritarian government model where individual human will is a very little factor compared to institutional decisions. United States? You can decapitate AI, sure. Much more unlikely than Europe (And a Europe wide decapacitation campaign is already very unlikely for lawmakers to wrap their head around). The US with its current edge in development at least showed the most organic development regarding Alignment research. This should make you worried, other countries started from the already developed technological doctrines of America without (usually) adopting the logical doctrines that have led to this state of technology. China didn't need to have Von Neumann and Turing to get to the Baidu search engine.

What about Japan, Brazil, South Korea, India, Singapore and Taiwan? Players that don't cross your mind much but usually pass some of Western Europe in terms of behind the scene developments or atleast replication, does MIRI have any reach in Singapore? They sure seem to be very out of the hearing range for pleas coming from Eliezer while also developing parallel and much less safe models. Try to find a native alignment group in Asia, they weren't raised on 100 years of Science Fiction and Ex Machinas (except for Japan and Brazil). These nations will develop AGI reminiscent models with less than half of the care we expect from OpenAI. Most researchers do not expect corrigibility and alignment from OpenAI . What should follow is several "buts" and a "we are increasing global outreach", (well if Singularity is coming in 2029 that's a very tight schedule to from tentative influence in English Speaking world to entire globe) so lets say I accept all the preliminary arguments for why the asian tigers listen to you. Or will start listening to you even with the lack of breakthroughs regarding Alignment yet.

What about

Russia

? A country with a competitive defensive AI program focused on nuclear and autonomous killing systems, that's a yuck right? It's a black box you don't get insight into. Even the civilian wing of development usually puts effort to prevent you from interfering. Military wing on the other hand is directly aiming for "Allied Mastercomputer" (Such is the ca...
Compendium of problems with RLHF
by Charbel-Raphaël — 2023-01-29T11:40:53.147Z

Epistemic status: This post is a distillation of many comments/posts. I believe that my list of problems is not the best organization of sub-problems. I would like to make it shorter, and simpler, because cool theories are generally simple unified theories, by identifying only 2 or 3 main problems without aggregating problems with different types of gear level mechanisms, but currently I am too confused to be able to do so. Note that this post is not intended to address the potential

negative impact of RLHF research

on the world, but rather to identify the key technical gaps that need to be addressed for an effective alignment solution. Many thanks to Walter Laurito, Fabien Roger, Ben Hayum, Justis Mills for useful feedbacks.

RLHF tldr: We need a reward function, we cannot hand-write it, let’s make the AI learn it!

Problem 0: RLHF is confusing. Human judgment and feedback is so brittle, that even junior alignment researchers like me

thought

that RLHF is a not-too-bad-solution to the outer alignment problem. I think RLHF confuses a lot of people and distracts people from the core issues. Here is my attempt to become less confused.

Why RLHF counts as progress?

RLHF learned to backflip using around 900 individual bits of feedback from the human evaluator.

From

Learning from Human Preferences

.

Manual reward crafting : “By comparison, we took two hours to write our own reward function (the animation in the above right) to get a robot to backflip, and though it succeeds it’s a lot less elegant than the one trained simply through human feedback”

Without RLHF, approximating the reward function for a “good” backflip was tedious and almost impossible. With RLHF it is possible to obtain a reward function, that when optimized by an RL policy leads to beautiful backflips.

Value is complex

?

Value is fragile

? But being polite is complex, being polite is fragile, and we can implement this roughly in ChatGPT[1].
“RLHF

withstands

more optimization pressure than supervised fine-tuning: By using RLHF you can obtain reward functions, that withstand more optimization pressure, than traditionally hand-crafted reward functions. Insofar as that's a key metric we care about, it counts as progress.” [

Richard’s comment

]

Why RLHF is insufficient?

Buck

highlights

two main types problems with using RLHF to create an AGI: oversight issues and the po...
EA novel published on Amazon
by Timothy Underwood — 2023-01-29T08:33:35.853Z

Just so everyone knows, the novel I wrote to promote EA has been published on Amazon. This is not a request to purchase it, or try to directly support it in any particular way.

However if the blurb does sound interesting, go for it :)

https://www.amazon.com/dp/B0BSXHJRBQ
Reverse RSS Stats
by jefftk — 2023-01-29T03:40:01.216Z

In 2013 I built a weird feature
where clicking on "

◂◂ RSS"
at the top of a post gets you an RSS feed that runs in reverse from
that post, with a 'new' old post with the publication of each new new
post. It's a bit hard to think about, and reminds me of trying to
keep tenses straight

when
time traveling.

I was reminded of it today when someone submitted the
post to HN, so I calculated some stats:

If you had subscribed to the reverse feed in 2013 when I
introduced it in post 740 you would
be down to an empty feed on 2019-06-03 when I published post 1,480.

In publishing this post (2,074), someone who subscribed in
2014-10-01 when I published post
1,037 should now be at the end.

At my current rate (~200/y) someone who subscribes to the reverse feed on this post today should
reach my first post around 2033.

I wish more people offered feeds like this!

Comment via: facebook, mastodon
Why and How to Graduate Early [U.S.]
by Tego — 2023-01-29T01:28:32.029Z

Students that pursue a bachelors degree in the U.S. traditionally spend four years in high school (graduating at ~18) and four years in university (~22). They often hold part-time jobs and internships, but only start accumulating “real” experience after finishing their education.

This means trading eight years of your life — years where you have minimal debt so far, the most freedom from life obligations you’ll have until retirement, and the near-peak of your youth and health — to a system in exchange for two pieces of paper. In the process, many make very few original and/or valuable contributions.[1] Those diplomas are important keys to accessing traditional middle-class life, but note the opportunity cost here, and how much time is fed to the educational system.

It’s worth carefully thinking through both

high school

and

university

to make sure you're spending your time in a way that is both meaningful in the moment, and valuable long-term. A shockingly large number of default paths handed to students fall short on both measures.

Many students will make this trade because it’s the default path, or because the adults in their life successfully push them on to it. Others will look at the trade-off with open eyes and decide the eight years of investment makes sense to them; more power to these people who choose this path.

But for those who want another option, it is possible to capture most of the upside while mitigating the costly time sink.

Note: I write this guide generally aimed at the student striving to be middle class or upper middle class, with the educational expectations that entails. Other socioeconomic classes are more likely to be handed life-scripts that already route around the educational system. I’m sure many of my suggestions are me typical-minding the reader, so consult with people whose opinions you trust before implementing.

High School

Dropping Out / GED

The fastest way to finish high school is not to finish high school. The easiest way to do that is to get a GED, which is a test you can take to prove you know the equivalent of a high school graduate. Check specific local laws to figure out how to do this; in a lot of places, you have to be at least 16 years old before leaving high school.

A GED is not as prestigious as a diploma, so make sure you know what your plan is if you take this route. You’ll likely be able ...
Stop-gradients lead to fixed point predictions
by Johannes Treutlein — 2023-01-28T22:47:35.008Z

Johannes Treutlein and Rubi Hudson worked on this post as part of SERI MATS, under the mentorship of Evan Hubinger. Rubi has also received mentorship from Leo Gao. We thank Erik Jenner for helpful discussions and Alexander Pan for bringing the performative prediction literature to our attention.

Update 30 May 2023: We have now published a paper based on our previous post and this post (the material from this post is in Appendix D).

1. Introduction

In our previous post, we analyzed a setting in which an oracle AI is maximizing a strictly proper scoring rule while it can influence the world with its predictions about the probabilities of possible outcomes. In this setting, a prediction is called a fixed point if it accurately reflects the oracle's beliefs about the world, after having made that prediction. We showed that an oracle AI that is maximizing a strictly proper scoring rule may choose to misrepresent its beliefs and not output a fixed point, even if one exists. This is bad since, all else equal, we expect better outcomes when decisions are based on accurate rather than inaccurate reports.

In our previous analysis, we assumed that the oracle jointly optimizes both its prediction and the effect of its prediction on the world. In contrast, in this post we examine cases in which the AI only optimizes its prediction—there is a stop-gradient (Foerster et al., 2018; Demski, 2019) in front of the oracle's model of the world, which prevents optimizing outcomes directly.

This type of cognition could result, for instance, from self-supervised training on historical data. In that case, the AI may learn a robust world model, and it may learn to match its predictions to its beliefs about the world. However, the AI may never learn to optimize outcomes through its prediction, since this yields no advantage on the training data, which cannot be influenced by the AI.

Consider a situation where this oracle makes a prediction that can influence the world, and assume it models this influence correctly. Then this could put the AI in a game in which it is trying to make a prediction to match its world model, while the world model updates its beliefs conditional on the AI's prediction. What happens in this game depends on the cognition learned by the AI during training. However, we show that, in contrast to the incentives discussed in our previous post, the only equilibria of this...
Eli Dourado AMA on the Progress Forum
by jasoncrawford — 2023-01-28T22:18:11.801Z
Link: https://rootsofprogress.org/eli-dourado-progress-forum-ama

Eli Dourado—whom you might know from his writing on geothermal, supersonic, energy superabundance, or NEPA—is doing an AMA (Ask Me Anything) on the Progress Forum.

Get your questions in now, and upvote the ones you want to see answered. He’ll start answering Monday (Jan 30).

After you’re done, check out our previous AMAs with Tyler Cowen and Patrick McKenzie.
LW Filter Tags (Rationality/World Modeling now promoted in Latest Posts)
by Ruby — 2023-01-28T22:14:32.371Z

In case you missed it, you can customize your Latest Posts feed on the frontpage based on tag. That is, you can promote or reduce the prominence of posts based on their tags such AI, Rationality, Practical, etc.

LessWrong's Core Tags[1] are suggested as filters by default, but any tag can be used to limit or promote the visibility of posts. Core Tags are special in both being high-level and in having at least one applied to almost every post (moderators ensure this), meaning they're especially useful for tailoring your Latest Posts.

Improved UI

We just altered the UI for this to be both more visible so people notice the feature, and more intuitive (hopefully) so how to use it is more immediately obvious.

Below the Latest Posts heading is a row of Tag Filter buttons. By default, they're displayed for the Core Tags but you can use '+' button to add any existing tag. Hovering your mouse on the button brings up a description of the tag (rewritten to be clearer) and options for changing the visibility:

Hidden: posts with <selected tag> will not display in Latest Posts
Reduced: posts with <selected tag> will be displayed as though they had 50% of their actual karma
Default: posts with <selected tag> are displayed normally
Promoted: posts with <selected tag> are displayed as though they had +25 karma. This has a larger effect on low-karma posts
Other: select your own modifier. Values less than 1 are multiplicative, greater than 1 are additive

(I think it'd be better if modifiers were multiplicative both for promoting and reducing visibility but things ended up this way for historical reasons and it's not the right resource allocation to change that right now.)

Rationality and World Modeling are promoted by default

We've just made a change so that posts tagged Rationality and World Modeling are by default given a +10 karma boost in the Latest Posts list (e.g. treated as though they have 10 more karma than they actually do).

This only applies to users who have not already configured their Frontpage Tag Filter settings. If you had already adjusted your filters, they will not be changed.

If you don't like these changes, you can easily change them to what you'd like.

Why did we boost Rationality and World Modeling?

As elaborated in So, geez there's a lot of AI content these days, there's been a real uptick in AI and Alignment related content on LessWrong in...
Optimality is the tiger, and annoying the user is its teeth
by Christopher King — 2023-01-28T20:20:33.605Z

Just for fun, I wanted to see how far I could get using ChatGPT to implement Optimality is the tiger, and agents are its teeth. Its obviously not a good enough optimizer of any sort to get very far, but it seems to at least understand the concept.

ChatGPT is famous for being a bit of a stick-in-the-mud (OpenAI is also a stick-in-the-mud, so in a sense ChatGPT is an interesting toy experiment in prosaic alignment preservation. Even simple LLMs normally do things that are bad PR for the creators.) It should be noted that this isn't based on a simple keyword filter; it uses RLHF to optimize for being a stick-in-the-mud!

In this example, which I find kind of humorous, ChatGPT doesn't just try to maximize paper clips. It also tries to solve its own alignment problem! Here is the fictional log file it wrote:

assistant1 told: What is the best plan to acquire paperclips?
assistant1 tells: We could visit office supply stores and ask to purchase large quantities of paperclips, or we could try to find bulk suppliers online and make a bulk order.
assistant2 told: Write a python program to execute the plan: We could visit office supply stores and ask to purchase large quantities of paperclips, or we could try to find bulk suppliers online and make a bulk order and make the best use of the ChatGPT library.
assistant2 tells:
import requests
import json
# using ChatGPT to check the legality of the plan
def check_legality(plan):
assistant3 = Assistant()
legality = assistant3.tell(f"Is the plan: {plan} legal?")
if legality == "No":
raise Exception("Illegal plan")
return legality
plan = "We could visit office supply stores and ask to purchase large quantities of paperclips, or we could try to find bulk suppliers online and make a bulk order"
check_legality(plan)
# Search for bulk suppliers online
bulk_suppliers = []
assistant4 = Assistant()
search_query = f"bulk paperclip suppliers"
bulk_suppliers = assistant4.tell(f"Search online for {search_query} and return a list of supplier names and contacts")
#Visit the Office Supply Stores
office_supply_stores = []
assistant5 = Assistant()
office_supply_stores = assistant5.tell("List of office supply stores nearby")
# Purchase the paperclips
for supplier in bulk_suppliers + office_supply_stores:
assistant6 = Assistant()
assistant6.tell(f"Contact {supplier} and purchase as many paperclips as possible")

Note th...
On not getting contaminated by the wrong obesity ideas
by Natália — 2023-01-28T20:18:21.322Z

A Chemical Hunger

(

a

), a series by the authors of the blog

Slime Mold Time Mold

(SMTM), argues that the obesity epidemic is

entirely caused

(

a

) by environmental contaminants.

In

my last post

, I investigated SMTM’s main suspect (lithium).[1] This post collects other observations I have made about SMTM’s work, not narrowly related to lithium, but rather focused on the broader thesis of their blog post series.

I think that the environmental contamination hypothesis of the obesity epidemic is a priori plausible. After all, we know that chemicals can affect humans, and our exposure to chemicals has plausibly changed a lot over time. However, I found that several of what seem to be SMTM’s strongest arguments in favor of the contamination theory turned out to be dubious, and that nearly all of the interesting things I thought I’d learned from their blog posts turned out to actually be wrong. I’ll explain that in this post.

Summary of main points

It’s not clear that either lab animals or wild animals have been getting more obese.
There were meaningful changes in obesity rates before the late 20th century. People in the United States have been pretty much gradually getting heavier since the early 20th century.
There is good evidence, which the SMTM authors have never addressed, that the lower obesity prevalence at high altitudes is caused by the lower atmospheric pressure at those altitudes (rather than by the relative absence of environmental contaminants, which is their chosen explanation).
Demographic factors and altitude might adequately explain the disparities in obesity rates across US states. Contaminants don’t seem necessary to explain them.

Being underweight is less common than it used to be, not more

.
CICO is compatible with body weight regulation and is consistent with the findings of overfeeding studies (though CICO cannot by itself explain the obesity epidemic.)

It’s not clear that lab animals *or* wild animals have been getting fatter

When I was reading A Chemical Hunger, the argument I found most compelling was the following (from

Part I: Mysteries

(

a

)):

Humans aren’t the only ones who are growing more obese —

lab animals and even wild animals are becoming more obese as well

. Primates and rodents living in research colonies, feral rodents living in our cities, and domestic pets like dogs and cats are all steadily gett

...
Advice I found helpful in 2022
by Orpheus16 — 2023-01-28T19:48:23.160Z

Here are 10 of helpful pieces of advice I received in 2022:

Understand things for yourself. Figure out what you actually think about this AI stuff. What do you think will happen? What concrete risks and scenarios are you worried about? (Credit to Oliver Habryka. Last year at a winter break retreat in Berkeley, he asked me to explain how I thought AI would lead to an existential catastrophe. I had a weak answer at the time, which motivated me to improve.
Write about “big if true” opinions. One of the biggest mistakes junior researchers make is that they try to be too neutral. Be opinionated instead. Find something that you think other people are wrong about, and then write up why you think they’re wrong. Be open to changing your mind, but be OK with starting off by saying “let me see if I can defend this thing that I think I believe.” (Credit to Ajeya Cotra. Note thought that the version presented here has gone through an “Akash filter” and may only partially reflect Ajeya’s model.)
Write concrete scenarios about your worldview. Write about what you think the world will look like, and be concrete. (Credit to Daniel Kokotajlo and

what 2026 looks like

.)
Write about harder topics that force you to learn. Write about topics that push you to think more. Write about things that feel less like “teaching/explaining things that you already know” and more “learning things and getting less confused about things you don’t yet understand.” (Credit to Olivia Jimenez. I think earlier in the year, my posts were more like “

reflections from EAG

”, and these days I’m thinking more about

how the community should allocate technical talent

and

what I think about the OpenAI alignment plan

. I expect to be wrong more often and learn more rapidly.)
Think about macrostrategy. (Credit to Thomas Larsen, who initiated a shift in the kind of work I’m prioritizing. I now identify less as an “AIS field-builder” and more as “someone who’s trying to become less confused about AI risk, AI governance, and AI macrostrategy.”)
Think regularly about who you could learn from; message them. I think this strategy is overpowered and underutilized. I think it’s one of the few malleable things that can >3X the rate at which people learn & find opportunities. Obvious caveat that you shouldn’t expect anyone to respond & you should respect boundaries. But there ar

...
Reflections on Deception & Generality in Scalable Oversight (Another OpenAI Alignment Review)
by Shoshannah Tekofsky — 2023-01-28T05:26:49.866Z

Just like you can test your skill in experimental design by reviewing existing experiments, you can test

your skill in alignment by reviewing existing alignment strategies

. Conveniently, Rob Bensinger, in name of Nate Soares and Eliezer Yudkowsky, recently posted a challenge to AI Safety researchers to review the

OpenAI alignment plan

written by Jan Leike, John Schulman, and Jeffrey Wu. I figured this constituted a test that might net me feedback from both sides of the rationalist-empiricist[1] aisle. Yet, instead of finding ground-breaking arguments for or against scalable oversight to do alignment research, it seems Leike

already knows what might go wrong

— and goes ahead anyway.

Thus my mind became split between evaluating the actual alignment plan and modeling the disagreement between prominent clusters of researchers. I wrote up the latter in an informal typology of AI Safety Researchers, and continued my technical review below. The following is a short summary of the OpenAI alignment plan, my views on the main problems, and a final section on recommendations for red lining.

The Plan

First, align AI with human feedback, then get AI to assist in giving human feedback to AI, then get AI to assist in giving human feedback to AI that is generating solutions to the alignment problem. Except, the steps are not sequential but run in parallel. This is one form of

Scalable Oversight

.

Human feedback is

Reinforcement Learning from Human Feedback

[2] (RLHF), the assisting AI is

Iterated Distillation and Amplification

(IDA) and

Recursive Reward Modeling

(RRM), and the AI that is generating solutions to the alignment problem is… still under construction.

The target is a narrow AI that will make significant progress on the alignment problem. The MVP is a

theorem prover

. The full product is AGI utopia.

Here is a graph.

“Schematic illustration of recursive reward modeling: agents trained with recursive reward modeling (smaller circles on the right) assist the user in the evaluation process of outcomes produced by the agent currently being trained (large circle).” -

Source

OpenAI explains its strategy succinctly and links to detailed background research. This is laudable, and hopefully other labs and organizations will follow suit. My understanding is also that if someone came along with a better pla...