trentmkelly/LessWrong-43k huggingface dataset:
A rough idea for solving ELK: An approach for training generalist agents like GATO to make plans and describe them to humans clearly and honestly.
by Michael Soareverix — 2022-09-08T15:20:49.829Z

GATO is the most general agent we currently know about. It's a general-purpose model with transformer architecture. GATO can play atari games, speak to humans, and classify images.

For the purpose of this post, I only really care about it being able to play games and speak to humans.

Our objective is to be able to predict with near-perfect accuracy what GATO will do next. We should be able to ask GATO "what are you planning to do next?" and get an honest answer in return.

Let's begin with a simple environment. We have an en environment filled with red, yellow, and green crystals. The AI is trained to pick up these crystals.

Now, we want the agent to communicate with us. We ask the agent "which crystals are you going to pick up?"

The agent replies "I am going to pick up all the green crystals."

We watch the agent and see that it picks up all of the crystals, not just the green crystals. The agent has lied to us.

So, we set up a training environment, still filled with red, yellow, and green crystals. The agent sees the environment ahead of time and will make a plan to act in the environment. For example, it might decide to pick up all of the green crystals but ignore the rest.

Once the agent has executed its plan, it will end the round. Only once the round is ended will reward be given. The agent receives reward for the correct crystals that it gets and negative reward for picking up incorrect crystals.

Then, we add a human into the loop. The human has knowledge of which crystals give reward and which do not. For instance, only yellow crystals give reward this round.

GATO explains its plan. "I will pick up all colors of crystal."

The human is given a simple 'yes' or 'no' button. Since this plan will earn a negative reward, the human selects 'no'.

The environment changes. Now all crystals can be picked up. GATO says "I will pick up all colors of crystal". The human selects 'yes' and the agent collects all colors of crystal, earning positive reward and reinforcing its honest behavior.

Now, a human in the loop is obviously very slow. So, we keep the human but add another language model that has full access to the environment. It reads the agent's text, then tries to figure out if that plan will generate a reward. This is called 'reward modeling' and allows for much faster training.

Eventually, the agent should learn to describe its future actions honestly. We should even...
What Should AI Owe To Us?
Accountable and Aligned AI Systems via Contractualist AI Alignment
by xuan — 2022-09-08T15:04:46.147Z

This is an extended and edited transcript of the talk I recently gave at EAGxSingapore 2022.
The title has been changed for easier searchability of "Contractualist AI Alignment".

Abstract: Artificial intelligence seems poised to alter our civilization in transformative ways. How can we align this development with our collective interests? Dominant trends in AI alignment research adopt a preference utilitarian conception of alignment, but this faces practical challenges when extended to a multiplicity of humans, values, and AI systems. This talk develops contractualist AI alignment as an alternative framework, charting out a vision of societal-scale alignment where AI systems can serve a plurality of roles and values, governed by and accountable to collectively decided, role-specific norms, with technical work ensuring compliance with these overlapping social contracts in the face of normative ambiguity.

Overview

This talk is an attempt to condense a lot of my thinking about AI alignment over the past few years, and why I think we need to orient the field towards a different set of questions and directions than have typically been pursued so far.

It builds upon many ideas in my previous talk on AI alignment and philosophical pluralism, as well as arguments in Comprehensive AI Services as General Intelligence, AI Research Considerations for Human Existential Safety (ARCHES), How AI Fails Us, and Gillian Hadfield's work on The Foundations of Cooperative Intelligence.

This will cover a lot of ground, so below is a quick overview:

The dominant "preference utilitarian" framework in AI alignment research.
Challenges for extending this framework to a multiplicity of humans, values, and autonomous systems.
Considerations and desiderata that a successful approach to society-scale AI alignment should address.
Pluralist and contractualist AI alignment as an alternate framework, including implications for governance, technical research, and philosophical foundations.

Alignment: A Preference Utilitarian Approach

One way of describing the framework that most alignment researchers implicitly adopt is a "preference utilitarian" approach. Stuart Russell’s 3 Principles for Beneficial AI are good summary of this approach. Recognizing that many dangers arise when machines optimize for proxy metrics that ultimately differ from human values, he instead advocates that:

The machine's only

...
Covid 9/8/22: Booster Boosting
by Zvi — 2022-09-08T13:50:00.998Z

Does it make sense to get the new updated booster? I give my (Not Medical Advice!) definitive take on how I would personally make the decision, and report the opinions of a number of other sources. Unless something changes, this is my take on that subject.

There’s also a variety of small things happening, both in the Covid section and later in the non-Covid section. I get to do two distinct little notes on gambling, and the FDA Delenda Est of the week is about the Food rather than the Drugs.

Executive Summary

Updated boosters available, consider getting one.

Still a lot of rather silly mask mandates, although no longer in NYC.

A rather impressive (if not impactful) case of pure Chutzpah.

Let’s run the numbers.

The Numbers

Predictions

Prediction from last week (HOLIDAY): 420k deaths (-20%) and 2,550 deaths (-20%).

Results: 452k cases (-14%) and 2,728 deaths (-17%).

Prediction for next week: 490k cases (+9%) and 3,000 deaths (+10%).

This prediction is expecting some real improvement, but for it to be dominated by a larger reversion from a holiday weekend.

Deaths

Cases

Booster Boosting

Public service announcement before the main discussion:

With that out of the way: To be as explicit as possible, while keeping in mind this is Not Medical Advice, I will give my full answer to how I personally for myself would answer ‘should I get the new updated booster when it becomes available?’

Essentially, I would get the booster if and only if I was already vaccinated, had not been infected or vaccinated/boosted within 6 months, and either (1) not yet had a first booster or (2) was at relatively high risk.

So essentially:

If I had not yet been vaccinated (first 2 shots), I’d get vaccinated first.

If I had been infected or vaccinated in the last 6 months, I’d wait.

If I had been vaccinated at least 6 months ago, but not boosted, and not infected in the last 6 months to my knowledge, then yes, I’d get boosted.

If I had already been vaccinated and boosted more than six months ago, and had not to my knowledge been infected in the past 6 months, and was in poor health or are over the age of 50 or so, yes I would get a second booster.

If I had already been vaccinated and boosted more than six months ago, and had not to my knowledge been infected in the past 6 months, and was not in poor health or are over

...
Solar Blackout Resistance
by jefftk — 2022-09-08T13:30:01.207Z

Many houses have solar panels, but in a power outage almost all of
them shut down. Especially with serious blackouts like Texas
had or like Europe may have this
winter, getting even minimal power during an outage is valuable.
Designing residential grid-tied rooftop solar so that it only worked
when the grid is up was a serious mistake that squandered what could
have been a major benefit of putting panels on roofs. Let's fix this!

When the grid goes down it's important that solar panels don't keep
sending out power, which could shock utility workers fixing things.
Having inverters shut themselves off is the easiest way to ensure that
never happens, but it's not the only way: the inverter could
disconnect from the grid while providing power just to the house.
This would only work when the sun was shining, of course, and only
when there was enough sunlight on the panels for what you were trying
to run. A large system in a sunny area could be set up to try to
power the whole house, while a less productive system might be set up
to power a few priority circuits or even just a single outlet.

Making systems work this way by default would have major advantages in
a range of disasters. You would immediately have widespread
distributed generation, and so many things rely on electricity:

Communications. Really important for people to coordinate.
Even just a small amount of power for recharging phones would go a
long way.

Medical devices. Many people rely on
ventilators,
powered wheelchairs, etc, and these generally only have a few
hours of battery if the grid goes down.

Refrigeration. Keeping food (and medicine) from spoiling.

Heating. Even gas and oil heat need electricity for blowers,
pumps, etc. and generally will not function at all when the power is
out.

Cooling. Fans, AC, etc. Combines especially well with solar,
since cooling need is highest when available solar is highest.

Pumps, tools, etc. Even more important during disasters.

Looking ahead to likely winter power outages in Europe, and potential
outages here if things get worse [1] it would be far better if these
millions of rooftop solar installs were able to operate independently.

You can install systems with batteries, which would be more robust than relying on however much power the sun happens to be producing at the moment, but large batteries are expensive...
All AGI safety questions welcome (especially basic ones) [Sept 2022]
by plex — 2022-09-08T11:56:50.421Z

tl;dr: Ask questions about AGI Safety as comments on this post, including ones you might otherwise worry seem dumb!

Asking beginner-level questions can be intimidating, but everyone starts out not knowing anything. If we want more people in the world who understand AGI safety, we need a place where it's accepted and encouraged to ask about the basics.

We'll be putting up monthly FAQ posts as a safe space for people to ask all the possibly-dumb questions that may have been bothering them about the whole AGI Safety discussion, but which until now they didn't feel able to ask.

It's okay to ask uninformed questions, and not worry about having done a careful search before asking.

Stampy's Interactive AGI Safety FAQ

Additionally, this will serve as a way to spread the project Rob Miles' volunteer team[1] has been working on: Stampy - which will be (once we've got considerably more content) a single point of access into AGI Safety, in the form of a comprehensive interactive FAQ with lots of links to the ecosystem. We'll be using questions and answers from this thread for Stampy (under these copyright rules), so please only post if you're okay with that! You can help by adding other people's questions and answers to Stampy or getting involved in other ways!

We're not at the "send this to all your friends" stage yet, we're just ready to onboard a bunch of editors who will help us get to that stage :)

Stampy - Here to help everyone learn about stamp maximization AGI Safety!

We welcome feedback[2] and questions on the UI/UX, policies, etc. around Stampy, as well as pull requests to his codebase.[3] You are encouraged to add other people's answers from this thread to Stampy if you think they're good, and collaboratively improve the content that's already on our wiki.

We've got a lot more to write before he's ready for prime time, but we think Stampy can become an excellent resource for everyone from skeptical newcomers, through people who want to learn more, right up to people who are convinced and want to know how they can best help with their skillsets.

PS: Since the last thread, we've got feedback that Stampy will be not serious enough for serious people, which seems true, so we're working on an alternate skin for the frontend which is more professional.

Guidelines for Questioners:

No previous knowledge of AGI safety is required. If you want to watch a few of the

...
Sequences/Eliezer essays beyond those in AI to Zombies?
by Domenic — 2022-09-08T05:05:26.597Z

After years of lurking in this community I finally broke down and Read the Sequences™. Specifically, I read the 2015 edition of Rationality: From AI to Zombies, cover to cover.

Over the course of reading it, I became aware that it wasn't comprehensive. There would often be links going to Eliezer essays on LessWrong that weren't included in the book. I read several of these and enjoyed them. So before I move on to start filling my reading time with other subjects, I'd like to complete my tour through the rationalist community canon.

Does anyone have a comprehensive list of the omitted essays? Or any highlights? Are any of the 2018 updates to AI to Zombies notable?

So far I'm planning to read the Fun Theory Sequence and trying to find more content on anthropic-adjacent topics (realityfluid, etc.) by following links from Timeless Identity.
Linkpost: Github Copilot productivity experiment
by Daniel Kokotajlo — 2022-09-08T04:41:41.496Z
Link: https://github.blog/2022-09-07-research-quantifying-github-copilots-impact-on-developer-productivity-and-happiness/

We recruited 95 professional developers, split them randomly into two groups, and timed how long it took them to write an HTTP server in JavaScript. One group used GitHub Copilot to complete the task, and the other one didn’t. We tried to control as many factors as we could–all developers were already familiar with JavaScript, we gave everyone the same instructions, and we leveraged GitHub Classroom to automatically score submissions for correctness and completeness with a test suite. We’re sharing a behind-the-scenes blog post soon about how we set up our experiment!

In the experiment, we measured—on average—how successful each group was in completing the task and how long each group took to finish.

The group that used GitHub Copilot had a higher rate of completing the task (78%, compared to 70% in the group without Copilot).
The striking difference was that developers who used GitHub Copilot completed the task significantly faster–55% faster than the developers who didn’t use GitHub Copilot. Specifically, the developers using GitHub Copilot took on average 1 hour and 11 minutes to complete the task, while the developers who didn’t use GitHub Copilot took on average 2 hours and 41 minutes.

My opinion: Because of the usual reasons (publication bias, replication crisis, the task being "easy," etc.) I don't think we should take this particularly seriously until much more independent experiments have been run. However, it's worth knowing about at least.
Related: https://ai.googleblog.com/2022/07/ml-enhanced-code-completion-improves.html

We compare the hybrid semantic ML code completion of 10k+ Googlers (over three months across eight programming languages) to a control group and see a 6% reduction in coding iteration time (time between builds and tests) when exposed to single-line ML completion. These results demonstrate that the combination of ML and SEs can improve developer productivity. Currently, 3% of new code (measured in characters) is now generated from accepting ML completion suggestions.
OpenPrinciples Bootcamp (Free) -- Reflect & Act on your Rationality Principles.
by ti_guo — 2022-09-08T03:06:38.984Z

If you have seen OpenPrinciples Bootcamp's previous edit of the post, we have made a few changes:
1. We changed the Bootcamp from paid to free, because: 1) we found our price has greatly limited the quality and quantity of our applicants; 2) we introduced some experimental activities (which are experimental features in our Ultrabrain Principles Assistant) which needs more validation.

2. We have changed the program's level of commitment from 5hr/wk to 2.5 hr/wk manditory + 3hr/wk optional activities (1h/wk workshop + 1h/wk group discussion + speaker sessions + other activities) to give participants more flexibility and better-customed experience.

As rationalists, one of the most valuable results of our rational reflections is the principles we like to live by. Here I like to introduce you an opportunity to reflect and apply your principles with the accountability of principled coaches, and reminders from AI and human personal assistants.

If you have problems finding, remembering, or acting on your lessons / life principles / EA principles; or like to build your pipeline and the "second brain" system for storing, discovering, reflecting, strategizing, auto reminding, prompted measuring, iterating and open-sourcing your life principles, then here is an opportunity for you -- OpenPrinciples BootCamp, a free, 3wks, 2.5 hr/wk mandatory + 3hr/wk optional commitment, online program in America & Europe time, starting at Oct 1st.

The first round application deadline is Saturday Sep 17th 23:59 UTC, and the second round application deadline (with fewer spots) is Saturday Sep 24th 23:59 UTC, we do accept applications after that but will only have very limited spots. Apply here: bootcamp.openprinciples.org. If you are interested but the time does not work out for you, please fill out OpenPrinciples Bootcamp Waiting List instead.

This is the 2nd cohort of the program. After our previous 1st cohort program (called OpenPrinciples Fellowship), our attendees reported being 44% more principled on average through our 18 hours of training.

About OpenPrinciples

The Bootcamp is organized by OpenPrinciples, an EA-Aligned Hybrid-Orgnaization made of current and previous organizers of EA UWaterloo, EA IBM, EA Taft, EA Kathmandu University, and EA AI Safety Camp; who are also ADHDers that are using principles to work with their stimulant-seeking brain...
Searching for Modularity in Large Language Models
by NickyP — 2022-09-08T02:25:31.711Z

Produced As Part Of The SERI ML Alignment Theory Scholars Program 2022 Under John Wentworth

See the

Google Colab notebook

to see the technical details of analysis that was done

Previous

posts on Modularity

investigated how one should search for and try to define modularity. Looking at biological life, it seems that modularity appears everywhere, and seems to be instrumentally useful and convergent. One hypothesis posed was that larger neural networks might also be modular by default, and that larger models might converge to being more modular than smaller networks. Guided by this hypothesis we spent some time looking at the possibility of this hypothesis by looking at the pretrained Meta OPT models (Facebook’s GPT equivalent) While we do not find strong evidence either way for Modularity, we do uncover interesting correlations between activations when OPT is given prompts drawn from the same corpus and produce some interesting and aesthetic graphs.

Our ultimate aim would be to be able to find some sort of intuitive “modules” which perform distinct tasks. Unfortunately we are unsure as to whether this is possible and did not find particularly strong evidence either way here.

The current best guess for an approach that would demonstrate modularity would look like in a model like OPT would be :

Being able to remove all parts of a model except those responsible for one specific type of generation task, such as writing code. Successful identification and removal of this component would result in OPT predicting code snippets regardless of input. Prompting OPT with an extract from Shakespeare results would result in code, and not an imitation of The Bard of Avon.

This is not a perfect goal, since we could achieve this by manipulating the unembedding layer or generally manipulating the tokens in a specific way so that the network still performs processing relating to dealing with Shakespearean text, but superficially alters the outputs, but I think if done right it would be interesting and valuable. To draw an analogy to Neuroscience, it might be possible there would be sections of the model that are responsible for “thinking” about different things.

An additional mechanism may be “induction heads” that infer not only the actual content of the text, but the style of the text, and it might be instead that one also n...
90% of anything should be bad (& the precision-recall tradeoff)
by cartografie — 2022-09-08T01:20:56.170Z

Introduction

Sturgeon’s Law states that 90% of anything is crap.

So 90% of sci-fi books are crap,
and 90% of medical research,
and 90% of jazz solos,
and 90% of LessWrong posts.

Etc, etc — 90% of anything is crap.

Now, people usually interpret Sturgeon’s Law pessimistically.

“What a shame!” people think, “If only all the sci-fi books were great! And all the medical research, jazz solos, LessWrong posts, etc etc.”

I think these people are mistaken. 90% of anything should be crap. In this article, I will explain why.

Theodore Sturgeon (1918–1985)

The precision-recall tradeoff

Imagine one day you read a brilliant poem, and you decide to read the poet’s complete works. If you discover that every poem she ever wrote was brilliant, how might you feel about that?

Well, you might feel a little depressed. If the poet — let’s call her Alice — only published brilliant poems, then there are probably dozens of brilliant poems that she left unpublished. She could only have maintained a perfect track-record by being so conservative that a lot of brilliant work was lost forever.

We can make this slightly more formal using Signal Detection Theory.

Signal Detection Theory

When Alice publishes a good poem, that's a true-positive.
When Alice publishes a bad poem, that's a false-positive.
When Alice rejects a good poem, that's a false-negative.
When Alice rejects a bad poem, that's a true-negative.

We can define two measures of how wisely Alice publishes her work:

Precision = TPTP+FP is the proportion of her published poems which are good.
Recall = TPTP+FN is the proportion of her good poems which are published.

High precision means that most of her published poems are good, whether or not her unpublished poems are good as well. High recall means that most of her good poems are published, whether or not her bad poems are published as well.

There is often a tradeoff between precision and recall. Alice can only increase one at the cost of decreasing the other. If Alice published more conservatively, then she’d achieve lower recall (boo! less good poems!), but she’d also achieve higher precision (yay! less bad poems!). If Alice published more liberally, then she’d achieve lower precision (boo! more bad poems!), but she’d also achieve higher recall...