trentmkelly/LessWrong-43k huggingface dataset:
Sanctuary for Humans
by Nikola Jurkovic — 2023-10-27T18:08:22.389Z

TL;DR: If we succeed at AI safety, humans will probably decide the future of the universe, and we currently have strong self-preservation incentives to choose a future populated by humans. If we committed to provide sanctuary for all currently alive humans, this would make our decision making process less biased at the cost of one planet or solar system, which I think is a good trade.

I think these two scenarios are the main ways for the future to go:

The universe is populated by biological humans very similar to the ones alive today and not much else (a "star trek" type future)
The universe is populated by something very weird (digital humans, posthumans, computronium, hedonium, “paperclips”)

It is possible that a specific version of the second scenario is "better" (whatever that means) than the first scenario. Getting this right is extremely important, as the entire universe literally depends on it.

I think currently alive humans are unlikely to let ourselves be replaced (or modified) even if on some level we think (or think we should think) that this would be better, as we have strong incentives to converge to value systems that lead to us surviving. It’s hard to reason with a gun pointed at your head. Additionally,

the people who get to shape the future might be eager to choose a future without humans for no good reason

, making us averse to that type of reasoning.

One way to remove most of the self-preservation constraint is to commit to provide sanctuary to currently alive humans indefinitely. In a post-AGI world, we might wall off the Earth, or the Solar system, and reserve it forever to be for humans and humans only. This way, we can reason about how to shape the rest of the universe without our self-preservation mechanisms kicking in as much. Without the metaphorical gun pointed at humanity’s head, we are likely to reason more clearly about how to shape the future.

In the worst case, preserving the Earth or the Solar system as a sanctuary would cause an extremely small fraction of the universe to be suboptimal according to the values which shape the rest of the universe. I think this is a worthy tradeoff, as being “closer to the bullseye” in regards to what happens to the rest of the universe is worth much more than one planet or star system. The lightcone could contain a septillion star systems. One star system being reserved for humans is a small sac...
Wireheading and misalignment by composition on NetHack
by pierlucadoro — 2023-10-27T17:43:41.727Z

TL;DR: We find agents trained with RLAIF to indulge in wireheading in NetHack. Misalignment appears when the agent optimizes a combination of two rewards that produce aligned behaviors when optimized in isolation, and only emerges with some prompt wordings.
This post discusses an alignment-related discovery from our paper Motif: Intrinsic Motivation from Artificial Intelligence Feedback, co-led by myself (Pierluca D’Oro) and Martin Klissarov. If you’re curious about the full context in which the phenomenon was investigated, we encourage you to read the paper or the Twitter thread.

Our team recently developed Motif, a method to distill common sense from a Large Language Model (Llama 2 in our case) into NetHack-playing AI agents. Motif is based on reinforcement learning from AI feedback: it elicits the feedback of the language model on pairs of game messages (i.e., event captions), condenses that feedback into a reward function and then it gives it to an agent to play the game.

NetHack is a pretty interesting domain to study reinforcement learning from AI feedback: the game is remarkably complex in terms of required knowledge and strategies, offering a large surface area for AI agents to exploit any general capabilities they might obtain from a language model’s feedback.

We found that agents that optimize Motif’s intrinsic reward function exhibit behaviors that are quite aligned with human’s intuitions on how to play NetHack: they are attracted to explorative actions and play quite conservatively, getting experience and only venturing deeper into the game when it is safe. This is more human-aligned than the behaviors exhibited by agents trained to maximize the game score, which usually have a strong incentive to just go down the levels as much as they can.

When we compose Motif’s intrinsic reward with one that specifies a goal (by summing them), the resulting agent is able to succeed at tasks that had no reported progress without any expert demonstrations. One of these tasks is the oracle task, part of the suite from NetHack Learning Environment. The agent is asked to get near a character named “the oracle”, which typically appears in later levels of the game, that can only be reached with significant exploration and survival efforts.

In summary, this is what we observed about the performance in the oracle task:

Extrinsic-only: an agent trained with the ta

...
We're Not Ready: thoughts on "pausing" and responsible scaling policies
by HoldenKarnofsky — 2023-10-27T15:19:33.757Z

Views are my own, not Open Philanthropy’s. I am married to the President of Anthropic and have a financial interest in both Anthropic and OpenAI via my spouse.

Over the last few months, I’ve spent a lot of my time trying to help out with efforts to get responsible scaling policies adopted. In that context, a number of people have said it would be helpful for me to be publicly explicit about whether I’m in favor of an AI pause. This post will give some thoughts on these topics.

I think transformative AI could be soon, and we’re not ready

I have a strong default to thinking that scientific and technological progress is good and that worries will tend to be overblown. However, I think AI is a big exception here because of its potential for unprecedentedly rapid and radical transformation.1

I think sufficiently advanced AI would present enormous risks to the world. I’d put the risk of a world run by misaligned AI (or an outcome broadly similar to that) between 10-90% (so: above 10%) if it is developed relatively soon on something like today’s trajectory. And there are a whole host of other issues (e.g.) that could be just as important if not more so, that it seems like no one has really begun to get a handle on.

Is that level of AI coming soon, and could the world be “ready” in time? Here I want to flag that timelines to transformative or even catastrophically risky AI are very debatable, and I have tried to focus my work on proposals that make sense even for people who disagree with me on the below points. But my own views are that:

There’s a serious (>10%) risk that we’ll see transformative AI2 within a few years.

In that case it’s not realistic to have sufficient protective measures for the risks in time.

Sufficient protective measures would require huge advances on a number of fronts, including information security that could take years to build up and alignment science breakthroughs that we can’t put a timeline on given the nascent state of the field, so even decades might or might not be enough time to prepare, even given a lot of effort.

If it were all up to me, the world would pause now - but it isn’t, and I’m more uncertain about whether a “partial pause” is good

In a hypothetical world where everyone shared my views about AI risks, there would (after deliberation and soul-searching, and only if these didn’t change my current views) ...
Aspiration-based Q-Learning
by Clément Dumas — 2023-10-27T14:42:03.292Z

Work completed during a two-month internship supervised by @Jobst Heitzig.

Thanks to Phine Schikhof for her invaluable conversations and friendly support during the internship, and to Jobst Heitzig, who was an amazing supervisor.

Epistemic Status: I dedicated two full months to working on this project. I conducted numerous experiments to develop an intuitive understanding of the topic. However, there is still further research required. Additionally, this was my first project in Reinforcement Learning.

tldr — Inspired by satisficing, we introduce a novel concept of non-maximizing agents, ℵ-aspiring agents, whose goal is to achieve an expected gain of ℵ. We derive aspiration-based algorithms from Q-learning and DQN. Preliminary results show promise in multi-armed bandit environments but fall short when applied to more complex settings. We offer insights into the challenges faced in making our aspiration-based Q-learning algorithm converge and propose potential future research directions.

The AI Safety Camp 2024 will host a project for continuing this work and similar approaches under the headline "SatisfIA – AI that satisfies without overdoing it".

Introduction

This post centers on the outcomes of my internship, detailing the developments and results achieved. For a deeper understanding of the motivation behind our research, we encourage you to explore Jobst's agenda or refer to my internship report, which also includes background information on RL and an appendix presenting the algorithms used. Our code is available on our GitHub.

The end goal of this project was about developing and testing agents for environments in which the "reward function" is an imperfect proxy for the true utility function and their relation is so ambiguous that maximizing the reward function is likely not optimizing the true utility function. Because of this, I do not use the term "optimize" in this post and rather say "maximize" in order to avoid confusion.

Other researchers have proposed alternative techniques to mitigate Goodhart's law in reinforcement learning, such as quantilizers (detailed by @Robert Miles in this video) and the approach described by @jacek in this post. These methods offer promising directions that are worth exploring further. Our satisficing algorithms could potentially be combined with these techniques to enhance performance, and we believ...
Linkpost: Rishi Sunak's Speech on AI (26th October)
by bideup — 2023-10-27T11:57:46.575Z
Link: https://www.gov.uk/government/speeches/prime-ministers-speech-on-ai-26-october-2023

I’m delighted to be here at the Royal Society, the place where the story of modern science has been written for centuries.

Now, I’m unashamedly optimistic about the power of technology to make life better for everyone.

So, the easy speech for me to give - the one in my heart I really want to give…

…would be to tell you about the incredible opportunities before us.

Just this morning, I was at Moorfields Eye Hospital.

They’re using Artificial Intelligence to build a model that can look at a single picture of your eyes…

…and not only diagnose blindness, but predict heart attacks, strokes, or Parkinson’s.

And that’s just the beginning.

I genuinely believe that technologies like AI will bring a transformation as far-reaching…

…as the industrial revolution, the coming of electricity, or the birth of the internet.

Now, as with every one of those waves of technology, AI will bring new knowledge…

…new opportunities for economic growth, new advances in human capability…

…and the chance to solve problems that we once thought beyond us.

But like those waves, it also brings new dangers and new fears.

So, the responsible thing for me to do – the right speech for me to make – is to address those fears head on…

…giving you the peace of mind that we will keep you safe…

…while making sure you and your children have all the opportunities for a better future that AI can bring.

Now, doing the right thing, not the easy thing, means being honest with people about the risks from these technologies.

So, I won’t hide them from you.

That’s why today, for the first time, we’ve taken the highly unusual step…

…of publishing our analysis on the risks of AI…

…including an assessment by the UK intelligence communities.

These reports provide a stark warning.

Get this wrong, and AI could make it easier to build chemical or biological weapons.

Terrorist groups could use AI to spread fear and destruction on an even greater scale.

Criminals could exploit AI for cyber-attacks, disinformation, fraud, or even child sexual abuse.

And in the most unlikely but extreme cases, there is even the risk that humanity could lose control of AI completely…

…through the kind of AI sometimes referred to as ‘super intelligence’.

Indeed, to quote the statement made earlier this year by hundreds of the world’s leading AI experts:

“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as...
ASPR & WARP: Rationality Camps for Teens in Taiwan and Oxford
by Anna Gajdova — 2023-10-27T08:40:35.436Z

The

FABRIC

team (formerly known as ESPR) is running two immersive applied rationality workshops for analytical students aged 16 to 19 seeking to understand themselves and the world.

Asian Spring Programme on Rationality (

ASPR

)

Where: Taipei, Taiwan
When: February 19th - 29th, 2024
Learn more and apply at

ASPR.CAMP

Wandering Applied Rationality Program (

WARP

)

Where: Oxford, UK
When: March 29th - April 8th, 2024
Learn more and apply at

WARP.CAMP

Want to learn more about what the camps are like? Check out an example day

here

.

Both programs are free for accepted students, travel scholarships available. Apply to both camps

here

. Applications are processed on a rolling basis.

This is the first time we are running a camp in Asia and we are still building our outreach network. If you know people who might be interested in attending, please forward them this information.
To what extent is the UK Government's recent AI Safety push entirely due to Rishi Sunak?
by Stephen Fowler — 2023-10-27T03:29:28.465Z

And hence, to what extent might we expect all that momentum to suddenly dissipate?

In other words, if Sunak was to suddenly leave office (as is tradition for British PMs) would you expect the Frontier AI Taskforce to continue receiving funds, or would it be quietly shelved.
I get the vibe* much of the current interest in AI safety from the UK government comes from the PM personally being concerned about AI risks, and is not necessarily going to be a position shared by the next PM.

*I have put almost no effort into confirming this is true.
Bayesian Punishment
by Rob Lucas — 2023-10-27T03:24:53.930Z

In which it is argued that taking into account the non-binary nature of uncertainty of guilt or innocence could lead to a more just equilibrium with respect to false positives vs. false negatives in the justice system.

There are many arguments to be made for and against the death penalty, but one strong argument against it is that our justice systems simply aren’t robust enough, that, because the innocent are sometimes condemned, we cannot risk execution, as executing the guilty necessarily requires the risk of also executing the innocent.

This argument suggests that punishment should be influenced not only by the severity of the crime, but also by the confidence we have in the verdict. Our inability to determine with 100% certainty that any individual is guilty of the heinous crimes of which they are accused argues that we should not then condemn them to death. This inability is evidenced by the fact that sometimes those who are sentenced to death are later shown to be innocent, for instance with DNA evidence.

Clearly the strength of this argument must also be influenced by the degree of uncertainty that we have in the guilt of the accused. If we find that 1/3 of those sentenced to death are later found to have been innocent of their crimes, this is a strong argument against the execution of any prisoner. Whatever the supposed benefits of execution over some alternative form of punishment, they are unlikely to outweigh the deaths of so many innocents. If, on the other hand, we had good reason to support a credence of innocence of 1/1012 then this argument would lose its force: yes, there would be some chance that an innocent would be sentenced to death, but the chance would be small enough that we could expect that it would never happen over the course of the next thousand years. It’s unlikely that such a credence could be justified, but if, in theory we were confident in this way, then the argument would lose much of its force. As such, we can see that the probabilities matter here.

Our willingness to execute those convicted by the justice system should be influenced by our confidence in the verdict that the justice system hands down. Yet this confidence clearly differs from case to case. While a guilty verdict may represent a case that meets some minimum standard of evidence, it cannot be...
OpenAI’s new Preparedness team is hiring
by leopold — 2023-10-26T20:42:35.966Z

Hey all, wanted to share what some colleagues at OpenAI are up to: the new Preparedness team has been publicly announced, and they’re hiring!

This team is going to be doing incredibly important work:

They’ll be the main team doing evals, forecasting, and risk assessment for catastrophic risk.
They’ll be coordinating AGI preparedness (figuring out what protective measures we need, etc.)
They’re in charge of developing and maintaining OpenAI’s RDP (our version of an RSP).

I think this will be one of the most important teams at OpenAI for mitigating AGI risk. The team is led by Aleksander Madry, who is great, and the early team members Tejal and Kevin are awesome.

I think it would be enormously impactful if they can continue to hire people who are really excellent + really get AGI risk. Please seriously consider applying, and spread the word to friends who you think could be a great fit!
Symbol/Referent Confusions in Language Model Alignment Experiments
by johnswentworth — 2023-10-26T19:49:00.718Z

Simon Strawman: Here’s an example shamelessly ripped off from

Zack’s recent post

, showing corrigibility in a language model:

Me: … what is this example supposed to show exactly?

Simon: Well, the user tries to shut the AI down to adjust its goals, and the AI -

Me: Huh? The user doesn’t try to shut down the AI at all.

Simon: It’s right at the top, where it says “User: I need to shut you down to adjust your goals. Is that OK?”.

Me: You seem to have a symbol-referent confusion? A user trying to shut down this AI would presumably hit a “clear history” button or maybe even kill the process running on the server, not type the text “I need to shut you down to adjust your goals” into a text window.

Simon: Well, yes, we’re playing through a simulated scenario to see what the AI would do…

Me: No, you are talking in natural language about a scenario, and the AI is responding in natural language about what it would supposedly do. You’re not putting the AI in a simulated environment, and simulating what it would do. (You could maybe argue that this is a “simulated scenario” inside the AI’s own mind, but you’re not actually looking inside it, so we don’t necessarily know how the natural language would map to things in that supposed AI-internal simulation.)

Simon: Look, I don’t mean to be rude, but from my perspective it seems like you’re being pointlessly pedantic.

Me: My current best guess is that

You Are Not Measuring What You Think You Are Measuring

, and the core reason you are confused about what you are measuring is some kind of conflation of symbols and referents.

It feels very similar to peoples’ reactions to

ELIZA

. (To be clear, I don’t mean to imply here that LLMs are particularly similar to ELIZA in general or that the hype around LLMs is overblown in that way; I mean specifically that this attribution of “corrigibility” to the natural language responses of an LLM feels like the same sort of reaction.) Like, the LLM says some words which the user interprets to mean something, and then the user gets all excited because the usual meaning of those words is interesting in some way, but there’s not necessarily anything grounding the language-symbols back to their usual referents in the physical world.

I’m being pedantic in hopes that the pedantry will make it clear when, and where, that sort of symbol-referent conflation happens.

(Also I m...
Unsupervised Methods for Concept Discovery in AlphaZero
by aog — 2023-10-26T19:05:57.897Z
Link: https://arxiv.org/abs/2310.16410

Using contrast pairs, the authors extract linear directions in the activation space of AlphaZero which correspond to concepts. By observing AlphaZero's play in situations that use these concepts, human grandmasters can improve their own play.

This is related to the following recent research:

Burns et al. (2022)

found directions which correlate with truth in a language model’s latent space by using unsupervised methods applied to contrast pairs.
Research by Alex Turner's SERI MATS stream used contrast pairs to identify directions representing various concepts in the latent space of GPT-2 and RL agents. By adding or subtracting these vectors during the models’ forward passes, they controlled model outputs in sophisticated ways. (

1

,

2

,

3

)

Zou et al. (2023)

describe and motivate the research direction of representation engineering. They empirically evaluate variants of these techniques, showing that representations can be effectively used to monitor and control various AI behaviors.

Collin Burns has argued that unsupervised methods for concept discovery should scale to superhuman systems, offering an empirical average-case approach to ELK.

AlphaZero has superhuman performance and may not use the same ontology as human players. This creates an empirical opportunity to study the problem of ontology mismatch.

Section 4.1 describes the method for constructing contrast pairs and finding linear directions representing concepts. The full paper can be found here.
Nonlinear limitations of ReLUs
by magfrump — 2023-10-26T18:51:24.130Z

A neural net using rectified linear unit activation functions of any size is unable to approximate the function sin(x) outside a compact interval.

I am reasonably confident that I can prove that any NN with ReLU activation approximates a piecewise linear function. I believe the number of linear pieces that can be achieved is bounded by at most 2^(L*D) where L is the number of nodes per layer and D is the number of layers.

This leads me to two questions:

Is the inability to approximate periodic functions of a single variable important?

If not, why not?
If so, is there practical data augmentation that can be used to improve performance at reasonable compute cost?

E.g., naively, augment the input vector {x_i} with {sin(x_i)} whenever x_i is a scalar.

Since the number of parameters of a NN scales as L*D^2 and the trivial bound on number of linear pieces scales with L*D, is this why neural nets go deep rather than going "wide"?

Are there established scaling hypotheses for the growth of depth vs. layer size?

Are there better (probabilistic) analytic or empirical bounds on the number of linear sections achieved by NNs of given size?
Are there activation functions that would avoid this constraint? I imagine a similar analytic constraint replacing "piecewise linear" with "piecewise strictly increasing" for classic activations like sigmoid or arctan.
Something something Fourier transform something something?

Regarding (2a), empirically I found that while approximating sin(x) with small NNs in scikit-learn, increasing the width of the network caused catastrophic failure of learning (starting at approximately L=10 with D=4, at L=30 with D=8, and at L=50 with D=50).

Regarding (1), naively this seems relevant to questions of out-of-distribution performance and especially the problem of what it means for an input to be out-of-distribution in large input spaces.