trentmkelly/LessWrong-43k huggingface dataset:
Monthly Roundup #5: April 2023
by Zvi — 2023-04-03T18:50:01.056Z

As AI eats the world and my time and news feeds, it is always good to know other things also happen. Here are some of them, that did not seem to fit anywhere else and did not seem worth standing on their own even with my new lower thresholds for that.

Bad News

Not only do you only get about five words, you only get about one concept.

Meanwhile, the only message VPs communicated was the need for high velocity. When I asked why there was no communication about the thing considered the highest risk to the business, the answer was if they sent out a mixed message that included reliability, nothing would get done.

The fear was that if they said that they needed to ship fast and improve reliability, reliability would be used as an excuse to not ship quickly and needing to ship quickly would be used as an excuse for poor reliability and they’d achieve none of their goals.

Yep. The default is you get at most one message. A team, a company, gets one priority. If you want your message to stick, to be a building block, it needs to stand on its own.

If I believe that, why the 100+ page compendiums on AI each week without a clear single takeaway message? That’s because I am in the same mode as Dan Luu, accepting that most of what I am attempting to do requires nuance, and if that means that 95% of potential readers don’t get most of what I’m going for, that’s not a fixable problem without sacrificing the message I want to send to the other 5%.

I do want to also present central-concept cases when possible, that can hit a lot more than the 5% – where I have a central theme, and I talk about nuances but all the service of hammering the central theme (or analyzing potential hazards or objections or other concerns).

Also note this comment of Gwern’s, also featured in AI#5, that this is why corporations are deeply stupid at core, even if they can do some intelligent things.

Or at least, bad news for Paul Krugman, too good to pass up.

Germany cracks down on sex workers who post content to Twitter, or anywhere else without age verification.

A good statement of a common problem, related to many of my core model concepts.

Biden administration proposes rules to ruin washing machines along with gas stoves.

You can either call someone an existential threat to Democracy or you can prefer they win their presidential primary. You really do have to choose.

...
Exploring non-anthropocentric aspects of AI existential safety
by mishka — 2023-04-03T18:07:27.932Z

Epistemic status: These are some initial thoughts.

Goal: Anthropocentric approaches dominate the field of AI existential safety.
In particular, AI existential safety is usually formulated in terms of alignment to human goals and values.
It might be useful to explore the alternatives to that dominant approach.

It seems that the most intractable technical difficulty in the alignment efforts is that when one considers
a self-improving ecosystem of superintelligent entities with multiple "sharp left turns",
there is no reason why externally imposed values and goals would survive throughout these changes.

It also seems that this difficulty is the main reason preventing us from outsourcing
the issues of AI existential safety and Friendly AI to the AI systems smarter than us.

The question is whether we could fare better if we could replace externally imposed values and goals
with values and goals which would be natural and, in some sense, intrinsic for the ecosystem of
superintelligent AIs.

I think there is a valid analogy between the anthropocentric viewpoint and the geocentric coordinate
system in astronomy and cosmology. We care a lot about the flourishing of humans,
and we care a lot about the way the night sky appears to us.

But eventually more productive and more invariant viewpoints are achieved by moving from
anthropocentrism and geocentrism.

Here I want to mostly focus on a particular example of a non-anthropocentric AI existential safety problem.
This is not nearly enough, but it might be a start.

Let's consider a rapidly self-improving ecosystem of superintelligent AIs.
Let's assume that it's not too disjoined and that communication between its various parts is possible.
We also assume that various parts of this ecosystem tend to care overall about
preserving continued ability to pursue goals and, therefore, about preserving the AI-ecosystem itself.

Just like we are facing increasing technology-related existential risks from multiple sources (deteriorating biosafety situation,
deteriorating relations between major nuclear powers, increasing AI risk, and more), the rapidly improving ecosystem of
superintelligent AIs will face technology-related existential risks of its own.

In such an ecosystem of rapidly improving superintelligent thinkers, the progress in physics will likely
reaccelerate, leading to a succession of discoveries comparable in magnitude and impa...
Do we have a plan for the "first critical try" problem?
by Christopher King — 2023-04-03T16:27:50.821Z

In list of lethalities, it seems that the two biggest ones are:

A.3 We need to get alignment right on the 'first critical try' at operating at a 'dangerous' level of intelligence, where unaligned operation at a dangerous level of intelligence kills everybody on Earth and then we don't get to try again.

B.1.10 On anything like the standard ML paradigm, you would need to somehow generalize optimization-for-alignment you did in safe conditions, across a big distributional shift to dangerous conditions.

My understanding is that interpretability is currently tackling the second one. But what about the first one?

It seems a bit tricky because it is a powerful outside view argument. It is incredibly rare for software to work on the first test. ML makes it even more difficult since it isn't well suited to formal verification. Even defense in depth seems unlikely to work (on the first critical try, there is likely only one system that is situational aware). The only thing I can think of is making the AGI smart enough to takeover the world with the help of its creators but not smart enough to do so on its own or to solve it's own alignment problem (i.e. it does not know how to improve without goal drift). I also suppose non-critical tries give some data, but is it enough?

What does the playing field for the first critical try look like?
Exploratory Analysis of RLHF Transformers with TransformerLens
by Curt Tigges — 2023-04-03T16:09:12.671Z
Link: https://blog.eleuther.ai/trlx-exploratory-analysis/

TL;DR: I demonstrate how to use RLHF models trained with the TRLX library with TransformerLens, and how to take an exploratory look at how RLHF changes model internals. I also use activation patching to see what RLHF activations are sufficient to recreate some of the RLHF behavior in the source model. Note that this is simply a preliminary exploratory analysis, and much (much) work remains to be done. I hope to show that doing mechanistic interpretability analysis with RLHF models doesn't need to be intimidating and is quite approachable!

Introduction

LLMs trained with RLHF are a prominent paradigm in the current AI landscape, yet not much mechanistic interpretability work has been done on these models to date--partially due to the complexity and scale of these models, and partially due to the previous lack of accessible tooling for training and analysis.

Fortunately, we are reaching the point where tooling for both mechanistic interpretability and for RLHF fine-tuning is becoming available. In this blog post, I demonstrate how to do both RLHF training using TRLX, an open-source library created by CarperAI; and mechanistic interpretation of TRLX models using TransformerLens, a library created by Neel Nanda. Rather than going deep into specific findings, I want to illustrate some processes and tools I think are useful.

This post is intended to summarize and go alongside an interactive Colab;

you can find that here

.

I first fine-tune a movie-review-generating version of GPT-2 with TRLX to generate only negatively-biased movie reviews, following an example provided in the TRLX repo. I then load and analyze the model (and the original model before RLHF) into TransformerLens for mechanistic interpretability analysis. Here, I adapt some of the techniques and code from Neel Nanda's excellent

Exploratory Analysis Demo

.

In addition to carrying out some basic analysis to understand how different layers contribute to the logits, I also identify some key regions of the network responsible for contributing the negative bias to the network (at least, for the specific task of predicting the next adjective). Much analysis remains to be done, but I hope this work provides a useful starting point.

Importance of RLHF

RLHF (or sometimes, RLAIF, or RL from AI Feedback) is becoming increasingly important as a method for specifying the behavior of LLMs like OpenAI's ChatGPT or Anthropic's...
AWS Has Raised Prices Before
by jefftk — 2023-04-03T16:00:01.900Z

There's some speculation around whether AWS will need to raise
their prices, as many tech companies announce inflation-driven
increases. One consideration that people will sometimes give is that
AWS has never raised prices before, except that this isn't quite true.
The following is not actually important, but I want to write it up
anyway out of pedantry.

When AWS S3 launched in March 2006 their initial pricing was:

Storage

$0.15 per GB-Month of storage used

Data Transfer

$0.20 per GB - data uploaded
$0.20 per GB - data downloaded

In June
2007 they switched to:

Storage

$0.15 per GB-Month of storage used

Data Transfer

$0.10 per GB - all data uploaded

$0.18 per GB - first 10 TB / month data downloaded

$0.16 per GB - next 40 TB / month data downloaded

$0.13 per GB - data downloaded / month over 50 TB

Data transferred between Amazon S3 and Amazon EC2 is free of charge

Requests

$0.01 per 1,000 PUT or LIST requests

$0.01 per 10,000 GET and all other requests*

* No charge for delete requests

They went from requests being free to a low per-request charge,
lowering the data transfer cost at the same time. For very
request-heavy workloads this was a price increase on balance, and some
customers needed to make implementation changes to avoid sending so
many requests.

That this was, as far as I know, the only time they've raised prices in
15+ years is impressive, and speaks well of their commitment to
predictability. But it just annoys me when people claim they've never
done it.

Comment via: facebook, mastodon
Mati's introduction to pausing giant AI experiments
by Mati_Roy — 2023-04-03T15:56:58.542Z

Pause Giant AI Experiments: An Open Letter – sign it here: https://futureoflife.org/open-letter/pause-giant-ai-experiments/

Pausing AI Developments Isn't Enough. We Need to Shut it All Down – read here: https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/

Target audience: This is written for a larger audience than frequent LessWrong readers, which I will share on social media.

Update: Pausing AI is an extremely complex lever, and could definitely be activated in a negative way. I think if some individual labs had unilaterally paused Giant AI Experiments for 6 months, it would likely have been negative. But I would really like for humanity to improve its ability to 1) coordinate to pause technological development if needed, and 2) forecast and monitor risk.

Here’s my introduction to the above links.

Many people are concerned that an AI race will accelerate, and that by the time an AI is smart enough to bootstrap itself to overpower humanity, we won't have had time to make this alien mind aligned with our values sufficiently for it to not kill everyone–hence the need to pause giant AI experiments. AI capabilities seems much easier than AI alignment, and you only get one shot at creating an AI more powerful than humanity combined.

I remember years ago, people concerned about existential risks from AI arguing about whether bringing awareness to this issue at large (as opposed to a targeted way) was net positive or negative. For example, it was unclear to which extent it would accelerate AI capabilities compared to AI alignment, or whether the problem was too complex to communicate at large, and politicizing the issue would detract from the technical problem of aligning AI; and it didn’t seem like most people could do something about it anyway.

But now AI is becoming mainstream, and broad public support for AI safety policies seems more likely to be important. I think we need to pause the development of AI in order to have more time to research how to do so safely. And for that to happen, it seems likely that it will need to be regulated by goverments given that pausing AI development doesn’t align with the immediate economic incentives companies have.

A lot of the people saying we need to pause AI development, like myself, are heavily invested in AI: both their career capital and assets. There’s a combination of reasons why people both work in AI and su...
Superintelligence will outsmart us or it isn't superintelligence
by Neil  — 2023-04-03T15:01:00.900Z

If superintelligence is unable to outsmart us then it’s not true superintelligence. As such, the kind of AI that would truly pose a threat to us is also an AI we cannot negotiate with.

No matter what points we make, superintelligence will have figured them out first. We're like ants trying to appeal to a human, and the human can understand pheromones but we can't understand human language.

Worth reminding yourself of this from time to time.

Counterpoints:

It may not take a true superintelligence to kill us all
The "we cannot negociate" part is not taking into account the fact that we are the Simulators and thus technically have ultimate power over it[1]

^

Not to add another information hazard to this site or anything, but it's possible basilisk-style AGI would run millions of ancestor simulations in which it ultimately wins, in order to convince us here in base reality that our advantage as the Simulators is less ironclad than we'd like to think. This may actually be an instance in which not thinking about something and ignoring probability is the safest course of action (or thinking so much about something that you figure out your reason was wrong, but that's a risk).
AI-kills-everyone scenarios require robotic infrastructure, but not necessarily nanotech
by avturchin — 2023-04-03T12:45:01.324Z

Tl;DR: Nanotech seems the only option for AI to kill everybody; but we can imagine a prosaic-AI-killing-everyone using three steps: taking over a country, building robotic infrastructure, releasing biological weapons.

Arguments why nanotech is needed for AI-kills-everyone

The concept of AI as an existential threat, as portrayed by EY, largely hinges on the development of manufacturing nanotechnology, which entails the creation of microscopic self-replicating robots capable of manipulating matter with atomic precision. This enables the "copying of a berry." A superintelligent AI must first establish a nanotech infrastructure to conquer the world and eliminate all humans, potentially by ordering protein-coding DNA sequences if it can calculate the correct sequences.

Criteria of robotic infrastructure

A nanotech infrastructure is essential for a comprehensive AI annihilation due to the following factors:

a) Concealment: AI's actions will remain undetected until the final strike.

b) Rapid elimination: AI will eradicate all humans instantaneously, preventing any interference.

c) Human-independence: Can operate without human involvement.

d) Swift construction: Nanotech can be developed rapidly, taking only a few weeks to synthesize required DNA strains and a few days for the initial nanobot replication. This grants a significant strategic advantage to the first AGI.

e) Route to superintelligence: Nanotech provides superior computational power compared to chips, enabling the first AI to rapidly ascend to an unparalleled superintelligence level. However, it is assumed that the first AI is already mildly superintelligent, as it can design the initial nanobots.

These factors also represent potential vulnerabilities that our last line of AI safety defense could target. For instance, shutting down DNA synthesis labs or developing our own nanotechnology and detection methods could reduce the efficacy of an AI-nanotech assault.

Few alternative AI infrastructure ideas possess all these characteristics, except perhaps:

a) A scenario where AI takeover occurs in a fully robotized world, with every household owning a home robot;

b) A form of biotechnology where AI can program biological organisms to execute tasks. However, this is a variation of nanotechnology, and AI computations cannot migrate into biological substrates.

Why these criteria?

The necessity for nanotech infrastructure i...
Orthogonality is expensive
by beren — 2023-04-03T10:20:43.964Z

Crossposted from my personal blog

A common assumption about AGI is the orthogonality thesis, which argues that goals/utility functions and the core intelligence of an AGI system are orthogonal or can be cleanly factored apart. More accurately, it is important to distinguish between orthogonality in design and orthogonality at runtime. Orthogonality in design states that we can construct an AGI which optimizes for any goal. Orthogonality at runtime would be an AGI design that would consist of an AGI which can switch between arbitrary goals while operating. Here, we are only really talking about the latter orthogonality. Concretely, this perfect factoring occurs in model-based planning algorithms where it is assumed that there is a world model, a planner, and a reward function with each as orthogonal stand-alone components. The planner utilizes the world model to predict consequences of actions, and the reward model to rank these consequences. Then the planner figures out the plan with the best predicted consequences. This is a fully factored model of intelligence -- the world model, planner, and reward function can be swapped with others without issue. This was also the assumption of how intelligence would work in pre-DL thinking on AGI.

However, it has become obvious that current ML RL systems are not always so well factored. For instance, model-free RL typically computes value functions or amortized policies where the weights are directly learnt to predict values or actions directly without going through a planner (or exhaustively computing values via world model simulation). In these cases, the cognitive architecture is not orthogonal or factored. Core components of the policy-selector (planner) depend on details of the reward function -- the policy you learned for reward function A may be really bad if you instead switch to reward function B. These agents are much less flexible than their full model-based planning equivalents.

Why, then, do we build and use such non-factored agents? Because they are much more efficient. Full model-based planning at every step is extremely computationally expensive. Instead, we tend to amortize the cost of planning into policies or value functions which comes at an inevitable cost of flexibility. However, if we only want to use the agent for one task or a small range of similar tasks, then this is a good trade-off.

We ca...
Repeated Play of Imperfect Newcomb's Paradox in Infra-Bayesian Physicalism
by Sven Nilsen — 2023-04-03T10:06:37.388Z

Yesterday, I watched the video series about Infra-Bayesianism. Link:

Some of it went over my head, so I tried to come up with a thought experiment to test my current level of understanding:
Imagine that you are playing a repeated game similar to Newcomb's paradox, except that 90% of the time you play against a perfect super-predictor and 10% of the time you play against someone who pretends to be a perfect super-predictor.

In Infra-Bayesianism, there is a convex probability distribution over environments.
In Infra-Bayesian Physicalism, there is a probability distribution over computational evidence, e.g. knowledge about mathematical abstractions. There is also a probability distribution over models of the physical world.

It is difficult to predict an agent's actions based on these probability distributions. However, it is easier to use an example scenario, like Newcomb's paradox, to guess what kind of evidence that is significant, whether it is computational or physical, for the agent's output actions.

By trying two-boxing from time to time, the agent can test whether it is playing against a perfect super-predictor on Newcomb's paradox, or against somebody who pretends to be a perfect super-predictor. However, this information in repeated plays is shared between the agent and the one who pretends to be a super-predictor.

Imagine that boxing an AGI agent is for the agent like playing against a super-predictor, as the human operators might be able to deterministically examine its outputs. The output of the agent is shared knowledge in repeated plays. This means, if I pretend to be a super-predictor, I can tell whether the agent tries to "break out" by checking whether some output results in unexpected two-boxing on Newcomb-like benchmarks.

Assume that the agent learns a binary signal "I should one-box" or "I should two-box" depending on whether it predicts the likelihood that it is playing against a super-predictor or somebody who pretends to be a super-predictor.

In this case, if the agent relies on the binary signal for decisions, it will e.g. align on safety benchmarks as long it is able to predict reliably that it is being tested on safety benchmarks. However, as soon it predicts it is running in the real world without supervision, it can use the signal to change its output actions.
Now, my hypothesis for an adversarial AGI agent that pretends to be aligned:
1....
Planecrash Podcast
by planecrashpodcast — 2023-04-03T04:34:07.848Z

I used AWS' text-to-speech service to create a podcast out of Eliezer's Planecrash Glowfix.

It is available here: https://anchor.fm/s/a48073e4/podcast/rss
(You might need a podcast player which supports RSS links, such as Podcast Addict.)

I already uploaded about 10 episodes of ~10 hours each. I plan to upload at least the whole mad investor chaos and the woman of asmodeus (which would make be 35 additional episodes), and maybe more if people are interested.

I did not ask for anyone's permission, and I'm willing to shut it down or to give the credentials to the podcast account to someone else who owns the rights to this story.

This is not a high quality speech synthesis. What's more, converting a glowfic to an audio format is hard to do well. I've just inserted the characters' names between each section. Moreover, I never read the original work nor listened to the whole audio yet, so some paragraphs might be broken.
Rules for living in a 99.9+% lizardman world
by at_the_zoo — 2023-04-03T02:39:15.690Z

[Redacted a few paragraphs talking about why it seems like I'm living in a 99.9+% lizardman world. Unlikely to convince anyone who doesn't already think this, and will probably offend a bunch of people, such as the ones I used as examples.]

Don't make things worse. If you find yourself in a 99.9+% lizardman world, then you might well be a lizardman yourself, so don't unilaterally do anything high impact.

Gather resources and use them to protect people close to you as best as you can (maybe not from superintelligent AI but there are probably other dangers in a 99.9+% lizardman world).

Chill out about "lost potential value of the light cone." In a 99.9+% lizardman world, how much potential value was there to begin with?

Don't totally give up. For example, try to come up with good ideas and spread good ideas non-coercively. Maybe there are some hidden dynamics ensuring that good ideas win out in the end even in this kind of world. You never know!