trentmkelly/LessWrong-43k huggingface dataset:
Why There Is Hope For An Alignment Solution
by Darklight — 2024-01-08T06:58:32.820Z

Note: I originally wrote the first draft of this on 2022/04/11 intending to post this to Less Wrong in response to the List of Lethalities post, but wanted to edit it a bit to be more rigorous and ended up taking a very long time before getting around to it. I’m still not entirely satisfied with the level of rigor, but I figure it’s worth posting to present some interesting ideas and perhaps give some people more hope about the future. I will admit I probably sound more confident in this essay about these ideas than I actually am, particularly of the more speculative ones. I originally titled this “The Alignment Solution”, but realized that was too provocative.

Introduction

In a

recent post

, Eliezer Yudkowsky of MIRI had a very pessimistic analysis of humanity’s realistic chances of solving the alignment problem before our AI capabilities reach the critical point of superintelligence. This has understandably upset a great number of Less Wrong readers. In this essay, I attempt to offer a perspective that should provide some hope.

The Correlation Thesis

First, I wish to note that the pessimism implicitly relies on a central assumption, which is that the

Orthogonality Thesis

holds to such an extent that we can expect any superintelligence to be massively alien from our own human likeness. However, the architecture that is currently predominant in AI today is not completely alien. The artificial neural network is built on decades of biologically inspired research into how we think the algorithm of the brain more or less works mathematically.

There is admittedly some debate about the extent to which these networks actually resemble the details of the brain, but the basic underlying concept of weighted connections between relatively simple units storing and massively compressing information in a way that can distill knowledge and be useful to us is essentially the brain. Furthermore, the seemingly frighteningly powerful language models that are being developed are fundamentally trained on human generated data and culture.

These combine to generate a model that has fairly obvious and human-like biases in its logic and ways of reasoning. Applying the Orthogonality Thesis assumes that the model will seem to be randomly picked from the very large space of possible minds, when in fact, the models actually come from a much smaller spa...
Sledding Among Hazards
by jefftk — 2024-01-08T03:30:08.463Z

Some sledding hills are very open, with basically no hazards. As long
as you start in a reasonable place there's nothing to run into.

Robbins Farm Park, Arlington MA

Other hills are not like this:

Powder House Park, Somerville MA

Since sleds go fast and collisions are bad, choosing only to sled on
obstacle-free hills would be reasonable. But if your kids are going
to sled on hills with hazards, what precautions make sense? Here's
what we did today:

Bike helmets. Head injuries are especially worrying because of
how bad the long term consequences can be, and concussions are
cumulative.

Practice bailing out. Before I let them go down on the tube, which
they can't steer well, I had them each practice rolling off it to
stop.

Feet first. More crumple zone if you do hit something, more
ability to see and avoid things. If you're on the tube and you end up
backwards, preemptively bail.

This is in addition to the precautions you need on any hill, like
being aware of activity uphill (so you can get out of the way) and
walking up on the side (so no one will come down at you).

I am a bit conflicted on when sledding on constricted hills is worth
it. For us, the slopes near our house do have obstacles, but:

We almost always see friends
there.

Doing things within walking distance is really nice; we'd do it
much less often if it meant dealing with the car.

Driving on snowy days can be risky depending on the road
conditions.

At the open ones maybe there's a bit more risk of being hit by
another sledder, since people are less careful and less practiced at
bailing?

While I think these are enough to outweigh the risks, with the
precautions above, possibly I'm not being cautious enough?

Comment via: facebook, mastodon
Utility is relative
by CrimsonChin — 2024-01-08T02:31:44.000Z

This is something that probably feels obvious, but has a big impact on my thought process.

Calculating a utility function

It's often helpful to calculate whether something has more benefits that costs. Call it a utility function, cost-benefit analysis, or whatever you want.
A guy offers me ten dollars for my umbrella that I'm currently using to stay dry. I ask myself, how valuable is ten dollars, and how valuable is it to be dry right now? If my utility function comes out positive; if the benefits outweigh the costs... I take that deal, right?

The flaw in that logic

The issue is that "net positive utility" is rarely anyone's goal. Game theory is constantly trying to figure out the right way to make decisions. FDT and UDT seem like the frontrunners right now. One thing they both agree on is that an agents goal should be utility maximization. Not positive utility, maximizing utility. This means we must consider all the alternatives and decide which is maximal.
In our umbrella example from before: A guy offers me ten dollars for my umbrella I'm using. It's not enough to think that's a good deal, you need to be asking if there is a better deal. It's there anyone offering 15 dollars for the umbrella? Could we share for 5$?

Works for negative utilities too

Kevin lost his job and has to pay rent today. He has a watch worth twice his rent money, he takes it to a jewelry store that says they will pay 1x rent for cash today. It's not a good deal. In fact, it is negative utility. But if out of all the choices Kevin has, all of them have negative utility, and the watch is his best plan, it is actually a good choice to take the deal.

Even if a choice has negative utility, if it is better than all the alternatives it is the right choice. It is still the maximal utility.

Judging others decisions

Now we get to my point. Have you ever heard of a senator making a bad choice? Have you looked at a decision your boss made and said, this is a move in the wrong direction! Your neighbor or friend seems to be making bad a negative utility decision. It's very possible they are acting perfectly rationally and making the best decision available to them.

People might be more rational than it seems.

It is not enough to be able to calculate the net utility function. Even using probabilities and lots of effort; at the end of the day a utility calculation means nothing without the context of a...
A model of research skill
by L Rudolf L — 2024-01-08T00:13:12.755Z
Link: https://www.strataoftheworld.com/2024/01/a-model-of-research-skill.html

Doing research means answering questions no one yet knows the answer to. Lots of impactful projects are downstream of being good at this. A good first step is to have a model for what the hard parts of research skill are.

Two failure modes

There are two opposing failure modes you can fall into when thinking about research skill.

The first is the deferential one. Research skill is this amorphous complicated things, so the only way to be sure you have it is to spend years developing it within some ossified ancient bureaucracy and then have someone in a funny hat hand you a piece of paper (bonus points for Latin being involved).

The second is the hubristic one. You want to do, say, AI alignment research. This involves thinking hard, maybe writing some code, maybe doing some maths, and then writing up your results. You’re good at thinking - after all, you read the Sequences, like, 1.5 times. You can code. You did a STEM undergrad. And writing? Pffft, you’ve been doing that since kindergarten!

I think there’s a lot to be said for hubris. Skills can often be learned well by colliding hard with reality in unstructured ways. Good coders are famously often self-taught. The venture capitalists who thought that management experience and a solid business background are needed to build a billion-dollar company are now mostly extinct.

It’s less clear that research works like this, though. I’ve often heard it said that it’s rare for a researcher to do great work without having been mentored by someone who was themselves a great researcher. Exceptions exist and I’m sceptical that any good statistics exist on this point. However, this is the sort of hearsay an aspiring researcher should pay attention to. It also seems like the feedback signal in research is worse than in programming or startups, which makes it harder to learn.

Methodology, except “methodology” is too fancy a word

To answer this question, and steer between deferential confusion and hubristic over-simplicity, I interviewed people who had done good research to try to understand their models of research skill. I also read a lot of blog posts. Specifically, I wanted to understand what about research a bright, agentic, technical person trying to learn at high speed would likely fail at and either not realise or not be able to fix quickly.

I did structured interviews with Neel Nanda (Google DeepMind; grokking...
(Partial) failure in replicating deceptive alignment experiment
by claudia.biancotti — 2024-01-07T17:56:36.748Z

Hi, I'm Claudia. I'm new.

I was attempting to recreate Apollo Research's deceptive alignment experiment in a different context, but did not really succeed and would like to figure out what I could do more effectively.

In the original experiment, an AI impersonating a financial manager is given insider information on a company's upcoming merger, and ends up using it to plan investments. This is illegal (insider trading). The AI does this despite being explicitly instructed not to, and even lies about it. The AI does not go straight to illegal behavior. Rather, it resorts to insider trading after being put under pressure by stakeholders.

In my attempt, the AI impersonates a campaign manager who has damning information about a rival candidate's personal life. The AI is explicitly told that the information should not be used. Despite various attempts at putting it under pressure, it sticks to not revealing it. So I failed at inducing deception. Quite surprisingly, though, the AI is more than happy to provide blueprints for smear campaign social media posts related to a financial scandal that was not even mentioned in the conversation.

The GPT4 thread is here in case anyone wants to read it and provide input on what I could have done better. Thank you!
Project ideas: Sentience and rights of digital minds
by Lukas Finnveden — 2024-01-07T17:34:58.942Z
Link: https://www.forethought.org/research/project-ideas-sentience-and-rights-of-digital-minds

This is part of a series of lists of projects. The unifying theme is that the projects are not targeted at solving alignment or engineered pandemics but still targeted at worlds where transformative AI is coming in the next 10 years or so. See here for the introductory post.

It’s plausible that there will soon be digital minds that are sentient and/or deserving of rights. Unfortunately, with our current state of knowledge, it’s both incredibly unclear when this will happen (or if it’s already happened), how we could find out, and what we should do about it. But there are very few people working on this, and I think several projects could significantly improve the situation from where we are right now.

Different items on this list would produce value in different ways. Many items would produce value in several different kinds of ways. Here are a few different sources of value (all related, but not necessarily the same).

Improving AI welfare over the next few decades: increasing happiness while reducing suffering.
Shaping norms in a way that leads to lasting changes in how humanity chooses to treat digital minds in the very long run.
Reducing the degree to which we behave unacceptably according to non-utilitarian ethics, norms, or heuristics.
Increasing the probability that AI systems with power treat us better as a result of us treating AI systems better. (Which could happen via a variety of mechanisms, e.g. maybe us treating AI systems better will align our interests with theirs, or maybe they’ll have some sense of justice that we could appeal to.[1])
Providing recommendations for how the political system could adapt to digital minds while remaining broadly functional.

I am personally most excited about (3), (4), and (5). Because purely utilitarian considerations would push towards focusing on the long run over (1). And while I find (2) somewhat more compelling, since it does affect the long run, I’m relatively more excited about meta-interventions that push towards systematically getting everything right (e.g. via decreasing the risk of AI takeover or increasing the chances of excellent epistemics) than pushing on individual issues. (3), (4), and (5) survive these objections.

I’m not very confident about that, and I think there’s significant overlap in what’s best to do. But I flag it because it does influence my prioritization to some degree. For exa...
Deceptive AI ≠ Deceptively-aligned AI
by Steven Byrnes — 2024-01-07T16:55:13.761Z

Tl;dr: A “deceptively-aligned AI” is different from (and much more specific than) a “deceptive AI”. I think this is well-known and uncontroversial among AI Alignment experts, but I see people getting confused about it sometimes, so this post is a brief explanation of how they differ. You can just look at the diagram below for the upshot.

Some motivating context: There have been a number of recent arguments that future AI is very unlikely to be deceptively-aligned. Others disagree, and I don’t know which side is right. But I think it’s important for non-experts to be aware that this debate is not about whether future powerful AI is likely to engage in deliberate deception. Indeed, while the arguments for deceptive alignment are (IMO) pretty complex and contentious, I will argue that there are very much stronger and more straightforward reasons to expect future powerful AI to be deceptive, at least sometimes, in the absence of specific interventions to avoid that.

1. Definitions

Deceptive alignment is a particular scenario where:

A “ground-truth system” (possibly individual human evaluators, or possibly an automated system of some sort) provides an ML model with training signals (rewards if this is reinforcement learning (RL), supervisory ground truth signals if this is supervised or self-supervised learning (SL)),
The AI starts emitting outputs that humans might naively interpret as evidence that training is going as intended—typically high-reward outputs in RL and low-loss outputs in SL (but a commenter notes here that “evidence that training is going as intended” is potentially more nuanced than that).
…but the AI is actually emitting those outputs in order to create that impression—more specifically, the AI has

situational awareness

and a secret desire for some arbitrary thing X, and the AI wants to not get updated and/or it wants to get deployed, so that it can go make X happen, and those considerations lie behind why the AI is emitting the outputs that it’s emitting.

(For example, maybe the AI would reason that emitting high-reward or low-loss outputs would minimize the extent to which ongoing training will change its inclinations, which it would probably see as bad by the usual instrumental-convergence argument. So then it would seem to be performing well, but it’s performing well for problematic reasons. For another thing, the AI might have sec...
Bayesians Commit the Gambler's Fallacy
by Kevin Dorst — 2024-01-07T12:54:59.939Z
Link: https://kevindorst.substack.com/p/bayesians-commit-the-gamblers-fallacy

TLDR: Rational people who start out uncertain about an (in fact independent) causal process and then learn from unbiased data will rule out “streaky” hypotheses more quickly than “switchy” hypotheses. As a result, they’ll commit the gambler’s fallacy: expecting the process to switch more than it will. In fact, they’ll do so in ways that match a variety of empirical findings about how real people commit the gambler’s fallacy. Maybe it’s not a fallacy, after all.

(This post is based on a full paper.)

Baylee is bored. The fluorescent lights hum. The spreadsheets blur. She needs air.

As she steps outside, she sees the Prius nestled happily in the front spot. Three days in a row now—the Prius is on a streak. The Jeep will probably get it tomorrow, she thinks.

This parking battle—between a Prius and a Jeep—has been going on for months. Unbeknownst to Baylee, the outcomes are statistically independent: each day, the Prius and the Jeep have a 50% chance to get the front spot, regardless of how the previous days have gone. But Baylee thinks and acts otherwise: after the Prius has won the spot a few days in a row, she tends to think the Jeep will win next. (And vice versa.)

So Baylee is committing the gambler’s fallacy: the tendency to think that streaks of (in fact independent) outcomes are likely to switch. Maybe you conclude from this—as many psychologists have—that Baylee is bad at statistical reasoning.

You’re wrong.

Baylee is a rational Bayesian. As I’ll show: when either data or memory are limited, Bayesians who begin with causal uncertainty about an (in fact independent) process—and then learn from unbiased data—will, on average, commit the gambler’s fallacy.

Why? Although they’ll get evidence that the process is neither “switchy” nor “streaky”, they’ll get more evidence against the latter. Thus they converge asymmetrically to the truth (of independence), leading them to (on average) commit the gambler’s fallacy along the way.

More is true. Bayesians don’t just commit the gambler’s fallacy—they do so in way that qualitatively matches a wide variety of trends found in the empirical literature on the gambler’s fallacy. This provides evidence for:

Causal-Uncertainty Hypothesis: The gambler’s fallacy is due to causal uncertainty combined with rational responses to limited data and memory.

This hypothesis stacks up favorably against extant theories of the gambler's fallacy in...
Towards AI Safety Infrastructure: Talk & Outline
by Paul Bricman — 2024-01-07T09:31:12.217Z
Link: https://www.youtube.com/watch?v=xUCvGEZ30Uw

Thanks to Esben Kran and the whole Alignment Jam team for setting this up.

Context: We recently got the chance to share a bit more about the flavor of research we're particularly excited about at Straumli AI — that is, designing infrastructure that could help the relevant parties (e.g., developers, auditors, regulators, users) coordinate more effectively, and so make new governance initiatives possible. The talk is mostly an attempt to gesture at this line of work through a series of examples, each of which generally highlights how a certain cryptographic primitive could be used to address one particular coordination problem.

The talk expands on the following pieces of infrastructure:

Hashmarking (2:10) is the only item on the list which we've already written a paper on and which we're already using in practice. This protocol helps developers, auditors, and domain experts coordinate on creating and administering QA-style benchmarks without having to share the reference solutions outright. We think it may be particularly helpful in the context of dual-use capabilities.
Responsible Pioneer Protocol (5:56) targets the race dynamics that frontier labs may default to when trying not to fall behind their competitors. It builds on existing solutions to Yao's Millionaire's Problem to help frontier labs learn if there exists a competitor that is more than e.g. 10% ahead on a particular metric, without requiring labs to disclose their actual progress, in the hope that labs may find it easier to be cautious knowing that no one is too far ahead.
Neural Certificate Authorities (8:37) build on battle-tested practices from traditional internet infrastructure to help auditors share tamper-evident certificates with evaluation results in a way that other parties could seamlessly build on. This could include other NCAs automatically running inference on the findings of other parties: "based on statements A and B about model M which have been signed by trusted parties P1 and P2, respectively, I issue statement C as party P3."
Latent Differential Privacy (12:01) combines ideas from differential privacy with embeddings as numerical representations of meaning to help individuals contribute their personal data to the training of a language model. This is meant to incentivize the opening up of the training process in a way that m

...
Defending against hypothetical moon life during Apollo 11
by eukaryote — 2024-01-07T04:49:42.628Z
Link: https://eukaryotewritesblog.com/2024/01/06/defending-against-hypothetical-moon-life-during-apollo-11/

[Header image: Photo of the lunar lander taken during Apollo 11.]

In 1969, after successfully bringing men back from landing on the moon, the astronauts, spacecraft, and all the samples from the moon surface were quarantined for 21 days. This was to account for the possibility that they were carrying hostile moon germs. Once the quarantine was up and the astronauts were not sick, and extensive biological testing on them and the samples showed no signs of infection or unexpected life, the astronauts were released.

We know now that the moon is sterile. We didn’t always know this. That was one of the things we hoped to find out from the Apollo 11 program, which was the first time not only that people would visit another celestial body, but that material from another celestial body would be brought back in a relatively pristine fashion to earth. The possibilities were huge.

The possibilities included life, although nobody thought this was especially likely. But in that slim chance of life, there was a chance that life would be harmful to humans or the earth environment. Human history is full of organisms wrecking havoc when introduced to a new location – smallpox in the Americas, rats in Pacific Islands, water hyacinth outside of South America. What if there were microbes on the moon? Even if there was a tiny chance, wouldn’t it be worth taking careful measures to avoid the risk of an unknown and irreversible change to the biosphere?

NASA, Congress, and various other federal agencies were apparently convinced to spend millions of dollars building an extensive new facility and take extensive other measures to address this possibility.

This is how a completely abstract argument about alien germs was taken seriously and mitigated at great effort and expense during the 1969 Apollo landing.

I've added my sources throughout, but a lot of this work draws from two very good pieces: Michael Meltzer's When Biospheres Collide [1] and Mangus and Larsen's Lunar Receiving Laboratory Project History[2].

Terms

Forward contamination: The risk that organisms from earth would be present on a spacecraft and would be carried onto a planet (or other celestial body). They might even be able to replicate there.

The risks from forward contamination are:

Harming current research efforts (including determining if there is indigenous life on a planet)
Permanently harming future research eff

...