trentmkelly/LessWrong-43k huggingface dataset:
Effects are correlated, policy outcomes are not, and multi-factor explanations are hard
by Thomas Kehrenberg — 2022-03-09T15:30:53.251Z
Link: https://tm.kehrenberg.net/a/consequences-are-correlated-policy-outcomes-and-causes-are-not/

(Repeating some of “Policy debates should not appear one-sided”, but expanding upon it and then countering “Multiple Factor Explanations Should Not Appear One-Sided”.)

If asked directly, most people who consider themselves sophisticated thinkers would probably agree that “acknowledging trade-offs” regarding your proposed policy is the right thing to do. In other words, they would agree that policy debates should not appear one-sided. But if you only ever hear that side of the message, you might apply it to questions of facts as well, where it is not the correct stance. So, let us first investigate why questions of fact should actually appear one-sided.

The evidence for facts is one-sided

Consider the question of whether Earthly life arose by natural selection. If this were true, then it would have had a profound impact on everything in our world. That is, the answer to the question of whether or not the theory of natural selection is true, strongly affects the answer to other questions we can ask about the world. The following graphical model is meant to visualize this. The arrows indicate the direction of causality: if Earthly life arose by natural selection, then this is the cause (or one cause) for our finding fossils of extinct animals. It is not the fact that fossils came first and somehow caused natural selection.

(Evo: “Earthly life arose by natural selection”,
Fos: “there are fossils from extinct species”,
DNA: “the genetic code of all life on Earth is encoded by DNA”,
Nic: “species occupy their own peculiar ecological niche”,
Emb: “embryos from different species look very similar”,
Psy: “human psychology is adapted to the living environment during the pleistocene”)

Now, suppose you have heard of this theory but you haven't seen any evidence yet. (A highly unusual situation for a scientist – the hard part always is coming up with the hypothesis, and not finding the evidence.) So, you have the theory and the graph it induces, but you don’t yet know whether the graph actually corresponds to reality; you cannot tell with your current state of knowledge. One day, you find fossils. This increases the credence you give to this theory but it's not certain yet; there may yet be other plausible theories. Next, you observe Darwin's finches on the Galápagos Islands and it strikes you how well they are adapted to different food sources. Evo goes up in probability a...
Ask AI companies about what they are doing for AI safety?
by mic — 2022-03-09T15:14:29.352Z

Cross-posted from the EA Forum.

Today, I had the pleasure of attending a talk by Jeff Dean at my university (Georgia Tech). Titled “Five Exciting Trends in Machine Learning,” it was a fascinating, engaging presentation. Midway through the talk, I started to idly wonder, if there ends up being a Q&A after the talk, could I ask a question about AI safety? I started drafting a question on my phone.

After the talk, there was in fact time for questions. The moderator took one question from one person sitting a few rows in front of me, and I wished I had raised my hand earlier and sat closer to the front. Then, the moderator read aloud two questions from people watching the livestream, and they asked their own question. Jeff Dean was still available for questions after the talk, however. I ended up asking him something like the following:

“One major focus of DeepMind is aligning ML models to follow what humans want, rather than narrowly pursuing objectives that are easy to specify. You mentioned the trend of how AI is becoming increasingly capable and general. If this continues, and if we had a highly advanced general AI and want it to cure cancer, one solution to that objective would be to kill everyone, but that would be pretty bad. So it seems important to be able to figure out how to specify the objectives that we actually want, and some exciting approaches to this include reward modeling or iterated amplification. Is this a problem that Google AI is working on or plans to work on, like its sister company DeepMind?”

I don’t think that was the optimal way of asking about AI alignment, but that’s what I ended up asking. (If anyone has suggestions on how to talk about long-term AI risk in a better way, please leave a comment!)

His response was essentially, Google AI is doing some work on AI safety. Google AI focuses more on near-term stuff, while DeepMind starts from the perspective of thinking about super AGI. He’s more optimistic that we'll be able to solve these issues as we go and that we'll have the constraints necessary to prevent an AI from killing everyone, while he does appreciate that some people are approaching this from a more paranoid perspective.

I thanked him for his thoughts. In total, I think Jeff Dean was asked just around ten questions.

I remember reading a quote that went something like, “At every town hall, ask your representatives what they are...
[Intro to brain-like-AGI safety] 7. From hardcoded drives to foresighted plans: A worked example
by Steven Byrnes — 2022-03-09T14:28:19.729Z

(Last revised: July 2024. See changelog at the bottom.)

7.1 Post summary / Table of contents

Part of the

“Intro to brain-like-AGI safety” post series

.

(Much of this post had been originally published here.)

The previous post presented a big picture of how I think motivation works in the human brain, but it was a bit abstract. In this post, I will walk through an example. To summarize, the steps will be:

(Section 7.3) Our brains gradually develop a probabilistic generative model of the world and ourselves;
(Section 7.4) There’s a “credit assignment” process, where something in the world-model gets flagged as “good”;
(Section 7.5) There’s a “valence” signal roughly tracking the expected probability of the “good” thing, along with processes that systematically lead to higher-valence thoughts over lower-valence ones. This signal drives us to “try” to make the “good” thing happen, including via foresighted planning.

All human goals and motivations come ultimately from relatively simple, genetically-hardcoded circuits in the Steering Subsystem (hypothalamus and brainstem), but the details can be convoluted in some cases. For example, sometimes I’m motivated to do a silly dance in front of a full-length mirror. Exactly what genetically-hardcoded hypothalamus or brainstem circuits are upstream of that motivation? I don’t know! Indeed, I claim that the answer is currently Not Known To Science. I think it would be well worth figuring out! Umm, well, OK, maybe that specific example is not worth figuring out. But the broader project of reverse-engineering certain aspects of the human Steering Subsystem (see my discussion of “Category B” in

Post #3

)—especially those upstream of social instincts like altruism and status-drive—is a project that I consider desperately important for AGI safety, and utterly neglected. More on that in Posts #12–#13.

In the meantime, I’ll pick an example of a goal that to a first approximation comes from an especially straightforward and legible set of Steering Subsystem circuitry. Here goes.

Let’s say (purely hypothetically… 👀) that I ate a slice of

prinsesstårta

cake two years ago, and it was really yummy, and ever since then I’ve wanted to eat one again. So my running example of an explicit goal in this post will be “I want a slice of prinsesstårta”.

Prinsesstårta cake. I suggest eating some, in order to be

...
Against Thinking About The Unactionable
by George3d6 — 2022-03-09T11:55:07.756Z
Link: https://www.epistem.ink/p/against-thinking-about-the-unactionable?s=w

There's always something that bothered me about people that talk too much about politics, especially at a national or global scale, especially using abstract concepts like "capitalism" and "socialism" rather than pointing to specific laws and regulations.

There's always something that bothered me about people that talk too much about history. Especially the kind where they focus on specific regimes, or battles, or empires or rulers... the kind of people that will argue about what-ifs during the world wars.

There's always something that bothered me about the areas of science that deal with items that can't be proven experimentally. Such as cosmology and geology, making models 1001 inferential steps away from what can be observed, for the sake of "understanding" rather than for pragmatic purposes (predictive power).

i - Usefulness

I think the best way to summarize this is it bothers me when people talk about unactionable things.

There's obviously no hard line between the unactionable and the difficult. Two hundred years ago we may have said that a theory of evolution or genetics is "unactionable", but it turns out that with our modern tooling it's quite actionable, and in part, our modern tooling may have been developed due to those primitive theories. A hundred years ago we may have said relativity is "unactionable", but it turns out that it's rather essential once you want to launch and use thousands of satellites.

But some things do seem clearly unactionable. Talking about poorly documented battles 2000 years ago as though they bear any relevance to anything except a macabre enjoyment of territorial violence is just silly. Talking about "how the universe began" or "how the universe will end" is equally silly.

What about talking politics?

Let's take a simple example like "debating the benefits/drawbacks of socialism". If you are a countries president, doing this might be somewhat relevant. Yes, socialism is a vague concept, but going in front of a nation and saying "this government will implement/retract socialist reforms" is highly relevant, it gives your subordinates and your population a sense of direction without having to read 5000 pages of laws. Similarly, debating it with an opponent is useful, because, once again, neither of you can read the 5000 pages of laws your teams are proposing. Having it as a general idea when approving the laws your team is proposin...
Will Russia default, and if so, will it lead to financial crisis?
by sanxiyn — 2022-03-09T06:34:00.567Z

I live in South Korea. Apart from threat of nuclear war, which seems low so far, the primary impact of Ukraine war seems to be potential default of Russia and its possibility to lead to financial crisis.

"Will Russia default?" is a rare question with a liquid real money prediction market, called CDS(credit default swap). Bloomberg claims the market implies 80% chance of default. Sadly I don't know how to do that computation myself, so please tell us if you can explain it step by step.

Russia defaulted in 1998. As I understand, it led to collapse of LTCM and almost led to financial crisis, akin to bankruptcy of Lehman Brothers and Global Financial Crisis. In 1998, crisis was averted by bailout organized by Federal Reserve, but we may not be as lucky this time, as we weren't in 2008. I hope financial industry advanced in terms of risk management since 1998 and 2008, but in fact don't know whether it did. If you know, please tell us.
On presenting the case for AI risk
by Aryeh Englander — 2022-03-09T01:41:14.674Z

[Cross-posting from the EA Forum]

Epistemic status: Personal anecdotal evidence, not fully thought through to my own satisfaction. I'm posting this anyway because if I wait until I've thought it through to my satisfaction then I might never post it at all.

People keep telling me how they've had trouble convincing others to care at all about AI risk, or to take the concerns about misaligned AI seriously. This has puzzled me somewhat, because my experience has been significantly different than that.

In my case I mostly talk to engineers who deal with real-world applications of current or near-future AI systems, and when I talk to them about AI safety I actually don't focus on existential risks at all. Instead I just talk about the risks from the systems we have right now, and the challenges we face in making them safe for today's safety- and mission-critical systems. So I'll talk about things like specification gaming, negative side effects, robustness, interpretability, testing and evaluation challenges, security against adversaries, and social coordination failures or races to the bottom. And then at the end I'll throw in something about, oh yeah and obviously if we scale up AI to be even more powerful optimizers than they are now, then clearly these problems can have potentially catastrophic consequences.

And the thing is, I don't recall getting just about any pushback on this - including on the longer term risks part. In fact, I find that people tend to extrapolate to the existential risks on their own. It really is pretty straightforward: There are huge problems with the safety of today's AI systems that are at least partly due to their complexity, the complexity of the environment they're meant to be deployed in, and the fact that powerful optimizers tend to come up with surprising and unforeseen solutions to whatever objectives we give them. So as we build ever more complex, powerful optimizers, and as we attempt to deploy them in ever more complex environments, of course the risks will go way up!

Sometimes I'll start adding something about the longer term existential risks and they'll be like, "whoah, yeah, that's going to be a huge problem!" And sometimes they do the extrapolation themselves. Sometimes I'll actually immediately follow that up with reasons people have given why we shouldn't be worried... and the people I'm talking to will usually shoot thos...
Deriving Our World From Small Datasets
by Capybasilisk — 2022-03-09T00:34:51.720Z

A suggested solution to AGI safety is to ensure intelligent systems only work on abstract mathematical problems that have, as far as we know, little to no connection with the "real world". I'll sidestep some obvious problems (limited usefulness, temptation to formulate real world problems in mathematical terms) to explore some implications of this idea, and then tie it back to modelling physical reality efficiently.

Ideally, these powerful "math oracles" would be handed datasets related to purely theoretical problems, model the data, and send back some form of domain-relevant answer. These AIs would live solely in a universe of esoteric mathematical objects and their relations. They would neither know nor care about humans and all their machinations, the births and deaths of stars, the decay of bromine atoms, the combining of simple cells into wondrously complex structures, or the rise and fall of galactic empires. Indeed, they wouldn't even know or care about their own existence. There's no concept of "math oracle running on a planet named Earth" that can be derived from the data. The whole idea of physical reality gets thrown out here.

Or such is the hope.

Let's imagine one of these superintelligent math oracles that's tasked with modelling a single (carefully selected) integer sequence. An oracle that learns an efficient, predictive model of the primes (specifically, their distribution among the natural numbers) can be said to fully "understand" that sequence. But is that all it understands? Due to the fundamental nature of primes in mathematics, an AI that finds their underlying pattern would arguably "know" number theory (and related mathematical fields) on a far deeper level than any human. A task that seemed constrained in its implications ("Predict the next n primes") with a simple, associated dataset (e.g. the first million primes) gave the AI a far wider insight into hidden mathematical reality than was perhaps imagined at the outset.

Note that this doesn't necessarily have to be viewed negatively.

Added to that, this prime-predicting oracle might know more than the secret workings of numbers. The primes seem to bleed into our physical reality. For instance, patterns in primes are mirrored in certain forms of matter interactions. The hibernation cycle of cicadas follows a prime pattern. Whether these are coincidences or hint at some deeper connect...
ELK prize results
by paulfchristiano — 2022-03-09T00:01:02.085Z

From January - February the Alignment Research Center

offered prizes

for proposed algorithms for

eliciting latent knowledge

. In total we received 197 proposals and are awarding 32 prizes of $5k-20k. We are also giving 24 proposals honorable mentions of $1k, for a total of $274,000.

Several submissions contained perspectives, tricks, or counterexamples that were new to us. We were quite happy to see so many people engaging with ELK, and we were surprised by the number and quality of submissions. That said, at a high level most of the submissions explored approaches that we have also considered; we underestimated how much convergence there would be amongst different proposals.

In the rest of this post we’ll present the main families of proposals, organized by their counterexamples and covering about 90% of the submissions. We won’t post all the submissions but people are encouraged to post their own (whether as a link, comment, or separate post).

Train a reporter that is useful to an auxiliary AI: Andreas Robinson, Carl Shulman, Curtis Huebner, Dmitrii Krasheninnikov, Edmund Mills, Gabor Fuisz, Gary Dunkerley, Hoagy Cunningham, Holden Karnofsky, James Lucassen, James Payor, John Maxwell, Mary Phuong, Simon Skade, Stefan Schouten, Victoria Krakovna & Vikrant Varma & Ramana Kumar
Require the reporter to be continuous: Sam Marks
Penalize depending on too many parts of the predictor: Bryan Chen, Holden Karnofsky, Jacob Hilton, Kevin Wang, Maria Shakhova, Thane Ruthenis
Compress the predictor’s state: Adam Jermyn and Nicholas Schiefer, “P”
Use reporter to define causal interventions: Abram Demski
Train a sequence of reporters: Derek Shiller, Beth Barnes and Nate Thomas, Oam Patel

We awarded prizes to proposals if we thought they solved all of the counterexamples we’ve listed so far. There were many submissions with interesting ideas that didn’t meet this condition, and so “didn’t receive a prize” isn’t a consistent signal about the value of a proposal.

We also had to make many fuzzy judgment calls, had slightly inconsistent standards between the first and second halves of the contest, and no doubt made plenty of mistakes. We’re sorry about mistakes but unfortunately given time constraints we aren’t planning to try to correct them.

Honorable mentions

Strategy: reward reporters that are sensitive to what’s actually happening in the world

Co...
We're already in AI takeoff
by Valentine — 2022-03-08T23:09:06.733Z

Back in 2016, CFAR pivoted to focusing on xrisk. I think the magic phrase at the time was:

"Rationality for its own sake, for the sake of existential risk."

I was against this move. I also had no idea how power works. I don't know how to translate this into LW language, so I'll just use mine: I was secret-to-me vastly more interested in being victimized at people/institutions/the world than I was in doing real things.

But the reason I was against the move is solid. I still believe in it.

I want to spell that part out a bit. Not to gripe about the past. The past makes sense to me. But because the idea still applies.

I think it's a simple idea once it's not cloaked in bullshit. Maybe that's an illusion of transparency. But I'll try to keep this simple-to-me and correct toward more detail when asked and I feel like it, rather than spelling out all the details in a way that turns out to have been unneeded.

Which is to say, this'll be kind of punchy and under-justified.

The short version is this:

We're already in AI takeoff. The "AI" is just running on human minds right now. Sorting out AI alignment in computers is focusing entirely on the endgame. That's not where the causal power is.

Maybe that's enough for you. If so, cool.

I'll say more to gesture at the flesh of this.

What kind of thing is wokism? Or Communism? What kind of thing was Naziism in WWII? Or the flat Earth conspiracy movement? Q Anon?

If you squint a bit, you might see there's a common type here.

In a Facebook post I argued that it's fair to view these things as alive. Well, really, I just described them as living, which kind of is the argument. If your woo allergy keeps you from seeing that… well, good luck to you. But if you're willing to just assume I mean something non-woo, you just might see something real there.

These hyperobject creatures are undergoing massive competitive evolution. Thanks Internet. They're competing for resources. Literal things like land, money, political power… and most importantly, human minds.

I mean something loose here. Y'all are mostly better at details than I am. I'll let you flesh those out rather than pretending I can do it well.

But I'm guessing you know this thing. We saw it in the pandemic, where friendships got torn apart because people got hooked by competing memes. Some "plandemic" conspiracy theorist anti-vax types, some blind belief in provably incoherent authorities, ...
Value extrapolation, concept extrapolation, model splintering
by Stuart_Armstrong — 2022-03-08T22:50:00.476Z

Post written with Rebecca Gorman.

We've written before that model splintering, as we called it then, was a problem with almost all AI safety approaches.

There's a converse to this: solving the problem would help with almost all AI safety approaches. But so far, we've been posting mainly about value extrapolation. In this post, we'll start looking at how other AI safety approaches could be helped.

Definitions

To clarify, let's make four definitions, distinguishing ideas that we'd previously been grouping together:

Model splintering is when the features and concepts that are valid in one world-model, break down when transitioning to another world-model.

Value splintering (or reward splintering) is when the value function (or reward function, or goal, or preference...) becomes invalid due to model splintering.

Concept extrapolation is extrapolating a feature or concept from one world-model to another.

Value extrapolation is concept extrapolation when the particular concept to extrapolate is a value, a preference, a reward function, an agent's goal, or something of that nature.

Thus concept extrapolation is a solution to model splintering, while value extrapolation is a solution to value splintering specifically.

Examples

Consider for example Turner et al's attainable utility. It has a formal definition, but the reason for that definition is that preserving attainable utility is aimed at restricting the "power" of the agent, or at minimising its "side effects".

And it succeeds, in the typical situation. If you measure the attainable utility of an agent, this will give you an idea of its power, and how many side effects it may be causing. However, when we move to general situations, this breaks down: attainable utility preservation no longer restricts power or reduces side effects. So the concepts of power and side effects have splintered when moving from typical situations to general situations. This is the model splintering[1]. If we solve concept extrapolation for this, then we could extend the concepts of power restriction or side effect minimisation, to the general situations. And thus successfully create low impact AIs.

Another example is wireheading. We have a reward signal that corresponds to something we desire in the world; maybe the negative of the CO2 concentration in the atmosphere. This is measured by, say, a series of CO2 detectors spread over the Earth's s...
Prediction Market News/Ticker?
by Raelifin — 2022-03-08T21:44:20.013Z

Lately I’ve been finding myself wanting a way to track how prediction markets have been changing with time. Some markets are pretty stable, while others drift from week to week or even swing violently. These trends feel like relevant news that I want as part of my information diet.

Are there any good aggregators, feeds, tools, etc. that already do this? I read Zvi and ACX, but neither are broad/regular/systematic enough in their market commentary to have me satisfied.