trentmkelly/LessWrong-43k huggingface dataset:
Quantilizer ≡ Optimizer with a Bounded Amount of Output
by itaibn0 — 2021-11-16T01:03:34.849Z

(Crossposted from my blog)

In 2015 MIRI proposed quantilizers as a decision theory criterion that an AI may use that allows it strive towards a goal while not optimizing too hard at the goal, so that if a superintelligent were a quantilizer there's a smaller chance it will do something catastrophic like kill all humans and turn all matter on Earth into paperclips because it's trying really hard to maximize the productivity of a paperclip factory. For a real number 0≤p≤1, a p-quantilizer is an agent which, given a goal and a space of possible strategies, picks randomly out of the top proportion p, i.e., randomly among any strategy that is better than a proportion (1−p) of all strategies. In fact, a quantilizer is for all practical purposes equivalent to a a perfect optimizer (an agent that always picks the best of all its possible strategies) that is further restricted to have its strategy specifies by a string of bits whose length must be less than log2(1/p). Since the people at MIRI have already considered the possibility of controlling an AI by restricting its ability to perform input and output and have generally decided that they are not satisfied with this, I consider the idea of quantilizers to be a failure.

Roughly speaking, my claim is that for a natural number n, up to a constant additive factor, a perfect optimizer that is limited to strategies taking at most n bits performs as well in any goal as a 2−n-quantilizer. More precisely, to explain what I mean by "up to a constant additive factor", there is a reasonably-sized constant c such that an optimizer with n+c bits of outputs performs as well as a 2−n-quantilizer, and a 2−(n+c)-quantilizer performs as well as an optimizer with n bits of output. I also need to assume that the space of strategies is given a rich enough encoding that we can do things like specify a strategy by specifying a computer program whose output is that strategy, or that the environment is rich enough that this can be done implicitly with strategies like "Walk to a computer and type this program, and attach to the computer the accessories necessary to run the strategy specified by the program." To really make the result true up to a constant factor and not a logarithmic factor or anything like that you need to careful about encoding using ...
D&D.Sci Dungeoncrawling: The Crown of Command Evaluation & Ruleset
by aphyer — 2021-11-16T00:29:12.193Z

This is a follow-up to last week's D&D.Sci scenario: if you intend to play that, and haven't done so yet, you should do so now before spoiling yourself.

Full generation & evaluation code is available here if you are interested, or you can read on.

RULESET

THE PARTY

A party has a combined HP total. (This represents health but also spells, healing potions, and any other resources a party has). Each adventurer contributes (1 + level) HP to the total, so e.g. a team of 4 Level 3 characters has 16HP. This HP total drops as the party goes through encounters. If it hits 0, the party withdraws.

Adventurers come in 6 classes, each with a different combat ability:

Fighters have Melee Guard, which helps defend against Melee attacks. If you have no Fighters, enemies with Melee attacks will charge into melee with your squishy ranged characters and deal double damage.
Rangers have Range Guard, which helps defend against Range attacks similarly.
Mages have Magic Guard, which helps defend against Magic attacks similarly.
Clerics have Healing. This restores 0.5HP per cleric to the party after each encounter if the party hasn't already retreated, plus provides defense against Undead (see Enemies below).
Druids have Wild Empathy. This helps defend against Beasts (see Enemies below).
Rogues have Cowardice. This is useless to the party, but makes them tend to show up to lower-level dungeons and not to higher-level ones, so a rogue will usually be higher-level than the dungeon they are encountering.

Encounters have two types: Traps and Enemies.

TRAPS

All traps work the same way: they deal 1d6 damage, reduced by the level of the highest-level appropriate adventurer in your party (to a minimum of 0):

Fighters counter Boulder Traps, smashing boulders with their great strength.
Rangers counter Lever Puzzle Rooms, using their bows to shoot levers from afar and to fire grappling hooks across the room.
Mages counter Riddle Doors, solving the riddles with their keen intellect.
Clerics counter Cursed Altars, dispelling the evil magicks upon them.
Druids counter Snake Pits, making friends with the snakes.
Rogues counter Poison Dart Traps, disarming them in order to steal the poison darts.

So if your party has a Level 4 Fighter, a Level 3 Fighter, a Level 3 Cleric, and a Level 1 Mage:

A Boulder Trap will do 1d6-4 damage (the higher-level fighter is counted, the l

...
Streaming Science on Twitch
by A Ray — 2021-11-15T22:24:05.388Z

Recently I was watching a livestream of a poker professional. I was surprised and interested in how it wasn’t purely-gut calls, and also wasn’t purely technical decisions, but a blend of both (with some other stochasticity thrown in).

I’ve been thinking about how to get more good scientists, especially scientists earlier in their career.

I think it should be possible to stream science — the actual practice of it.

I think this would be disproportionally useful for younger/earlier-career people, who I expect would find twitch streams more appealing, and lack the access to advisement/mentorship that often comes later.

This probably only works first with sciences (like mine — I’m biased here) that can happen mostly/exclusively on computers. Software sciences, data sciences, and machine learning sciences seem like good candidates for this.

A bunch of it would probably be boring stuff. Debugging experiments, sorting/processing data, formatting and reformatting plots.

Also, a bunch of the good stuff probably happens internally in ways that wouldn’t be stream-able. There’s a lot of processing in my head that I don’t have access to, and I assume this is true for a lot of people who work on science.

There’s also probably tasks or modes of work that recording/streaming/broadcasting would be distracting. I assume this would also be true of video games, so maybe this effect is not that bad.

The stream could also make science better! It’s possible that (if people watch it and interact live) someone in the audience spots a bug or issue that would have been missed, or proposes a better method of doing something.

A big limitation here is that probably a lot of great science is not share-able (due to company secrets or academic results that cannot be shared before publication to avoid scoops, etc). This would necessitate working on projects of secondary quality/importance, and using a totally different code base and set of tools.

As I’m writing this, I notice that “science” feels a bit too small of a category. I would expect policy research, governance and other sorts of topics would also fit in this category. I expect these (and many others) are also fields where young/early career people have hard times finding advisement/mentorship.

What could be done in this direction soon:

I expect it would be worthwhile to go over and see ...
Ngo and Yudkowsky on alignment difficulty
by Eliezer Yudkowsky — 2021-11-15T20:31:34.135Z

This post is the first in a series of transcribed Discord conversations between Richard Ngo and Eliezer Yudkowsky, moderated by Nate Soares. We've also added Richard and Nate's running summaries of the conversation (and others' replies) from Google Docs.

Later conversation participants include Ajeya Cotra, Beth Barnes, Carl Shulman, Holden Karnofsky, Jaan Tallinn, Paul Christiano, Rob Bensinger, and Rohin Shah.

The transcripts are a complete record of several Discord channels MIRI made for discussion. We tried to edit the transcripts as little as possible, other than to fix typos and a handful of confusingly-worded sentences, to add some paragraph breaks, and to add referenced figures and links. We didn't end up redacting any substantive content, other than the names of people who would prefer not to be cited. We swapped the order of some chat messages for clarity and conversational flow (indicated with extra timestamps), and in some cases combined logs where the conversation switched channels.

Color key:

Chat by Richard and Eliezer	Other chat	Google Doc content	Inline comments

0. Prefatory comments

[Yudkowsky][8:32] (Nov. 6 follow-up comment)

(At Rob's request I'll try to keep this brief, but this was an experimental format and some issues cropped up that seem large enough to deserve notes.)

Especially when coming in to the early parts of this dialogue, I had some backed-up hypotheses about "What might be the main sticking point? and how can I address that?" which from the standpoint of a pure dialogue might seem to be causing me to go on digressions, relative to if I was just trying to answer Richard's own questions. On reading the dialogue, I notice that this looks evasive or like point-missing, like I'm weirdly not just directly answering Richard's questions.

Often the questions are answered later, or at least I think they are, though it may not be in the first segment of the dialogue. But the larger phenomenon is that I came in with some things I wanted to say, and Richard came in asking questions, and there was a minor accidental mismatch there. It would have looked better if we'd both stated positions first without question marks, say, or if I'd just confined myself to answering questions from Richard. (This is not a huge catastrophe, but it's something for the r

...
Dan Luu on Persistent Bad Decision Making (but maybe it's noble?)
by Elizabeth — 2021-11-15T20:05:21.442Z
Link: https://danluu.com/bad-decisions/

Dan walks through some extremely clear-cut sports examples where people continued making decisions that were wrong costing them wins in ways that were easily calculable. They did eventually change, but even that took years, sometimes decades. Given that this was happening in something as easy to measure as sports, he suggests we should expect bad decision making in corporations or governments to take even longer to correct.

My major concern with this post is that the goal of pro sports is a mix of winning and making money via attention from fans. This first means that the feedback loop isn't as tight as it would be otherwise. Additionally, many of the optimizations that lead to more wins make games more boring, which ultimately costs the entire league money. Using these optimizations is in some ways defecting in a prisoner's dilemma: you win more, which your fans like and other fans don't, but if everyone does it the game becomes more boring, which drives away everyone's fans. Under this light, base stealing attempts that cost you counterfactual wins could be viewed as a prosocial sacrifice to the entire league's benefit.
The poetry of progress
by jasoncrawford — 2021-11-15T19:24:42.030Z
Link: https://rootsofprogress.org/poetry-of-progress

I’ve said that Western culture, before the World Wars, had a positive philosophy of progress. How do we know that people in that era saw glory, romance, even poetry in progress?

One way we know is that they literally wrote poetry about it. Take, for example, Erasmus Darwin (grandfather of Charles). By vocation a doctor, he also wrote poems. In 1791, he published The Botanic Garden, of which Part 1, titled “The Economy of Vegetation,” “celebrates scientific progress and technological innovation, such as the forging of steel, the invention of the steam engine and the improvements to gunpowder. It depicts scientists and inventors, such as Benjamin Franklin, responsible for this progress as the heroes of a new age.” (Wikipedia) In one remarkable passage, he praises the power of steam and predicts the steamboat, the locomotive, and the airship:

Soon shall thy arm, Unconquer’d Steam! afar
Drag the slow barge, or drive the rapid car;
Or on wide-waving wings expanded bear
The flying-chariot through the fields of air.
—Fair crews triumphant, leaning from above,
Shall wave their fluttering kerchiefs as they move;
Or warrior-bands alarm the gaping croud,
And armies shrink beneath the shadowy cloud.

Or take this stanza from “The Engine-Driver to His Engine”, by engineer William J. M. Rankine, which reads less like a classical epic and more like a popular ballad:

My blessing on old George Stephenson! Let his fame for ever last,
For he was the man that found the plan to make you run so fast.
His arm was strong, his head was long, he knew not guile nor fear;
When I think of him it makes me proud that I am an engineer!

Samuel Morse, telegraph inventor, was also praised in poetry. A poem read at his memorial service in 1872 begins by describing mankind as having conquered all of nature except for “the Ocean, the Mountain, and Time and Space”—until Morse came along:

But one morning he made him a slender wire,
As an artist’s vision took life and form,
While he drew from heaven the strange, fierce fire,
That reddens the edge of the midnight storm;
And he carried it over the Mountain’s crest,
And dropped it into the Ocean’s breast;
And Science proclaimed, from shore to shore,
That Time and Space ruled man no more.

It wasn’t only amateurs and enthusiasts who wrote poetry praising machines and inventions. No less than Rudyard Kipling wrote of the beauty and...
Worst Commonsense Concepts?
by abramdemski — 2021-11-15T18:22:31.465Z

Perhaps the main tool of rationality is simply to use explicity reasoning where others don't, as Jacob Falcovich suggests:

New York Times reporter Cade Metz interviewed me and other Rationalists mostly about how we were ahead of the curve on COVID and what others can learn from us. I told him that Rationality has a simple message: “people can use explicit reason to figure things out, but they rarely do”

However, I also think a big chunk of the value of rationality-as-it-exists-today is in its corrections to common mistakes of explicit reasoning. (To be clear, I'm not accusing Jacob of ignoring that.) For example, bayesian probability theory is one explicit theory which helps push a lot of bad explicit reasoning to the side.

The point of this question, however, is not to point to the good ways of reasoning. The point here is, rather, to point at bad concepts which are in widespread use.

For example:

Fact vs opinion. There are several reasons why this is an awful concept.

The common usage suggests that there are "matters of fact" vs "matters of opinion"; eg, I like hummus (opinion) but 1+1=2 (matter of fact). But common usage also suggests that probabilistic reasoning gives mere opinions, while other modes of reasoning (such as direct observation, and logical reasoning) yield facts. This is inconsistent; it suggests that we can tell whether a belief is an opinion or a fact by examining what it is about (beliefs about subjective things = opinions; beliefs about objective things = facts), while also seeming to need the mode of reasoning by which we arrived at the belief (eg, if I saw a black hole myself, it would be a fact, but if I derived one's existence from unproven physics, it would be opinion).
Calling something a fact generally indicates that others are epistemically obligated to believe it. But if it is contentious, then this is precisely what's at issue. So calling something a fact like this is generally useless.
We could take "fact" to mean something like "true opinion". But from the inside, this is no different from a strong belief. So again, to call something a fact rather than a strong opinion seems to add no information (whereas, I take it, it's supposed to according to common usage).

"Purpose" as an inherent property. In common usage, it makes sense to ask "the purpose of life" because a purpose is a property which lots of objects have. In real

...
My understanding of the alignment problem
by danieldewey — 2021-11-15T18:13:19.913Z

I've been clarifying my own understanding of the alignment problem over the past few months, and wanted to share my first writeups with folks here in case they're useful:

https://www.danieldewey.net/risk/

The site currently has 3 pages:

The case for risk: how deep learning could become very influential, training problems that could lead models to behave in systematically harmful ways, and what I think we should do about it. Inspired mainly by What failure looks like.
Fermi estimate of future training runs: a short AI timelines estimate inspired by Forecasting transformative AI.
Applications of high-capability models: some notes on how high-capability models could actually be trained, and how their behavior could become highly influential.

None of the ideas on the site are particularly new, and as I note, they're not consensus views, but the version of the basic case I lay out on the site is very short, doesn't have a lot of outside dependencies, and is put together out of nuts-and-bolts arguments that I think will be useful as a starting point for alignment work. I'm particularly hoping to avoid semantic arguments about "what counts as" inner vs outer alignment, optimization, agency, etc., in favor of more mechanical statements of how models could behave in different situations.

I think some readers on this forum will already have been thinking about alignment this way, and won't get a lot new out of the site; some (like me) will find it to be a helpful distillation of some of the major arguments that have come out over the past ~5 years; and some will have disagreements (which I'm curious to hear about).

I thought about posting all of this directly on the Alignment Forum / LessWrong, but ultimately decided I wanted a dedicated home for these ideas.

Out of everything on the site, the part I'm most hoping will be helpful to you is my (re)statement of two main problems in AI alignment. These map roughly onto outer and inner alignment, though different people use those terms differently, so not everyone will agree:

As models become more capable, it looks like currently known training methods will run into fundamental safety problems, and become increasingly likely to produce models that behave in systematically harmful ways:

1. Evaluation breakdown: As a model’s behavior becomes more sophisticated, it will reach a point where an automated reward function or human evaluato

...
How Humanity Lost Control and Humans Lost Liberty: From Our Brave New World to Analogia (Sequence Introduction)
by Justin Bullock — 2021-11-15T14:22:53.130Z

In this sequence of posts, we will examine how our global civilization has found itself wandering, staggering, and rocketing through, and now beyond Our Brave New World.

Superintelligences exist and they are misaligned with the interests of both humanity and individual humans. These superintelligences have their own emergent goals and patterns of behavior. In fact, humans have already lost much meaningful control over these superintelligences and their behavior. This is the birth of Analogia.

Across ~25 posts gathered across ~6 major parts, this sequence will explore how humans, machines, and civilization have coevolved to present a world of increasing complexity, one dominated by hybrid superintelligences, and one in the process of developing machine superintelligences.

In Part I: The Pathway to Analogia, we will map how over-digitizalization, when coexistent with over-population and over-organization, has brought us to the fulfillment of Aldous Huxley’s Brave New World of mass manipulation of the general population by the more narrow interests of both individual groups of humans and of machines. It is from Our Brave New World, that Analogia has arisen. Analogia is a world where much of humanity has embraced over-digitalization and its continual encroachment into our analog, natural world, the world that, throughout all of human history has always been considered the “real world”, the one our biological minds and biological bodies inhabit. It is a world where the digital world has begun to assert control over the analog world. As a brief detour in Part I, we visit an abridged aversion of Olaf Stapledon’s “The Other Earth.” An early cautionary tale. (Chapters 1-7, planned titles with planned posting dates)

Ch 1: Machine Agents, Hybrid Superintelligences, and The Loss of Human Control

Ch 2: The Other Earth

Ch 3: Revisiting A Brave New World Revisited

Ch 4: The Age of Over-population

Ch 5: Over-organization

Ch 6: Over-digitalization

Ch 7: Welome to Analogia!

In Part II: The Machines, we will step away from the diagnosis of Our Brave New World and the troubling rise of Analogia. Here, we will turn to speculative fiction for some inspiration on the range of outcomes, challenges and opportunities, for living in Analogia. First, I turn to Samuel Butler's description of The Book of The Machines for a speculation on how a society settl...
Re: Attempted Gears Analysis of AGI Intervention Discussion With Eliezer
by lsusr — 2021-11-15T10:02:54.153Z

TL;DR

I think AGI architectures will be quirky and diverse. Consequently, there are many possible futures. My thoughts on the AI alignment problem are far more optimistic than (my best measure of) mainstream opinions among AI safety researchers (insofar as "AI safety" can be considered mainstream).

Disclaimer

In Attempted Gears Analysis of AGI Intervention Discussion With Eliezer, Zvi attempts to explain Elizer's perspective on AGI from a conversation here as explained by Rob Besinger. My post here is the 5th node in a game of telephone. The opinions I quote in this post should not be considered Eliezer Yudkowsky's or Rob Besinger's. I'm just using Zvi's post as a reference point from which I can explain how my own views diverge.

Individual Contentions

Nate rather than Eliezer, but offered as a preface with p~0.85: AGI is probably coming within 50 years. Rob notes that Eliezer may or may not agree with this timeline, and that it shortens if you condition on ‘unaligned’ and lengthens conditional on ‘aligned.’

I don't know whether AGI is coming within 50 years. Zvi says he would "at least buy 30% and sell 80%, or something like that." I would at least buy 2% and sell 90% except, as Zvi noted, "betting money really, really doesn’t work here, at all". (I'm ignoring the fact that "copying Zvi" is a good strategy for winning bets.)

My exact probabilities depend on how we define AGI. However, as we'll get to in the subsequent points, it doesn't really matter what the exact probability is. Even a small chance of AGI is of tremendous importance. My uncertainty about AGI is high enough that there is at least a small chance of AGI being built in the next 40 years.

By default this AGI will come from something similar to some of today’s ML paradigms. Think enormous inscrutable floating-point vectors.

I disagree for reasons, but it depends on what we mean by "today's ML paradigms".

AGI that isn’t aligned ends the world.

If we define AGI as "world optimizer" then yes, definitely. But I can imagine a couple different kinds of superintelligences that aren't world optimizers (along with a few that naturally trend toward world optimizing). If you built a superintelligent machine that isn't a world optimizer then it need not necessarily end the world.

For example, MuZero separates value from policy from reward. If you built just the value network and cranked it up to ...