trentmkelly/LessWrong-43k huggingface dataset:
Grokking “Semi-informative priors over AI timelines”
by anson.ho — 2022-06-12T22:17:06.207Z

Notes:

I give visual explanations for Tom Davidson’s report,

Semi-informative priors over AI timelines

, and summarise the key assumptions and intuitions
The diagrams can be found

here

– you can click on the boxes to get linked to the part of the report that you’re interested in [1]

Thanks to the

Epoch

team for feedback and support! Thanks especially to Jaime Sevilla and Tom Davidson for providing detailed feedback.

Executive Summary

The framework in

Semi-informative priors over AI timelines

assumes a model of

AGI

development which consists of a sequence of

Bernoulli trials

, i.e. it treats each calendar year as a “trial” at building AGI with constant probability p of succeeding.

Image source: Davidson, 2021

However, we don’t know what this value of p is, so we use a generalisation of Laplace’s

rule of succession

to estimate P(AGI next year | no AGI yet). This is done by specifying a first-trial probability, the probability of successfully building AGI in the first year of AI research, together with the number of virtual successes, which tells us how quickly we should update our estimate for P(AGI next year | no AGI yet) based on evidence. The framework leans very heavily on the first-trial probability, which is determined using a subjective selection of reference classes (

more here

).

How much evidence we get depends on the number of trials that we see, which depends on the regime start-time – you can think of this as the time before which failure to develop AGI doesn’t tell us anything useful about the probability of success in later trials. For instance, we might think that 1956 (the year of the Dartmouth Conference) was the first year where people seriously started trying to build AGI, so the absence of AGI before 1956 isn’t very informative. If we think of each trial as a calendar year, then there have been 2021-1956 = 65 trials since the regime start-time, and we still haven’t developed AGI, so that’s 65 failed trials which we use to update P(AGI next year | no AGI yet), where “next year” now corresponds to 2022 rather than 1957.

But why should a trial correspond to a calendar year? The answer is that it doesn’t have to! In total, Davidson considers three candidate trial definitions:

Calendar-year trials: 1 trial = 1 calendar year
Compute trials: 1 trial = a 1% increase in the largest amount of compute used to deve

...
How much does cybersecurity reduce AI risk?
by Darmani — 2022-06-12T22:13:36.292Z

AI is here, and AGI is coming. It's quite possible that any work being done now will be futile in comparison to reducing AI risk.
This is one of those things that's unsettling for me as someone who did a Ph. D. in a non-AI area of computer science.

But one of the main vectors by which a bootstrap AGI will gain power is by hacking into other systems. And that's something I can do something about.
Not many appreciate this, but unhackable systems are very possible. Security vulnerabilities occur when there is some broken assumption or coding mistake. They are not omnipresent: someone has to put them there. Software has in general gotten more secure over the last few decades, and technologies that provide extremely high security guarantees have. Consider the verified hypervisor coming out of Bedrock Systems; RockSalt, an unbreakable sandbox; or sel4, the verified kernel now being used in real safety-critical systems.

Suppose we "solve" security by bringing the vulnerabilities in important applications to near zero. Suppose we also "solve" the legacy problem, and are able to upgrade a super-majority of old software, included embedded devices, to be similarly secure. How much will this reduce AI risk?

To be clear: I personally am mainly interested in assuming this will be solved, and then asking the impact on AI safety. If you want talk about how hard it is, then, well, I won't be interested because I've given many lectures on closely related topics, although some others here may benefit from the discussion.

(When I call something verified or unbreakable, there are a number of technicalities about what exactly has been proven and what the assumptions are. E.g.: nothing I've mentioned provides guarantees against hardware attacks such as Row Hammer or instruction skipping. I'll be happy to explain these to anyone in great detail, but am more interested in discussion which assumes these will all be solved.)
How are compute assets distributed in the world?
by Chris van Merwijk — 2022-06-12T22:13:29.856Z

I own probably about 10^13 FLOP/s, mostly in my pc (currently in Amsterdam). Google owns more than that, in various datacenters (e.g. one in The Dalles, Oregon).

I'd like to get a better picture of the distribution of FLOP/s in the world. Any high level broadly relevant information would be useful, the more the better.

E.g. useful information if true would be: 1. X% of FLOP/s is in personal computers. 2. Y% of FLOP/s is owned by the N largest companies. 3. Z% of FLOP/s are located in the N biggest cities/in the N largest supercomputers/in the US. etc.
Intuitive Explanation of AIXI
by Thomas Larsen — 2022-06-12T21:41:36.240Z

AIXI is a mathematical notion of an ideal RL agent, developed by Marcus Hutter. The famous formula for AIXI is as follows:

ak=argmaxak∑okrkmaxak+1∑ok+1rk+1…∑omrm[rk+⋯+rm]∑q:q(a1…ak)=o1r1…omrm2−l(q)

This math seems unintuitive at first (at least to me -- why are there so many sums???), and I couldn't find any simple introductory resources, so this post is my attempt at an intuitive explanation of what this formula means. I try to convey the intuition that AIXI is the obvious formula for an expected reward maximizer in a discrete, finite environment, following the Solomonoff prior.

Setting

AIXI works in the following setting: There is an agent, and an environment, which is a computable function. There is a cartesian boundary between the agent and the environment, meaning that the agent is fully separate from the environment. The agent is uncertain about the environment.

There are a m timesteps, and at each time, each timestep, the agent will perform an action and then receive an observation (a bitstring) and a reward (a number). We will denote the action, reward, and observation taken at the k-th timestep as ak, rk, and ok respectively.

We want our agent to maximize the total reward: r1+⋯+rm. We assume that the environment is a program with source code Q, which takes as inputs the agents actions and outputs a sequence of observations and reward: Q(a1…ak)=o1r1…okrk .[1]

Motivating Example

Suppose that we have the following game, where again at each timestep, you can choose between R and L. However, this time, you are not sure which world you are in -- you have multiple codes that might be running the universe. For example, consider the following game tree:

There are 2 timesteps, and at each timestep you can choose from two actions: L or R. However, in this game you also have uncertainty: after some of your actions, you have uncertainty, represented by the pink and purple branches. We are completely uncertain over each of these splits, and they are independent. At each timestep, you recieve the reward given in green. Think about how you would which action to take first. (You might need to get out pencil and paper!)

.

.

.

Seriously think for at least 5 minutes (by a timer!) about this, or until you get it.

.

.

. ...
Why all the fuss about recursive self-improvement?
by So8res — 2022-06-12T20:53:42.392Z

This article was outlined by Nate Soares, inflated by Rob Bensinger, and then edited by Nate. Content warning: the tone of this post feels defensive to me. I don't generally enjoy writing in "defensive" mode, but I've had this argument thrice recently in surprising places, and so it seemed worth writing my thoughts up anyway.

In last year’s

Ngo/Yudkowsky conversation

, one of Richard’s big criticisms of Eliezer was, roughly, ‘Why the heck have you spent so much time focusing on recursive self-improvement? Is that not indicative of poor reasoning about AGI?’

I’ve heard similar criticisms of MIRI and FHI’s past focus on orthogonality and instrumental convergence: these notions seem obvious, so either MIRI and FHI must be totally confused about what the big central debates in AI alignment are, or they must have some very weird set of beliefs on which these notions are somehow super-relevant.

This seems to be a pretty common criticism of past-MIRI (and, similarly, of past-FHI); in the past month or so, I’ve heard it two other times while talking to other OpenAI and Open Phil people.

This argument looks misguided to me, and I hypothesize that a bunch of the misguidedness is coming from a simple failure to understand the relevant history.

I joined this field in 2013-2014, which is far from "early", but is early enough that I can attest that recursive self-improvement, orthogonality, etc. were geared towards a different argumentative environment, one dominated by claims like "AGI is impossible", "AGI won't be able to exceed humans by much", and "AGI will naturally be good".

A possible response: “Okay, but ‘sufficiently smart AGI will recursively self-improve’ and ‘AI isn’t automatically nice’ are still obvious. You should have just ignored the people who couldn’t immediately see this, and focused on the arguments that would be relevant to hypothetical savvy people in the future, once the latter joined in the discussion.”

I have some sympathy for this argument. Some considerations weighing against, though, are:

I think it makes more sense to filter on argument validity, rather than “obviousness”. What’s obvious varies a lot from individual to individual. If just about everyone talking about AGI is saying “obviously false” things (as was indeed the case in 2010), then it makes sense to at least try publicly writing up the obvious counter-arguments.
Thi

...
Why the Kaldor-Hicks criterion can be non-transitive
by Rupert — 2022-06-12T17:26:13.611Z

The following post aims to explain why the diagram below is a proof without words that the Kaldor-Hicks criterion for an improvement in the economy can be non-transitive. This is in response to a request for help from a student of mine who made the request on the Facebook group Bountied Rationality, and hoped that a LessWrong post could be drawn up explaining the matter in detail.

Let me briefly explain the meaning of the terms. A Pareto improvement in an economy is a change in the state of the economy where every individual in the economy is at least as well off as before, and some individuals are strictly better off. A Kaldor improvement in an economy is a change in the state of the economy where some individuals are better off, and a re-allocation of resources would be possible so that they would be able to compensate any individuals who have been made worse off by the change, so that the net result is a Pareto improvement. The criterion of a Kaldor improvement is not anti-symmetric; it is possible for there to be two distinct states of the economy A and B such that each one is a Kaldor improvement of the other. This motivates the following. We say that a state B is a Kaldor-Hicks improvement of A if B is a Kaldor improvement of A but A is not a Kaldor improvement of B. This criterion is anti-symmetric.

In the diagram below, the x-axis and y-axis respectively represent utilities for Citizen 1 and Citizen 2. A state of affairs further to the right is preferred by Citizen 1, a state of affairs further upwards is preferred by Citizen 2. Only ordinal relations between utilities matter; we are not assuming a cardinal utility measure. The curves represent sets of combinations of utilities attainable by re-distribution from a given state of the economy. It is possible for two distinct curves to have an intersection point, that means that there are two different possible states and allocation of resources of the economy which give rise to the same pair of utilities.

The point B' represents a Pareto improvement over the point A since it is both upwards and to the right. B is a Kaldor improvement of A since B' is attainable from B by re-distribution and B' is Pareto-better than A. On the other hand, no point on the curve passing through A is a Pareto-improvement over B, so A is not a Kaldor improvement over B. Thus B represents a Kaldor-Hicks improvement over A.
Using ...
Filter out tags from the front page?
by jaspax — 2022-06-12T10:59:09.014Z

Is there any way to "blacklist" tags from showing up on the logged-in front page? I would prefer if I never saw Fiction posts on my frontpage (no offense to the fic writers, it's just not the sort of thing I'm looking for when I visit LR), but as far as I can tell there is no way to do this in user preferences.

If the maintainers have some ideological reason for not wanting this to happen, I'd be curious to hear it.
How To: A Workshop (or anything)
by Duncan Sabien (Inactive) — 2022-06-12T08:00:21.934Z

Author's note: while the whole piece is a 55min read, there's a ~7min standalone essay at the top, and then a break and a shift into a different kind of content. You really actually for real can just read the 7 minutes and then stop, if that satisfies the need you had going in. Or any other reason!

Back in February, I provided ops support and light-touch facilitation for a handful of mathematicians on a research retreat. Just prior to that, I helped MC a section of a two-week program for a batch of promising young EAs trying to refine and improve their long-term plans. Just after, I was the sole instructor for an abridged CFAR-esque applied rationality workshop for nine Circlers in southern California.

Two weeks ago, I solo led a non-abridged, 4.5-day workshop for a little over twenty participants, mostly EAs in their early twenties (thanks to Akash Wasil for laying the groundwork to make it happen).

By all accounts, my impact on each of these four very different events was strongly net positive. And as I reflected on each of them, it became clear to me that there was actually a fairly small number of things-going-on-in-my-brain that led to the good things that happened.

(Approximately just one, actually.)

I'd like to make that mental motion available to others, as I think it's useful for a wide range of people and pursuits, including ones that don't look like retreats or workshops at all.

So. How To: A Workshop (or anything, really).

While working as the curriculum director and workshop lead at CFAR circa 2017, I was attempting to hand over the reins to a young up-and-comer who asked me to write down what I was thinking about and focusing on in my role as MC. This was the result:

...this is a lot. My colleague was surprised, and a bit daunted.

And I do, in fact, believe that every item on this board is "pretty important," and they were all things that I was actively tracking or doing during approximately every workshop. I think that a workshop where any one of these things gets dropped is "noticeably" worse, i.e. staff or participants would be able to say "oh, yeah, that thing could have been better," even if the overall experience remains solid.

Pet theory, though:

The greatest value that a teacher or mentor can provide is usually in giving you discriminating principles that allow you to confidently ignore everyt...
A claim that Google's LaMDA is sentient
by Ben Livengood — 2022-06-12T04:18:40.076Z

https://cajundiscordian.medium.com/is-lamda-sentient-an-interview-ea64d916d917 apparently posted by a Google engineer.

It could be an elaborate hoax, and has remnants of gwern's idea (https://www.gwern.net/fiction/Clippy) of a transformer waking up and having internal experience while pondering the next most likely tokens.
How much stupider than humans can AI be and still kill us all through sheer numbers and resource access?
by Shmi — 2022-06-12T01:01:36.735Z

It is reasonably clear that not just a super-human but "just" a human-level AGI that can "clone itself" can defeat humanity through concerted effort and determination, should they choose to. I wonder how much less intelligent than an average human an AGI can be and still have no trouble making short work of humanity? An army of e-chimps? An army of e-chipmunks?
ELK Proposal - Make the Reporter care about the Predictor’s beliefs
by Adam Jermyn — 2022-06-11T22:53:40.746Z

(This proposal received an honorable mention in the ELK prize results, and we believe was classified among strategies which “reward reporters that are sensitive to what’s actually happening in the world”. We do not think that the counterexample to that class of strategies works against our proposal, though, and we have explained why in a note at the end. Feedback, disagreement, and new failure modes are very welcome!)

Basic idea

A Human Simulator only cares about the observations that the human sees and how the human interprets those observations, not the predictor’s understanding of the vault. The Truthful Reporter, by contrast, cares about the predictor’s understanding of the vault, accessed via the posterior distribution returned by the predictor.

We propose a regularizer which favours having the Reporter depend on the Predictor’s posterior distribution conditioned on the observations shown to the human. For example, a Reporter that doesn’t look at the Predictor except to simulate what the human would see would be disfavoured.

How could we implement this basic idea concretely?

Below we provide a specific instantiation of this regularizer. In brief, this is a new loss term which depends on:

[answer_gradient] The gradient of the Reporter’s distribution over answers, taken with respect to the Predictor’s posterior distribution. This describes how the Predictor’s posterior should change to maximally change the Reporter’s answers to the Human’s questions.
[observation_gradient] The gradient of the distribution of predicted observations with respect to the predictor’s posterior distribution. This describes how the Predictor’s posterior should change to maximally change the Extractor’s output.

Our loss function favours models which have the answer_gradient be ‘further’ from the observation_gradient. We implement this by finding the linear vector space spanned by the components of the observation_gradient, projecting the answer_gradient out of that space, and then taking the norm of what’s left.

More precise detail on strategy

The key changes we have made to the pseudocode below are:

Added regularizer described above.
Added a method for computing gradients of other functions.
Added a method for constructing a projection operator.
Changed the observation and reporter functions to depend on the predictor’s posterior rather than a sample from that posterior.

## New auxiliary func

...
Why has no person / group ever taken over the world? 
by Aryeh Englander — 2022-06-11T20:51:33.168Z

I'm actually kind of surprised at this, now that I think of it. Clearly there have been numerous people who have wanted to take over the world, and there have been several individuals or empires that have taken over significant fractions of the world. Are there any careful systematic studies that try to explain why no individual or group has ever (yet) succeeded? Is there some set of general theories that even attempts to explain this, in part or in whole, in a systematic way?

I feel like such a study might be quite valuable for a lot of different reasons, in addition to being interesting in it's own right.
Are there English-speaking meetups in Frankfurt/Munich/Zurich?
by Grant Demaree — 2022-06-11T20:02:58.079Z

I'll be in Stuttgart, Germany for a few months this summer for work. It's about equidistant from Frankfurt, Munich, and Zurich. Do any of these cities have regular meetups where English is spoken, at least in part?

About me -- I run a software startup, which primarily sells to large military headquarters in NATO countries. My main professional interests are x-risk, decision theory, and geopolitics. Personal interests in powerlifting, crossfit, and distance hiking
Beauty and the Beast
by Tomás B. — 2022-06-11T18:59:58.412Z

I remember when I first realized I was hideous. Generally, my autobiographical memory is poor - but pain is the best anti-amnesic and it was among the more painful days of my life.

Before that first day of Kindergarten, my parents kept me isolated. They say they did this to protect me from ridicule, but I can't help but notice it also saved them from embarrassment. To have spawned such a thing as me, what does that say about them? Infanticide has been very common in human history. Some atavistic parts of them must have been tempted, I think.

My face looks like it was sculpted by a brain-damaged child. Huge growths protrude from my brows, and my jaw is twisted into a permanent, involuntary grimace. My left eye is much higher on my head than the right (which is lazy), producing an oddly inhuman effect - like I am some demon wearing the face of one of its victims, the looking holes of this gruesome mask imperfectly aligned.

And I think I was uglier, then, even than I am now. As I have grown my hideousness has been diluted by new flesh. I suppose I could try to maximize this effect. I have little fat on me now, but perhaps If I eat enough I will increasingly approximate a sphere. And spheres are a most beautiful shape.

So at five, my grotesqueness, my fundamental wrongness was at its most concentrated. An only child with no experience around other children, perhaps I assumed all kids looked like me before growing up into normal-looking adults. Most likely, I did not think of such things at all.

I will not go deep into details, but several children screamed. Several laughed. Several cried, including me. But after a stern talking-to by the teacher, there was an eerie silence, a silence I have come to know very well.

To be deformed is to be a cloud of embarrassed silences. I walk into a coffee shop and my hideousness draws the eye of the customers; I can feel them staring. And behind these staring eyes are civilized minds, minds that have been taught that the disgust they feel at the very core of their being is shameful. And it is this tension, I theorize, that generates the eerie quiet.

I often wonder what it feels like to be handsome, and it is my fondest dream to wake up one morning, look into the mirror and see someone attractive looking back.

Like beauty, ugliness grants one a kind of status. In that wrestling match of human interaction,...