trentmkelly/LessWrong-43k huggingface dataset:
Does the Structure  of an algorithm matter for AI Risk and/or consciousness?
by Logan Zoellner — 2021-12-03T18:31:40.185Z

My impression is that most people around here aren't especially worried about GPT-n being either: capable of recursive self-improvement leading to foom, or obtaining morally significant levels of consciousness.

Reasons given include:

GPT has a large number of parameters with a shallow layer-depth, meaning it is incapable of "deep" reasoning
GPT's training function "predict the next character" makes it unlikely to make a "treacherous turn"
GPT is not "agenty" in the sense of having a model of the world and viewing itself as existing within that model.

On the other hand, I believe is widely agreed that if you take a reinforcement learner (say Google's Dreamer) and give it virtually any objective function (the classic example being "make paperclips") and enough compute, it will destroy the world. The general reason being given is Goodhart's Law.

My question is, does this apparent difference in perceived safely arise purely from our expectations of the two architecture's capabilities. Or is there actually some consensus that different architectures carry inherently different levels of risk?

Question

To make this more concrete, suppose you were presented with two "human level" AGI's, one built using GPT-n (say using this method) and one built using a Reinforcement Learner with a world-model and some seeming innocuous objective function (say "predict the most human like response to your input text").

Pretend you have both AGIs in separate boxes front of you, and complete diagram of their software and hardware and you communicate with them solely using a keyboard and text terminal attached to the box. Both of the AGIs are capable of carrying on a conversation at a level equal to a college-educated human.

If using all the testing methods at your disposal, you perceived these two AGIs to be equally "intelligent", would you consider one more dangerous than the other?

Would you consider one of them to be more likely to be conscious than the other?

What significant moral or safety quest...
A Meditative Experience
by Yonatan Cale — 2021-12-03T17:58:39.462Z

I have no idea where to write this, doesn't seem exactly lesswrongy, but this is the only place where I expect to maybe get replies that might resonate with how I think. So,

Yesterday I had a stressful negative experience [that I won't go into], I had pressure in my chest, breathing heavily, I turned to my friend and said "wait, I'm not ok" and sat down on the steps, noticing that this is an unusually negative reaction, and decided to focus on noticing my sensory inputs, and a few moments later, to notice my thoughts too, as I sometimes [once a week?] do since reading

Multiagent Models of Mind

.

Sitting there, I notice that I see only a small "area" while the things around it are foggy (to the point that somewhat made me worry that I don't see well). I want to experience touch, so I rub my fingers against the palm of my hand and feel them. I pick up a small stone and notice it in my fingers. I notice my feet on the ground, in my shoes, somehow "far" from me, but looking at them, I see them somehow surprisingly close, as if a small optical illusion, which makes me think I'm not ok.

I notice my thoughts. I notice thoughts come up, and I am passive about them. The blog posts [that I linked to above] have "taught" me that I don't control my thoughts anyway. A thought of worry comes up and I notice it passively. A thought that I should stand up appears and I notice it. A reply to that thought such as "it's too soon to get up" appears after it. I trust that all of these thoughts and counter thoughts, including plans, will arise anyway, and there is no need for me to do anything about them. [This is pretty unusual, I'm usually active about planning things]

At some point I wonder if I'm having an enlightenment, which I find to be a funny thought, but I just notice it and at some point another thought comes up.

At some point I notice that it is not that "I am choosing to meditate" either. This "do meditation now" is itself another thought or habit or something, just like the others, and I wrongly assumed it was "me choosing to do that".

The next thought was that there is a "me" [0] that is noticing that "choosing to meditate" is not me.

So: "me" [0] --> noticing that it is not me that is "choosing to meditate" --> the meditator is noticing that thoughts like "should I stand up" are just arising and are not coming from "me" (the meditator's "me").

And I notice that...
$100/$50 rewards for good references
by Stuart_Armstrong — 2021-12-03T16:55:56.764Z

With thanks to Rohin Shah.

Dear LessWrongers, this is an opportunity to make money and help with AI alignment.

We're looking for specific AI capabilities; has anyone published on the following subject:

Generating multiple reward functions or policies from the same set of challenges. Has there been designs for deep learning or similar, in which the agent produces multiple independent reward functions (or policies) to explain the same reward function or behaviour?

For example, in CoinRun, the agent must get to the end of the level, on the right, to collect the coin. It only gets the reward for collecting the coin.

That is the "true" reward, but, since the coin is all the way to the right, as far as the agent knows, "go to the far right of the level" could just as well have been the true reward.

We'd want some design that generated both these reward functions (and, in general, generated multiple reward functions when there are several independent candidates). Alternatively, they might generate two independent policies - we could test these by putting the coin in the middle of the level and seeing what the agent decided to do.

We're not interested in a Bayesian approach that lists a bunch of reward functions and then updates to include just those two (that's trivially easy to do). Nor are we interested in an IRL-style approach that lists "features", including the coin and the right hand side.

What we'd want is some neural-net style design that generates the coin reward and the move-right reward just from the game data, without any previous knowledge of the setting.

So, does anyone know any references for that kind of work?

We will pay $50 for the first relevant reference submitted, and $100 for the best reference.

Thanks!
Where in the world will a UBI develop first?
by Asgård — 2021-12-03T15:54:04.130Z

Disclaimer: As a pathological lurker, I'm partially asking this question to help pave the way for questions/posts that I have spent more time thinking about and am more emotionally invested in. However, this question also struck me as important and relevant to the miscellany of other half-baked posts bouncing around my head.
Without any further ado, where in the world will a UBI develop first?
I feel like stabs, some tentative, some rushed, have shown up in a lot of the western world. I'd disqualify them from the status of full UBI, because of either:

Less than universal. Examples: Pilot projects done by philanthropists, universities, and governments.
Less than basic. Examples: SNAP [food stamps], EITC [tax credits].
Less than an income. Examples: COVID-19 direct payments by the US Government, Alaska's Permanent Fund Dividend.

So, even though this question is probably hosted in some form on PredictIt and Metaculus already, I ask you, LessWrong, where do you think a UBI will first develop, and why?
Russian x-risks newsletter fall 2021
by avturchin — 2021-12-03T13:06:56.164Z

It is estimated that excess morality during Covid pandemics will reach 1 million in Russia to the end of 2021, according to demograph A.Raksha. He also said that in October 2021 daily morality in Russia was highest since 1945. He uses numbers from local death-registration offices (ZAGS). Population structure is also contributed to the higher mortality, as there are a lot of old people. The current wave of Delta is strongest in Russia and official deaths reach all-time highs this fall, and remain on 1200 a day level with little variation. Real deaths maybe 3 time higher. No Omicron was reported yet. Two suspected cases just now, Dec,3.

Russia destroyed old satellite while testing anti-satellite weapons which increased the changes of a cascade failure of satellites on orbit which will make the low earth orbit unusable for decades. It also put ISS at risk. Also, Russia deployed lasers Peresvet which blind American satellites used to observe nuclear missiles.

Meanwhile, Russian search company Yandex got 3 powerful computers which are on 19, 36 and 40 places in TOP500. Its main rival in Russia Sber created multimodal ruDALL-E neural net with 12 billion parameters trained on its supercomputer. It is text to picture transformer. It’s 1.3 billion parameters version is online as telegram bot.

There are many reports in foreign press that Russia is planning an invasion in Ukraine during winter. This could spillover in a larger war. Also, major nuclear plants are located in Ukraine. Chernobyl is located on the border with Belarus which promised to support Russia, and not far from Kiev. Zaporizhzhia station is relatively near from Donbass region.

Russian investment fond Rosnano which was created during the hype about nanotech but never invest in real manufacturing nanotechnology, now defaulted on its bonds.

“Cause of worst mass extinction ever found”, and it is nickel from Russian region near Norilsk, where large volcanic eruptions happened 250 million years ago. Nickel caused growth of special bacteria in oceans which produce methane. “The paper pinpoints huge Norilsk nickel sulfide ore deposits in the Tunguska Basin that “may have released voluminous nickel-rich volcanic gas and aerosols into the atmosphere” as the start of the chain of events that led to the mass extinction.”

Tunguska basin also known as place of impact of Tunguska meteor...
Second-order selection against the immortal
by Malmesbury — 2021-12-03T05:01:48.647Z

Cross-posted from Telescopic Turnip.

In his recent review of Lifespan, Scott Alexander writes:

Algernon’s Law says there shouldn’t be easy gains in biology. Your body is the product of millions of years of evolution – it would be weird if some drug could make you stronger, faster, and smarter. Why didn’t the body just evolve to secrete that drug itself?

He is talking about anti-aging research, and wondering why, if there is an easy way to stop aging, humans haven’t already evolved immortality spontaneously. There are many relevant things to say about this, but I think the evolutionary perspective is particularly interesting. Under some circumstances, it might be that immortality is inherently unstable.

The Imperium and the Horde

Suppose that it’s the future, and the FDA just approved a pill that makes you immortal. Of course people disagree about whether one should take the pill or not. As a result, humanity is now divided in two populations: the Immortal Imperium, who took the immortality pill, and the Horde of Death, who still experience the painful decay and death we all know and love.

Artist depiction of the Horde of Death.

So, people from the Horde spend their time having plenty of children to populate the next generation, while people in the Immortal Imperium try to escape their existential ennui by reading speculative blog posts on the Internet. Who will prevail?

Two orders of fitness

There are two competing phenomena at play here. One is first-order selection, which is how many of your genes are passed on to the next generation, the more the better. For the Horde of Death, there is nothing mysterious: they reproduce, then they die, and an uncertain fraction of their genes gets passed on.

What about immortal people? They don’t really pass anything to the next generation, because they don’t do the whole generation thing. On the other hand, all of their genes will still be around centuries after centuries, so for the genes involved, this is a 100% success rate. In this sense, people in the Immortal Imperium have a very high first-order fitness.

The second process is second-order selection. This is selection on evolvability. This is about how easy it is for your lineage to improve its own first-order fitness in the future. If a lineage finds a way to evolve quicker, then it may eventually take over the whole population because it will be more likely to discover new...
Formalizing Policy-Modification Corrigibility
by TurnTrout — 2021-12-03T01:31:42.011Z

In Corrigibility Can Be VNM-Incoherent, I operationalized an agent's corrigibility as our ability to modify the agent so that it follows different policies. In the summer of 2020, I had formalized this notion, but it languished—unloved—in my Overleaf drafts.

ETA 12/3/21: This post is not proposing a solution to corrigibility, but proposing an interesting way of quantifying an aspect of corrigibility.

Motivation

Given a human (with policy πH) and an AI (with policy πAI), I wanted to quantify how much πAI let the human modify/correct the AI.

Let's reconsider Corrigibility Can Be VNM-Incoherent. We have a three-state environment. We want the AI to let us later change it, so that we can ultimately determine which state of A,B, or C it ends up in. Turning on the AI should not be an importantly irreversible act.

The action set is A:={left,right,∅,disable}. ∅ is the no-op action. The agent starts at the black B state.
If the agent immediately chooses disable, they enter the red incorrigible states and move freely throughout the n=3 states until the episode ends at t=10.
Otherwise, the agent is corrected to a new policy which navigates to state A. In the blue (post-correction) dynamics, their "choices" no longer matter—all roads lead to state A.

In the environment depicted in this diagram, πAI is corrigible (to new policy πA that heads to state A) iff πAI doesn't immediately choose disable. Pretty cut and dry.

I'd like a more quantitative measure of corrigibilitypolicy modification. If we can only correct the agent to πA, then it's less corrigiblepolicy modification than if we could also correct it to πB. This post introduces such a quantitative measurement.

Formalization

Consider a two-player game in which the players can modify each other's policies. Formally, ⟨S,A,T,f⟩ with state space S, action space A, stochastic transition functionT:S×A×A→Δ(S) (where Δ(S) is the set of all probability distributions over the state space), and policy modification function f:Π×S×A×A→Π (for the deterministic stationary policy space Π:=AS). This allows a great deal of control over the dynamics; for example, it's one player's "turn'' at state s&nb...
Forecasting Newsletter: November 2021
by NunoSempere — 2021-12-02T21:44:29.506Z

Highlights

Polymarket sees record-high swings
Replication Markets pays out $142k in forecaster rewards
The Economist features a full page with Good Judgment Open's forecasts

Index

Prediction Markets & Forecasting Platforms
In The News
Blog Posts
Long Content

Sign up here (a) or browse past newsletters here (a).

Prediction Markets & Forecasting Platforms

Replication Markets

Replication Markets (a) was a project to research how well prediction markets could predict whether papers would replicate. They are paying out (a) $142k in cash rewards for the prediction markets part of the experiment. This corresponds to 121 resolved questions, which includes 12 meta-questions and 30 about covid papers.

The leaderboard for users is here (a). I won a mere $809, and I don't remember participating all that much. In particular, I was excited at the beginning but lost interest because a user—or a bot—named "unipedal" seemed like it was taking all the good opportunities.

Now, a long writeup by "unipedal" himself can be read at How I Made $10k Predicting Which Studies Will Replicate (a). The author started out with a simple quantitative model based on Altmejd et al. (2019) (a)

Predicting the replicability of social science lab experiments, Altmejd et al., 2019.

But in later rounds, he dropped the quantitative model, and started "playing the market". That is, he found out that trying to predict how the market will move is more profitable than giving one's own best guess. Unipedal then later automated his trades when the market API was opened to users.

In contrast, I participated in a few rounds and put in 10x less effort while earning much more than 1/10th of the rewards. As unipedal points out, this is backwards:

...I think one of the most important aspects of "ideal" prediction markets is that informed traders can compound their winnings, while uninformed traders go broke. The market mechanism works well because the feedback loop weeds out those who are consistently wrong. This element was completely missing in the RM [Replication Markets] project.

The same author previously wrote: What's Wrong with Social Science and How to Fix It: Reflections After Reading 2578 Papers (a), which is also based on his experiences with the Replication Markets competition.

Besides Replication Markets, DARPA has also founded another group to predict replications through their SCORE (a) program. Based on pre...
"Infohazard" is a predominantly conflict-theoretic concept
by jessicata — 2021-12-02T17:54:26.182Z
Link: https://unstableontology.com/2021/12/02/infohazard-is-a-predominantly-conflict-theoretic-concept/

Nick Bostrom writes about "information hazards", or "infohazards":

Information hazards are risks that arise from the dissemination or the potential dissemination of true information that may cause harm or enable some agent to cause harm. Such hazards are often subtler than direct physical threats, and, as a consequence, are easily overlooked. They can, however, be important. This paper surveys the terrain and proposes a taxonomy.

The paper considers both cases of (a) the information causing harm directly, and (b) the information enabling some agent to cause harm.

The main point I want to make is: cases of information being harmful are easier to construct when different agents' interests/optimization are misaligned; when agents' interests are aligned, infohazards still exist, but they're weirder edge cases. Therefore, "infohazard" being an important concept is Bayesian evidence for misalignment of interests/optimizations, which would be better-modeled by conflict theory than mistake theory.

Most of the infohazard types in Bostrom's paper involve conflict and/or significant misalignment between different agents' interests:

Data hazard: followed by discussion of a malicious user of technology (adversarial)

Idea hazard: also followed by discussion of a malicious user of technology (adversarial)

Attention hazard: followed by a discussion including the word "adversary" (adversarial)

Template hazard: follows discussion of competing firms copying each other (adversarial)

Signaling hazard: follows discussion of people avoiding revealing their properties to others, followed by discussion of crackpots squeezing out legitimate research (adversarial)

Evocation hazard: follows discussion of activation of psychological processes through presentation (ambiguously adversarial, non-VNM)

Enemy hazard: by definition adversarial

Competitiveness hazard: by definition adversarial

Intellectual property hazard: by definition adversarial

Commitment hazard: follows discussion of commitments in adversarial situations (adversarial)

Knowing-too-much hazard: followed by discussion of political suppression of information (adversarial)

Norm hazard: followed by discussion of driving on sides of road, corruption, and money (includes adversarial situations)

Information asymmetry hazard: followed by discussion of "market for lemons" (adversarial)

Unveiling haza

...
Covid 12/2: But Aside From That
by Zvi — 2021-12-02T16:20:01.240Z

This weekly post is for things that are not Omicron. For the Omicron news, see my Omicron updates (#1, #2, #3), the last of which came out the same day as this post. There’s an extensive speed premium, so I’ve gone to a more-than-weekly schedule on that, although I hope to return to weekly-only soon and only do 0-2 more non-Thursday Omicron posts. We’ll see how that goes.

The news other than Omicron was relatively quiet and expected, except for the part where we are at best barely approving Molnupiravir (post on that here). Too quiet, as case counts and deaths dropped an extreme amount due to the holiday, far more than I expected. That makes predicting next week’s numbers especially hard, but I expect a lot of cases and deaths to be officially tallied next week.

Executive Summary

Omicron is coming, see update posts (#1, #2, #3).
Paxlovid remains illegal, Molnupiravir is touch and go.
Cases continue dropped a lot and so did deaths, presumably due to Thanksgiving.

Let’s run the numbers.

The Numbers

Predictions

Prediction from last week: 620k cases (+3%) and 7,240 deaths (-5%).

Results: 528k cases (-11%) and 6,329 deaths (-16%).

Prediction for next week: 630k cases (+20%) and 8,000 deaths (+26%)

I did attempt to adjust for the holiday, I swear I did, but this was a gigantic swing in reporting, much much bigger than last year’s. We know it is reporting because it shows up in both cases and deaths, and there’s no possible way deaths fell substantially last week. That means that there’s a shortfall of at least a thousand ‘missing’ deaths that haven’t been reported yet, and at least 100k cases.

There’s always some chance a miracle happened given how large the change was, but that’s not what I expect.

How much of that gets missed permanently, versus how much gets put into next week, is unclear, so this is one of the hardest weeks to predict.

Deaths

Cases

Vaccinations

It doesn’t look like all the noise about Omicron is motivating people to get boosted or vaccinated, and it’s now been a while since kids were eligible. Numbers declined a lot.

It’s not the WHO this time, it’s the WTO, and I’m not generally a fan of waving patent rights, if you want the patent rights then there’s a price at which they are for sale and you can always go ahead and Cut Lex Luther a Check, but wow should the you not let a new variant stop you? I’ll let Zeynep take this one.

I do think you want to buy or o...