trentmkelly/LessWrong-43k huggingface dataset:
Limits to the Controllability of AGI
by Roman_Yampolskiy — 2022-11-20T19:18:12.782Z

Roman Yampolskiy

(Summary by Karl von Wendt and Remmelt Ellen)

This is a summary of the paper

“On the Controllability of Artificial Intelligence: An Analysis of Limitations”

, which shows that advanced artificial general intelligence (AGI) with the ability to self-improve cannot be controlled by humans. Where possible, we have used the original wording of the paper, although not necessarily in the same order. For clarity, we have replaced the term AI in the original paper with “AGI” wherever it is referring to general or superintelligent AI. Literature sources and extensive quotes from previous work are given in the original paper.

Introduction

The invention of artificial general intelligence is predicted to cause a shift in the trajectory of human civilization. In order to reap the benefits and avoid pitfalls of such powerful technology it is important to be able to control it. Unfortunately, to the best of our knowledge no mathematical proof or even rigorous argumentation has been published demonstrating that the AGI control problem may be solvable, even in principle, much less in practice.

It has been argued that the consequences of uncontrolled AGI could be so severe that even if there is a very small chance that an unfriendly AGI happens it is still worth doing AI safety research because the negative utility from such an AGI would be astronomical. The common logic says that an extremely high (negative) utility multiplied by a small chance of the event still results in a lot of disutility and so should be taken very seriously. But the reality is that the chances of misaligned AGI are not small. In fact, in the absence of an effective safety program that is the only outcome we will get. So in reality the statistics look very convincing to support a significant AI safety effort. We are facing an almost guaranteed event with potential to cause an existential catastrophe. This is not a low risk high reward scenario, but a high risk negative reward situation. No wonder that this is considered by many to be the most important problem ever to face humanity.

Types of control

The AGI Control Problem can be defined as: How can humanity remain safely in control while benefiting from a superior form of intelligence?

Potential control methodologies for superintelligence have been classified into two broad categories, namely Capability Control and Motivational Control-bas...
Career Scouting: Dentistry
by koratkar — 2022-11-20T15:55:12.431Z
Link: https://careerscouting.substack.com/p/dentistry

As a high school student, I worry a great deal over my future profession. According to Cal Newport, career satisfaction for any choice of occupation often won't materialize until you've become "so good they can't ignore you" at what you do. Based on this, Newport recommends directing your nervous energy towards building skill in whatever you choose, rather than choosing the "right job". While I found his advice useful for framing the issue, it opened a new box of concerns: What if you chose something you have little innate ability in – or what if the point at which your improvement slows down is not exceptional? How do you even know whether you have talent in something before you've invested effort into shadowing or interning? Sometimes talent doesn't appear initially – what if the thing you have the greatest potential in is something you'll have to struggle at for a long time? And there are so many jobs! We have to narrow the search space!

There's another panoply of concerns related to your worth to the world: if you care about the world and your impact in it, don't you owe it to those who worked hard to give you the opportunities you have to find the thing you'll be the best at? But what if the thing you'll end up being the best at is something that doesn't always scale well – like medicine? (Of course, you could go into research, but what if you're only mid-tier at that?) You'll end up positively affecting the lives of many fewer people than you could have! And how do you avoid choosing work dry of meaning?

People tell me I over-think these things – the answer to most (if not all) of my what-if's is "nothing interesting will happen in this case or the counter-factual one - you are ultimately insignificant in the greater procession of the world, and you live a cushy live in the developed world, meaning that no matter how badly you screw things up, as long as you don't get addicted to heroine, you will still have access to food, water, and shelter." While I think that answer is probably the right one, it looks like most young people don't think about this much at all! I have a seriously useful bone to throw them from my side of the anxiety fence!

To provide more information, I'm compiling a database of rationalist-inspired interviews with members of various professions over at careerscouting.substack.com.

Below is my first interview with a general dentist...
Decision Theory but also Ghosts
by eva_ — 2022-11-20T13:24:51.552Z

Spoiler Warning: The Sixth Sense (1999) is a good movie. Watch it before reading this.

A much smaller eva once heard of Descartes' Cogito ergo sum as being the pinnacle of skepticism, and disagreed. "Why couldn't I doubt that? Maybe I just think 'I think' → 'I am' and actually it doesn't and I'm not." This might be relevant later.

FDT has some problems. It needs logical counterfactuals, including answers to questions that sound like "what would happen if these logically contradictory events co-occurred?" and there is in fact no such concept to point to. It needs logical causality, and logic does not actually have causality. It thinks it can control the past, despite admitting that the past has already happened and believing its own observations of the past even when these contradict its claimed control over the past.

It ends up asking things like "but what would you want someone in your current epistemic state to do if you were in some other totally contradictory epistemic state" and then acting like that proves you should follow some decision policy. Yes, in a game of Counterfactual Mugging, someone who didn't know which branch they were going to be in would want that they pay the mugger, but you do know what branch you are in. Why should some other less informed version of yourself get such veto power over your actions, and why should you be taking actions that you don't expect to profit from given the beliefs that you actually have?

I have an alternative and much less spooky solution to all of this: ghosts.

Ghosts

In contrast to Philosophical Zombies, which are physically real but have no conscious experience, I define a Philosophical Ghost to be something that has an experience but is not physically instantiated into reality, although it may experience the belief that it is physically instantiated into reality. Examples include story characters, simulacra inside of hypothetical or counterfactual predictions, my mental model of the LessWrong audience that I am bouncing thoughts off of as I write this post, your mental model of me that you bounce thoughts off of as you try to read it, and so on.

That Ghosts can be agentic follows directly from any computational theory of mind. This universe seems purely computational, so to believe that we exist at all we must subscribe to some such theory, by which the ghosts exist as well.

That you might genuinely be...
ARC paper: Formalizing the presumption of independence
by Erik Jenner — 2022-11-20T01:22:55.110Z
Link: https://arxiv.org/abs/2211.06738

(I did not have anything to do with this paper and these are just my own takes.)

The Alignment Research Center recently published their second report, Formalizing the presumption of independence. While it's not explicitly about AI alignment, it's probably still interesting for some people here.

Summary

The paper is about "heuristic arguments". These are similar to proofs, except that their conclusions are not guaranteed to be correct and can be overturned by counterarguments. Mathematicians often use these kinds of arguments, but in contrast to proofs, they haven't been formalized. The paper mainly describes the open problem of finding a good formalization of heuristic arguments. They do describe one attempt, "cumulant propagation", in Appendix D, but point out it can behave pathologically.

So what's the "presumption of independence" from the title? Lots of heuristic arguments work by assuming that some quantities are independent to simplify things, and that's what the paper focuses on. Such an argument can be overturned by showing that there’s actually some correlation we initially ignored, which should then lead to a more sophisticated heuristic argument with a potentially different conclusion.

What does this have to do with alignment?

The paper only very briefly mentions alignment (in Appendix F), more detailed discussion is planned for the future. But roughly:

Avoiding catastrophic failures. Heuristic arguments can let us better estimate the probability of rare failures, or failures which occur only on novel distributions where we cannot easily draw samples. This can be used during validation to estimate risk, or potentially during training to further reduce risk.

Eliciting latent knowledge. Heuristic arguments may let us see “why” a model makes its predictions. We could potentially use them to distinguish cases where similar behaviors are produced by very different mechanisms—for example distinguishing cases where a model predicts that a smiling human face will show up on camera because it predicts there will actually be a smiling human in the room, from cases where it makes the same prediction because it predicts that the camera will be tampered with. [...]

Neither of these applications is straightforward, and it should not be obvious that heuristic arguments would allow us to achieve either goal. [...]

Heuristic arguments can be seen as somewhere between ...
Update to Mysteries of mode collapse: text-davinci-002 not RLHF
by janus — 2022-11-19T23:51:27.510Z

I (and many others) did not realize this before, but: text-davinci-002 and text-davinci-001, the InstructGPT models on the OpenAI API, were not trained with RLHF (reinforcement learning from human feedback) as described in the InstructGPT paper, but a "similar but slightly different"[1] method that uses the same human feedback data. Apparently, this other method is not technically RLHF.

Since this update has potentially nontrivial implications for interpreting the phenomena exhibited by text-davinci-002 described in Mysteries of mode collapse (formerly titled "Mysteries of mode collapse due to RLHF"), I'm making this separate post for a signal boost.
I have not corrected the original text of "Mysteries of mode collapse due to RLHF", but I've added a section at the beginning with further details on this update, copied here:

I have received evidence from multiple credible sources that text-davinci-002 was not trained with RLHF.

The rest of this post has not been corrected to reflect this update. Not much besides the title (formerly "Mysteries of mode collapse due to RLHF") is affected: just mentally substitute "mystery method" every time "RLHF" is invoked as the training method of text-davinci-002. The observations of its behavior otherwise stand alone.

This is kind of fascinating from an epistemological standpoint. I was quite surprised to learn that text-davinci-002 was probably not trained with RLHF. I don't remember exactly how "text-davinci-002 is RLHF" got elevated to an unquestioned assumption in my mind. I might have mistook not being contradicted by people who I assumed were in the know as confirmation. I certainly did not expect to talk for months to dozens of people about odd behaviors I've observed in a well-known model "due to RLHF" without being contradicted in a world where the model in question wasn't trained with RLHF, but that's what happened.[2] It wasn't just me either: the assumption that text-davinci-002(/text-davinci-001) is InstructGPT is RLHF seems ambient (e.g. search "text-davinci-002 rlhf" on Twitter, this LW post, this article, and many others). I contributed to perpetuating this misinformation cascade, and for that I apologize.

text-davinci-002's behaviors described in this post also contributed to my confidence because RLHF seemed to be a likely and potentially satisfying explanation. Its apparently unsubstantiated confidence

...
Make the Drought Evaporate!
by AnthonyRepetto — 2022-11-19T23:41:29.352Z

~ desalinate with <5% the energy & equipment ~

TL;DR — We need to stop droughts, especially as climate change makes them worse. Desalination is nice, capturing clean water from the sea, yet it requires expensive equipment and lots and lots of energy. We can, instead, accelerate evaporation OUT into the air above our heads! We won’t be ‘capturing’ that clean water — it just rains-down miles away. Perfect, because we don’t need pipes and pumps to transport it! Using this sort of design, we can end droughts and re-take the land captured by deserts in recent decades, cheaply and quickly.

The Concept

Imagine you have a bowl of salty sea water, with a tiny hole at the bottom. The sea water is slowly dribbling out of that hole, into another bowl beneath, to then get pumped back up to the top. At that dribbling spout, you hang a loose, fuzzy length of yarn, straight down. The water is held against that yarn by surface tension, scuttling down along its surface.

As the wind blows past your wet-yarn-wick, it will cause evaporation from that surface, adding humidity to the air, to come back down as rain further downwind. (Place the evaporators geographically so that the wind carries the water to the general spot you plan to use.) With wide troughs, thousands of yarns packed per square yard, stacked a dozen feet high in layers, you can pump vast amounts of water into the air.

The cost of pumping sea water up onto those troughs, only a few dozen meters above sea-level, is less than 1 Megajoule per ton of water. A Megajoule generated on-site from solar, only pumping when the sun shines, costs about 1 or 2 cents, for a ton of water to evaporate and return as rain on farms and snowpack in the mountains. For anyone in Ag, this translates to “$12 per acre-foot of irrigation, with no pumping or well-drilling, maintenance, or ditches… because it just rains.” That sounds darn good to me!

Compare that to the top-of-the-line desalination plants, spending 10 to 20 Megajoules per ton of fresh water produced! If you want to irrigate with that, you need to pump it to the site, with the associated barrier to entry and lag to scaling, maintenance and risk, while the water itself still costs you $240 per acre-foot! Some regions of California have seen peak prices at $2,000 an acre-foot, which is crippling. At $12 a pop, we can get green again — in flora and finances.

Getting into Details

S...
Elastic Productivity Tools
by Simon Berens — 2022-11-19T21:59:39.913Z
Link: https://simonberens.me/blog/elastic-productivity-tools

Like most tech bros, I’m a little too interested in productivity and optimizing my life. I’ve even made a few of my own tools to help me stay focused and efficient. In the process of trying to find and build the best productivity tools possible, I discovered common elements among all the productivity tools I found most effective.

I call such tools elastic, because of their similarities to rubber bands. And so, elastic tools

stretch, meaning that they’re designed to allow for some leeway
snap back, i.e. after stretching they automatically go back to their intended form
don’t break, in the sense that you can never stretch them too far

Elastic tools are a marked improvement over standard tools, which typically

are brittle, allowing no leeway
don’t help you get back on track
end up getting ditched

Inelastic Tool Examples

Freedom: gives no way of getting out of a focus session, so when I really needed to access a blocked app I would disable it, and it would take me days to re-enable it.

Blocklist: when I needed to go to a blocked site, it gave me no choice but to disable it.

I had 3 tasks in the windows task scheduler that would shut my computer down at 9, 10, and 11pm respectively in an attempt to enforce a consistent bedtime. However, when I knew I needed to get work done I would disable them, and then go to bed at 2am the days following that, until I would turn them back on.

Elastic Tool Examples

After Freedom didn’t work out for me, I switched to LockMeOut, where I could pay $2 to unlock it for the day. Unlocking for the day is a little too much for my taste, but much better than disabling it altogether.

As for Blocklist, I made my own alternative, where if you really want to go on a blocked website, you have to effectively stare at your screen for ~1 minute. After that, you can go on previously blocked websites for ~1 hour, after which it blocks the websites again. I found that a little bit of boredom is the perfect amount of deterrent.

To replace my shutdown tasks, I made my own tool to shut down my computer every 30 minutes after 9pm. If I need to do work, I can skip the next shutdown. This effectively gives me a maximum of 30 minutes to slack off after I finish working before my computer shuts down.

Things That Should Be Made Elastic

All my lights are connected to smart plugs, which I schedule to turn on at 7am (and off at 9pm) with Alexa. However, someti...
A Short Dialogue on the Meaning of Reward Functions
by Leon Lang — 2022-11-19T21:04:30.076Z

Produced as part of the

SERI ML Alignment Theory Scholars Program

- Winter 2022 Cohort.

The following is a short slack dialogue between Leon Lang, Quintin Pope, and Peli Grietzer that emerged as part of the SERI-MATS stream on shard theory. Alex Turner encouraged us to share it.

To follow the dialogue, it seems beneficial to have read Reward is not the optimization target before.

Leon:

I was thinking about what the "limits" of the idea "reward is not the optimization target" is. Playing devils advocate: Maybe the reward doesn't matter at all! We spend all this time thinking about how to design reward functions that somehow capture our values. But maybe the only thing we need to do is have some kind-of-dumb reward, and an environment where the agent can only reach this reward by acting in accordance to our values.
E.g.: We might reward the agent for winning some board game which is (partially) cooperative. But then in the environment, real people only cooperate with the agent if it displays cooperative/friendly/nice behavior, and other behaviors are a losing strategy. Never did the reward itself say anything about being cooperative, but this still emerged in accordance with the environment, and so the agent ends up pretty nice.
Someone might argue against this view: Being nice is, in this environment, only instrumentally valuable for achieving reward, and a smart agent knows this, plays nice to win games and, once it has enough power, will overthrow humanity. This is a version of the problem of deceptive alignment.
However, I would argue (still playing devil's advocate): The agent's learning process doesn't care about whether something is an instrumental value or the actual reward! The behavior to be friendly will be reinforced just as much, and the trained agent doesn't necessarily reason about whether something is instrumentally or finally valuable, it just wants to be nice if this was reinforced.
Not playing devil's advocate anymore, I'm a bit skeptical of this type of reasoning. Human's are generally pretty social, but we also know that powerful humans often lose quite a lot of their morality.

Peli:

I like this a lot! I think there's a novel and true insight here, which I'd maybe paraphrase as 'the reinforcement schedule is a function of the reward function and environment together, and it's the reinforcement schedule that matters.'

Qui...
By Default, GPTs Think In Plain Sight
by Fabien Roger — 2022-11-19T19:15:29.591Z

Epistemic status: Speculation with some factual claims in areas I’m not an expert in.

Thanks to Jean-Stanislas Denain, Charbel-Raphael Segerie, Alexandre Variengien, and Arun Jose for helpful feedback on drafts, and thanks to janus, who shared related ideas.

Main claims

GPTs’ next-token-prediction process roughly matches System 1 (aka human intuition) and is not easily accessible, but GPTs can also exhibit more complicated behavior through chains of thought, which roughly matches System 2 (aka human conscious thinking process).
Human will be able understand how a human-level GPTs (trained to do next-token-prediction) complete complicated tasks by reading the chains of thought.
GPTs trained with RLHF will bypass this supervision.

System 2 and GPTs’ chains of thought are similar

A sensible model of the human thinking process

Here is what I feel like I’m doing when I’m thinking:

Repeat

Sample my next thought from my intuition
Broadcast this thought to the whole brain[1]

When you ask me what is my favorite food, it feels like some thoughts “pop” into consciousness, and the following thoughts deal with previous thoughts. This is also what happens when I try to prove a statement: ideas and intuitions come to my mind, then new thoughts about these intuitions appear.

This roughly matches the model described in Consciousness and the Brain by Stanislas Dehaene, and I believe it’s a common model of the brain within neuroscience.[2]

How GPTs “think”

Autoregressive text models are performing the same kind of process when they generate text. Sampling text is using the following algorithm:

Repeat:

Do a forward pass, and sample the next token from the output distribution
Add the generated token to the input. It makes it part of the input for the next forward pass, which means it can be used by lots of different attention heads, including specialized heads at earlier layers.

This looks similar the human thinking process, and the rest of the post will be exploring this similarity and draw some conclusion we might draw from this.

System 2 and GPTs’ chains of thought have similar strengths

A theoretical reason

The sample + broadcast algorithm enables the execution of many serial steps. Neurons take about 1ms to fire[3], which means that if a thought takes 200ms to be generated, the brain can only do 200 serial operations. This is similar to the hundredths of serial matrix multiplications GPT-3 ...