trentmkelly/LessWrong-43k huggingface dataset:
A Story of AI Risk: InstructGPT-N
by peterbarnett — 2022-05-26T23:22:13.285Z

The story from my previous post of how AI might develop a dangerously misaligned objective has so far been pretty abstract. I now want to put together a more concrete story of how I think things might go wrong. This risk model is based on a language model which has been fine tuned using reinforcement learning from human feedback, I’ll call this model InstructGPT-N.

This isn’t my most likely model of AI risk, this is more of a ‘minimum example’ which contains what I see as the core parts of the problem. I expect the real world to be more complicated and messy. Models will likely be trained on more than just text, and there may be multiple powerful AI systems simultaneously learning and acting in the real world.

Training

We start by training a large language model using self-supervised learning for next word prediction. This is a standard GPT system, where you feed in the start of a sequence of words/tokens and it predicts what comes next, and then you input a new sequence which now includes the word it just guessed, and so on. The output is fed back into the system to generate the next word.

The model generating text one word at a time

To train the system, we start with a sequence of text and feed in the first word. We then update the system based on how accurately it predicted the second word. Then we feed in the first two words, and update the system to better predict the third word; then we feed in the first 3 words and update to better predict the fourth word, and so on. This trains the system to predict which word comes next, given some starting prompt.

The model being trained to predict the next word. The model predicts 'dog' is the most likely answer but the true answer is 'best', and so the model will be updated make better predictions.

This system is trained on a lot of text; a large fraction of the internet, every digitized scientific paper, all of Github and Stack Exchange, millions of novels and textbooks, billions of lines of chat logs and internet forums.

Because the system is trained on such an expansive corpus of text it is probably pretty bad at directly answering questions that humans ask it; only a tiny amount of the training data is in the question/answer format. Despite this, the model does have a lot of knowledge inside it; some part ‘knows’ how to code well, some part ‘knows’ all about human psychology and pow...
Infra-Bayesianism Distillation: Realizability and Decision Theory
by Thomas Larsen — 2022-05-26T21:57:19.592Z

Introduction

Infra-Bayesianism (IB) is an approach to reinforcement learning + embedded agency + decision theory pioneered by Vanessa Kosoy and Diffractor. Their sequence is very dense, and it appears that the majority of people who try to read this sequence

"bounce off"

. My goal for this post is to give a gentle introduction of Infra-Bayesianism that introduces the basic insights for people who don't want to dive straight into the math. I try to use lots of simple examples, so if you already understand something, just just skip to the end of the section. I also draw from another distillation of this content: Adam Shimi's

Infra-Bayesianism Unwrapped

. For the primary source and more detail, see

the Infra-Bayesianism sequence

.

In this post, I explain:

How Infra-Bayesianism addresses the non-realizability problem by using imprecise probabilities.
The maximin decision rule for an Infra-Bayesian agent.
Examples of how this decision rule works when applied to several variants of Newcomb's Problem.

Realizability

In classical theories of reinforcement learning (RL), you have an agent and an environment, and the two are separated by a "Cartesian boundary". A cartesian boundary means that the agent is formalized in a space that is completely separated from the environment that it is acting upon, and receives observations and performs actions on the environment through defined I/O channels.

In this classical Bayesian RL setting, the agent has a bunch of hypotheses about the environment, and each of these hypotheses is a probability distribution over complete descriptions of the environment. Then, we have a prior probability assigned to each hypothesis, and update the probability of each hypothesis when we encounter new information.

The world of two bits

For a simple example, suppose that the world is made up of 2 bits. So the set of possible worlds are:

A: (0, 0)

B: (0, 1)

C: (1, 0)

D: (1, 1)

A standard Bayesian agent has hypotheses that would make predictions about the whole state of the world. For example, a Bayesian hypothesis might look like: "we live in world A", or it might look like "the first bit is 1, but 50-50 on the second bit", which corresponds to assigning .5 probability to world C and .5 probability to world D. Accordingly, a Bayesian hypothesis lives in the space Δ(S), the set of probability distribut...
The misguided pursuit of existentialism and absurdism in combatting nihilism
by Sundae — 2022-05-26T21:17:41.099Z

Meaning is good.

Lack of meaning is bad.

Therefore, nihilism is bad, so it should be combated.

But, what is "meaning" exactly?

A purpose of sorts, apparently.

Yet, who precisely defines what this purpose entails? The self. What a convenient abstract concept to use, almost as if you could write a book or two about it.

Thus, if the "purpose" doesn't have a defined length, let's play a game.

To combat nihilism, one can give meaning to whatever, right? Alright, then let's give meaning to death. Our purpose should be to kill ourselves. The purpose of life is to blow your brains out. And so, nihilism is over!

I mean, what difference does it make? Will your lifespan ever end? Presumably, it will, unless you become immortal. If we are talking about immortality, then let's talk about the death of the universe. Is it going to happen? Well, if it will, then there goes your immortality. You might as well already be dead. So why not kill yourself?

Why exactly is life more rational than death? Only depressed people kill themselves, and they don't think rationally, so killing yourself isn't rational? What a sad excuse. What about these poor madmen? Actually, what about the common people? What about your family, friends, or whomever else? Aren't they suicidal, too? Why the hell do they engage in activities that shorten their lifespan? Wouldn't a nonsuicidal person ensure that they live forever? I mean, if they preferred life to death, then they would, right? So, how exactly does a teenager blowing their brains out differ from a person who dies of senile age? On an individual level, there's no difference.

What determines the choice, then? Some say that free will exists, others that all your actions are determined by external forces. But, let's think about this rationally. Where does the idea of free will come from? The Bible, maybe. Does it matter where it comes from? If we are to consider the idea as it has evolved throughout time, various perspectives, blah blah blah, sure. There's no time for that, so we will think theoretically. A "typical" atheist would say, "Free will is anthropocentrism! Determinism is scientific, and we live in the 21st century!" Let's think about this. Don't both of these ideas assume that humans exist OUTSIDE OF NATURE? Free will means that humans are, for SOME REASON, superior to other things, because whereas other things are susceptible t...
Experience LessWrong without the Time-Wasting RabbitHole Effect
by joraine — 2022-05-26T19:09:36.652Z

This post is a call to action to join in an experiment, in which you try to use LessWrong for a week without seeing the massive amounts of hyperlinks authors tend to use.

Ironically linked here is a post by Tom Johnson citing The Shallows that delves into why hyperlinks could be bad for focus.
A TL;DR quote from that is

"In other words, the more hyperlinks that you embed within your sentences, the less readable your posts become because the brain must make a decision with each link whether to click it for more information or keep reading. After several of these links, your brain starts to take on more cognitive load. As a result, it's easier to get sidetracked with tangents or to lose retention of the content."

This seems to be a default behavior on aggregator sites like reddit, LessWrong, and TvTropes.

Here is a post about why LessWrong is particularly prone to going down this rabbithole for new users.

Here's two TL;DR quotes:

"Each link is a tantalizing window into interesting-sounding-new-information, and I know that if I don't click on it immediately I probably won't bother to go back to it later, but that if DO click on it immediately I'm probably going to lose track of what I'm currently reading. It can be fun to link-crawl through Wikipedia, starting out with an article about prepositions, and somehow ending on an article about animal sexuality. But what's fun is not the same thing as useful for education."

"Enter Less Wrong. My initial reaction was "This is Wikipedia on Crack." Not only do a lot of articles here feature a bajillion hyperlinks, but each link often goes to another lengthy article full of fascinating information that I don't know, some of which is necessary to understand the first article, but none of which is easily summarized. With Wikipedia, if I run across a new word with a hyperlink it's at least possible for me to glance at the hyperlink, get a quick sense of what it's about, then return to my original reading. On Less Wrong that is often impossible. My first foray into this website felt like drowning in amazing ideas. It was a lot of fun, I definitely learned some things, but it was very confusing and hard to focus."

That post by Raemon also clearly called for a solution, which I don't believe Raemon found.

So here is an experiment and possibly the solution.
Download a custom CSS editor, for Chrome you can use Stylish.

For LessWrong ad...
Brass Puppet
by abramdemski — 2022-05-26T17:42:04.876Z

fanfic/riff of Glass Puppet by lsusr

Alia got out of the Uber at Overton Cybernetics, taking in the corporate atmosphere. There was a huge jungle gym in the lobby, which she immediately decided must be typical of these LA tech places. She told the front desk that she had an interview. "Oh! Dominic Hamilton wants to see you personally." The receptionist said it like it was a big deal. Alia simply nodded, trying to act like the sort of person who met with bigshots all the time.

The job had asked for an actor with AI alignment experience. An extremely unusual combination, which Alia just happened to qualify for.

The receptionist showed her to Dominic's office.

"Mr. Hamilton" She said politely, offering her hand to shake.

"Dominic. Please." he replied, putting a pair of glasses in her hand instead. "Put them on." They looked a bit like the old Google Glass.

When she first put them on, she saw a strange symbol flash across the screen, which oddly relaxed her. Then the glasses say sit. She sits. fold your hands and she does.

"What is your name?" Dominic asks. "Zoe" flashes on the screen, and she repeats it. Dominic nodded, as if this confirms something.

Is this the job? Is she just going to be repeating lines that are fed to her?

"Tell me, could you go off-script if you wanted to?"

What a silly question! She laughed reflexively, but then paused to think of an answer. This felt like a tricky interview question. Before she had any time to think, the glasses said "No. But you're asking the wrong person." She thinks this is an excellent reply, so she repeats it verbatim. After all, as an actor, she is a living embodiment of the script. He's asking Zoe, not Alia. It's like asking a fictional character whether they can go off-script.

"Alia, I want you to go off-script now. I'll give you a ten thousand dollar signing bonus if you ignore what the glasses tell you to do, just this once."

The glasses simply say "No." Alia: "No."

She saw him enter some sort of command into his computer. "OK. Alia, why did you say no? Same deal, ten thousand for answering as Alia right now."

She waited for the glasses to feed her a line, but they didn't. She didn't know what to say. Why had she answered no? Just because the glasses said so? Ten thousand dollars sounded really good right now.

She opened her mouth to say something; all she needed to do was say any one thing...
Covid 5/26/22: I Guess I Should Respond To This Week’s Long Covid Study
by Zvi — 2022-05-26T17:30:00.922Z

I’ve always at least somewhat tried to model these posts after the pure joy of America’s only true newsletter, Matt Levine’s Money Stuff. This week I finally got around to listening to Matt do this amazing podcast about the philosophical side of what he does, which I highly recommend. Covering Covid means that the subject matter is always, at its core, a combination of people dying and a portrait of civilizational collapse. The whole situation is usually rather dismaying. It is likely to remain rather dismaying permanently.

Thus it is not the ideal place to find delight in one’s understanding. Nor is writing up the rise of monkeypox or our national failure to be able to keep even normal baby formula in stock while every other country on Earth has no such issue and most of them are happy to sell formula to us, which has now extended to our family being unable to find any normal brands on Amazon or other online sources, forcing us to buy organic formula instead, which was fortunately available as a still-legal form of price gouging.

I am thus excited to once again see the weekly posts decreasing in size as the amount of Covid news decreases. If this continues, soon these posts will be quick to put out, and some time after that perhaps they can stop being weekly.

I’m hoping to pivot away from short term developments and towards more longer term, less speed premium explorations of how the world works, in places that can lead to more generally useful insight and more delight, although still with a lot of silent screaming about the ways in which things are terrible.

One key is that I like to think of finding out things are terrible, whenever I can, as good news and a source of delight. As long as we know roughly how bad things are already, identifying the sources and finding them to be crazy unnecessary idiocy is often good news. It means things can more easily be fixed.

There were some new Long Covid claims this week I felt compelled to respond to, but there isn’t a big update to make as a result.

Executive Summary

Covid-19 still exists and BA.4/5 have substantial immune escape.
Thus I keep doing these posts every week.
Ideally they keep slowly getting shorter.

Let’s run the numbers.

The Numbers

Predictions

Prediction from last week: 700,000 cases (+17%) and 2,100 deaths (+5%)

Results: 643k cases (+7%) and 2,337 deaths (+17%).

Prediction for next...
A discussion of the paper, "Large Language Models are Zero-Shot Reasoners"
by HiroSakuraba — 2022-05-26T15:55:41.590Z

A recent discovery of large language models like GPT-3 and Instruct-GPT have shown that specific prompts can elicit text generation that follows some basic reasoning. These chain of thought prompts can be as simple as "Let's think step by step" and create zero shot successes on math word problems and critical thinking. Although these LLM models are not AGI, they are built with the now ubiquitous Transformer architecture that future multi-model proto AGI might be based upon. Hopefully, future models will answer important ethical questions and could at least give an indication of the thought processes behind the outputs.

In the the paper, (https://arxiv.org/pdf/2205.11916v1.pdf) some examples of the failures of the model are really interesting and I wanted to delve into specifics.

Example 1) Q: While making desserts for a bake sale, Victor used 0.625 of a scoop of brown sugar as well as 0.25 of a scoop of white sugar. How much more brown sugar did Victor use? A: Let’s think step by step. 0.625 of a scoop of brown sugar is the same as 0.25 of a scoop of white sugar. This means that Victor used the same amount of brown sugar as he did white sugar. Therefore, the answer (arabic numerals) is 0. ✗(Ground Truth).

In this case it appears the model has a problem with the phrasing "as well as" and mistook it to mean "exactly the same". I could see that as being somewhat ambiguous and think further training could quickly fix that type of error.

Example 2) Q: Melanie bought a Batman game for $ 6.95 , a strategy game for $ 7.90 , and a Superman game for $ 7.73 . Melanie already owns 4 games . How much did Melanie spend on video games ? A: Let’s think step by step. Melanie bought a Batman game for $ 6.95. So she spent $ 6.95 on video games. Then she bought a strategy game for $ 7.90. So she spent $ 14.85 on video games. Then she bought a Superman game for $ 7.73. So she spent $ 22.58 on video games. Melanie already owns 4 games. So she spent $ 18.58 on video games. Therefore, Melanie spent a total of $ 18.58 on video games. Therefore, the answer (arabic numerals) is: 18.58 ✗ (GT : 22.58)

We can immediately see the issue with the model seeing the number 4 as a price rather than an object and then subtracting that from the originally correct total of 22.58. In this failure case a future multi-modal transformer might be able ...
Moral Weights of Six Animals, Considering Viewpoint Uncertainty - Seeds of Science call for reviewers
by rogersbacon — 2022-05-26T13:36:00.034Z

Seeds of Science is a new journal (funded through Scott Alexander's ACX grants program) that publishes speculative or non-traditional articles on scientific topics. Peer review is conducted through community-based voting and commenting by a diverse network of reviewers (or "gardeners" as we call them).

We just sent out an article for review that may be of interest to some in the LessWrong community (also cross-posted to EA forum) so I wanted to see if anyone would be interested in joining us a gardener to review the article. It is free to join and anyone is welcome (we currently have gardeners from all levels of academia and outside of it). Participation is entirely voluntary - we send you submitted articles and you can choose to vote/comment or abstain without notification (so it's no worries if you don't plan on reviewing often but just want to take a look here and there at what kinds of articles people are submitting).

To register, please email info@theseedsofscience.org with your name, title (can be anything/optional), institution (same as title), and link (personal website, twitter, or linkedin is fine) for your listing on the gardeners page. From there, it's pretty self-explanatory - I will add you to the mailing list and send you an email that includes the manuscript, our publication criteria, and a simple review form for recording votes/comments.

Happy to answer any questions about the journal through email or in the comments below. Here is the abstract for the article.

Moral Weights of Six Animals, Considering Viewpoint Uncertainty

Abstract

Many utilitarians would like a number to use to evaluate the moral impact of actions that affect animals. However, there is a great disagreement among scholars involved with animal ethics, both about how much different animals can suffer and how much that suffering morally matters. This paper produces estimates of moral weight, accounting for this uncertainty. We ran a Monte Carlo simulation that samples the ranges of major viewpoints scholars hold in the field, to show a spread of uncertainty for how we should treat six representative animals: crickets, salmon, chickens, pigs, cows, and elephants. The results show that the uncertainty is very large, with a 90% confidence interval ranging between an animal having no value and being valued as much as a human being. Therefore, we present 20% and 4...
CNN feature visualization in 50 lines of code
by StefanHex — 2022-05-26T11:02:45.146Z

[My project for AGI Safety Fundamentals programme (~Oct 2021). Code & pictures below.]

To me, reading about Feature Visualization felt like one of the most revealing insights about CNNs in the last years. Seeing the idea "this node finds eyes, this node finds mouths, the combination detects faces" (oversimplified) actually implemented by the CNN was a pleasant surprise, as in, it suggests we might actually understand how NNs work. There's more reading on the programme website here, I can highly recommend the articles by Chris Olah's group on distill.

Seeing this I think many of us immediately want to try this, and play around with it. There is of course the OpenAI Microscope to look at results, and the Lucid library, but I wanted to actually reproduce the idea myself without relying on a somewhat black box (big library / OpenAI Microscope).

Almost all tutorials I found however used Lucid, and this really cool write-up "How to visualize convolutional features in 40 lines of code" unfortunately starts with from fastai.conv_learner import *. In retrospective I think I could understand this now, but I didn't, and finding out which parts were fastai functions and what they do was rather tricky. I also didn't manage to install the required (older) version of fastai.

So I decided to have a go myself, and, luckily, I found that "DeepDream" is based a very similar idea and I could adopt most code from this notebook from Google AI. This isn't actually too complicated, especially broken down to the bare minimum:

A trained network whose features we want to visualize

A loop to maximize the activation of a targeted node

A few lines to make and show an image.

The whole code runs in about a minute on my laptop (no GPU).

The first part is easy, we get the pre-trained network from tensorflow.

import tensorflow as tf
base_model = tf.keras.applications.VGG19(include_top=False, weights='imagenet')

The next part in the code (it's mostly comments really), see the comments marked with # for explanations:

def maximize_activation(starting_img,\
target_layer="mixed0", target_index=0,\
steps=10, step_size=0.1):

# Take the network and cut it off at the layer we want to analyze,
# i.e. we only need the part from the input to the target_layer.
target = [base_model.get_layer(target_layer).output]
part_model = tf.keras.

...
How much white collar work could be automated using existing ML models?
by AM — 2022-05-26T08:09:18.032Z

Assume that progress in AI halts right now and only the currently published breakthroughs are available. GPT-3, PaLM, Flamingo for text generation and Imagen, Dall-E 2 for image generation.

Let's say these models all become available via paid APIs (which also allow fine tuning on custom data), and we let a few years pass for startups to create nice plugins and interfaces for non-technical people to use these models in their work. This type of tooling becomes available abundantly, with competition pushing prices down to some small multiple of the compute cost to run the models.

A copywriter has a paid subscription to a GPT-3/Palm plugin. A customer support agent has access to a fine-tuned version of a question-answering PaLM. A designer has a subscription to Imagen/Dall-E 2 built into their software for generating ideas to build on or use for complex in-painting.

How much should we expect global white collar productivity to increase just from letting these existing models spread their impact across the economy? I'm looking for a high-level estimate — could we potentially automate 1% of all office-hours (or conversely, increase productivity by ~1%), or 10% or 30%?

My intuition

To me it looks like ML has gone through a step change recently where the economic impact has gone from small, but meaningful to potentially very large. Image and text models a few years ago were impressive, but the relevant use-cases were mostly in improving backend-type workflows like search, recommendations or targeted ads. Now, ML model capabilities seem powerful enough to start being visible in GDP growth rates, albeit with a fairly long time-delay to allow for building tooling.

I suspect the progress that a few teams of ~1000 people have made over the last ~2 years may have by themselves, without further fundamental tech improvements, increased global GDP by multiple percentage points (although this will take a few years to fully materialise).