trentmkelly/LessWrong-43k huggingface dataset:
Consequentialism & corrigibility
by Steven Byrnes — 2021-12-14T13:23:02.730Z

Background 1: Preferences-over-future-states (a.k.a. consequentialism) vs Preferences-over-trajectories other kinds of preferences

(Note: The original version of this post said "preferences over trajectories" all over the place. Commenters were confused about what I meant by that, so I have switched the terminology to "any other kind of preference" which is hopefully clearer.)

The post

Coherent decisions imply consistent utilities (Eliezer Yudkowsky, 2017)

explains how, if an agent has preferences over future states of the world, they should act like a utility-maximizer (with utility function defined over future states of the world). If they don’t act that way, they will be less effective at satisfying their own preferences; they would be “leaving money on the table” by their own reckoning. And there are externally-visible signs of agents being suboptimal in that sense; I'll go over an example in a second.

By contrast, the post

Coherence arguments do not entail goal-directed behavior (Rohin Shah, 2018)

notes that, if an agent has preferences over universe-histories, and acts optimally with respect to those preferences (acts as a utility-maximizer whose utility function is defined over universe-histories), then they can display any external behavior whatsoever. In other words, there's no externally-visible behavioral pattern which we can point to and say "That's a sure sign that this agent is behaving suboptimally, with respect to their own preferences.".

For example, the first (Yudkowsky) post mentions a hypothetical person at a restaurant. When they have an onion pizza, they’ll happily pay $0.01 to trade it for a pineapple pizza. When they have a pineapple pizza, they’ll happily pay $0.01 to trade it for a mushroom pizza. When they have a mushroom pizza, they’ll happily pay $0.01 to trade it for a pineapple pizza. The person goes around and around, wasting their money in a self-defeating way (a.k.a. “getting money-pumped”).

That post describes the person as behaving sub-optimally. But if you read carefully, the author sneaks in a critical background assumption: the person in question has preferences about what pizza they wind up eating, and they’re making these decisions based on those preferences. But what if they don’t? What if the person has no preference whatsoever about pizza? What if instead they’re an asshole restaurant customer who derives pure joy from ...
Enabling More Feedback for AI Safety Researchers
by  — 2021-12-13T20:10:07.047Z

Cross-posted from the EA forum.

Acknowledgements

Thank you to JJ Hepburn for creating this initiative & providing the ideas described in this post, and thank you to Daniel Braun for his comments.

Introduction

AI Safety has a pool of research talent working via three main avenues: industry, academia, and independent research. Unfortunately, independent researchers and, depending on the lab, those in academia can be physically isolated from others in the community. Thus, they lose out on incidental feedback: bumping into coworkers at lunch or labmates in the office and having continuing, casual discussions on research ideas, progress updates, points of uncertainty, etc. Alternative sources of feedback are largely constrained and intimidating: you can share drafts with people, write a blog post, email/message back and forth, or schedule video calls with other researchers. These sources can be difficult to find and typically follow a, “provide context, receive feedback” structure that prevents deep exploration of particular ideas or avenues, and unless scheduled regularly, will lack the familiarity that can elevate the quality of a discussion.

Building community through feedback events

AI Safety Support would like to try hosting feedback/discussion events. Ideally, these could run for a longer duration (maybe 90 minutes) with participants allowed to pop out at any time (there could be a scheduled break after 45 minutes so those who feel uncomfortable exiting can do so at that time). The event would involve a host individual or group briefly presenting their work, followed by free-form discussion between participants and the hosts, rather than one-way feedback. If there is some rough continuity in the group of people who frequent such discussions, there is increased potential for familiarity between participants and hosts. Additionally, AISS relieves the burden of requesting feedback and can help encourage a friendly, collaborative atmosphere.

The challenge of feedback events

In particular, AISS would like participants to have a working knowledge of AI Safety research to facilitate high-level discussions. One challenge with this is convincing such participants that feedback events are worth their time. This is difficult because:

Participants are providing most of the value in these events. Though AI Safety researchers may provide in

...
Language Model Alignment Research Internships
by Ethan Perez — 2021-12-13T19:53:32.156Z

I'm

Ethan Perez

, a final year PhD student at NYU, working on aligning language models with human preferences. I'm looking to hire research interns to work on projects in this space, starting early 2022. I expect candidates to have strong software engineering ability, for ML engineering (e.g., to finetune GPT2 to good performance on a new task) or data engineering (e.g., to quickly find high quality subsets of text within petabytes of Common Crawl data). Ideal candidates will have experience doing ML and/or NLP research, reading papers, and coming up with ideas to test. I'm looking for people who'd be able to work full-time (remotely), with compensation of $70/hour. I expect each project to take 4-8 months to complete and lead to a first-author publication at a top-tier ML or NLP conference.

Below are a few examples of projects I have in mind:

Pretraining language models on more factual, high-quality, and value-aligned text, so that they are easier to control and

more truthful

.
Developing algorithms for training language models to learn from different forms of human feedback, beyond simple scalar/numerical feedback as in

prior work

.
Finding tasks that exhibit inverse scaling, where better performance on the language modeling pretraining objective leads to worse downstream task performance. Examples of such tasks include

TruthfulQA

and RealToxicityPrompts.

If what I've described sounds like a good fit, I'd love to hear from you over email! Just send me your website, resume, LinkedIn, GitHub, Google Scholar, or anything else that might be helpful for understanding your background (my email is perez at nyu dot edu).
Omicron Post #6
by Zvi — 2021-12-13T18:00:01.098Z

Previous posts: #1, #2, #3, #4, #5, last weekly post

Well, that escalated quickly. Omicron has already escalated quickly in log terms, and will soon escalate quickly in linear terms. The core questions remain the same, and we continue to continuously get better data to allow us to make better guesses and projections.

There are three central places where population-level information is available first:

South Africa, because Omicron has taken over already and they have good data.
Denmark, because they do all the sequencing.
United Kingdom, because of S-gene-deletion data and good sequencing.

We also have a few African countries like Zimbabwe with giant spikes. But in other European countries and in the United States, the lack of sequencing makes it impossible to know how far along things are or how fast they are moving.

Thus, we must rely on what data we have, and give thanks to those who provide it.

Uncertainty abounds but there is no known link between ‘has more Omicron infections’ and ‘has ability and willingness to monitor Omicron cases’ so we should start with the assumption that Denmark and the United Kingdom are typical western countries for this, adjusted for the state of Covid-19 under Delta there relative to other places.

All of that can and should be complemented by experimentation and lab results. Alas, as per usual, challenge trials and most non-natural experiments remain illegal, so we have orders of magnitude less knowledge and certainty from these sources, and everyone is much worse off, than we could if we cared more about saving lives and making people’s lives better and actual ethics, and prioritized those considerations over ‘biomedical ethics.’ So mostly the useful findings lie elsewhere.

Here’s two charts of overall cases and positive test percentages in various places, for context. The rise in Denmark does reflect some Omicron cases, they’d still be under 1,000 on this graph without it, and the UK rise as well.

South Africa

The most hopeful news would be if South African cases peaked all of a sudden for no clear reason, the same way cases of Delta in India did, and started heading downward. We did see a blip for a few days, but was it more than a blip? My read on it is no, and there are signs of issues with data reporting to explain it.

The news on this front is still good, because it’s possible things are improvin...
Analysis of Bird Box (2018)
by TekhneMakre — 2021-12-13T17:30:15.045Z

[Caveat lector: hastily written, speculative. Also banal maybe.]

[SPOILER ALERT for the film Bird Box (2018) on Netflix. IMO it's not amazing but is fairly good, possibly worth watching spoiler-free. This post will spoil most of the events of the movie. Be warned that it's a horror movie so it might be horrifying.]

Very short summary: There some Things going around, and when people look at them, they get extremely sad and kill themselves. The protagonist tries to avoid looking at the Things.

This post will discuss the question: What are the Things? What's the deal with them?

Data

Here's some of the data to be explained:

--When someone looks at the Thing:

--Their eyes are visibly altered; the irises become jagged or marked.

--Most people:

--Their eyes tear up.

--They become extremely suicidal and kill themselves immediately if possible, including by creative means (e.g. Douglas, bound to his chair, tipping over and smashing his head on something hard).

--Some people:

--They don't kill themselves.

--They say the Thing is beautiful.

--They want others to look at the Thing.

--They say the Things will cleanse the world.

--These people include: people who were in an insane asylum, a guy who was a criminal and was "kind of weird but nice" and used others's pity for him to trick them into Looking, some people who go around looting, a guy who tries to violently force peotle to Look, and a businessman who was (he claims) forced to look and who then very deliberately treacherously forces people to Look.

--Birds get agitated when a Thing is around.

--Even just looking at a digital image of a Thing is sufficient to have the effect.

--When a Thing is around, a person may hear voices of people (especially dead people, but also live ones) they know well (perhaps, people they love or trust) calling to them, asking for help, and asking them to come closer and look at the Things.

--Kids can hear voices from the Thing, and also kill themselves if they Look.

So, the question: What's the deal with the Things? What's this about?

Interpretation

Vaguely, it seems like it's about inner demons, and/or about Death. Or about the unthinkable. Doug looks at a Thing through a camera and then kills himself, so the effect is mediated by the information in the Thing's appearance. It's Lovecraftian in a specific sense: it's about things that ...
Solving Interpretability Week 
by Logan Riggs — 2021-12-13T17:09:12.822Z

For original motivation, see solving corrigibility week. I'll state what I learned from corrigibility week, why I didn't post one last week, and the updated format for interpretability.

What I Noticed

The open call to co-work with my calendly link didn't work. Only met with people when I messaged them specifically. I'm changing my availability from "all day Saturday" to "two hours/day" to better meet other's schedule and not have a single point of failure (see next section). It also seems great to ask specific people to meet when I have a question on their research.

Going through previous work, it was good to write out my thoughts in the google doc, spruce it up, and make it a comment on the original post. This also connects with messaging people to meet with if I feel there's a large disconnect or I have a lot of confusion regarding their work.

The google doc was also less long-term collaborative, which may have been from a lack of notifications that someone responded to what you wrote. So I'm moving the research direction part to the comment section here.

No Post Last Week

I committed to posting 3 of these this month, but didn't last week. I had a bad Saturday (which was my designate work day for the corrigibility post) and a bad week until I talked to Shay, my therapist, and worked out a plan (Shay is great and has free sessions for those working on alignment; highly recommend). I am not ashamed or embarrassed, but I do put less stock in my public commitments and more aware of my failure modes. I personally still recommend people try (and possibly fail) ambitious projects.

I do want to write a post on corrigibility, but it's too time-consuming to both work out my own thoughts going through the literature and meet with people to understand their work & distill those conversations. Both are important and not mutually exclusive. Some possible solutions are to circle back to corrigibility next year or take two weeks per topic.

Format This Week

Here is the google doc for interpretability. In the comments are top-level comments for:

Research directions you want discussed with any questions you have
Your meeting schedule for co-working and how you'd like to be contacted
Suggestions for future weeks or changes to the format

In the doc are:

Literature Review
Tasks to do for further research

Again, in the google doc, it is socially acceptable to write low-quality b...
Understanding and controlling auto-induced distributional shift
by L Rudolf L — 2021-12-13T14:59:40.704Z

This post was written under Evan Hubinger’s direct guidance and mentorship, as a part of the Stanford Existential Risks Initiative ML Alignment Theory Scholars (MATS) program.

TL;DR: The input distribution of a supervised machine learning system is modeled as an unchanging thing, and even in reinforcement learning there are types of distributional shift we want to avoid. However, ML systems are part of the world, and often have incentives for changing the distribution of data they act on (for example, changing the set of users or the preferences of those users in an online content recommendation system such that the users become easier to satisfy). A paper by Krueger, Maharaj, and Leike explores such concerns, and especially whether ML systems end up acting on incentives for distributional shift. They invent simple scenarios and see what training setups lead to agents non-myopically act to shift the future distribution, and find that introducing meta-learning alone is enough to cause this, and that sometimes the choice of learning algorithm (e.g. policy gradient vs Q-learning) has perhaps surprising effects on whether or not such incentives are taken. The paper proposes shuffling agents between environments as a solution, but this has its flaws. The topic relates to some concerns with embedded agents and especially myopia.

Introduction

A machine learning system receives inputs and produces outputs. When we train it, we want the system to become good at producing the right output given the right input.

Consider the wider context in which the model is used. If the model is deployed, then its outputs are generally used to do something in the world (and even if not deployed, they have some effect on the world: the ML engineers look at them, their calculation heats up the computer circuits a bit, and so on).

We can visualise the chain of influence like this (where the blue arrows denote causality):

How does the model decrease its loss? It's subject to the whims of the input distribution X and the loss function, so it is incentivised to somehow learn to do well, as measured by the loss, on inputs that are likely given the distribution (though perhaps deceptively, or otherwise in an unaligned, non-robust way).

However, the above diagram isn't really correct. Instead it's more like this:

The model is part of the world too. The agent's options include not just ma...
What’s the backward-forward FLOP ratio for Neural Networks?
by Marius Hobbhahn — 2021-12-13T08:54:48.104Z

Summary:

Classic settings, i.e. deep networks with convolutional layers and large batch sizes, almost always have backward-forward FLOP ratios close to 2:1.
Depending on the following criteria we can encounter ratios between 1:1 and 3:1

Type of layer: Passes through linear layers have as many FLOP as they use to do weight updates. Convolutional layers have many more FLOP for passes than for weight updates. Therefore, in CNNs, FLOP for weight updates basically play no role.
Batch size: Weights are updated after the gradients of the batch have been aggregated. Thus, FLOP for passes increase with batch size but stay constant for weight updates.
Depth: The first layer has a backward-forward ratio of 1:1 while all others have 2:1. Therefore, the overall ratio is influenced by the fraction of FLOP in first vs. FLOP in other layers.

We assume the network is being optimized by stochastic gradient descent (w += ɑ⋅dw) and count the weight update as part of the backward pass. Other optimizers would imply different FLOP counts and could create ratios even larger than 3:1 for niche settings (see appendix B). However, the ratio of 2:1 in the classic setting (see point 1) should still hold even when you use momentum or Adam.

Compute-intensity of the weight update

Most compute-intensive layers

Backward-forward ratio

Large batch size OR compute-intensive convolutional layer

First layer

1:1

Other layers

2:1

Small batch size AND no compute-intensive convolutional layers

First layer

Other layers

3:1

Introduction:

How many more floating-point operations (FLOP) does it take to compute a backward pass than a forward pass in a neural network? We call this the backward-forward FLOP ratio.

This ratio is useful to estimate the total amount of training compute from the forward compute; something we are interested in the context of our study of

Parameter, Compute and Data Trends in Machine Learning

.

In this post, we first provide a theoretical analysis of the ratio, and we then corroborate our findings empirically.

Theory:

To understand where the differences in ratios come from, we need to look at the classical

equations of backpropagation

.

Let’s start with a simple example---a neural network with 2 hidden layers.

In this example, we have the following computations for forward and backward pass assuming linear layers with ReLU activations. The “@”-symbols denote matrix multiplications.

Operation

...
Summary of the Acausal Attack Issue for AIXI
by Diffractor — 2021-12-13T08:16:26.376Z

Attention conservation notice: To a large extent, this is redundant with Paul's previous post about this, but I figured that some people might be interested in my restatement of the argument in my own words, as I did not start out believing that it was an issue, or start out agreeing with Vanessa about the attack operating via bridge rules.

Solomonoff induction/AIXI runs all possible computations, so it's possible in theory to alter which predictions a particular Turing machine outputs, by making stuff happen in your own universe, and this would then influence any process running that Turing machine, such as an AIXI agent. Of course, doing such a thing is subject to the obvious limitation where, if, say, you make the Turing machine that's reading your universe output a 0, you'll have less ability to influence the predictions and decisions of AIXI-like agents in the worlds where they see a 1 instead, because the Turing machine that you're controlling got eliminated for mispredictions.

Taking the attacker's perspective, if you were trying to influence one particular universe (this assumption will be loosened later) containing an AIXI or sufficiently AIXI-like target agent, and you had sufficiently high predictive abilities, you could try finding two low description complexity spots in your universe, one to check the state of, and one to write data to, and committing to the strategy "if the input data from this simple spot looks like the data I'd predict to receive from the world I'm interested in, I will submit output data accordingly (mostly accurate predictions of the victim's environment, but with whatever tweaks are needed), in order to influence what the target agent does in the targeted universe."

Basically, if you learn that there's an input channel to your universe, it's worthwhile to try to hack the output channel. It doesn't even take much effort to do, you just need to keep an eye on the conjectured input channel, commit to responding accordingly if it looks like it's transmitting data, and do other things in the meantime, and the Turing machine "run your universe, signal in via this channel, read output via this channel" is now under your control.

So... from the perspective of the targeted agent/universe, how well would your hacking attempt work? Well, it critically depends on whether the complexity of specifying "your universe + the I/O channels" is m...
Hard-Coding Neural Computation
by MadHatter — 2021-12-13T04:35:51.705Z

Previously: Teaser: Hard-coding Transformer Models

Introduction

Transformer models are incredibly powerful for natural language tasks (and they are starting to find uses in many other fields of machine learning). Unfortunately, it is nigh-impossible to interpret what goes on inside them. OR IS IT???

I have found that I can, with a fair amount of effort, hard-code the weights of a transformer model in order to perform some very crude versions of linguistic tasks. So far I have achieved English-to-French translation (on a toy corpus of about 150 sentences), text classification (is a sentence grammatical or not? on a toy corpus of a couple hundred sentences), and sentiment analysis (again on a limited corpus). These results are obviously not impressive compared to the state of the machine learning field, but I am pretty sure that they can all be drastically scaled up with the investment of some time and energy. Unfortunately, I have a fairly demanding day job, and haven't found the time and energy yet.

All of this is done by inspection (no gradient descent!). The process is a lot like programming, although it is more difficult than programming, at least right now for me. I am fairly certain that better tools and better notation can be developed to make the process easier. It is also almost certainly possible to combine hard-coding with gradient descent approaches to be able to scale these methods up in a slightly less labor-intensive way.

I think that these ideas could prove useful in alignment research - if we understand how a language model works in excruciating detail, it seems drastically more likely that we will be able to reason about and predict various misunderstandings rooted in the ambiguity of language. Given that language is (arguably) a fully general means of interacting with an artificial intelligence, it seems plausible to me that this work is on the critical path to alignment.

Doneness Status

This post is a work-in-progress. I will be editing it as I go, mostly appending more content to the end, but I will also try to fix any errors or unclear parts as I notice them or commenters point them out.

So let's hard-code some neural computation! I have a very, very messy github repository where I've done my initial experiments, if you prefer to just jump into semi-working code. Otherwise, I will do my very best to explain the ideas from scratch in this post. I...