trentmkelly/LessWrong-43k huggingface dataset:
Inner Alignment via Superpowers
by JamesH — 2022-08-30T20:01:52.129Z

Produced As Part Of The SERI ML Alignment Theory Scholars Program 2022 Under John Wentworth

The Problem

When we train RL agents, they have many opportunities to see what makes actions useful (they have to locate obstacles, navigate around walls, navigate through narrow openings etc.) but they can only learn what they should actually care about from how the goal appears in training. When deployed, their capabilities often generalize just fine, but their goals don't generalize as intended. This is called goal misgeneralization.

Usually we conceptualize robustness as 1-dimensional, but to talk about goal misgeneralization, we need to use vlad_m's 2-dimensional model:

1D Robustness above; 2D Robustness below, with the Line of Doom in grey. Source.

“There’s an easy solution to this,” you might say. “Just present a whole boatload of environments where the goals vary along every axis, then they have to learn the right goal!”

“Our sweet summer child,” we respond, “if only it were so simple.” Remember, we need to scale this beyond simple gridworlds and Atari environments, where we can just change coin position and gem color, we’re going all the way to AGI (whether we like it or not). Can we really manually generate training data that teaches the AGI what human values are? We need a method that’ll be robust to huge distribution shifts, things we aren't able to even think of. We need a method that’ll allow this AGI to find what humans value. We need superpowers!

Proposed Solution

Our solution is ‘giving the AI superpowers.’

Oh, that's not clear enough?

Alright then: during training, we occasionally let the RL agent access an expanded action-space. This lets it act without the restrictions of its current abilities. We also encourage it to explore states where it’s uncertain about whether it’ll get reward or not. The aim is that these ‘superpowers’ will let the AI itself narrow down what goals it ought to learn, so that we won’t need to be as certain we’ve covered everything in the explicit training data.

Through this we hope to combat the two principal drivers of goal misgeneralization:

Instrumental Goals

When you were a child, you were happy every time you ate a lollipop. But you realized you needed money to buy lollipops, so eventually you started becoming happy whenever you made money. And look at you now, hundreds of dollars in the bank and not a lollipop in...
ML Model Attribution Challenge [Linkpost]
by aog — 2022-08-30T19:34:39.007Z
Link: mlmac.io

From the ML Model Attribute Challenge website:

Large language models (LLMs) may enable the generation of misinformation at scale, especially when fine-tuned on malicious text. However, it is unclear how widely adversaries use LLMs to generate misinformation today, partly because no general technical method has been widely deployed to attribute a fine-tuned LLM to a base LLM.

The ML Model Attribution Challenge (MLMAC) challenges contestants to develop creative technical solutions for LLM attribution. Contestants will attribute synthetic text written by fine-tuned language models back to the base LLM, establishing new methods that may provide strong evidence of model provenance. Model attribution would allow regulatory bodies to trace intellectual property theft or influence campaigns back to the base model. This competition works toward AI assurance by developing forensic capabilities and establishing the difficulty of model attribution in natural language processing and could contribute to the maturation of AI forensics as a subfield of AI security.

This competition incentivizes open research in this area. Each winning contestant must publish their method, and after the competition concludes, we will release query logs for each fine-tuned model that can enable post-hoc analysis of blind attribution methods. We have planned subsequent activity in this area to institutionalize practical model attribution research.

The contest concludes next Friday, September 9th. $10,000 in prizes will be awarded amongst the winning submissions. MLMAC is a joint effort of organizers including Schmidt Futures, Hugging Face, Microsoft, and MITRE.

For more analysis of risks from automated misinformation, see "Risks from Automated Persuasion" (Barnes, 2021), "Persuasion Tools" (Kokotajlo, 2020), and my summary. For a broader discussion of language model misuse, see "Ethical and social risks of harm from Language Models" (Weidinger et al., 2021).
How likely is deceptive alignment?
by evhub — 2022-08-30T19:34:25.997Z

The following is an edited transcript of a talk I gave. I have given this talk at multiple places, including first at Anthropic and then for ELK winners and at Redwood Research, though the version that this document is based on is the version I gave to SERI MATS fellows. Thanks to Jonathan Ng, Ryan Kidd, and others for help transcribing that talk. Substantial edits were done on top of the transcription by me. Though all slides are embedded below, the full slide deck is also available here.

Today I’m going to be talking about deceptive alignment. Deceptive alignment is something I'm very concerned about and is where I think most of the existential risk from AI comes from. And I'm going to try to make the case for why I think that this is the default outcome of machine learning.

First of all, what am I talking about? I want to disambiguate between two closely related, but distinct concepts. The first concept is dishonesty. This is something that many people are concerned about in models, you could have a model and that model lies to you, it knows one thing, but actually, the thing it tells you is different from that. So this happens all the time with current language models—we can, for example, ask them to write the correct implementation of some function. But if they've seen humans make some particular bug over and over again, then even if in some sense it knows how to write the right function, it's going to reproduce that bug. And so this is an example of a situation where the model knows how to solve something and nevertheless lies to you. This is not what I'm talking about. This is a distinct failure mode. The thing that I want to talk about is deceptive alignment which is, in some sense, a subset of dishonesty, but it's a very particular situation.

So deceptive alignment is a situation where the reason that your model looks aligned on the training data is because it is actively trying to look aligned for instrumental reasons, which is very distinct. This is a situation where the thing that is causing your model to have good performance is because it is trying to game the training data, it actively has a reason that it wants to stick around in training. And so it’s trying to get good performance in training for the purpose of sticking around.

Ajeya Cotra has a really good analogy here that I think is helpful for understanding the difference between these two c...
Built-In Bundling For Faster Loading
by jefftk — 2022-08-30T19:20:03.145Z

I used to work on

mod_pagespeed, which would
rewrite web pages so they would load faster. A key component was
bundling: combining multiple small resources into a single larger one
to reduce overhead. For example, it had tools to combine multiple

CSS,

JS, and

images.

There were serious downsides to our approach, however, because there
was no built-in support for bundling:

Combining many small images required spriting, with CSS to
identify which area of the image you want for each usage. This is a
very different syntax, and is awkward to do by hand. Our ability to
do it automatically was limited, because it needed to understand the
site's css and the publisher needed to have intentionally set their
css up to minimize the changes required.

Combining many small CSS or JS files worked better but still
required hacks to prevent errors in one file from breaking the others.

We had to change the URLs in the page, which was mostly a
problem for images. JS that modified image URLs, like changing
foo.png to foo_hover.png on hover, could be
broken, and images wouldn't have the right names if you downloaded
them. This also sometimes broke JS that expected it could derive the
location of related files from its URL.

We needed separate files for the CSS, JS, and images.

With the release of Chrome/Edge/Opera/Vivaldi 104, however, there's a new
API that makes bundling subresources very natural. Here's a demo: wbn-demo.

The core of the demo is:

<script type="webbundle">
{
"source": "subresources.wbn",
"resources": ["script.js", "sheet.css"]
}
</script>

<link rel=stylesheet href="sheet.css">
<script async src="script.js"></script>

This tells the browser to fetch
subresources.wbn, and
then if it needs to load script.js
or sheet.css it will
get them from the bundle instead of making separate requests. Of
course this is overkill for my little demo, but if I had a page with a
large number of small resources the savings would add up.

You can read more about this API by looking at the spec
or the Chrome explainer.
I'm hoping that Firefox and Safari decide to add support for this
functionality!

Above I listed Chrome, Edge, Opera, and Vivaldi, four out of the five main
Chromium-based browsers, as supporting this feature. What about
Brave? Brave objected
to Web Bundles, arguing that t...
A bayesian updating on expert opinions
by amarai — 2022-08-30T11:56:19.925Z

I am trying to model a bayesian agent who updates their credence in some proposition p based on a set of experts on p, who just respond in a binary way to p (i.e. "belief"/"disbelief"). Do you know of attempts (inside or outside academic literature) to model something similar?

My search wasn't very successful. Thanks a lot for any leads (even if they are only vaguely similar projects)!
Any Utilitarianism Makes Sense As Policy 
by George3d6 — 2022-08-30T09:55:00.346Z
Link: https://www.epistem.ink/p/any-utilitarianism-makes-sense-as

There’s been a lot of discussions around utilitarianism as of late in the blogoblob I hang out in. So fine, I’ll bite:

Most issues with any system of ethics arise when you try to make it absolute. Be it the repugnant conclusion, theodicy, solipsism, preference-towards non-existence; Or any other sort of driving axioms to infinity causing you to conclude seagulls should pluck out your eyeballs.

The problem is that all of these issues arise under at least 3 of the following assumptions:

The system of ethics is to be implemented by an omnipotent being
The system of ethics is to be agreed upon by all people and acted in accordance with at all times
The actors implementing the system of ethics have perfect knowledge of the future
The system of ethics is not to be revisited, modified, or replaced in the future

i - Negative Utilitarianism

Take as an example negative utilitarianism, which can be taken to say something like:

Suffering is the only thing we ought to concern ourselves with, our goal is to minimize suffering for all conscious and potentially conscious beings.

If one applies assumptions (4), (3) and (1) or (2), this leads to the main counter-argument to negative utilitarianism, which is:

Then, our ultimate goal ought to be eliminating all conscious life from the universe in such a way as to cause no suffering in the process.

This sounds horrible, but that’s fine because we are NOT talking about how to implement morality for an omnipotent, omniscient, and unchanging god. We are talking about how to implement morality for… a governing institution of some sort.

So how would a negative utilitarian government’s constitution sound like, starting from the same definition of the ideology?

We, the legislators, hereby agree that our main goal will be reducing suffering for every conscious being we know of. Under all currently commonly agreed upon definitions, such as minimizing the infliction of unwanted pain, trying to fulfill all fundamental needs (hunger, shelter, friendship) or needs which will be identified as fundamental.

In order to ensure this charter gets enacted, we shall prioritize our own constituents, as they are the ones giving us the power to act, but shall direct a portion of our effort towards those beings we think most underprivileged even if they are not under our governance.

We don’t care about preference fulfillment, so all forms of business and re

...
A gentle primer on caring, including in strange senses, with applications
by Kaarel — 2022-08-30T08:05:12.333Z

Introduction

Firstly, I will define a straightforward model of caring about other people. I think it is a good model for understanding (and predicting) people's actions. I also think it's pretty meta-good to have one's own ethics effectively be of approximately this form. However, I will not argue for either of these claims in this post.

I will make a few observations about the relationship between Pareto improvements and social welfare in this model. I will then clarify the model in a crucial way, leading to a low-complexity way to think about caring across time. I will use this framework to make some observations on value drift, to give a justification of (some) sin taxes, and to state an insight into parenting.

I see this post as mostly not having ground-breaking insights, but instead as thinking through basic things carefully. I think it's surprising how much one can get out of such a simple model. I will mention a number of open directions for further thought along the way. It was my intention to keep the main text relatively modular and approachable[1], with further discussion in the footnotes.

The model

Let I be the set[2] of all moral patients.[3] For a moral patient i∈I, let xi denote the personal utility of i. One might also call this the welfare of i, or how much fun it's having, or the total amount of joy its experiences spark, or (the integral of) valence or pleasantness, or personal pleasure.[4][5]

The idea we would like to implement here is that i is not necessarily (just) trying to maximize xi – it's possible that i cares about (certain) other people, and not just instrumentally because of e.g. some contract, but terminally.[6] In particular, we will assume that the terminal utility of i, denoted ui, takes the following linear form[7]:

ui=∑j∈Iwijxj.

ui is what agent i strives to maximize.[8] For simplicity, we will assume WLOG that for every i, we have wii=1.[9] We will also assume all wij≥0.[10] If one likes, one can think of the weights wij as living in a matrix with rows and columns indexed by I. One can treat this as the adjacency matrix of a weighted directed graph.[11] The weights wij capture how much i cares about each element of I.[12] For example:

i being

...
Modified Guess Culture
by konstell — 2022-08-30T02:30:05.990Z
Link: https://konstell.com/modified_guess_culture/modified_guess_culture/

[Epistemic status: rough heuristics, probably needs refinement, comments are always welcome.]

It has recently occurred to me that, to build close-knit communities, a modified Guess Culture would work better than pure Guess Culture, which is better than Ask Culture.

Guess Culture, at its best, places a heavy emphasis on being considerate. The essence of being considerate is putting effort into building up an accurate mental model of another person, and using that model as a guide to choose actions that would improve the other person's life.

Selfless Dating by Jacob Falkovich is an example of taking thoughtfulness seriously. It doesn't have to apply to dating -- there are many ways to be thoughtful to family, friends, roommates...

Ask Culture, at its best, removes the awkwardness of error correction. When we encounter a situation in territory unfamiliar to our mental map, asking the other person about their preference does not imply that we don't care enough to make a prediction. Similarly, when our mental model of another person makes a prediction error, Ask Culture makes it easier for them to point it out.

My understanding of friendship (which is a foundation of all good interpersonal relationships) is an iterative process of building up more accurate mental models of each other, and using that model to guide decision-making when relevant. For example, if I have told a friend that I'm allergic to X when we go out for food once or twice, I would expect this friend to update their mental of me as "having dietary restrictions." If the next time we go out for food they suggest something that clearly has X in it, I would read that as them having failed to update their mental model of me -- either because they are not good at keeping track of mental models of people, or that they simply don't care about me.

Why is Guess Culture better than Ask Culture in a community context? Compared to Ask Culture, it enables an additional dimension on which to signal -- how accurate my mental model is of you is a reliable signal of how much I care about you. This is an essential signal in a high-trust context.

The downside of pure Guess Culture is that a wrong guess incurs high social cost, since the accuracy of the guesses is a signal of thoughtfulness. The modification here, is to lower the cost for the initial several wrong guesses. Rather than assessing the accuracy of the guesses,...
How to plan for a radically uncertain future?
by Kerry — 2022-08-30T02:14:47.244Z

While playing around with Stable Diffusion last week I had an epiphany. I realized that while I believe with high probability that the world will be radically different in 15 years, all my investments, living arrangements, and other long term plans implied a belief that the world would be roughly similar to now. This shook me to my core.

More concretely, I have investments in suburban American property, a 401k plan that I maximize and invest in a broadly diversified low cost fund, many children whom I educate in roughly conventional subjects (from college age down to toddler age), and a job that I intend to keep for the long term. But I believe there is a less than 1% probability that my 401k will mean anything by the time I can withdraw it. I don't believe that my property (or any suburban American property) will be valuable in 15 years under most scenarios I consider likely. I don't think my children's education will serve them particularly well during their adulthood.

What are my beliefs about the future? I believe that AI and biology at least (two subjects I know well and follow closely) are accelerating quickly, with logarithmic increases of capability and similar log decreases of cost. I believe that it is early days for both fields, and these log curves will be extended further for years to come.

Given those beliefs, the thing I expect most is radical change, in a direction that I can't predict well. I'm not biased towards a doom scenario (AI takeover, engineered unstoppable plague), nor towards a utopian scenario (beneficent AGI, biological immortality). But I am biased strongly away from the continuation of the status quo, away from any regime where I can sell my house, live off my 401k, watch my children take jobs that exist today and have their own families.

I suspect others here hold similar beliefs about the future. How are you preparing now? How do you invest, how do you rear your young, what actions do you take that let you sleep well, knowing that whatever tomorrow brings your rationality has armed you in advance?
EA & LW Forums Weekly Summary (21 Aug - 27 Aug 22')
by Zoe Williams — 2022-08-30T01:42:39.309Z

This is also posted on the EA forum: see here.
Supported by Rethink Priorities

Sunday August 21st - Saturday August 27th

The amount of content on the EA and LW forums has been accelerating. This is awesome, but makes it tricky to keep up with! The below hopes to help by summarizing popular (>40 karma) posts each week. It also includes announcements and ideas from Twitter that this audience might find interesting. This will be a regular series published weekly - let me know in the comments if you have any feedback on what could make it more useful!
If you'd like to receive these summaries via email, you can subscribe here.

Methodology

This series originated from a task I did as Peter Wildeford’s executive research assistant at Rethink Priorities, to summarize his weekly readings. If your post is in the ‘Didn’t Summarize’ list, please don’t take that as a judgment on its quality - it’s likely just a topic less relevant to his work. I’ve also left out technical AI posts because I don’t have the background knowledge to do them justice.

My methodology has been to use

this

and

this

link to find the posts with >40 karma in a week for the EA forum and LW forum respectively, read / skim each, and summarize those that seem relevant to Peter. Those that meet the karma threshold as of Sunday each week are considered (sometimes I might summarize a very popular later-in-the-week post in the following week’s summary, if it doesn’t meet the bar until then). For twitter, I skim through the following lists:

AI

,

EA

,

Forecasting

,

National Security

(mainly nuclear),

Science

(mainly biosec).

I’m going through a large volume of posts so it’s totally possible I’ll get stuff wrong. If I’ve misrepresented your post, or you’d like a summary edited, please let me know (via comment or DM).

EA Forum

Philosophy and Methodologies

Critque’s of MacAskill’s ‘Is it Good to Make Happy People?’

Discusses population asymmetry, the viewpoint that a new life of suffering is bad, but a new life of happiness is neutral or only weakly positive. Post is mainly focused on what these viewpoints are and that they have many proponents vs. specific arguments for them. Mentions that they weren’t well covered in Will’s book and could affect the conclusions there.

Presents evidence that people’s intuitions tend towards needing significantly more happy people than equivalent level of suffering people for a...