trentmkelly/LessWrong-43k huggingface dataset:
Techniques for enhancing human feedback
by abergal — 2021-10-29T07:27:46.700Z

By Ajeya Cotra

Training powerful models to maximize simple metrics (such as quarterly profits) could be risky. Sufficiently intelligent models could discover strategies for maximizing these metrics in perverse and unintended ways. For example, the easiest way to maximize profits may turn out to involve stealing money, manipulating whoever keeps records into reporting unattainably high profits, capturing regulators of the industry to be allowed to ship shoddy products or avoid taxes, etc. More generally, the most effective path to maximizing a simple metric may involve acquiring enough power to tamper directly with whatever instruments or sensors are used to evaluate the metric, effectively deceiving and disempowering humans to do so.

It seems significantly safer if powerful models could be trained using something like

human feedback

, where human evaluators inspect a model’s action and rate how good that action is likely to be all-things-considered, and the model is trained to take actions that humans would rate highly. Human feedback could potentially disincentivize some obviously-perverse strategies like “blatantly stealing money to maximize profits,” and incentivize practices which could help maintain or improve human control like “explaining why a proposed action will be beneficial.”

However, human feedback isn’t fully adequate for supervising powerful models, especially if they take actions that are too complex for humans to understand. For example, even if blatant forms of theft are disincentivized, a sufficiently intelligent model trained with human feedback may still e.g. participate in various abstruse and complicated financial contracts which effectively constitute theft. On the other hand, if human evaluators simply penalize any action they don’t understand, the models they train would be significantly less valuable than they could have been and may be outcompeted by models trained with outcome metrics like profit.

We are interested in ways to enhance or improve upon human feedback, so that humans can provide adequate feedback even in domains where models are more capable or knowledgable than humans, without resorting to training on outcome metrics. For example, projects in this space could explore questions like:

How could human evaluators that are unfamiliar with a certain subject (like computer science or economics) effectively give feedback th

...
Measuring and forecasting risks
by abergal — 2021-10-29T07:27:32.836Z

By Jacob Steinhardt, with input from Beth Barnes

As machine learning pervades more and more sectors of society, it brings with it many benefits, but also poses risks, especially as systems become more powerful and difficult to understand and control. It is important to understand these risks as well as our progress towards addressing them. We believe that systematically measuring these risks is a promising route to improving understanding and spurring progress. In addition, measuring safety-related qualities of ML systems (e.g. alignment) allows us to hold models to certain safety standards and to compare the safety performance of different systems. Both of these help incentivise AI developers to invest more heavily in safety.

This RFP solicits ideas for measuring several safety-related properties:

concrete risks, such as “objective hacking”, “competent misgeneralization”, or “intent misalignment”, that scale with ML capabilities; and
unintended or unexpected emergent capabilities that may pose new risks.

The three concrete risks constitute problems that could get worse, rather than better, as capabilities improve, and thus lead to a negative long-term trajectory from ML. Meanwhile, measuring emergent capabilities guards against new unknowns, where we care most about capabilities that could pose new risks or rapidly increase the scope or impact of AI systems. Below we describe several categories of work that relate to measuring the above risks.

A measurement is any reproducible quantity or set of quantities (such as an ROC or learning curve) associated with a phenomenon of interest. While one type of measurement is the accuracy on a benchmark dataset, other types of measurement include probing accuracy, disagreement rate, or adversarial robustness, to name a few examples. Others include plotting accuracy vs. model width to understand the phenomenon of double descent (

Belkin et al., 2018

), plotting phase transitions in learning curves to understand grokking (

Power et al., 2021

), or collecting a few hundred examples of common misconceptions to measure imitative deception (

Lin et al., 2021

). Many subjective judgments can be turned into measurements by employing human raters.

Non-examples of measurements include a single anecdote of a phenomenon, a thought experiment, a method to improve performance, or a theorem. The primary motivation for focusing on measurements is ...
Request for proposals for projects in AI alignment that work with deep learning systems
by abergal — 2021-10-29T07:26:58.754Z

As part of our work on reducing

potential risks from advanced artificial intelligence

, Open Philanthropy is seeking proposals for projects working with deep learning systems that could help us understand and make progress on

AI alignment

: the problem of creating AI systems more capable than their designers that robustly try to do what their designers intended. We are interested in proposals that fit within certain research directions, described below and given as posts in the rest of this sequence, that we think could contribute to reducing the risks we are most concerned about.

Anyone is eligible to apply, including those working in academia, industry, or independently. Applicants are invited to submit proposals for up to $1M in total funding covering up to 2 years. We may invite grantees who do outstanding work to apply for larger and longer grants in the future.

Proposals are due January 10, 2022.

Submit a proposal

here

.

If you have any questions, please contact

ai-alignment-rfp@openphilanthropy.org

.

Our view of alignment risks from advanced artificial intelligence

This section was written by Nick Beckstead and Asya Bergal, and may not be representative of the views of Open Philanthropy as a whole.

We think the research directions below would be pursued more fruitfully by researchers who understand our background views about alignment risks from advanced AI systems, and who understand why we think these research directions could help mitigate these risks.

In brief:

We believe it is plausible that later this century, advanced AI systems will do the vast majority of productive labor more cheaply than human workers can.
We are worried about scenarios where AI systems more capable than humans acquire undesirable objectives that make them pursue and maintain power in unintended ways, causing humans to lose most or all influence over the future.
We think it may be technically challenging to create powerful systems that we are highly certain have desirable objectives. If it is significantly cheaper, faster, or otherwise easier to create powerful systems that may have undesirable objectives, there may be economic and military incentives to deploy those systems instead.
We are interested in research directions that make it easier to create powerful systems that we are highly certain have desirable objectives.

In this request for proposals, we are focused on scenarios whe...
My current thinking on money and low carb diets
by Adam Zerner — 2021-10-29T06:50:38.543Z

Epistemic status: Writing to think. Ie. I'm trying to figure it out. Feedback/conversation would be appreciated.

Background

Recently, I have been learning about low carb diets and how harmful carbs are. Mostly through Peter Attia, Tim Ferris, and What I've Learned.

So there's that, and then there's the fact that low carb seems to be the thing that works for me. I am 28 years old and am 5'6. At my peak, I was 165 lbs. That would put my BMI at 26.6, which would be considered overweight, and I suspect that a healthy weight is even lower than what those BMI calculators would indicate. I also have been gaining steadily since college. I think I was around 120 lbs in the beginning of college and maybe 130 at the end. Plus, my cholesterol and stuff is high.

But my diet has always been pretty healthy, I think. I only eat out maybe twice a month. I rarely eat processed meat, processed foods in general, deserts, or things with a lot of sugar, including fruit. I do a lot of cooking my own meals using raw ingredients. Chicken, beef, veggies, rice, pasta, beans, etc. That's always been what I assumed was healthy.

On the other hand, I eat a lot. I'm not really sure how to describe this, but one indicative thing is that I often eat more than twice as much food as my girlfriend. And when I am eating meals with guy friends, I also often eat maybe 1.5x as much food as them. This is very ballparky, it's hard to describe since I don't have actual data on calories or grams. Another data point is that friends and family all notice and point out that I eat crazy amounts of food, and joke with me about it.

After learning about low carb stuff, maybe it makes sense. My plates are often 1/2 carb, 1/4 meat, 1/4 veggies. Maybe the carbs are making me hungry. I'm also a pretty good cook, so my meals are often pretty appetizing, making me want to eat more.

On top of this, I tried Tim Ferris' slow carb diet once, and it actually worked! I lost a little over five pounds in, maybe two weeks? I forget. Unfortunately I ended up getting off the rails and giving up on it, but I really felt like it was working. So I get the sense that, at least for me personally, low/no/slow carb is what I need to lose weight and improve my blood numbers.

Expensive?

But here's the thing: that seems to be rather expensive. It involves eating lots of meat, and meat is expensive.

Does it have to be? Is there a way to do th...
How to generate idea/solutions to solve a problem?
by warrenjordan — 2021-10-29T00:53:15.941Z

Are any rationality techniques one could leverage to generate ideas or solutions towards solving a specific problem?

This is probably the biggest obstacle for me to have effective goal factoring. I sometimes cannot come up with good ideas or alternatives.

I found this post, however it's 9 years old and wanted to see if there are any more recent techniques.
Forecasting progress in language models
by Matthew Barnett — 2021-10-28T20:40:59.897Z
Link: https://www.metaculus.com/notebooks/8329/human-level-language-models/

Note: this post was cross-posted to Metaculus over a week ago as part of their new Metaculus journal.

Here, I describe a way of measuring the performance of language models, and extrapolate this measure using publicly available data on benchmarks. The result is a (surprisingly) short timeline to "human-level"—within one decade from now. Since posting to Metaculus, I have realized the need to clarify a few things first,

I am not saying that this model represents my inside view. I merely used linear regression to extrapolate performance metrics, and I think there is credible reason to doubt these results, not least of which because I may have made a mistake. Check out the comment section on Metaculus for more discussion.
That said, it may still be interesting to consider what might be true about the world if we just take these results literally. Predicting that a trend will soon inexplicably slow down is easy. What's harder is proposing an alternative model.
There are probably errors in the Jupyter notebook, my graphs, and my equations. Just let me know in the comments if you see any.

Language models have received a lot of attention recently, especially OpenAI’s GPT-3 and Codex. While the possibility of a human-level language model remains out of reach, when might one arrive? To help answer this question, I turn to some basic concepts in information theory, as pioneered by Claude Shannon.

Shannon was interested in understanding how much information is conveyed by English text. His key insight was that when a text is more predictable, less information is conveyed per symbol compared to unpredictable texts. He made this statement more precise by introducing the concept of entropy. Roughly speaking, entropy measures the predictability of a sequence of text, in the limit of perfect prediction abilities.

Since Shannon’s work, a popular hobby of computational linguists has been to invent new ways of measuring the entropy of the English language. By comparing these estimates with the actual performance of language models at the task of predicting English text, it is possible to chart the progress we have made toward the goal of human-level language modeling. Furthermore, there are strong reasons to believe that entropy is a more useful metric for tracking general language modeling performance when compared to performance metrics on extrinsic tasks, such as those...
[AN #168]: Four technical topics for which Open Phil is soliciting grant proposals
by Rohin Shah — 2021-10-28T17:20:03.387Z

Alignment Newsletter is a weekly publication with recent content relevant to AI alignment around the world. Find all Alignment Newsletter resources here. In particular, you can look through this spreadsheet of all summaries that have ever been in the newsletter.

Please note that while I work at DeepMind, this newsletter represents my personal views and not those of my employer.

HIGHLIGHTS

Request for proposals for projects in AI alignment that work with deep learning systems (Nick Beckstead and Asya Bergal) (summarized by Rohin): Open Philanthropy is seeking proposals for AI safety work in four major areas related to deep learning, each of which I summarize below. Proposals are due January 10, and can seek up to $1M covering up to 2 years. Grantees may later be invited to apply for larger and longer grants.

Rohin's opinion: Overall, I like these four directions and am excited to see what comes out of them! I'll comment on specific directions below.

RFP: Measuring and forecasting risks (Jacob Steinhardt) (summarized by Rohin): Measurement and forecasting is useful for two reasons. First, it gives us empirical data that can improve our understanding and spur progress. Second, it can allow us to quantitatively compare the safety performance of different systems, which could enable the creation of safety standards. So what makes for a good measurement?

1. Relevance to AI alignment: The measurement exhibits a failure mode that becomes worse as models become larger, or tracks a potential capability that may emerge with further scale (which in turn could enable deception, hacking, resource acquisition, etc).

2. Forward-looking: The measurement helps us understand future issues, not just those that exist today. Isolated examples of a phenomenon are good if we have nothing else, but we’d much prefer to have a systematic understanding of when a phenomenon occurs and how it tends to quantitatively increase or decrease with various factors. See for example scaling laws (AN #87).

3. Rich data source: Not all trends in MNIST generalize to CIFAR-10, and not all trends in CIFAR-10 generalize to ImageNet. Measurements on data sources with rich factors of variation are more likely to give general insights.

4. Soundness and quality: This is a general category for things like “do we know that the signal isn’t overwhelmed by the noise” and “are there any reasons that the measurem...
Better and Worse Ways of Stating SIA
by dadadarren — 2021-10-28T16:04:22.333Z

This post is motivated by Joe Carlsmith's post in which he argued some ways of understanding SIA undersells its appeal. He purposed a better way of stating SIA. I disagree with some of the assessments.

Before I start, it should be pointed out that all different ways of stating SIA are computationally equivalent. So it is unlikely that one interpretation is "correct" while the other is "wrong". But it may help us in evaluating SIA's validity.

The Two Ways:

First is the ordinary SIA statement. If you check on Wikipedia or the Lesswrong Wiki you can find SIA as:

All other things equal, an observer should reason as if they are randomly selected from the set of all possible observers.

This is actually slightly different from Nick Bostrom's original statement of SIA in his book "Anthropic Principle". But it is logically the same. And since then it has been most widely used including Bostrom himself in some interviews (if my memory is correct). Furthermore, the criticism presented by Carlsmith applies to this formulation. So I will use this statement.

Carlsmith presented SIA as:

You’re more likely to exist in worlds with more people in your epistemic situation. Or update the prior of objective worlds in proportion to n (the number of people in your epistemic situation).

I intend to compare the two and argue why the ordinary statement in my opinion is still better.

The Reference Class Problem

Carlsmith argued the ordinary statement uses the notion of reference-class. Yet its definition is arbitrary. E.g. who are the "observers"? How do you define them? Even though the choice of reference-class will not affect the answer in many anthropic problems, due to its effect cancels out, this "inflate-and-claw-back" method is undesirable. The appeal of SIA is not "you can use whatever reference class you like" but you don't have to think in terms of any made-up reference class at all. You only need to consider yourself a member of "people in your epistemic situation".

But "people in your epistemic situation" is not a clear-cut definition either. An epistemic situation is a subjective state, and the only subjective experience you have is that of your own. So what qualifies “people in your epistemic situation” has to be a judgment call, or in another word, made up. Do brains in jars that are being fed with similar neural signals count? as Adam Elga...
Recommending Understand, a Game about Discerning the Rules
by MondSemmel — 2021-10-28T14:53:16.901Z

I play a lot of video games. Enough to occasionally find a rare gem like the puzzle game Understand. Its inspirations include the games The Witness, Baba is You, and Zendo, as well as the 2-4-6 task in psychology. Understand costs 4$ (3.3€) and is available for Windows and macOS on the Steam game client. (EDIT: For Linux compatibility, see this discussion.)

But before I get to the game itself and why I recommend it on Less Wrong, a brief aside:

Why Recommend a Game on LW?

I've easily played >1000 video games, including most critically acclaimed puzzle games irrespective of popularity. That's a lot of time spent on leisure. When gamers compliment a game, they call it "addicting", which is a pretty insane state of affairs. In any case, I would not recommend most puzzle games or even programming-style optimization games to a community interested in understanding and improving the world. Skills learned in any particular game rarely generalize, and the opportunity cost of spending time on potentially addicting games is high.

With that said, this is the one game I recommend here. I think it does something genuinely unique which could be used to practice a few core rationalist techniques, and I consider its design and presentation far from addicting.

Anyway, with that out of the way, let's get to the game.

What is Understand?

The game's aesthetic befits one-man studio Artless Games. There's no music, either.

The premise in Understand is delightfully simple: each level consists of a couple of screens with a common but hidden ruleset. Your task is to draw a path on each screen (controlled with the mouse) which satisfies these rules (represented by one or more dots below the screen). So early on the challenge is almost entirely in discerning the hidden rules.

For this you get various bits of feedback. The game tells you how many rules there are, and when you draw a path it tells you which specific rules it fulfills. The first screen in each level shows you a valid path which fulfills all rules, and the next few screens are cleverly designed to teach by example. So you'll use deduction and trial & error to draw paths and uncover the rules.

The game consists of ~130 levels à 5-10 screens, which makes for a lot of game, but you can already get something out of it within the first hour of play, and there's no need to play through it in its entirety.

(My more in-depth rev...
Covid 10/28: An Unexpected Victory
by Zvi — 2021-10-28T14:50:01.072Z

The story of this week is what went down at the port of Los Angeles. The port had been brought to a standstill, because there was nowhere to put empty containers and trucks were sitting around with empty containers on them instead of moving freight, and the problem kept getting worse and worse. By taking a carefully planned boat ride, creating a finely crafted Tweetstorm designed to persuade, and getting the right people to assist, Ryan Peterson was able within one day to get the container stacking rule at the port changed from a limit of two to a limit of five, and get cargo moving again. It’s hard to tell exactly how much of a difference this will make, but from everything I’ve seen it will be a big difference. This is not only a huge story for its object level implications, it’s a story of how someone can make a difference, and how we were able to actually do something.

Yet mainstream media is ignoring the story. Shout it from the rooftops! I’ve split that story off into its own post. If you have to choose between reading that one and reading this one, read that one, it’s far more important.

Meanwhile in the land of Covid-19, things continue as you might expect.

Executive Summary

Ryan Peterson managed to get them to change the container stacking rules at the port of Los Angeles, which had otherwise reached a standstill, potentially saving Christmas and/or the economy. This got its own post.
Covid-19 conditions continue to slowly improve.
Child vaccinations for age 5-11 on track for November.

Let’s run the numbers.

The Numbers

Predictions

Prediction from last week: 410k cases (-13%) and 9,600 deaths (-10%).

Results: 431k cases (-9%) and 9,506 deaths (-11%).

Prediction for next week: 400k cases (-9%) and 8,600 deaths (-10%).

Slightly disappointing case counts, very slightly good news on deaths. Trends continue and I see no reason for that to change in the next few weeks. Children getting vaccinated soon should help somewhat as well, although that will take a while.

Deaths

Nothing here should be surprising, given what we saw in cases several weeks ago, and it lined up with the predicted outcome.

Cases

We see clear improvement continuing in all regions, with an especially good number coming out of the Northeast. It’s a disappointing number since we had reason to expect better, but there’s nothing here that causes me to worry.

Polimath is back with another state by...