trentmkelly/LessWrong-43k huggingface dataset:
Brute-forcing the universe: a non-standard shot at diamond alignment
by Martín Soto — 2022-11-22T22:36:36.599Z

This is an expanded version of my answer to application problem 2 for Nate Soares and Vivek Hebbar's SERI MATS stream. This Alignment idea is somehow non-standard. See Section C for a discussion of that, along with some general ideas on Alignment. The text is long because I’ve tried to include all details relevant to the discussion.

Alignment idea: For any possible AGI design, run a physics simulation calculating how much diamond it ends up producing in the universe. Build the one maximizing it.

1. How do we get an (approximately) accurate Physics simulation, and the right world model?

Build a simulation environment with the best current guess of the physical laws governing the evolution of the macroscopic[1] universe. Now, since we don't have a Theory of Everything (Quantum Mechanics and General Relativity are incompatible), we can't for instance model everything in terms of elementary particles. But we can model everything (although with a lower granularity) using the macroscopic (and approximate) laws of physics which we employ daily for dealing with molecules or electricity (including General Relativity) (this is not looking good for maximizing diamond, but see Problem 3 below). For instance, instead of specifying the quantum fluctuations governing molecule vibration, just implement some empirical facts that determine their behavior as correctly as possible. Of course, building this model (if possible) would require huge amounts of work from many physicists and engineers. Also the use of unbounded memory and compute for testing it accurately.[2] It is possible that the physicists get stuck, or can't put together a coherent macroscopic simulator without a paradigm change, but for these concerns see Problem 2.

This simulator can run many different universes. We still need to specify which universe we're in (the "initial conditions" of our world model). For this, combine two approaches:

Have Physicists put in (approximate) known facts. For example, if we know with certainty that on Earth there's an amount between X and Y of a certain material, then specify that between so and so coordinates[3] (where Earth should be in the simulation) there's such an amount of it. This fact will rule out a vast amount of possible simulations (universes). Of course, it will still leave open a huge amount of possible universes (some of which don't even have somethin

...
Announcing AI Alignment Awards: $100k research contests about goal misgeneralization & corrigibility
by Orpheus16 — 2022-11-22T22:19:09.419Z

We’re grateful to our advisors Nate Soares, John Wentworth, Richard Ngo, Lauro Langosco, and Amy Labenz. We're also grateful to Ajeya Cotra and Thomas Larsen for their feedback on the contests.

TLDR: AI Alignment Awards is running two contests designed to raise awareness about AI alignment research and generate new research proposals. Prior experience with AI safety is not required. Promising submissions will win prizes up to $100,000 (though note that most prizes will be between $1k-$20k; we will only award higher prizes if we receive exceptional submissions.)

You can help us by sharing this post with people who are or might be interested in alignment research (e.g., student mailing lists, FB/Slack/Discord groups.)

What are the contests?

We’re currently running two contests:

Goal Misgeneralization Contest (based on

Langosco et al., 2021

): AIs often learn unintended goals. Goal misgeneralization occurs when a reinforcement learning agent retains its capabilities out-of-distribution yet pursues the wrong goal. How can we prevent or detect goal misgeneralization?

Shutdown Problem Contest (based on

Soares et al., 2015

): Given that powerful AI systems might resist attempts to turn them off, how can we make sure they are open to being shut down?

What types of submissions are you interested in?

For the Goal Misgeneralization Contest, we’re interested in submissions that do at least one of the following:

Propose techniques for preventing or detecting goal misgeneralization
Propose ways for researchers to identify when goal misgeneralization is likely to occur
Identify new examples of goal misgeneralization in RL or non-RL domains. For example:

We might train an imitation learner to imitate a "non-consequentialist" agent, but it actually ends up learning a more consequentialist policy.
We might train an agent to be myopic (e.g., to only care about the next 10 steps), but it actually learns a policy that optimizes over a longer timeframe.

Suggest other ways to make progress on goal misgeneralization

For the Shutdown Problem Contest, we’re interested in submissions that do at least one of the following:

Propose ideas for solving the shutdown problem or designing corrigible AIs. These submissions should also include (a) explanations for how these ideas address core challenges raised in the corrigibility paper and (b) possible limitations

...
Human-level Full-Press Diplomacy (some bare facts).
by Cleo Nardo — 2022-11-22T20:59:18.155Z

Key links?

Here's the paper: https://www.science.org/doi/10.1126/science.ade9097
Here's the blog post: https://ai.facebook.com/blog/cicero-ai-negotiates-persuades-and-cooperates-with-people/
Here's the website: https://ai.facebook.com/research/cicero/diplomacy/

What is Diplomacy?

Diplomacy is a negotiation-based board game. There are 7 players corresponding to the 7 major European powers during WW1. In each round, the players write down their moves, which are then executed simultaneously. The moves might be attacking/defending territory, supporting/opposing another player's action, etc.
Unlike Risk, there are no dice or other random mechanism.
In "Full Press Diplomacy", players can also hold private conversations between rounds. The dialogue is used to establish trust and coordinate actions with other players. Players can make agreements, but all these agreements are non-binding.
In "No Press Diplomacy", players can't communicate.
Because the moves are simultaneous, game-play requires recursive theory of mind. The players must reason about the other players, and reason about what others are reasoning about them, and so on.
Diplomacy is notorious for ending friendships, probably because it erodes trust. Anecdotally, I've played Diplomacy twice in my life. Each game lasted at least 6 hours. The first game was played with slightly LessWrong-ish friends and wasn't friendship-destroying. The second was played with un-LessWrong-ish friends and it did cause outside-the-game distrust.
Diplomacy was the favourite game of John F. Kennedy and Henry Kissinger.

How well did CICERO perform?

On Nov 22 (today!) Meta AI presented an AI which achieved human-level performance at Full Press Diplomacy.
CICERO achieved more than double the average score of the other players and ranked in the top 10% of players.
The evaluation consisted of anonymous blitz Diplomacy games against humans on webDiplomacy.net

40 games
82 human players
5,277 messages
72 hours of gameplay

Here is commentary by a Diplomacy expert playing against six CICEROs: https://www.youtube.com/watch?v=u5192bvUS7k
"CICERO is so effective at using natural language to negotiate with people in Diplomacy that they often favoured working with CICERO over other human participants." (blog)
They are open-sourcing the code and models, and "interested researchers can submit a proposal to the CICERO RFP to gain access to the data." (?!!)

How doe

...
How does late-2022 COVID transmissibility drop over time?
by Daniel Dewey — 2022-11-22T19:54:50.422Z

This seems like the kind of thing LW folks might have good information on (better than my random googling).

I tested positive 7 days ago, and have been on Paxlovid for 5 days; symptoms are mostly gone, and we'll see if they bounce back when I finish the course of Pax. I would like to avoid giving my wife Killian COVID, but also it would be nice to be able to help out around the house more than is possible with strict isolation. I'm considering switching to "mask around the house, but not holed up in my office all day every day."

Anyone have a lead on good info about transmissibility during days 7-14, given positive or negative antigen tests on those days?

Thanks!
AI will change the world, but won’t take it over by playing “3-dimensional chess”.
by boazbarak — 2022-11-22T18:57:29.604Z

By Boaz Barak and Ben Edelman

[Cross-posted on Windows on Theory blog; See also Boaz’s posts on

longtermism

and

AGI via scaling

as well as other “philosophizing” posts.]

[Disclaimer: Predictions are very hard, especially about the future. In fact, this is one of the points of this essay. Hence, while for concreteness, we phrase our claims as if we are confident about them, these are not mathematically proven facts. However we do believe that the claims below are more likely to be true than false, and, even more confidently, believe some of the ideas herein are underrated in current discussions around risks from future AI systems.]

[To the LessWrong audience: we realize this piece is stylistically different from many posts here, and is not aimed solely at regular readers, for which various terms and claims below might be very familiar. Our original impetus to write this piece was a suggestion from the organizers of the Future Fund AI worldview competition; while we are not holding our breath for any reward, we thought it would be a good idea to engage with the LessWrong audience in this discussion. This post could be described as somewhere between an ordinary collaboration and an adversarial one—the views expressed are a closer fit for Boaz's current views than Ben's.]

In the past, the word “computer” was used to denote a person that performs calculations. Such people were highly skilled and were crucial to scientific enterprises. As described in the book “

Hidden Figures

”, until the 1960s, NASA still used human computers for the space mission. However, these days a $10 calculator can instantly perform calculations beyond the capabilities of every human on earth.

On a high level, the situation in Chess and other games is similar. Humans used to be the reigning champions in Chess and Go, but have now been surpassed by computers. Yet, while the success of computers in performing calculations has not engendered fears of them “taking over the world,” the growing powers of AI systems have more people increasingly worried about their long-term implications. Some reasons why the success of AI systems such as

AlphaZero

in Go and Chess is more concerning than the success of calculation programs include

Unlike when working with numerical computation programs, it seems that in Chess and Go humans are entirely “unnecessary.” There is no need to ha

...
Progress links and tweets, 2022-11-22
by jasoncrawford — 2022-11-22T17:39:06.498Z
Link: https://rootsofprogress.org/links-and-tweets-2022-11-22

The Progress Forum

Welcome to the Progress Forum, the online home for the progress community
How fear of the ‘Mega-Machine’ helped end America’s postwar Golden Age
Anton Howes on why we waited so long for the optical telegraph
The curse of plenty (by @mattsclancy)
How Karl Compton believed a research department should be run

Announcements

Andy Matuschak launches a research fellowship for “transformative tools for thought” (via @andy_matuschak)
Britain Remade, a new campaign focused on British economic growth (@samrichardswebb)
Waymo and Cruise are now both operating robotaxis in SF (@kfury)
The American Dynamism 50, from A16Z: “companies tackling our nation’s most pressing problems” (via @rmcentush)
The Progress Dashboard, a collection of progress links and resources
Facebook AI launched Galactica, a large language model for science (via @paperswithcode). Looked cool but did not actually work super-well and got pulled quickly
We hit 8 billion people (@yishan)

Links

For Thanksgiving, why we should be grateful for modern textiles (Virginia Postrel)
Planes are still decades away from displacing most bird jobs (by @alexeyguzey)

Queries

Intuition for why you can knock off a piece of a rock by hitting it with a piece of wood/bone?
If you know Bret Victor’s “Inventing on Principle,” what’s your principle? (@Prigoose). (If you don’t know it, see here)
A good writeup of the Gates Foundation’s role in covid response? (@benlandautaylor)

Quotes

Incandescent light is “so wasteful… as to render its practical application impossible” -Nature, 1878 (via @_brianpotter). I added some context
“Toys with as much power as the great IBM computers” (@arbesman)
Von Neumann on existential risk (@IvanVendrov)

Tweets & retweets

My meta-level take on “why did everything take so long?” (thread version of this previous post)
AGI will get built sooner than most people think, but take longer to “change everything” (@sama)
10 things underrated about America (@zachtratar)
Patterns across bubbles
RIP Fred Brooks, who wrote the classic book on software engineering, The Mythical Man-Month
A misheard word caused a $500M nuclear cleanup (@curiouswavefn)
Tyranny of the Epistemic Majority
by Scott Garrabrant — 2022-11-22T17:19:34.144Z

This post is going to mostly be propaganda for Kelly betting. However, the reasons presented in this post differ greatly from the reasons people normally use to argue for Kelly betting.

The Steward of Myselves

The curse of uncertainty is that I must make decisions that simultaneously affect many different versions of myself. When I close my eyes and then flip a coin, there are two potential versions of me: one sitting in front of a coin showing heads, the other sitting in front of a coin showing tails. Both of these potential versions of me are stakeholders in my current decisions. How can I make decisions on behalf of these multiple stakeholders?

If it is a fair coin, then we can think of these two potential selves as equal stakeholders in my decisions. However, I know that it is not a fair coin. It has a 60 percent chance of coming up heads. Thus, heads-me is a 60 percent stakeholder in my current decisions, and tails-me is a 40 percent stakeholder. They amount of each one's stake is naturally in proportion to the probability that they actually exist.

You, however, do not know if it is a fair coin, and are offering me a fair bet. I only have 100 dollars to my name, and I am can bet as much as I want (up to 100 dollars) in either direction at even odds.

If I bet 100 dollars on heads, heads-me gets 200 dollars, and tails-me gets nothing. If I bet 100 dollars on tails, tails-me gets 200 dollars, and heads-me gets nothing. If I bet nothing, both versions of me get 100 dollars.

However, every dollar in the hands of heads-me is worth 1.5 times as much as a dollar in the hands of tails-me, since heads-me exists 1.5 times as much. (I am ignoring here any diminishing returns in my value of money.)

Thus, to maximize value I should bet 100 dollars on heads. However, maybe it is better to think of tails-me as the rightful owner of 40 percent of my resources. When I bet 100 dollars on heads, I am seizing money from tails-me for the greater good, since heads-me has the (proportionally greater) existence necessary to better take advantage of it.

Alternatively, I could say that since 60 percent of me is heads-me, heads-me should only control 60 dollars, which can be bet on heads. Tails-me should control 40 dollars, which can be bet on tails. These two bets partially cancel each other out, and the net result is that I bet 20 dollars on heads.

If you are especially fast at max...
A Walkthrough of In-Context Learning and Induction Heads (w/ Charles Frye) Part 1 of 2
by Neel Nanda — 2022-11-22T17:12:02.562Z
Link: https://www.youtube.com/watch?v=dCkQQYwPxdM

New paper walkthrough: In-Context Learning and Induction Heads. This is the second paper in Anthropic's Transformer Circuits thread, a series of papers trying to reverse engineer transformer language models. I read through it with Charles Frye (from Full-Stack Deep Learning), and we discuss the paper, and give takes and intuitions. See the original paper and a Twitter thread of my paper takeaways

This is pitched so that it's hopefully accessible to people who haven't read the paper (very interested in feedback on this!), but I expect you to get more out of it if you understand transformers, and especially if you've read A Mathematical Framework. We only got partway through the paper, so there's a more in-the-weeds Part 2 in the works where we finish it off - let me know if you're interested in seeing it!

Disclaimer: I worked on this paper, along with Catherine Olsson, Nelson Elhage and Chris Olah, when I was at Anthropic, but I have since left and everything in this video is purely my own takes!

If you find this useful, check out my previous walkthroughs: A Mathematical Framework for Transformer Circuits and Interpretability in the Wild

And I'd be excited to see other researchers do these kinds of walkthroughs! The effort to usefulness ratio is way better than writing papers (and to my tastes, it's much more fun!)
Simple Improvement to College Football Overtime Rules
by Zvi — 2022-11-22T17:00:00.487Z

For a long time I have thought that a simple tweak to the NCAAF overtime rules would fix several problems with no downsides. The Alabama-LSU game reinforced my impression of this by showing what we could have every time.

PROPOSED: In overtime, teams cannot kick an extra point that ties the game.

This replaces the new ‘teams cannot kick extra points in the second and later overtime periods’ rule.

This does three things.

Dramatically improves the chances the game will end quickly, avoiding the need for two-point-try-only periods. If either team gets a touchdown, unless both miss extra points, the game is over.

Reduces the advantage of winning the coin toss. Under current rules, knowing what you need to do, and thus having the ability to use four downs, is a big advantage.

Ends games on a great moment. Everyone loves all-or-nothing final plays.

If you want an additional safeguard against too many overtimes, rather than successive two point tries, one can (starting with the third or fourth overtime) add a rule that one cannot kick a field goal to tie the game, either. That all but assures that the next overtime period will be the last one.

Compare this to the current rule that requires both teams to go for two starting with the second overtime. This rule is so bad it is essentially a bug. It is harder to explain, and half the time it fails to achieve its goal of ending the game.

Is a pair of offensive possessions a better way to resolve a tie than a timed overtime period? I don’t know. However, once you do use a pair of possessions, this is a strictly better way to do that.
Meta AI announces Cicero: Human-Level Diplomacy play (with dialogue)
by Jacy Reese Anthis — 2022-11-22T16:50:20.054Z
Link: https://www.science.org/doi/10.1126/science.ade9097

Abstract
Despite much progress in training AI systems to imitate human language, building agents that use language to communicate intentionally with humans in interactive environments remains a major challenge. We introduce Cicero, the first AI agent to achieve human-level performance in Diplomacy, a strategy game involving both cooperation and competition that emphasizes natural language negotiation and tactical coordination between seven players. Cicero integrates a language model with planning and reinforcement learning algorithms by inferring players' beliefs and intentions from its conversations and generating dialogue in pursuit of its plans. Across 40 games of an anonymous online Diplomacy league, Cicero achieved more than double the average score of the human players and ranked in the top 10% of participants who played more than one game.
Meta Fundamental AI Research Diplomacy Team (FAIR)†, Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, et al. 2022. “Human-Level Play in the Game of Diplomacy by Combining Language Models with Strategic Reasoning.” Science, November, eade9097. https://doi.org/10.1126/science.ade9097.
Austin LW meetup notes: The FTX Affair
by jchan — 2022-11-22T14:01:15.625Z

This article is a writeup of the conversation at a meetup hosted by Austin Less Wrong on Saturday, November 19, 2022. The topic was the collapse of the FTX cryptocurrency exchange and its implications. There were a total of 28 participants, although people came and went throughout the meetup, so the number present at any given time was less than that. This was not an EA meetup, but it attracted more EA-involved people than usual.

Epistemic disclaimer: This writeup should not be read like a normal, well-polished article written by a single author. It contains speculative, off-the-cuff remarks which the speakers may have regarded as tentative even at the time, and which may be easily revised by new evidence. Each statement given here represents the views only of the particular person speaking (which may change from one sentence to the next), and not of the other participants. Below the section break, "I" refers to the current speaker.

General disclaimer: I took pains to make it clear before, during, and after the meetup that I was taking notes for posting on LessWrong later. I do not endorse posting meetup writeups without the knowledge and consent of those present!

What was going on at FTX?

Initial impressions

When I first heard the news about FTX being insolvent, I thought they would still honor withdrawals. I figured I'd be fine because I only have money on FTX.us which is separate from the insolvent entity, but it turned out they were commingling funds—FTX.us is part of the bankruptcy and so its funds will be distributed to non-US customers. (But FTX bought LedgerX and they seem to be fine—why were they not also included in the bankruptcy?)

I expected Sam Bankman-Fried ("SBF") to have his act together, because he had been rescuing other exchanges.

Early on, a bunch of employees quit Alameda Research, and if these people have overlap with the EA community, I want to know more about whatever scandal caused them to quit. Tara Mac Aulay, the co-founder of Alameda, was one of them. She later co-founded Lantern, and previously worked for the Centre for Effective Altruism (CEA). She wrote a Twitter thread [archive] saying that the people who left Alameda did so "in part due to concerns over risk management and business ethics."

I think FTX would've still been successful without the scamming—they had a reliable business, and the team had widely-applicable skills. Gemi...