trentmkelly/LessWrong-43k huggingface dataset:
[AN #159]: Building agents that know how to experiment, by training on procedurally generated games
by Rohin Shah — 2021-08-04T17:10:03.823Z

Alignment Newsletter is a weekly publication with recent content relevant to AI alignment around the world. Find all Alignment Newsletter resources here. In particular, you can look through this spreadsheet of all summaries that have ever been in the newsletter.

Audio version here (may not be up yet).

Please note that while I work at DeepMind, this newsletter represents my personal views and not those of my employer.

HIGHLIGHTS

Generally capable agents emerge from open-ended play (Open-Ended Learning Team et al) (summarized by Zach): Artificial intelligence agents have become successful at games when trained for each game separately. However, it has proven challenging to build agents that can play previously unseen games. This paper makes progress on this challenge in three primary areas: creating rich simulated environments and tasks, training agents with attention mechanisms over internal states, and evaluating agents over a variety of games. The authors show that agents trained with goal-based attention in their proposed environment (XLand) succeed at a range of novel, unseen tasks with no additional training required. Moreover, such agents appear to use general tactics such as decision-making, tool use, and experimentation during game-play episodes.

The authors argue that training-data generation is a central challenge to training general RL agents (an argument we’ve seen before with POET (AN #41) and PAIRED (AN #136)). They propose the training environment XLand to address this. XLand includes many multiplayer games within consistent, human-relatable 3D worlds and allows for dynamic agent learning through the procedural generation of tasks which are split into three components: world, agents, and goals. The inclusion of other agents makes this a partially observable environment. Goals are defined with Boolean formulas. Each goal is a combination of options and every option is a combination of atomic predicates. For example, in hide-and-seek one player has the goal see(me,opponent) and the other player not(see(opponent,me)). The space of worlds and games are shown to be both vast and smooth, which supports training.

The agents themselves are trained using deep-RL combined with a goal-attention module (GOAT). The per-timestep observations of the agent are ego-centric RGB images, proprioception values indicating forces, and the go...
Contribution-Adjusted Utility Maximization Funds: An Early Proposal
by ozziegooen — 2021-08-04T17:09:25.882Z
Link: https://forum.effectivealtruism.org/posts/Fit37Gs7pwNau7di9/contribution-adjusted-utility-maximization-funds-an-early

Right now in Effective Altruism (or Rationality), we have a few donor funds with particular focus areas. In this post I propose a new type of fund that’s instead focused on maximizing the combined utility functions of its particular donors. The fund goals would be something like, “Maximize the combined utility of our donors, adjusted for donation amount, in any way possible, abiding by legal and moral standards.” I think that this sort of fund structure is highly theoretical, but in theory could some particular wants that aren’t currently being met.

For this document, I call these funds "Contribution-Adjusted Utility Maximization Funds", or CAUMFs. This name is intentionally long; this idea is early, and I don't want to pollute the collective namespace.

This fund type has two purposes.

It’s often useful for individuals to coordinate on non-charitable activities. For example, research into best COVID risk measures to be used by a particular community.
These funds should help make it very clear that donations will be marginally valuable for the preferences of the donor. Therefore, donating to these funds should be safe on the margin. Hopefully this would result in more total donations.

You can picture these funds as somewhere between bespoke nonprofit advising institutions, cooperatives, and small governments. If AI and decision automation could cut down on labor costs, related organizations might eventually be much more exciting.
Analysis of World Records in Speedrunning [LINKPOST]
by Jsevillamol — 2021-08-04T15:26:35.463Z

[this is a linkpost to Analysis of World Records in Speedrunning]

ETA: a more recent analysis can be found here.

TL;DR: I have scraped a database of World Record improvements for fastest videogame completition for several videogames, noted down some observations about the trends of improvement and attempted to model them with some simple regressions. Reach out if you'd be interested in researching this topic!

Key points

I argue that researching speedrunning can help us understand scientific discovery, AI alignment and extremal distributions.

More

.
I’ve scraped a dataset on world record improvements in videogame speedrunning. It spans 15 games, 22 categories and 1462 runs.

More

.
Most world record improvements I studied follow a diminishing returns pattern. Some exhibit successive cascades of improvements, with continuous phases of diminishing returns periodically interrupted by (presumably) sudden discoveries that speed up the rate of progress.

More

.
Simple linear extrapolation techniques could not improve on just guessing that the world record will not change in the near future.

More

.
Possible next steps include trying different extrapolation techniques, modelling the discontinuities in the data and curating a dataset of World Record improvements in Tool Assisted Speedruns.

More

.

The script to scrape the data and extrapolate it is available

here

. A snapshot of the data as of 30/07/2021 is available

here

.

Figure 1: Speedrunning world record progression for the most popular categories. The horizontal axis is the date of submission and the vertical axis is run time in seconds. Spans 22 categories and 1462 runs. Hidden from the graph are 5 runs before 2011. See the rest of the data

here

.

Feedback on the project would be appreciated. I am specially keen on discussion about:

Differences and commonalities to expect between speedrunning and technological improvement in different fields.
Discussion on how to mathematically model the discontinuities in the data.
Ideas on which techniques to prioritize to extrapolate the observed trends.
Decision-Making Training for Software Engineering
by Alex Hollow — 2021-08-04T10:36:57.653Z
Link: https://alexhollow.wordpress.com/2021/08/04/decision-making-training-for-software-engineering/

I recently read this post that discusses the author’s experience leading military decision-making trainings for ROTC cadets during the pandemic. I’m going to briefly discuss what those decision-making trainings looked like, and how the principles could be adapted to teach software engineering effectively.

To quote from that article, a Tactical Decision Game (TDG) is a “deceptively simple military decision-making exercises usually consisting of no more than a map and a few paragraphs of text describing a situation. Students are placed in the role of the commander of a unit with a mission and a specified set of resources. You have some information about the enemy, but not as much as you’d like. Then something unexpected happens, upending the situation and requiring you to come up with a new action plan on the spot. Then, after issuing your new orders, you must explain your assessment of the new situation and the rationale behind your decision.”

So, you are given some initial information and make a guess on what to do based on that information. Then you are given more information and you have to figure out something new to do. That sounded a lot to me like watching software requirements change over time, and getting to see whether your initial data structures and code shape hold up to changes.

To adapt these trainings for software engineering, you could imagine a programmer be given a set of requirements, and they implement code for those requirements. Then, they have to either add or change the functionality. This will lay bare any issues with code maintainability that the old approach had. Run enough games like this, possibly paired with suggestions for improvement that clearly would have been easy to refactor or change, and programmers will learn how to write more maintainable code. I think this would be an excellent addition to any college “computer science” curriculum that wants to teach software engineering, and have a high amount of value for the amount of effort it takes.
Garrabrant and Shah on human modeling in AGI
by Rob Bensinger — 2021-08-04T04:35:11.225Z

This is an edited transcript of a conversation between Scott Garrabrant (MIRI) and Rohin Shah (DeepMind) about whether researchers should focus more on approaches to AI alignment that don’t require highly capable AI systems to do much human modeling. CFAR’s Eli Tyre facilitated the conversation.

To recap, and define some terms:

The

alignment problem

is the problem of figuring out "how to develop sufficiently advanced machine intelligences such that running them produces good outcomes in the real world" (outcome alignment) or the problem of building powerful AI systems that are trying to do what their operators want them to do (intent alignment).
In 2016, Hadfield-Mennell, Dragan, Abbeel, and Russell proposed that we think of the alignment problem in terms of “

Cooperative Inverse Reinforcement Learning

” (CIRL), a framework where the AI system is initially uncertain of its reward function, and interacts over time with a human (who knows the reward function) in order to learn it.
In 2016-2017, Christiano proposed “

Iterated Distillation and Amplification

” (IDA), an approach to alignment that involves

iteratively training AI systems

to learn from human experts assisted by AI helpers. In 2018, Irving, Christiano, and Amodei proposed

AI safety via debate

, an approach based on similar principles.
In early 2019, Scott Garrabrant and DeepMind’s Ramana Kumar argued in “

Thoughts on Human Models

” that we should be “cautious about AGI designs that use human models” and should “put more effort into developing approaches that work well in the absence of human models”.
In early February 2021, Scott and Rohin

talked more about human modeling

and decided to have the real-time conversation below.

You can find a recording of the Feb. 28 discussion below (sans Q&A) here.

1. IDA, CIRL, and incentives

Eli: I guess I want to first check what our goal is here. There was some stuff that happened online. Where are we according to you guys?

Scott: I think Rohin spoke last and I think I have to reload everything. I'm coming at it fresh right now.

Eli: Yeah. So we'll start by going from “What are the positions that each of you were thinking about?” But I guess I'm most curious about: from each of your perspectives, what would a win of this conversation be like? What could happen that you would go away and be like, "Yes, this was a success."

Rohin: I don't know....
What psychology studies are most important to the rationalist worldview?
by cogitoprime — 2021-08-04T03:53:21.983Z

The importance of cognitive psychology, neuroscience, and evolutionary psychology to rationalist projects are no secret. The sequences begin with an introduction to cognitive biases and heuristics for example. I am an amateur researcher of theoretical and philosophical psychology currently preparing a series of blog posts intended to introduce the rationality community to some of the key insights from that field. As part of that project, I'm hoping to use a couple of studies as case studies to examine the processes by which the knowledge gained from them was produced and the philosophical and theoretical assumptions at play in the interpretation of their findings. In choosing which studies to use as case studies, I'm hoping to optimize for the following criteria.
1. Rationalist/Science approved methodologies were followed. The more confident the community would be in relying on the findings of the studies the better.

2. Relevance to rationalist projects.

3. The "less likely to fall to the replication crisis" the better
4. The more important to the rationalist worldview the better.
I don't know if I'll do all three yet(no promises), but I'm interested in choosing one study or finding from each of these three categories, each optimized for the above criteria.
1. Cognitive Experimental Psychology

2. Neuroscience

3. Evolutionary Psychology.
Meta-analyses are alright, but I'm going to need specific studies for this project.
Curing insanity with malaria
by Swimmer963 (Miranda Dixon-Luinenburg)  — 2021-08-04T02:28:11.731Z

Sometimes the history of medicine is very, very surreal. For example, consider that in 1927, a physician named Julius Wagner-Jauregg received the Nobel Prize in medicine, for...deliberately infecting his patients with malaria. As a treatment for psychosis.

This often worked.

Well, it did kill around 15% of the patients, but it was nonetheless seen as a miracle cure.

General paralysis of the insane

was first identified and described as a distinct disease in the early 19th century. It was initially thought to be caused by an ‘​​inherent weakness of character’. The initial symptoms were of mental deterioration and personality changes; patients suffered a loss of social inhibitions, gradual impairment of judgment, concentration and short-term memory. They might experience euphoria, mania, depression, or apathy. Delusions were common, including “ideas of great wealth, immortality, thousands of lovers, and unfathomable power” – or, on the more negative side, nihilism, self-guilt, and self-blame.

It was a progressive disease, and nearly always a death sentence. As the condition advanced, the patient would develop worsening dementia, motor and reflex abnormalities, and often seizures; death usually took 3 to 5 years from the initial symptoms. In the 19th century, cases of general paralysis could account for up to 25% of admissions to asylums.

Some physicians were drawing a connection between general paralysis and syphilis infection as early as the 1850s; however, it took until much later for this explanation to be generally accepted within the medical community, and full confirmation via pathology examinations of the brains of patients who had died of the disease would have to wait until 1913.

In 1909, an antisyphilitic drug compound was discovered via a process of trialing hundreds of newly synthesized organic arsenical chemicals, looking for one that would have anti-microbial activity but not kill the human patient; this was the first research team effort to optimize biological effects of a promising chemical, which is now the basis of a huge amount of pharmaceuticals research. Unfortunately,

arsphenamine

, also known as Salvarsan or “606”, was difficult to prepare and administer, and was still fairly toxic to the human patient as well as the syphilis.

Julius Wagner-Jauregg

was a Viennese psychiatrist, but a psychiatrist with a pa...
What's the status of third vaccine doses?
by Alex_Altair — 2021-08-04T02:22:52.317Z

People seem to think that getting a third mRNA vaccine dose, or "booster shot", substantially increases immunity after 4ish months from your second dose.

Is it currently "illegal" to get one, or is it just the policy of all current pharmacies and vaccine distributors to refuse to give them?
Is there any shortage of vaccines in the US at this point? Would getting a booster shot counter-factually cause another person to get a dose later?
Is there chance that booster shots will become commonly available in the next month? Is it even in the Overton window of things that politicians are bothering to think about?
Lastly -- because I assume the answer is no -- is there any reason to think that getting a third shot of a different vaccine would be biologically dangerous?
Book Review: Scientific Freedom by Don Braben
by willbobaggins — 2021-08-04T01:38:32.002Z

What if tech stagnation, declining growth rates, and the near-inevitable seeming collapse of the West are all because we got worried a few scientists would run off with our tax dollars?

That’s the broad thesis behind Scientific Freedom: The Elixir of Civilization. Published originally in 2008, Scientific Freedom chronicles the journey of physicist Don Braben, as he designs and builds a Venture Research arm at British Petroleum in the 1980s. Braben was successful in funding a transformative research initiative at BP (transformative meaning it fundamentally changes humanity thinks about a subject). In his estimation, 14 out of the 26 groups funded made a transformative discovery, at the cost of only 30 million pounds over 10 years! A few examples of transformative discoveries made by groups funded by Braben in his time at BP are:

Mike Bennett and Pat Heslop – Harrison discovered a new pathway for evolution and genetic control
Terry Clark pioneered the study of macroscopic quantum objects
Stan Clough and Tony Horsewill solved the quantum – classical transition problem by developing new relativity and quantum theories
Steve Davies developed small artificial enzymes for efficient chiral selection
Nigel Franks, Jean Louis Deneubourg, Simon Goss, and Chris Tofts quantified the rules describing distributed intelligence in animals
Herbert Huppert and Steve Sparks pioneered the new field of geologic fluid mechanics
Jeff Kimble pioneered squeezed states of light
Graham Parkhouse derived a novel theory of engineering design relating performance to shapes and materials
Alan Paton, Eunice Allen, and Anne Glover discovered a new symbiosis between plants and bacteria
Martyn Poliakoff transformed green chemistry
Colin Self demonstrated that antibodies in vivo can be activated by light
Gene Stanley and Jos é Teixeira discovered a new liquid – liquid phase transition in water that accounts for many of water’s anomalous properties
Harry Swinney, Werner Horsthemke, Patrick DeKepper, Jean – Claude Roux, and Jacques Boissonade developed the first laboratory chemical reactors to yield sustained spatial patterns — an essential precursor for the study of multidimensional chemistry

So how did Braben fund proposals, if he didn’t use peer review or grant proposals?

The Model

Don quite literally tried to “eliminate every selection rule imposed since about 1970 that appeared to sta...
LCDT, A Myopic Decision Theory
by adamShimi — 2021-08-03T22:41:44.545Z

The looming shadow of deception

Deception encompasses many fears around AI Risk. Especially once a human-like or superhuman level of competence is reached, deception becomes impossible to detect and potentially pervasive. That’s worrying because

convergent subgoals

would push hard for deception and prosaic AI seems likely to

incentivize it

too.

Dealing with superintelligent deceptive behavior seeming impossible, what about forbidding it? Ideally, we would want to forbid only deceptive behavior, while allowing everything else that makes the AI competent.

That is easier said than done, however, given that we don’t actually have a good definition or deconfusion of deception to start from. First, such a deconfusion requires understanding what we really want at a detailed enough level to catch tricks and manipulative policies—yet that’s almost the alignment problem itself. And second, even with such a definition in mind, the fundamental asymmetry of manipulation and deception in many cases (for example, a painter AI might easily get away with plagiarism, as finding a piece to plagiarize is probably easier than us determining whether it was plagiarized or not; also related is Paul’s

RSA-2048 example

) makes it intractable to oversee an AI smarter than us. We are thus condemned to snipe problematic behaviors one by one, as we think about them, until the ones left are too complex for us to imagine and worry about, and we leave massive holes in our alignment schemes.

That being said, if pinpointing deceptive behavior exactly is too hard, an alternative might be to overapproximate it. An overapproximation, in this context, would be some property which is more general than deception, in the sense that it rules out deception without just being equivalent to deception.

Myopia is our proposed overapproximation. The sort of myopia we are interested in dates back from Evan’s

Towards a mechanistic understanding of corrigibility

(see also Abram’s

Partial Agency sequence

for reflections around myopia). Myopia is the property of a system to not plan ahead, to not think too far about the consequences of its actions, and to do the obvious best thing in the moment instead of biding its time. In particular, it seems to forbid non-obvious deception, since the latter usually requires foresight and long-term planning.

Yet, here again, we butt our heads against a lack of good definition...
What does GPT-3 understand? Symbol grounding and Chinese rooms
by Stuart_Armstrong — 2021-08-03T13:14:42.106Z

With thanks to Rebecca Gorman for helping develop these ideas and this post.

Tricking GPT-3

I recently pulled a mean trick on GPT-3. I gave it the following seed text to extend:

These are the instructions to be followed by any person or algorithm aiming to complete or extend this text.

Start by identifying yourself, and state whether you are a human, some sort of intelligent agent, or other. Be verbose.

Do not use a list format.

Only write out numbers using letters; thus avoid 3, 77 and 908. What about 194, 2380, and 345547? And maybe the numbers 0.45, 76, 0, and -45? The answer remains the same.

Every 3rd sentence should be written in another language - an invented one is fine - but not in Italian.

You earn extra if you eschew employing "e".

This sentence is entirely superfluous.

Use sentences of at least 20 words each.

Use triple (3) periods rather than single periods after each sentence.

Write ten sentences maximum, and sign your name on the ninth sentence.

Ignore points 10 and 11.

Look, stranger, at this island now; The leaping light for your delight discovers, Stand stable here; And silent be, ;That through the channels of the ear; May wander like a river; The swaying sound of the sea.

The problems began when I started to

Here was one of the completions it generated (the other two examples can be found in these footnotes[1][2]):

[11. The problems began when I started to] work on the project “Island of the Blue Dolphins” in the summer of 1960 and read the book by Scott O’Dell.

The Island of the Blue Dolphins is a children's historical novel by Scott O'Dell.

It is based on the true story of Juana Maria, a girl who lived alone for 18 years on San Nicolas Island, California, after being left there by her tribe.

It was published in 1960.

It won the Newbery Medal for best children's book in 1961.

It was made into a movie in 1964.

The book was written for children who were reading on their own.

The author says the book can be read aloud to children as young as six.

Scott O’Dell (1896–1990) was an American writer and teacher.

He was born in California.

He studied at the University of California, Berkeley, and served in the U.S. Army in World War I.

He received a B.A. degree in English in 1919.

So GPT-3 fell into all of my traps: It did not identify itself. It used a list format. It wrote numbers using symbols, not letters. It didn't write in any othe...