trentmkelly/LessWrong-43k huggingface dataset:
[MLSN #6]: Transparency survey, provable robustness, ML models that predict the future
by Dan H — 2022-10-12T20:56:31.670Z

You can subscribe to the newsletter here, follow the newsletter on twitter here, or join the subreddit here.

Welcome to the 6th issue of the ML Safety Newsletter. In this edition, we cover:

A review of transparency research and future research directions
A large improvement to certified robustness
“Goal misgeneralization” examples and discussion
A benchmark for assessing how well neural networks predict world events (geopolitical, industrial, epidemiological, etc.)
Surveys that track what the ML community thinks about AI risks
$500,000 in prizes for new benchmarks
And much more…

Monitoring

Transparency Survey

A taxonomy of transparency methods. Methods are organized according to what part of the model they help to explain (weights, neurons, subnetworks, or latent representations). They can be intrinsic (implemented during training), post hoc (implemented after training), or can rely on a mix of intrinsic and post hoc techniques. ‘Hazards’ (in orange) are phenomena that make any of these techniques more difficult.

This survey provides an overview of transparency methods: what’s going on inside of ML models? It also discusses future directions, including:

Detecting deception and eliciting latent knowledge. Language models are dishonest when they babble common misconceptions like “bats are blind” despite knowing that this is false. Transparency methods could potentially indicate what the model ‘knows to be true’ and provide a cheaper and more reliable method for detecting dishonest outputs.

Developing rigorous benchmarks. These benchmarks should ideally measure the extent to which transparency methods provide actionable insights. For example, if a human implants a flaw in a model, can interpretability methods reliably identify it?

Discovering novel behaviors. An ambitious goal of transparency tools is to uncover why a model behaves the way it does on a set of inputs. More feasibly, transparency tools could help researchers identify failures that would be difficult to otherwise anticipate.

[Link]

Other Monitoring News

[Link] This paper discusses the sudden emergence of capabilities in large language models. This unpredictability is naturally a safety concern, especially when many of these capabilities could be hazardous or discovered after deployment. It will be difficult to make models safe if we do not know what they are capable of.

[Link] This work attributes e...
Previous Work on Recreating Neural Network Input from Intermediate Layer Activations
by bglass — 2022-10-12T19:28:39.058Z

Recently I've been experimenting with recreating a neural network's input layer from intermediate layer activations.

The possibility has implications for interpretability. For example, if certain neurons are activated on certain input, you know those neurons are 'about' that type of input.

My question is: Does anyone know of prior work/research in this area?

I'd appreciate even distantly-related work. I may write a blog post about my experiments if there is an interest and if there isn't already adequate research in this area.
Be more effective by learning important practical knowledge using  flashcards
by Stenemo — 2022-10-12T18:05:50.343Z

I made a collection of 200+ flashcards (Part 1 together with part 2) to help remember useful life lessons from hundreds of great books, videos, podcasts and other sources. It includes knowledge to

Make more informed decisions.

Determine if someone is drowning or having a stroke.

Have stronger more fulfilling relationships.

Succeed with your goals.

References including links to video explanations are included with each card, giving the option to watch or read an explanation to make learning easy.

I think more people share my interest in collecting important knowledge and making it available as flashcars. If we worked together we could make a comprehensive deck of what life skills everyone should know.

Feel free to

Provide corrections (ideally with sources).

Share this post with anyone who might find it useful.

Share your own deck of cards you think everyone should know and link to it below or send as private message, and I'll add it.
Article Review: Google's AlphaTensor
by Robert_AIZI — 2022-10-12T18:04:48.915Z

[Cross-posted from my Substack at https://aizi.substack.com/p/article-review-googles-alphatensor]

How AlphaTensor sees matrices, probably
Last week, Google announced AlphaTensor, “the first artificial intelligence (AI) system for discovering novel, efficient, and provably correct algorithms for fundamental tasks such as matrix multiplication”. In this post, I want to rephrase their

press release

and

Nature article

in terms that make sense to me (and hopefully you, the reader), and share some thoughts on whether this is significant and holds any AI safety lessons.

Background: What are Tensors?

Tensors are everywhere in our world: from press releases by Google, to Nature articles by Google, to AlphaTensor, the name of the agent trained by Google. But what are tensors?

Grids of numbers. Tensors are a grid of numbers. A tensor with two axes[1] is a rectangular grid of numbers, a.k.a. a matrix, and a tensor with one axis is just a vector. Python programmers will be familiar with tensors as ndarrays. You can add tensors of the same size componentwise, and multiply them by scalars, so they form a vector space. You can make tensors more complicated if you want, but “grids of numbers” is all you need to know for this paper.

The goal of Google’s AI agent is to find fast algorithms for multiplication of matrices. I’ll pretend we’re working with just n-by-n square matrices, but the agents in the article are trained to find algorithms for n-by-m times m-by-p matrix multiplication for n,m,p≤5. Matrix multiplication is one of the most common operations performed by a computer, but it can be slow, especially for large matrices. The speed of these algorithms are measured in multiplications, since that makes up the bulk of the runtime for any of these algorithms (addition is relatively fast). The intro-to-linear-algebra method for matrix multiplication takes n3 multiplications to multiply n-by-n matrices, but its been known since the 1969 Strassen algorithm that you can save multiplications by doing more additions (Strassen’s original algorithm multiplies 2-by-2s in 7 multiplications instead of 8, and a variant called Strassen-square multiplies n-by-ns in roughly n2.8 multiplications[2]).

So, what does matrix multiplication have to do with tensors? The first insight used in the paper is to think of matrix multiplication as a bilinear operator on&n...
Alignment 201 curriculum
by Richard_Ngo — 2022-10-12T18:03:03.454Z
Link: https://www.agisafetyfundamentals.com/alignment-201-curriculum

This curriculum, a follow-up to the

Alignment Fundamentals curriculum

(the ‘101’ to this 201 curriculum), aims to give participants enough knowledge about alignment to understand the frontier of current research discussions. It assumes that participants have read through the Alignment Fundamentals curriculum, taken a course on deep learning, and taken a course on reinforcement learning (or have an equivalent level of knowledge).

Although these are the basic prerequisites, we expect that most people who intend to work on alignment should only read through the full curriculum after they have significantly more ML experience than listed above, since upskilling via their own ML engineering or research projects should generally be a higher priority for early-career alignment researchers.

When reading this curriculum, it’s worth remembering that the field of alignment aims to shape the goals of systems that don’t yet exist; and so alignment research is often more speculative than research in other fields. You shouldn’t assume that there’s a consensus about the usefulness of any given research direction; instead, it’s often worth developing your own views about whether techniques discussed in this curriculum might plausibly scale up to help align AGI.

The curriculum was compiled, and is maintained, by Richard Ngo. For now, it’s primarily intended to be read independently; once we’ve run a small pilot program, we’ll likely extend it to a discussion-based course.

Curriculum overview

Week 1: Further understanding the problem
Week 2: Decomposing tasks for better supervision
Week 3: Preventing misgeneralization
Week 4: Interpretability
Week 5: Reasoning about Reasoning
Weeks 6 & 7 (Track 1): Eliciting Latent Knowledge
Weeks 6 & 7 (Track 2): Agent Foundations
Weeks 6 & 7 (Track 3): Science of Deep Learning
Weeks 8 & 9: Literature Review or Project Proposal

See the full curriculum here. Note that the curriculum is still under revision, and feedback is very welcome!
Progress links and tweets, 2022-10-12
by jasoncrawford — 2022-10-12T16:59:34.308Z
Link: https://rootsofprogress.org/links-and-tweets-2022-10-12

Announcements

Our World in Data is hiring a Senior Data Visualization Engineer
World’s Fair Co looking for space in San Francisco to host meetings with creatives; can anyone help? (@camwiese)

Links

“A well-known Dussehra tradition is to worship all weapons, tools, instruments… all articles that are used for progress and prosperity of mankind are worshipped”

Queries

Who should Dwarkesh have on his podcast to discuss The Power Broker? (@dwarkesh_sp)
Are drugs ever redesigned to make them easier to manufacture? (@_brianpotter)
Who are the “field strategists” in your domain? (@abiylfoyp)
Fiction that follows the same family throughout multiple centuries? (@FreshMangoLassi)

Tweets

Public health should be told as part of the story of the Second Industrial Revolution
Over 50% of the Earth’s fresh surface water is in just three lakes (@lewis_dartnell adds that two are rift lakes formed in tectonic faults)
“2 + 2 = 5” was literally a Soviet Communist slogan
“Sveriges Riksbank,” who endowed the econ Nobel, is not some rich dude
Augustus vs. Stalin
What dessert does a nuclear reactor want for its birthday?
The periodic table table, by Theo Gray

Retweets

Researchers are spending almost half of their time dealing with grants instead of researching (@Ben_Reinhardt)
Far-UVC light is a promising tool to fight airborne germs (@arpitrage)
Every plan to decarbonize assumes the power grid gets builds out (@JohnArnoldFndtn)
Claim: nearly 40% of Europe’s “carbon neutral” energy is obtained from burning wood (@DoombergT)
“Subjective measures of wellbeing but with vivid anchors” (@mbateman)
Captain Picard, prompt engineer (@rickasaurus)
Building a transformer from scratch - AI safety up-skilling challenge
by Marius Hobbhahn — 2022-10-12T15:40:10.537Z

It is not always obvious whether your skills are sufficiently good to work for one of the various AI safety and alignment organizations. There are many options to calibrate and improve your skills including just applying to an org or talking with other people within the alignment community.

One additional option is to test your skills by working on projects that are closely related to or a building block of the work being done in alignment orgs. By now, there are multiple curricula out there, e.g. the one by

Jacob Hilton

or the one by

Gabriel Mukobi

.

One core building block of these curricula is to understand transformers in detail and a common recommendation is to check if you can build one from scratch. Thus, my girlfriend and I have recently set ourselves the challenge to build various transformers from scratch in PyTorch. We think this was a useful exercise and want to present the challenge in more detail and share some tips and tricks. You can find our code

here

.

Building a transformer from scratch

The following is a suggestion on how to build a transformer from scratch and train it. There are, of course, many details we omit but I think it covers the most important basics.

Goals

From the ground up we want to

Build the attention mechanism
Build a single-head attention mechanism
Build a multi-head attention mechanism
Build an attention block
Build one or multiple of a text classification transformer, BERT or GPT. The quality of the final model doesn’t have to be great, just clearly better than random.
Train the model on a small dataset.

We used the

polarity dataset

for binary text sentiment classification.
We used the

AG_NEWS

dataset (PyTorch built-in) for BERT and GPT.

Test that the model actually learned something

We looked at the first batch of the test data to see if the model predicted something plausible.
We compared the test loss of a random network with the test loss of the trained network to see if our model is better.

Bonus goals

Visualize one attention head
Visualize how multiple attention heads attend to the words of an arbitrary sentence
Reproduce the grokking phenomenon (see e.g.

Neel’s and Tom’s piece

).
Answer some of the questions in Jacob Hilton's post.

Soft rules

For this calibration challenge, we used the following rules. Note, that these are “soft rules” and nobody is going t...
Instrumental convergence in single-agent systems
by Edouard Harris — 2022-10-12T12:24:58.546Z
Link: https://www.gladstone.ai/instrumental-convergence-1

Summary of the sequence

Over the past few months, we’ve been investigating instrumental convergence in reinforcement learning agents. We started from the definition of single-agent POWER

proposed

by

Alex Turner

et al., extended it to a family of multi-agent scenarios that seemed relevant to AI alignment, and explored its implications experimentally in several RL environments.

The biggest takeaways are:

Alignment of terminal goals and alignment of instrumental goals are sharply different phenomena, and we can quantify and visualize each one separately.
If two agents have unrelated terminal goals, their instrumental goals will tend to be misaligned by default. The agents in our examples tend to interact competitively unless we make an active effort to align their terminal goals.
As we increase the planning horizon of our agents, instrumental value concentrates into a smaller and smaller number of topologically central states — for example, positions in the middle of a maze.

Overall, our results suggest that agents that aren’t competitive with respect to their terminal goals, nonetheless tend on average to become emergently competitive with respect to how they value instrumental states (at least, in the settings we looked at). This constitutes direct experimental evidence for the instrumental convergence thesis.

We’ll soon be open-sourcing the codebase we used to do these experiments. We’re hoping to make it easier for other folks to reproduce and extend them. If you’d like to be notified when it’s released, email Edouard at

edouard@gladstone.ai

, or DM me here or on Twitter at

@harris_edouard

.

Thanks to Alex Turner and Vladimir Mikulik for pointers and advice, and for reviewing drafts of this sequence. Thanks to Simon Suo for his invaluable suggestions, advice, and support with the codebase, concepts, and manuscript. And thanks to David Xu, whose comment inspired this work.

Work was done while at Gladstone AI, which Edouard is a co-founder of.

🎧 This research has been featured on an episode of the Towards Data Science podcast. You can listen to the episode here.

1. Introduction

One major concern for AI alignment is instrumental convergence: the idea that an intelligent system will tend to pursue a similar set of sub-goals (like staying alive or acquiring resources), independently of what ...
A game of mattering
by KatjaGrace — 2022-10-12T08:50:18.097Z

When I have an overwhelming number of things to do, and insufficient native urge to do them, I often arrange them into a kind of game for myself. The nature and appeal of this game has been relatively stable for about a year, after many years of evolution, so this seems like a reasonable time to share it. I also play it when I just want to structure my day and am in the mood for it. I currently play something like two or three times a week.

The game

The basic idea is to lay out the tasks in time a bit like obstacles in a platformer or steps in Dance Dance Revolution, then race through the obstacle course grabbing them under consistently high-but-doable time pressure.

Here’s how to play:

Draw a grid with as many rows as there are remaining hours in your hoped for productive day, and ~3 columns. Each box stands for a particular ~20 minute period (I sometimes play with 15m or 30m periods.)

Lay out the gameboard: break the stuff you want to do into appropriate units, henceforth ‘items’. An item should fit comfortably in the length of a box, and it should be easy enough to verify completion. (This can be achieved through house rules such as ‘do x a tiny bit = do it until I have a sense that an appropriate tiny bit has been done’ as long as you are happy applying them). Space items out a decent amount so that the whole course is clearly feasible. Include everything you want to do in the day, including nice or relaxing things, or break activities. Drinks, snacks, tiny bouts of exercise, looking at news sites for 5 minutes, etc. Design the track thoughtfully, with hard bouts followed by relief before the next hard bout.

To play, start in the first box, then move through the boxes according to the time of day. The goal in playing is to collect as many items as you can, as you are forced along the track by the passage of time. You can collect an item by doing the task in or before you get to the box it is in. If it isn’t done by the end of the box, it gets left behind. However if you clear any box entirely, you get to move one item anywhere on the gameboard. So you can rescue something from the past, or rearrange the future to make it more feasible, or if everything is perfect, you can add an entirely new item somewhere.

I used to play this with tiny post-it stickers, which I would gather in a large moving pile, acting as a counter:

Now I just draw the who...
Calibration of a thousand predictions
by KatjaGrace — 2022-10-12T08:50:16.768Z

I’ve been making predictions in a spreadsheet for the last four years, and I recently got to a thousand resolved predictions. Some observations:

I’m surprisingly well calibrated for things that mostly aren’t my own behavior1. Here’s the calibration curve for 630 resolved predictions in that class:

I don’t know what’s up with the 80% category, but the average miscalibration of the eleven categories is <3%.

At risk of bragging, this seems wild to me. My experience of making these predictions is fairly well described as ‘pulling a number out of thin air’2. But apparently if you take all these conjured numbers, and look at the 45 of them that fell in the vicinity of 40%, then I implicitly guessed that 17.28 of those events would happen. And in fact 18 of them happened. WTF? Why wasn’t it eight of them or thirty-five of them? And that was only the fifth most accurate of the eleven buckets shown above! For predictions in the vicinity of 70%, I was off by 0.15%—I said 54.88 of those 80 things would happen, and in fact 55 of them happened.

Possibly people overall are just better calibrated than I thought. I had some remembered view that people’s asserted 90% confidence intervals were really 50% confidence intervals or something, but I can’t immediately find such evidence now, and I can find various graphs of groups of people being on average fairly calibrated. And the handful of PredictionBook users I could find with more than a thousand predictions are not hugely worse.

If you are curious about what I predicted, I put examples at the end of this post.

For the entire thousand predictions—the above plus 370 about my own behavior— I’m off by 6.25% on average (up from 2.95%) over the same eleven buckets.

As you may infer, I’m pretty bad overall at predicting my own behavior!

This is more what I expected of a calibration curve—broadly overconfident. And perhaps its general worseness is explained by the appeal of optimism in predicting oneself. But it’s a pretty weird shape, which seems less explicable. If I think there’s a 40% chance that I’ll do something, apparently it’s not happening. If you want it to happen, you should hope I change my mind and put 5% on it!

I’m not sure what is up with this particular strange shape. But note that making predictions about one’s own behavior has pa

...
My argument against AGI
by cveres — 2022-10-12T06:33:27.521Z

This is the third post about my argument to try and convince the Future Fund Worldview Prize judges that "all of this AI stuff is a misguided sideshow". My first post was an extensive argument that unfortunately confused many people.

(The probability that Artificial General Intelligence will be develop)

My second post was much more straightforward but ended up focusing mostly on revealing the reaction that some "AI luminaries" have shown to my argument
(Don't expect AGI anytime soon)

Now, as a result of answering many excellent questions that exposed the confusions caused by my argument, I believe I am in a position to make a very clear and brief summary of the argument in point form.

To set the scene, the Future Fund is interested in predicting when we will have AI systems that can match human level cognition: "This includes entirely AI-run companies, with AI managers and AI workers and everything being done by AIs." This is a pretty tall order. It means systems with advanced planning and decision making capabilities. But this is not the first time people predicted that we will have such machines. In my first article I reference a 1960 paper which states that the US Air Force predicted such a machine by 1980. The prediction was based on the same "look how much progress we have made, so AGI can't be too far away" argument we see today. There must be a new argument/belief if today's AGI predictions are to bear more fruit than they did in 1960. My argument identifies this new belief. Then it shows why the belief is wrong.

Part 1

Most of the prevailing cognitive theories involve classical symbol processing systems (with a combinatorial syntax and semantics, like formal logic). For example, theories of reasoning and planning involve logic like processes and natural language is thought by many to involve phrase structure grammars, like for example Python does.
Good old-fashioned AI was (largely) based on the same assumption, that classical symbol systems are necessary for AI.
Good old-fashioned AI failed, showing the limitations of classical symbol systems.
Deep Learning (DL) is an alternative form of computation that does not involve classical symbol systems, and its amazing success shows that human intelligence is not based on classical symbolic systems. In fact, Geoff Hinton in his Turing Award Speech proclaimed that "the success of machine translation is the last

...