trentmkelly/LessWrong-43k huggingface dataset:
Science of Deep Learning - a technical agenda
by Marius Hobbhahn — 2022-10-18T14:54:35.406Z

I have written down a long list of alignment ideas that I’d be interested in working on. The ideas roughly boil down to “To make progress on alignment, we need to understand Deep Learning models and the process by which they arrive at their final parameters in much more detail than we currently do”.

I’m not the first person to think of most of these ideas and it builds on a lot of other people’s work. You should think of it more as a collection of existing resources and ideas than a new agenda. I also haven’t come up with the term “Science of Deep Learning”. I have already heard it being used by multiple people within the alignment community and many researchers are already working on parts of this agenda.
I obviously don’t own this agenda. The questions in this list are sufficient to keep hundreds of researchers busy for a while, so feel free to hop on. If you’re interested in collaborating just reach out.
Some of this research has the potential to increase capabilities more than alignment and the results should, in some cases, be kept private and only discussed with a small group of trusted peers. However, I think that most of the projects have a “

defender’s advantage

”, i.e. they increase alignment more than capabilities.
Whenever possible, Science of DL projects should have a direct benefit for alignment but I think our current understanding of DL is so bad that just increasing our general understanding seems like a good start.

Here is the link to the full version (comments are on, please don’t abuse it):

https://docs.google.com/document/d/1AyuTphQ31rLHDtpZoEwEPb4fWbZna1H3hGx_YUACxk4/edit?usp=sharing

The rest of this post is an overview copied from the doc. Feedback is welcome.

Overview - Science of Deep Learning

By Science of DL, I roughly mean “understanding DL systems and how they learn concepts” better. The main goal is to propose a precise and testable hypothesis related to a phenomenon in DL and then test and refine it until we are highly confident in its truth or falsehood. This hypothesis could be about how NNs behave on the neuron level, the circuit level, during training, during fine-tuning, etc. This research will almost surely at some point include mechanistic interpretability but it is not limited to it.
The refined statement after investigation can but doesn’t have to be of mathematical form as long as it...
My search for a reliable breakfast
by tomdekan — 2022-10-18T09:42:21.726Z
Link: https://www.tomdekan.com/my-search-for-a-reliable-breakfast

Unless I'm eating with other people, food for me is fuel.

This applies to breakfast particularly. Like a scrambled fighter jet, I want the refuelling process in the morning to be as fast as possible.

Not eating is not a sustainable option for me. Higher functions slow as my brain begins to run out of glucose. I become slightly less human and slightly more lungfish. My mood drops.

Here are the major contenders who have competed in my search over the past few years for a fast and filling breakfast.

Fried eggs

Immediately flawed. Making fried eggs for breakfast everyday uses too much equipment.

The dish is also too slow to prepare, both when watching the eggs cook and when fishing for broken egg-shell fragments in a hot frying pan.

There is too much time spent cleaning up pans afterwards. As a hobby, I find scrubbing oil-and-egg stuck to a metal pan quite satisfying. But it's a hobby that I only allow myself to indulge in on weekends and public holidays, not at the careful moment of breakfast.

Huel

Huel is quick to prepare and easy to eat. You put powder and water in a bottle and drink it.

Unfortunately, Huel is not filling for me, regardless of the quantity. My time-until-very-hungry-again ('TUVHA') is about 90 mins with Huel. This is way too short. After 90mins post-Huel, my desk-side wooden clocks start looking appetising.

Blood sugar spikes are another problem.

During the period that I had a continuous blood sugar implant in my arm (for fun) this year, I watched Huel spiking my blood sugar in real-time. After adding Huel into the gastrointestinal mix, my blood sugar would climb like an impatient rollercoaster car, before dropping. For me, consuming Huel's "nutritionally complete" advanced drink-food - marketed at professional athletes - is sadly like eating cake.

I like the idea of Huel, and turning my eating experience into something like filling up a car at a gas station. But I don't want to eat Victoria sponge for breakfast.

Preparation time is also deceptively high. It is fast to dump powder and water into a shaker. But cleaning out the protein shaker bottle once it contains a half-liquid, half-solid slurry takes more time. This time increases if you delay washing the bottle for a morning, causing the Huel to behave like papier mâché. Such breakfast powders would make reasonable insulating materials if there is a construction supplies shortage.

Amazfit Complete

V...
Infinite Possibility Space and the Shutdown Problem
by magfrump — 2022-10-18T05:37:12.111Z
Link: https://www.magfrump.net/blog/infinite-possibility-space-and-the-shutdown-problem

This post is a response to the recent Astral Codex Ten post, “

CHAI, Assistance Games, And Fully-Updated Deference

”.

A brief summary of the context, for any readers who are not subscribed to ACX or familiar with the shutdown problem:

The Center for Human-Compatible Artificial Intelligence (CHAI) is a research group at UC Berkeley. Their researchers have published on the shutdown problem, showing that “propose an action to humans and wait for approval, allowing shutdown” strictly dominates “take that action unilaterally” as well as “shut self down unilaterally” for agents satisfying certain assumptions.

MIRI discusses a counterexample, using a toy example where the AI has a finite number of policy options available, and expresses that “learn which of those finite set of options is best according to humans, then execute without allowing humans to shut it down” can dominate the course of “propose action to humans and wait for approval.”

I claim that the fact that the AI is “larger” than its value-space seems to me to be a critical ingredient in the AI being able to conclude that it has reached its terminal point in value-space. I posit that given a value-space that is “larger” than the AI, the AI will accept shutdown. Here I present an argument that, for at least one AI architecture and structure of value-space, the “propose action and allow shutdown” option should dominate much of the time.

Assume that a current AI model A contains a deep neural net connected to some decision procedure, of a specified, finite size (such as 16 layers with 1024 nodes each).

Then assume we can specify that human values are best specified by some ideal AI Z with the same structure[1], but with unknown size (e.g. Z includes a neural net with currently unknown number of layers and nodes in each layer.)

Further assume that we can specify that A’s action-space is to set the weights of its existing nodes, then propose actions to humans who can allow the actions or veto them by shutting down A.

This search space over all possible neural net sizes is infinite-dimensional. In particular, the “correct” value can always be an architecture which has additional layers or additional nodes in each layer, beyond what A is capable of expressing.

This space admits a well-formed prior probability distribution, and also presents a well-formed alternative...
Audition to perform in Bay Secular Solstice
by mingyuan — 2022-10-18T03:10:28.112Z

We’re looking to get new people involved with Secular Solstice this year! We need orators to give speeches, and singers and instrumentalists of all kinds to lead singalongs.

Solstice will be on December 17th in SF or Oakland, and there will be a dress rehearsal in the week prior. Those are the only two times you need to be available; most of the preparation can be done on your own, so performing doesn't necessarily need to be a significant time commitment. (You don't even necessarily need to live in the Bay!)

Audition by recording a video and submitting it through this form 🙂
Decision theory does not imply that we get to have nice things
by So8res — 2022-10-18T03:04:48.682Z

(Note: I wrote this with editing help from Rob and Eliezer. Eliezer's responsible for a few of the paragraphs.)

A common confusion I see in the tiny fragment of the world that knows about logical decision theory (FDT/UDT/etc.), is that people think LDT agents are genial and friendly for each other.[1]

One recent example is Will Eden’s tweet about how maybe a molecular paperclip/squiggle maximizer would leave humanity a few stars/galaxies/whatever on game-theoretic grounds. (And that's just one example; I hear this suggestion bandied around pretty often.)

I'm pretty confident that this view is wrong (alas), and based on a misunderstanding of LDT. I shall now attempt to clear up that confusion.

To begin, a parable: the entity Omicron (Omega's little sister) fills box A with $1M and box B with $1k, and puts them both in front of an LDT agent saying "You may choose to take either one or both, and know that I have already chosen whether to fill the first box". The LDT agent takes both.

"What?" cries the CDT agent. "I thought LDT agents one-box!"

LDT agents don't cooperate because they like cooperating. They don't one-box because the name of the action starts with an 'o'. They maximize utility, using counterfactuals that assert that the world they are already in (and the observations they have already seen) can (in the right circumstances) depend (in a relevant way) on what they are later going to do.

A paperclipper cooperates with other LDT agents on a one-shot prisoner's dilemma because they get more paperclips that way. Not because it has a primitive property of cooperativeness-with-similar-beings. It needs to get the more paperclips.

If a bunch of monkeys want to build a paperclipper and have it give them nice things, the paperclipper needs to somehow expect to wind up with more paperclips than it otherwise would have gotten, as a result of trading with them.

If the monkeys instead create a paperclipper haplessly, then the paperclipper does not look upon them with the spirit of cooperation and toss them a few nice things anyway, on account of how we're all good LDT-using friends here.

It turns them into paperclips.

Because you get more paperclips that way.

That's the short version. Now, I’ll give the longer version.[2]

A few more words about how LDT works

To set up a Newcomb's problem, it's important that the predictor does not fill the box if they predict that the ag...
Vertical Flutes
by jefftk — 2022-10-18T01:40:01.254Z

The standard ("

transverse")
flute has pretty terrible ergonomics: your arms are way out to the
side. Most woodwinds use a much more natural playing position where
the arms are comfortably in front of you, including some ("

end-blown")
flutes. In the case of the flute, however, this gives a pretty
different sound. Could we give up on the simple cylindrical design of
the flute to get some thing that sounds like a transverse flute and
plays like a transverse flute but is more ergonomic?

Yes! By putting a bend in the head joint it's possible to do just
that. This is a common configuration for bass flutes and lower:

Jupiter
JBF1100E bass flute

Pearl PFC-905 contrabass flute

You can get a replacement head joint for a standard concert flute,
though it's pricy:

Flute
Lab Vertical Headjoint

I don't know why this isn't more common? Do people not like that it
looks unusual? Katherine Saenger looked into this some (pdf)
and claims that the main options for concert flute today don't sound
great but better-sounding ones are possible. Has anyone tried these?
Why Weren't Hot Air Balloons Invented Sooner?
by Lost Futures — 2022-10-18T00:41:43.156Z
Link: https://lostfutures.substack.com/p/why-werent-hot-air-balloons-invented

"A medieval European painting by Giotto Di Bondone of a hot-air balloon flying above a city as onlookers watch" by Dalle-2

TL;DR

Hot air balloons certainly could have arrived at least decades earlier and there's a high chance they could have arrived centuries, possibly even millennia earlier.

Jason Crawford asks, "Why did it take so long to invent X?" The post is an invitation. It asks the reader to identify low-hanging fruit, inventions that could have been invented far earlier but weren't. Now, most technologies don't fit the bill. Not everything could have been invented long before it was. Many relied on recent advancements. The airplane only became possible once engines became powerful enough to lift them into the sky. But for other inventions such as the cotton gin or flying shuttle, neither scientific nor technological barriers seem present. To this list of mysteriously absent inventions, I'd like to propose an addition: The hot air balloon.

Ancient Balloons?

The notion of the ancient or medieval world having hot air balloons makes for exciting counterfactual history. Floating Roman military lookouts, ornate balloons arriving with Zheng He’s fleet in South Asia, balloon cartographers in the age of exploration… It’s all very eye-catching, but is it realistic?

In actual history, there were no Roman, Umayyadan, or Imperial Chinese hot air balloons. In fact, highly dubious claims of ancient South American balloons aside, no one in the classical or medieval world is known to have constructed a manned balloon. The first verified flight occurred in 1783, amid the industrial revolution. This early balloon was created by the Montgolfier brothers, French industrialists in the paper industry.

Why Manned Balloons Could Have Been Invented Sooner

Previous Inventions/Historical Precedent

Mini Hot Air Balloons

The sky lantern was invented by the Chinese by at least the 3rd century AD. Constructed from paper and a small fuel source, sky lanterns operated using the same principles as manned hot air balloons, well over a thousand years prior to the invention of the latter.

Human Flight Before Balloons

Beyond this, the concept of human flight was by no means novel to Dynastic China. Man-lifting kites were likely developed shortly before the time of the Sui Dynasty (581–618). Such kites became widespread enough throughout East Asia for the Japanese government to have...
Is GPT-N bounded by human capabilities? No.
by Cleo Nardo — 2022-10-17T23:26:43.981Z

I recently talked with someone about large language models and the risks they pose. They suggested that LLMs would be bounded by human intelligence/knowledge because LLMs learn from the internet text that humans have written.

Now, this was a very casual conversation (small-talk at a housewarming party), and they changed their mind a few minutes later. But perhaps this is an assumption people often make about LLMs.

Anyway, I'll briefly explain why LLMs are not bounded by human capacities.

I think the picture they had in their head was something like this:

Let's say that all the internet text has been written by humans and we train GPT-N to model the text. There are many ways that GPT-N might do this, but at the very least, GPT-N can just model the humans writing the internet text. This strategy is overkill — like swatting a fly with a sledgehammer — but it suggests that GPT-N has no incentive for more-than-human intelligence or more-than-human knowledge.

Concretely, GPT-N doesn't need to "know" things that humans don't know.
For example, humans currently don't know the genome of the next pandemic virus, and therefore the genome of the next pandemic virus won't appear in the internet text, and therefore GPT-N doesn't need to "know" the genome of the next pandemic virus.
Similarly, GPT-N doesn't need any skills that humans don't possess.
For example, humans currently can't prove really tricky theorems in mathematics, and therefore proofs of really tricky theorems won't appear in the internet text, and therefore GPT-N won't need to prove really tricky theorems.

Here's the problem. If GPT-N wants to perfectly model the internet text, it must model the entire causal process which generates it. But this causal process doesn't just include the human, but rather includes the entire universe that the human interacts with.

For example, suppose the internet text includes many chess games between Stockfish engines. If GPT-N wants to minimise cross-entropy loss on internet text, then GPT-N would need to learn chess at the level of Stockfish. It doesn't matter that it was a human who physically typed the games onto the internet. A model which could play superhuman chess would achieve lower cross-entropy loss on the current internet text.

(What exactly do I mean by "learn chess"? Either of the following two definitions will work:

There is a circuit in the weights that instantiate

...
EA & LW Forums Weekly Summary (10 - 16 Oct 22')
by Zoe Williams — 2022-10-17T22:51:04.175Z

Supported by Rethink Priorities

This is part of a weekly series - you can see the full collection here. The first post includes some details on purpose and methodology.

If you'd like to receive these summaries via email, you can subscribe here.

Podcast version: prefer your summaries in podcast form? A big thanks to Coleman Snell for producing these! Subscribe on your favorite podcast app by searching for 'Effective Altruism Forum Podcast'.

Top / Curated Readings

Designed for those without the time to read all the summaries. Everything here is also within the relevant sections later on so feel free to skip if you’re planning to read it all.

Lessons learned from talking to >100 academics about AI safety

by mariushobbhahn

The author talked to 100-200 people in academia about AI safety, from bachelor’s students to senior faculty, over several years. This post summarizes the key learnings.

Author’s tl;dr (lightly edited): “Academics are increasingly open to arguments about AI risk and I’d recommend having lots of these chats. I underestimated how much work related to aspects of AI safety (eg. interpretability) already exists in academia - we sometimes reinvent the wheel. Messaging matters, e.g. technical discussions got more interest than alarmism and explaining the problem rather than trying to actively convince someone received better feedback.”

Measuring Good Better

by MichaelPlant, GiveWell, Jason Schukraft, Matt_Lerner, Innovations for Poverty Action

Transcripts of 5-minute lightning talks by various orgs on their approach to measuring ‘good’. A very short summary of each is below:

Givewell uses moral weights to compare different units (eg. doubling incomes vs. saving an under-5s life). These are 60% based on donor surveys, 30% from a 2019 survey of 2K people in Kenya and Ghana, and 10% staff opinion.

Open Philanthropy’s global health and wellbeing team uses the unit of ‘a single dollar to someone making 50K per year’ and then compares everything to that. Eg. Averting a DALY is worth 100K of these units.

Happier Lives Institute focuses on wellbeing, measuring WELLBYs. One WELLBY is a one-point increase on a 0-10 life satisfaction scale for one year.

Founder’s pledge values cash at $199 per WELLBY. They have conversion rates from WELLBYs to Income Doublings to Deaths Avoided to DALYs Avoided, using work from some of the orgs above. This means they can ge...
A pragmatic metric for Artificial General Intelligence
by lorepieri — 2022-10-17T22:07:25.108Z
Link: https://lorenzopieri.com/pgii/

Current intelligence definitions are either partial or do not provide an actionable evaluation of intelligence, something tangible which can be used to quantify the potential of a given AGI. In absence of this definition we commonly define intelligence as the result accuracy of an intelligence test, for instance an IQ test, but this is prone to evaluation overfitting. We review existing definitions and then we consider a new heuristic definition, which we call PGI index or Pragmatic General Intelligence index, which relates intelligence to the ability to perform arbitrary tasks, but in which task accuracy is just one of the factors contributing to intelligence. In a toy model, we compute the PGI Index of an average human and of the Gato AI.
They gave LLMs access to physics simulators
by ryan_b — 2022-10-17T21:21:56.904Z
Link: https://arxiv.org/pdf/2210.05359.pdf

Over at Google, large language models have been plugged into physics simulators to help them share a world model with their human interlocutors, resulting in big performance gains. They call it Mind's Eye.
This is how the authors describe the work:

Correct and complete understanding of properties and interactions in the physical world is not only essential to achieve human-level reasoning (Lake et al., 2017), but also fundamental to build a general-purpose embodied intelligence (Huang et al., 2022). In this work, we investigate to what extent current LMs understand the basic rules and principles of the physical world, and describe how to ground their reasoning with the aid of simulation. Our contributions are three-fold:

• We propose a new multi-task physics alignment dataset, UTOPIA, whose aim is to benchmark how well current LMs can understand and reason over some basic laws of physics (§2). The dataset contains 39 sub-tasks covering six common scenes that involve understanding basic principles of physics (e.g., conservation of momentum in elastic collisions), and all the ground-truth answers are automatically generated by a physics engine. We find that current large-scale LMs are still quite limited on many basic physics-related questions (24% accuracy of GPT-3 175B in zero-shot, and 38.2% in few-shot).

• We explore a paradigm that adds physics simulation to the LM reasoning pipeline (§3) to make the reasoning grounded within the physical world. Specifically, we first use a model to transform the given text-form question into rendering code, and then run the corresponding simulation on a physics engine (i.e., MuJoCo (Todorov et al., 2012)). Finally we append the simulation results to the input prompts of LMs during inference. Our method can serve as a plug-and-play framework that works with any LM and requires neither handcrafted prompts nor costly fine-tuning.

• We systematically evaluate the performance of popular LMs in different sizes on UTOPIA before and after augmentation by Mind’s Eye, and compare the augmented performance with many existing approaches (§4.2). We find Mind’s Eye outperforms other methods by a large margin in both zero-shot and few-shot settings. More importantly, Mind’s Eye is also effective for small LMs, and the performance with small LMs can be on par or even outperform that of 100× larger vanilla LMs.

This seems lik...