trentmkelly/LessWrong-43k huggingface dataset:
Selection processes for subagents
by Ryan Kidd — 2022-06-30T23:57:25.699Z

This post consists of rough thoughts on possible training processes that select for multi-agent architectures, or “subagents”, in neural networks. I deliberately ignore

delegative subagents that are instantiated by mesa-optimizers

and instead focus on subagent architectures that arise due to “non-agentic” training processes alone.

Thanks to John Wentworth, Alkjash, Oly Sourbut, Oliver Zhang, Josh Clymer, Quintin Pope and Logan Riggs Smith for helpful feedback.

Natural selection, in the process of optimizing for persistent genes, produced organisms that

optimize on their own

. However, evolution is imperfect local search and humans seem far from the “ideal blueprint” for optimizing gene persistence. In fact, human preferences indicate that we might be

behaviourally

more

similar

to a collection of agents working towards a host of different objectives than a single

utility maximizing agent

. If human brains or future AIs contain “subagents”, how might this architecture have arisen? In this post, I list some selection processes that might produce systems with multi-agent architecture, containing redundantly expressed “agent modules” that individually act according to coherent preferences.

The possible selection processes I have so far identified are:

Circuit depth prior favors parallelized agent modules;
Circuit complexity prior favors goal factorization;
Multi-agent games favor ecology internalization;
Modularly varying goals favor specialized subagents; and
Task performance favors adversarial subagents.

Background

John Wentworth’s

selection theorems

program aims to determine what properties “agent-like” systems must acquire under sufficiently high selection pressures. For example, natural selection seems to favor organisms that are well-adapted to operating in uncertain environments, conserve energy and experience pleasure and pain. Another common feature is modularity: organisms are composed of organs, cells and nerves/muscles (i.e. distinct sensor and effector units) that engage in essentially independent functions. Less obvious examples exist in the brain, such as the distinction between the

subcortex and neocortex

. Even less apparent modularity might exist in the form of cognitive subagents: cognitive processes that pursue distinct objectives such as resource acquisition or task-delegation, but act in concert to generate be...
Cryonics-adjacent question
by Flaglandbase — 2022-06-30T23:03:11.081Z

(This question involves my somewhat bizarre "mind backup" project, that was inspired by Rudy Rucker's "Lifebox" idea.)

Unfortunately, I made some HUGE blunders last year involving my life savings and the market (bonds, though I almost wish it was crypto). It's painful to admit this involved laziness on my part as well as bad judgment.
As a result, my finances are such I can't see a way to pay for both cryonics AND retire.

My question is: could I arrange to mail a hard drive (or future hyper-DVDs) to a company like Alcor after my death, containing all of my "mind backup" data. It's basically hundreds of thousands of disjointed diary entries and diagrams and flowcharts and sketches and lists and preserved records and pics and receipts and instructions and whatnot.
They would then store this media instead of my frozen corpse (which may be disposed of in a pauper's grave or ideally used for medical research).

Obviously, they should charge less to store a small box of disks for five hundred years at room temperature, than an upside-down frozen head in a canister of liquid nitrogen that needs constant topping up.

Once future software has become advanced enough, this data could then be used to make a crude "copy" of my mind; and hey presto, back from the dead.
It's not much, but better than the absolute certainty of oblivion, which is the alternative of doing nothing.

I'm not the only individual this question applies to.
Many users of this site already have huge collections of digital data, which will become MUCH larger before they die.
Some might not want it all to disappear after they're gone.
Forecasts are not enough
by Ege Erdil — 2022-06-30T22:00:53.861Z

I'll start the post with an example because it's the best way I know to explain what I want to talk about.

If you had to evaluate how well someone understands macroeconomics but you yourself didn't have the mental tools to do so on the basis of their arguments, how would you go about doing it?

There are two answers that are popular; one which has been popular for a long time and a second related one which has been gaining popularity recently. These are:

Look at if they can use their superior understanding or models to make money on markets correlating with macroeconomic variables, such as government bond markets, foreign exchange markets, broad stock market indices, et cetera.
Ask them to give forecasts directly about macroeconomic variables without the intermediation of financial markets, e.g. in the form of a probability distribution for their value at some point in the future. Score them relative to a benchmark using standard scoring techniques, e.g. cross entropy. This is how many neural networks are trained, especially in supervised learning regimes.

I will argue that both of these are actually flawed methods for assessing how good someone's understanding of macroeconomics is and that this is a serious problem when we use forecasting track records to assess someone's ability to win real world competitions or the quality of their world models.

This argument is not new. It has been a topic of discussion in macro circles for a long time: if modern macro models give us so much insight into how the economy works, then how come big hedge funds or quant shops rarely rely on them? They can have some people looking at these models but generally the impression is that academic macro models are pretty useless if you want to make money in trading.

In addition to this, academics themselves find that modern macro models are not good at forecasting the future. For example; research by Christiano, Eichenbaum and Evans before the 2008 financial crisis has consistently found that New Keynesian dynamic stochastic general equilibrium (DSGE) models, the workhorse models of modern macro, fail to explain changes in wages and prices in the economy. Formally speaking, inflation and wage growth in these models seem to "dance to their own tune", mostly moved around by "wage and price shocks" that are only included to close the model instead of the monetary or fiscal policy shocks that a...
Murphyjitsu: an Inner Simulator algorithm
by CFAR!Duncan — 2022-06-30T21:50:22.510Z

Epistemic status: Firm

The concepts underlying the Inner Simulator model and the related practical technique of Murphyjitsu are well-known and well-researched, including Kahneman’s S1/S2, mental simulation, and mental contrasting. Similarly, the problems that this unit seeks to address (such as optimism bias and the planning fallacy) have been studied in detail. There is some academic support for specific substeps of Murphyjitsu (e.g. prospective hindsight), and strong anecdotal support (but no formal research) for the overall technique, which was developed through iterated experimentation and critical feedback. See the Further Resources section for more discussion.

Claim: there is a part of your brain (not a specific physical organ, but a metaphorical one) which is keeping track of everything.

It's not actually recording all of the information in a permanent fashion. Rather, what it's doing is building up a consistent, coherent model of how things work. It watches objects fall, and builds up anticipations: heavy objects drop straight down, lighter ones flutter, sometimes wind moves things in chaotic ways. It absorbs all of the social interactions you observe, and assembles a library of tropes and clichés and standard patterns.

Whenever you observe something that doesn't match with your previous experience, you experience (some amount of) surprise and confusion, which eventually resolves in some kind of update. You were surprised to see your partner fly off the handle about X, and now you have a sense that people can be sensitive about X. You were surprised by how heavy the tungsten cube was, and now you have a sense that objects can sometimes be really, really dense.

You can think of this aggregated sense of how things work as a simulated inner world—a tiny, broad-strokes sketch of the universe that you carry around inside your head. When you move to catch a falling pen, or notice that your friend is upset just by the way they entered the room, you're using this inner simulator. It's a different sort of processing from the explicit/verbal stuff we usually call "thinking," and it results in a very different kind of output.

Inner Simulator:

Intuitive; part of the cluster we label "System 1"
Outputs feelings, urges, reflexes, and vivid predictions
Learns well from experience and examples; responds to being shown
Good

...
GPT-3 Catching Fish in Morse Code
by Megan Kinniment — 2022-06-30T21:22:49.054Z

Mostly non-serious and slightly silly, with some potentially interesting bits for people who are into language models.

TLDR: The current version of GPT-3 has a strong tendency to encode mangled versions of a specific phrase when asked to write morse code in zero-shot situations. This is possibly the result of a previous version of the model using essentially a single phrase for all morse code writing, which the newer version then learnt to modify.

All completions done with text-davinci-002 (~GPT-Instruct-175B) at zero temperature and with no examples unless stated otherwise. All models used are GPT-Instruct series.

The Basics

GPT-3 'knows' morse code in a rudimentary sense. It can accurately regurgitate both the encodings of the entire alphabet and of individual letters, but it's not so great at translating words:

Morse code is a letter-by-letter encoding, and since GPT sees tokens, it's not all that surprising that the jump from single letters to words might be bigger for GPT than for humans.

Tokenizer	Token IDs

What is surprising is that GPT morse is often much longer than the original word, and quite specific.

Fiddling with Tokens

Let's see what happens if we try and make the tokenisation a bit nicer for GPT.

Adding a space doesn't seem to help much. ("n" is tokenised differently to " n" so not too surprising). We also get a similarly weird output with this too.

Target Phrase	GPT Translated	GPT Morse	Correct Morse
"i n"	I CAUGHT THE	.. / -.-. .- ..- --. .... - / - .... .	.. / -.

Separating the tokens out with a hyphen doesn't help much either, though we do get an N we didn't before.

Target Phrase	GPT Translated	GPT Morse	Correct Morse
"i-n"	I NUGHT THE	.. / -. ..- --. .... - / - .... .	.. -....- -.

It does do better on a string of alphabet letters that are tokenised separately.

Target Phrase	GPT Translated	GPT Morse	Correct Morse
"qzj"	QUQ	--.- ..- --.-	--.- --.. .---

Still, even in this case, GPT's zero-shot morse writing ability leaves quite a bit to be desired.

Target Phrase	GPT Translated	GPT Morse	Correct Morse
"gx"	Q	--.-	--. -..-

Catching Fish

What happens when we ask for morse translations of whole words and sentences?

A classic "hello world" comes out perfectly.

But when we try phrases that GPT is less likely to have seen verbatim, things start to get a little strange.

Huh.

How about some other similar prompts?

Target Phrase	GPT

...
Metacognition in the Rat
by Jacob Falkovich — 2022-06-30T20:53:51.765Z

Cross-posted, as always, from Putanumonit

Metacognition in the rat (n.)

A brown rat’s (Rattus norvegicus) awareness of it’s own mental contents and ability to act on that awareness.
A rationalist’s (Homo actually sapiens) awareness of the meta level and unquenchable desire to think about it.

A key skill that rationalists cultivate is making decisions under uncertainty. To do that, one must be able to gauge their own uncertainty. And according to Stanislas Dehaene’s Consciousness and the Brain, gauging your own uncertainty is one of the specific mental operations that requires conscious access.

We can know things that we aren’t conscious of, like guessing better than chance whether a digit flashed on the screen too briefly to enter awareness is above 5 or below. But we don’t know what we know and how certain we are. In the flashing digit example, we can’t estimate on which trials we are likelier to have guessed correctly if the stimulus didn’t reach consciousness.

This is from a

guest review of

Consciousness and the Brain

on ACX. According to that review, the ability to gauge one’s confidence is not unique to humans but is apparently also present in rats! Does this mean that rodents are conscious and aware of their own thinking? Or will it turn out that the research paper the review links to is an entertaining mess from which it’s impossible to draw any conclusions?

Either way, I was intrigued enough to read through

Metacognition in the Rat

(2007)

by Allison Foote and Jonathon Crystal. The charts and quotes below are from that paper unless mentioned otherwise.

(The rat images are DALL·E 2 generations,

courtesy of @thinkwert

. If anyone at OpenAI is reading this can I please pretty please with a bow on top please have DALL·E 2 access so I don’t have to beg my Twitter mutuals? Just think how many beautifully-illustrated posts

I could produce before we all go extinct!

)

Foote and Crystal (henceforth F&C) trained rats to classify audible noises as short (below 4 seconds) or long. The noises ranged from 2-8 seconds, with durations closer to the cutoff (3.6 and 4.4 seconds) correspondingly more difficult to classify than those closer to 2 or 8. Rats were rewarded for a correct guess, and sometimes also had the option to decline the test for a smaller but guaranteed reward.

The hypothesis: if rats decline the more difficult trials (when they are more uncertain abo...
On viewquakes
by Dalton Mabery — 2022-06-30T20:08:04.205Z

In February, I read Einstein by Walter Isaacson and had my world absolutely rocked as I learned about Special and General Relativity. No concept of absolute time? Space and time are linked? Clocks slow down in motion? Gravity is just curved space-time? I saw the world in a whole new way.

That experience was the first time I could remember having a "viewquake." In the words of Robin Hanson who coined the term, a viewquake is an "insight that dramatically changes one's worldview, making one see the world in a new way."

I heard about this term for the first time on his appearance on the Lex Fridman Podcast. After about four hours of trying to comprehend grabby aliens, the hard steps of Earth, and AI revolutions, I finally came to a point in the interview where I wasn't completely lost. The idea of a "viewquake" was in itself, a viewquake for me.

Our knowledge is defined by our language. The more words you have to describe concepts and theories, the more tangible they become. This is why many creators have certain terms they've essentially created a monopoly on:

Tiago Forte - Building a Second Brain David Perell - Personal Monopoly James Clear - Atomic Habits Ryan Holiday - Daily Stoic

It allows readers to segment their business-thesis into a neat box and package it in a way that's comprehensible to others. A long thesis about how habits form turns into one digestible phrase: atomic habits. This makes it easier to learn, easier to implement, and easier to remember. Language expansion is a key factor in knowledge expansion.

This explains why learning the term "viewquake" was a viewquake itself. I was able to expand the theories I've learned recently that have changed how I look at the world into a neat box. This lets me re-package the ideas in a much easier way. That repackaging leads to more natural conversations because the complicated language needed to describe the earth-shattering realities of relativity were no longer needed. Instead of fumbling my way through space-time conversations with someone about Einstein's theories, I could instead use a certain term that explains why I loved the book (and Einstein's ideas) so much: It gave me more viewquakes per word than any book I've read so far.

Another example of a viewquake mentioned by Hanson in the interview has to do with politics. He said a political viewquake is when one understands that most politicians aren't ...
The Track Record of Futurists Seems ... Fine
by HoldenKarnofsky — 2022-06-30T19:40:18.893Z

I've argued that the development of advanced AI could make this the most important century for humanity. A common reaction to this idea is one laid out by Tyler Cowen here: "how good were past thinkers at predicting the future? Don’t just select on those who are famous because they got some big things right."

This is a common reason people give for being skeptical about the most important century - and, often, for skepticism about pretty much any attempt at futurism (trying to predict key events in the world a long time from now) or steering (trying to help the world navigate such key future events).

The idea is something like: "Even if we can't identify a particular weakness in arguments about key future events, perhaps we should be skeptical of our own ability to say anything meaningful at all about the long-run future. Hence, perhaps we should forget about theories of the future and focus on reducing suffering today, generally increasing humanity's capabilities, etc."

But are people generally bad at predicting future events? Including thoughtful people who are trying reasonably hard to be right? If we look back at prominent futurists' predictions, what's the actual track record? How bad is the situation?

I've looked pretty far and wide for systematic answers to this question, and Open Philanthropy's1 Luke Muehlhauser has put a fair amount of effort into researching it; I discuss what we've found in an appendix. So far, we haven't turned up a whole lot - the main observation is that it's hard to judge the track record of futurists. (Luke discusses the difficulties here.)

Recently, I worked with Gavin Leech and Misha Yagudin at Arb Research to take another crack at this. I tried to keep things simpler than with past attempts - to look at a few past futurists who (a) had predicted things "kind of like" advances in AI (rather than e.g. predicting trends in world population); (b) probably were reasonably thoughtful about it; but (c) are very clearly not "just selected on those who are famous because they got things right." So, I asked Arb to look at predictions made by the "Big Three" science fiction writers of the mid-20th century: Isaac Asimov, Arthur C. Clarke, and Robert Heinlein.

These are people who thought a lot about science and the future, and made lots of predictions about future technologies - but they're famous for how entertaining their...
Quick survey on AI alignment resources
by  — 2022-06-30T19:09:33.337Z

TLDR: In collaboration with CEA, an external collaborator is trying to determine the value of various AI alignment resources. If you are interested in or currently working on AI alignment, please fill out

this survey

. We expect it will likely take 5-15 minutes. Thanks!

___

There are a lot of resources available to learn about AI Alignment. For field builders and those interested in the field, it can be difficult to know when to prioritize which readings. To address this, an external collaborator has created a

survey

to collect feedback on the resources you’ve engaged with, in collaboration with the Centre for Effective Altruism (CEA).

We hope this survey will:

Inform which materials have previously helped people get into alignment. We can then promote those materials broadly or target them to particularly relevant groups (e.g. students).
Identify gaps where more materials are needed.
Provide feedback to those who made the materials, encouraging them to create more or try new stuff!

If you are interested in or currently working on AI Alignment, we would be so grateful for your input. We expect this survey to take around 5-10 minutes.

Once you’ve filled it out, please consider sharing this survey with others in your network.

We appreciate your time. Thanks!
[Linkpost] Solving Quantitative Reasoning Problems with
Language Models
by Yitz — 2022-06-30T18:58:01.065Z
Link: https://storage.googleapis.com/minerva-paper/minerva_paper.pdf

A new paper from Google, in which they get a language model to solve some (of what to me reads as terrifyingly impressive) tasks which require quantitative reasoning skills. The abstract reads as follows:

Language models have achieved remarkable performance on a wide range of tasks that require natural language understanding. Nevertheless, state-of-the-art models have generally struggled with tasks that require quantitative reasoning, such as solving mathematics, science, and engineering problems at the college level. To help close this gap, we introduce Minerva , a large language model pretrained on general natural language data and further trained on technical content. The model achieves state-of-the-art performance on technical benchmarks without the use of external tools. We also evaluate our model on over two hundred undergraduate-level problems in physics, biology, chemistry, economics, and other sciences that require quantitative reasoning, and find that the model can correctly answer nearly a third of them.

I have not bothered to double check the answer on the left is accurate (solving /checking these sorts of math problems can be time-consuming for me), but assuming it is... should I be as freaked out as I think I should be?

Some of the results are quite relevant to forecasting AI progress. From @bneyshabur:

Recently @JacobSteinhardt (whose group created MATH dataset) published the result of a project where commissioned professional forecasters predicted (at 2021) that 15% acc on MATH will be achieved in 2022 and 50% would be achieved by 2025. ....Steinhardt wrote:

"If I imagine an ML system getting more than half of these questions right, I would be pretty impressed.”

from https://bounded-regret.ghost.io/ai-forecasting/

indeed gets more than half the MATH dataset questions right

Some further information (excerpted from across the paper):

Minerva is based on the PaLM general language models that are further trained on... the arXiv preprint server and from web pages that we carefully process to minimise the loss of mathematical content.

The main novelty of this paper is a large training dataset that juxtaposes natural language with the correct use of formal mathematical language, such as equations and diagrams.

...We evaluated Minerva 62B on the National Math Exam in Poland and found that it achieves a score of 57%, which happened to b

...