trentmkelly/LessWrong-43k huggingface dataset:
Agency: What it is and why it matters
by Daniel Kokotajlo — 2021-12-04T21:32:37.996Z

[ETA: I'm deprioritizing completing this sequence because it seems that other people are writing good similar stuff. In particular, see e.g. https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality-is-the-tiger-and-agents-are-its-teeth and https://www.lesswrong.com/posts/pdJQYxCy29d7qYZxG/agency-and-coherence ]

This sequence explains my take on agency. I’m responding to claims that the standard arguments for AI risk have a gap, a missing answer to the question “why should we expect there to be agenty AIs optimizing for stuff? Especially the sort of unbounded optimization that instrumentally converges to pursuit of money and power.”

This sequence is a

pontoon bridge

thrown across that gap.

I’m also responding to claims that there are coherent, plausible possible futures in which agent AGI (perhaps better described as APS-AI) isn’t useful/powerful/incentivized, thanks to various tools that can do the various tasks better and cheaper. I think those futures are incoherent, or at least very implausible. Agency is powerful. For example, one conclusion I am arguing for is:

When it becomes possible to make human-level AI agents, said agents will be able to outcompete various human-tool hybrids prevalent at the time in every important competition (e.g. for money, power, knowledge, SOTA performance, control of the future lightcone...)

Another is:

We should expect Agency as Byproduct, i.e. expect some plausible training processes to produce agenty AIs even when their designers weren't explicitly aiming for that outcome.

I’ve had these ideas for about a year but never got around to turning them into rigorous research. Given my current priorities it looks like I might never do that, so instead I’m going to bang it out over a couple of weekends so it doesn’t distract from my main work. :/ I won't be offended if you don't bother to read it.

Outline of this sequence:

1. P₂B: Plan to P₂B Better - LessWrong

2. Agents as P₂B chain reactions

3. Interlude: Agents as automobiles

4. Gradations of agency

5. Why agents are powerful, & other conclusions

Incomplete list of related literature and comments:

Frequent arguments about alignment - LessWrong (A comment in which Richard Ngo summarizes a common pattern of conversation about the risk from agenty AI vs. other sorts of AI risk)

Joe Carlsmith, drawing on writings from others, had 20% credence that AI agents won't be powerful enough rel...
Misc. questions about EfficientZero
by Daniel Kokotajlo — 2021-12-04T19:45:12.607Z

Perhaps these can be thought of as homework questions -- when I imagine us successfully making AI go well, I imagine us building expertise such that we can answer these questions quickly and easily. Before I read the answers I'm going to think for 10min or so about each one and post my own guesses.

Useful links / background reading: The glorious EfficientZero: How it Works. Related comment. EfficientZero GitHub. LW discussion.

Some of these questions are about EfficientZero, the net trained recently; others are about EfficientZero the architecture, imagined to be suitably scaled up to AGI levels. "If we made a much bigger and longer-trained version of this (with suitable training environment) such that it was superhuman AGI..."

EfficientZero vs. reward hacking and inner alignment failure:

Barring inner alignment failure, it’ll eventually reward hack, right? That is, if it gets sufficiently knowledgeable and capable, it’ll realize that it can get loads of reward by hacking its reward channel, and then its core algorithm would evaluate that action/plan highly and do it. Right?
But fortunately (?) maybe there would be an inner alignment failure and the part of it that predicts reward would predict low reward from that action, even in the limit of knowledge and capability? Because it’s learned to predict proxies for reward rather than reward itself, and continued to do so even as it got smarter and more capable? (Would this happen? Why would this not be corrected by further training? Has some sort of proxy crystallization set in? How?)

EfficientZero approximates evidential decision theory, right?
EfficientZero is a consequentialist (in the sense defined here) architecture, right? It’s not, for example, updateless or deontological. For example, it has no deontological constraints except by accident (i.e. if its predictor-net mistakenly predicted super low reward for actions of type X, always, even in cases where actually a reasonable intelligent predictor would predict high reward.) Right?
What is the most complex environment AIs in the family of MuZero, EfficientZero, etc. have been trained on? Is it just some Atari game?
Roughly how many parameters does EfficientZero have? If you don’t know, what about MuZero? What about the biggest net to date from that general family? The EfficientZero paper doesn't give a direct answer but it describes the architecture in enough deta

...
Lars Doucet's Georgism series on Astral Codex Ten
by Lars Doucet — 2021-12-04T19:43:00.000Z
Link: https://astralcodexten.substack.com/p/does-georgism-work-is-land-really

[Lars Doucet won this year’s Book Review Contest with his review of Henry George’s Progress and Poverty. Since then, he’s been researching Georgism in more depth, and wants to follow up with what he’s learned. I’ll be posting three of his Georgism essays here this week, and you can read his other work at Fortress Of Doors]

Hi, my name's Lars Doucet (not Scott Alexander) and this is a guest post in an ongoing series that assesses the empirical basis for the economic philosophy of Georgism.

Part 0 - Book Review: Progress & Poverty
Part I - Is Land Really a Big Deal? 👈 (You are here)
Part II - Can Land Value Tax be Passed on to Tenants?
Part III - Can Unimproved Land Value be Accurately Assessed Separately from Buildings?

Extremely special thanks to Count Bla and Alexandra Elbakyan

For those of you wondering who this "Lars" guy is, I'm the Astral Codex Ten reader who reviewed Henry George's Progress & Poverty for the book review contest. Henry George is the founder of an economic philosophy known as Georgism which is principally concerned with the deprivations caused by unchecked rentiers. George is famous for promoting two specific policies, the Land Value Tax (LVT) and the Citizen's Dividend (what we would now call a Universal Basic Income).

I was shocked and humbled when this readership selected me as the winner. Even more shocking was how many people from around the world wrote to me about their interest in the article. Family, friends, and acquaintances for sure, but also a lot of total strangers–including business owners, activists, podcasters, online game designers, investors, even government officials from around the world. Scott's blog has way more reach than I realized.

This fills me with a sense of responsibility. If there's a chance people might make policy decisions based on my writing, I need to make sure I haven't been taken in by an argument that's just really persuasive; it had also better be true. What follows therefore is my best attempt at a fair, rigorous, and (where possible) empirical assessment of whether the claims of the Georgist movement stand up to scrutiny.

Let's admit some bias upfront. I'm a Georgist, and I would be happy to find this philosophy true and sad to find it false. But by George, what would make me really sad is to live in a world where Georgism is wrong but where I blissfully continue to believe in it anyway. In t...
What are the limitations on politically motivated relocation?
by Asgård — 2021-12-04T15:16:18.573Z

I know that this idea has been around for a very long time (probably since representative democracy), and that there are current attempts at it. The Free State Project considered Wyoming, Vermont and Alaska (amongst others), the least populous states in the Union, before settling on New Hampshire almost 20 years ago. They are maybe the most successful version of this, with a little over 5,000 members now residing in NH (out of the desired 20,000). It seems as if they have a total of 17 seats in the State House of Representatives, giving them a little over 4% of the 400 member House. This is an order of magnitude greater than (FSP population/NH pop), showing the outsized influence of a small group of politically oriented individuals who placed themselves strategically.
So, with remote work in vogue, with living costs climbing astronomically in liberal bastions, what do you see as the main limitations on politically motivated relocation?
And as a second question, what would your personal limitations be?
Should we postpone getting a booster due to Omicron, till there are Omicron-specific boosters?
by ChristianKl — 2021-12-04T12:46:09.431Z

Wai Dai writes over in the comments at Omicron Variant Post #1: We’re F***ed, It’s Never Over:

From your linked paper https://www.cell.com/trends/immunology/fulltext/S1471-4906(21)00177-0: [Wai Dai's quote from the paper in italics]

Experience from the IAV field indicates that multiple serial exposures to variant IAV strains might imprint (focus) the immune response to a narrow array of well-conserved viral epitopes relative to emerging subsequent strains and this might eventually reduce vaccine effectiveness. Randomizing multiple cohorts over time to either serial or partial immunization might also yield important neutralization data, although powering such studies for efficacy would be challenging. Larger observational studies of cohorts receiving serial immunizations might therefore also prove useful.

This suggests that it may be a good idea to avoid getting a booster now (with the ancestral spike protein) and wait for an Omicron-specific booster instead. (Use other precautions to avoid infection in the meantime.)

Given that this is an important question for many of us, I consider it useful to have the conversation on the top level instead of only in the comments of the other thread.
Can solipsism be disproven?
by  — 2021-12-04T08:24:00.340Z

I had a discussion with a find of mine via text messaging on whether solophisim can be proved or disproved. The conversation went a somewhat off track, but I believe it produced some conclusions that would be of interest to others. I shall refer to my friend as Thomas Aquinas. The conversation started with the following text message from Thomas.

TA) If solipsism were true then you wrote every song that you have ever heard including ones you don't like. Do you know enough about music to do so? Solipsism is a kind of perversion of "I think therefore I am" (cogito ergo sum). This assertion rests on specific presuppositions such as that the laws of logic work/are capable of revealing truth and that language has meaning. Why does logic work? Are numbers real? If logic and maths are real rather than being mere social constructs then what makes them real? TBH, he lost me after cogito ergo sum. But we'll get back to the last part.
NX) It rests on the assumption that the subconscious is much larger than we give it credit for. I often have dreams where I meet people who feel just as (if not more) real than people I meet in the waking world. I also end up going to strange wonderful places that feel more real than the waking places. It's an interesting coincidence you mention logic and math because I was just browsing this last night: http://us.metamath.org Metamath is a large database of hyperlinked formal math proofs
TA) Solipsism is only true if certain assumptions/presuppositions are granted correct? As for logic and maths there are two competing ideas. 1. they are human invented social constructs based on mutual agreement. 2. Maths and logic were discovered not invented. Arguing that these symbols mean these symbols because we agreed or claiming that it's just self evident breaks down into circularity. What I'm trying to do is to show that every argument, view, paradigm contains specific presuppositions that must be true for the system to work. The best we can do is compare the paradigms to see which ones hold up better under scrutiny. I'm still not following what he's getting at.
NX) My argument for solipsism being a possibility is the existence of dreams. Clearly everything we experience in a dream is from our own subconscious (or whatever you want to call it) after that we make an inductive leap to question wh

...
Shulman and Yudkowsky on AI progress
by Eliezer Yudkowsky — 2021-12-03T20:05:22.552Z

This post is a transcript of a discussion between Carl Shulman and Eliezer Yudkowsky, following up on a conversation with Paul Christiano and Ajeya Cotra.

Color key:

Chat by Carl and Eliezer	Other chat

9.14. Carl Shulman's predictions

[Shulman][20:30]

I'll interject some points re the earlier discussion about how animal data relates to the 'AI scaling to AGI' thesis.

1. In humans it's claimed the IQ-job success correlation varies by job, For a scientist or doctor it might be 0.6+, for a low complexity job more like 0.4, or more like 0.2 for simple repetitive manual labor. That presumably goes down a lot with less in the way of hands, or focused on low density foods like baleen whales or grazers. If it's 0.1 for animals like orcas or elephants, or 0.05, then there's 4-10x less fitness return to smarts.

2. But they outmass humans by more than 4-10x. Elephants 40x, orca 60x+. Metabolically (20 watts divided by BMR of the animal) the gap is somewhat smaller though, because of metabolic scaling laws (energy scales with 3/4 or maybe 2/3 power, so ).

https://en.wikipedia.org/wiki/Kleiber%27s_law

If dinosaurs were poikilotherms, that's a 10x difference in energy budget vs a mammal of the same size, although there is debate about their metabolism.

3. If we're looking for an innovation in birds and primates, there's some evidence of 'hardware' innovation rather than 'software.' Herculano-Houzel reports in The Human Advantage (summarizing much prior work neuron counting) different observational scaling laws for neuron number with brain mass for different animal lineages.

We were particularly interested in cellular scaling differences that might have arisen in primates. If the same rules relating numbers of neurons to brain size in rodents (

6

)

The brain of the capuchin monkey, for instance, weighing 52 g, contains >3× more neurons in the cerebral cortex and ≈2× more neurons in the cerebellum than the larger brain of the capybara, weighing 76 g.

[Editor’s Note: Quote source is “

Cellular scaling rules for primate brains

.”]

In rodents brain mass increases with neuron count n^1.6, whereas it's close to linear (n^1.1) in primates. For cortex neurons and cortex mass 1.7 and 1.0. In general birds and primates are outliers in neuron scaling with brain mass.

Note also that bigger brains with lower neuron density have longer communication times

...
[Linkpost] A General Language Assistant as a Laboratory for Alignment
by Quintin Pope — 2021-12-03T19:42:39.447Z

This is a linkpost for a recent paper from Anthropic: A General Language Assistant as a Laboratory for Alignment.

Abstract:

Given the broad capabilities of large language models, it should be possible to work towards a general-purpose, text-based assistant that is aligned with human values, meaning that it is helpful, honest, and harmless. As an initial foray in this direction we study simple baseline techniques and evaluations, such as prompting. We find that the benefits from modest interventions increase with model size, generalize to a variety of alignment evaluations, and do not compromise the performance of large models. Next we investigate scaling trends for several training objectives relevant to alignment, comparing imitation learning, binary discrimination, and ranked preference modeling. We find that ranked preference modeling performs much better than imitation learning, and often scales more favorably with model size. In contrast, binary discrimination typically performs and scales very similarly to imitation learning. Finally we study a `preference model pre-training' stage of training, with the goal of improving sample efficiency when finetuning on human preferences.

I think this is great work and shows encouraging results. I take the fact that alignability scales with model size as evidence for the most convenient form of the natural abstraction hypothesis:

Stronger AIs have better internal models of human values.
A few well chosen gradient updates can “wire up” the human values model to the AI’s output.

I’d be interested to see this sort of work extended to reinforcement learning systems, which I think are more dangerous than language models.

Edit: thanks to Daniel Kokotajilo's comment for prompting me to highlight some additional interesting results, now added here:

Another interesting result is the paper's investigation of "alignment taxes" (the potential loss in performance of aligned models compared with unaligned models):

The "HHH prompt" is a prompt they use to induce "Helpful, Honest and Harmless" output. Context distillation refers to training an unprompted model to imitate the logits generated when using the HHH prompt.

Their alignment tax results indicate the models suffer a small alignment penalty on Lambada, which is roughly constant with respect to model size (and therefore decreases in relative importance as model performance increases)....
Browser Engines
by jefftk — 2021-12-03T19:30:05.866Z

While there are many web browsers, they're almost all wrappers around
the only three actively developed full engines. Those are:

Gecko (source code):
Developed by Mozilla for Firefox. While it
previously powered several other browsers, at this point it looks like the Tor Browser and
Waterfox are the
only ones still tracking Gecko.

WebKit (source
code): Beginning as a fork of KHTML, it's primarily
developed by Apple for Safari.
It also powers GNOME
Web (Epiphany) and Otter.

On iOS, all browsers, including Firefox and Chrome, are powered by
WebKit. This is an Apple requirement
(2.5.6), and is controversial.

Blink,
part of Chromium
(source
code): Beginning as a fork of WebKit, it's primarily developed by
Google for Chrome. It also
powers Edge, Brave, Silk, Vivaldi,
Opera,
and most other browser projects (some via QtWebEngine). If someone
introduces a new browser and doesn't talk about its engine, it's very
likely they are using Blink.

You could also make the case for two others:

Flow: A
clean-slate closed-source browser and engine, designed to make
effective use of parallelism. This matters most on highly
resource-constrained devices, and if you have a Raspberry Pi you can
download and play with their demo. The reason I am
not including this engine above (yet!) is that while it's made
impressive progress, including handling Google
Docs, it's still a ways from done; the demo page has "we're not
quite ready to start receiving reports for rendering issues just yet,
these are mostly down to missing features, so as we add them you'll
see those issues steadily disappear over time." You can see which
features they have and haven't implemented on the web-platform-tests
dashboard. To get a very rough sense of where it stands today we
can compare the proportion of web platform tests that pass in each
browser:

Chrome
97%

Firefox
96%

Safari
94%

Flow
24%

Goanna / UXP:
A pre-Quantum
fork of Gecko, developed for Pale
Moon. There doesn't seem to be an official SCM: palemoon.org/sourcecode
links to a snapshot
archive and an archived
GitHub repo. It looks like it was on repo.palemoon.org
until sometime between May and October 2021 (wayback).
Probably ~2021-08-25, since that's the final
public commit. Based on the timing, probably deleted it as part of the MyPal
licensing
blowup? They
don't seem to participate in we

...
The Learning System
by Henrik Karlsson — 2021-12-03T19:08:01.019Z
Link: https://escapingflatland.substack.com/p/learningsystem

Civilization is kept afloat by a massive, decentralized body of often unseen knowledge.

This dark mass is made up of innumerable pieces of know-how accumulated by people mostly stumbling around, observing each other and the way things work. It's what's lodged in the head of the East German handyman that knows whom to bribe (and how) to get West German spare parts. It's the idiosyncratic thought patterns and norms picked up by the students of Gerty and Carl Cori, who won the Nobel Prize in 1947, six of which went on to win the prize in turn. It's your two-year-old learning the local language.

Since this body of knowledge is hard to quantify, and often even hard to spot, we tend to not think about it as deeply as it deserves. Talking about learning, and how to improve it, we often limit our discussions to the legible subset of knowledge transmission channels – that is, schools and universities. But as important as those institutions are, the education system plays only a minor part in knowledge reproduction.

As Lester Thurow argued in Education and economic equality: “Most actual job skills are acquired informally through on-the-job training after a worker finds an entry job and a position on the associated promotional ladder.”

This informal process, whereby job skills get picked up at work, social skills in friend groups, and language at home, is where virtually all knowhow is passed on. Education is the visible white tip kept up by this submerged iceberg; it is a formal order kept afloat by an informal one. Education cannot function without decentralized pathways of knowledge transfer, yet it often ignores them, and competes with them for resources.

As James Scott has observed:

The more highly planned, regulated, and formal a social or economic order is, the more likely it is to be parasitic on informal processes that the formal scheme does not recognize and without which it could not continue to exist, informal processes that the formal order cannot alone create and maintain.

In this essay, I'm going to look at the body of the iceberg – the decentralized processes that create and spread knowledge. To distinguish it from the education system, I'm going to refer to it as the learning system. Can we leverage it to more efficiently spread useful knowledge?

Decentralized Knowledge Reproduction

Whatever knowledge is spreading through society without a top-down mandate, is...