trentmkelly/LessWrong-43k huggingface dataset:
Bureaucracy of AIs
by Logan Zoellner — 2022-06-09T23:03:06.608Z

In my post about AI Alignment Strategies, I strongly endorsed an approach I call Bureaucracy of AIs.

Specifically, I made this claim:

In the first place, [Bureaucracy of AIs] is the only approach (other than aligned by definition) that is ready to go today. If someone handed me a template for a human-level-AI tomorrow and said "build a super-intelligent AI and it needs to be done before the enemy finishes theirs in 6 months", this is the approach I would use.

I would like to give some more details here about what I mean by "Bureaucracy of AIs", and why I think it is a promising approach

Edit: Huge thanks to @JustisMills for helping to edit this. I've included many of his comments within.

What is a Bureaucracy of AIs?

Bureaucracy of AIs refers not to a specific algorithm or implementation, but rather to a family of strategies having the following properties:

Given a "weakly aligned" human-level oracle AI
Produce a super-human oracle AI
That is not catastrophically misaligned

These strategies work by aggregating human-level AIs into a greater whole, by strictly controlling the communication channels between these AIs, and by using game theory to prevent any individual AI from being able to successfully perform a treacherous turn.

What do you mean by a weakly aligned oracle AI?

Roughly speaking, weak alignment means: do all of the things any competent AI researcher would obviously do when designing a safe AI.

For instance, you should ask the AI how it would respond in various hypothetical situations, and make sure it gives the "ethically correct" answer as judged by human beings.

The AI should be programmed to cooperate with its creators, including when asked to deactivate itself.

In addition, the AI should not be excessively optimized toward any particular objective function. An AI that is optimized using gradient descent to make as many paperclips as possible is not weakly aligned, even if it otherwise seems non-threatening.

To the extent that the AI is trained with any objective at all, its basic desire should be to "be useful by giving truthful answers to questions."

The AI should not have any manifestly dangerous capabilities, such as: the ability to modify its own hardware or software, the ability to inspect its own software or hardware[1], the ability to inspect or modify its utility function, unrestricted ability to communicate with...
You Only Get One Shot: an Intuition Pump for Embedded Agency
by Oliver Sourbut — 2022-06-09T21:38:23.577Z

This is a short attempt to articulate a framing which I sometimes find useful for thinking about embedded agency. I noticed that I wanted to refer to it a few times in conversations and other writings.

A useful stance for thinking about embedded agents takes as more primitive, or fundamental, 'actor-moments' rather than (temporally-extended) 'agents' or 'actors'. The key property of these actor-moments is that they get one action - one opportunity to 'do something' - before becoming simply part of the history of the world: no longer actual.

This is just one of the implications of embedded agency, but sometimes pulling out more specific consequences helps to motivate progress on ideas. It is an intuition pump, and, as with the exemplar archetype for intuition pumps, it does not tell the whole story and should be used with caution.

The Cartesian picture

It is often convenient to consider a decision algorithm to persist through time, separated from its environment by a Cartesian boundary. The agent receives (perhaps partial) observations from the environment, performs some computation, and takes some action (perhaps updating some internal state, learning from observations as it goes). The resulting change in the environment produces some new observation and the process continues.

This is convenient because it is often empirically approximately true for the actors we encounter[1].

Only one shot

In contrast, in reality, any act, and the concomitant changes in the environment, impinge on the actor (which is after all part of the environment), even if only in a minor way[2].

Taking an alternative stance where we imagine an actor only existing for a moment - having a single 'one shot' at action - can prompt new insights. In this framing, a 'state update' is just a special case of the more general perspective of 'self modification', which is itself a special case of 'successor engineering'. And all of these are part of the larger picture implied by the stance taking as more fundamental not 'actor' but 'actor-moment': wherein an actor makes a decision and so unfolds a particular future world trajectory, which may - or may not - contain relevantly-similar actor-moments.[3]

This stance is somewhat unnatural for many actors we encounter (and consider worthy of attention and conversation investment), because for the most part these have been selected for a semblance of self-in...
Forestalling Atmospheric Ignition
by Lone Pine — 2022-06-09T20:49:08.177Z

It's April, 1945 and you're a high ranking scientist in the Manhattan Project. You also happen to have the ear of President Truman, so your words and actions have a major impact on world outcomes. Everyone has been working hard on the bomb, determined to defeat the Nazis, except that the Nazis are presently being defeated and you're not nearly as concerned about the war in the Pacific. (You know Japan will lose eventually, regardless of the bomb.)

Another scientist points out to you a disturbing concept: It is possible that the forthcoming A-bomb test (scheduled for July) will cause the atmosphere itself to ignite in a self-sustaining fusion reaction that would consume the entire globe. The consequences would presumably be the extinction of the entire human race and most life on Earth. You don't know if this could really happen, so you do some back of the envelope Fermi estimates and you at least can't disprove the possibility. You might think it's a 0.01%, 1%, 10%, 50%, 90% or near-100% chance. [Pick whichever makes the thought experiment most interesting to you.]

How do you react? Do you resign? What would you tell the president in your next conversation with him? Do you advocate that nuclear testing and the Manhattan Project be immediately halted? If it is halted and the war ends without a test being done, do you advocate for an international ban on testing until the question of atmospheric ignition can be resolved? And how would you propose to resolve the question, from a physical and engineering perspective? (Maybe we can test fusion temperatures inside a device like a particle accelerator or tokamac. The tokamac would be invented a few years later, so you could probably conceptualize it. Or maybe an atomic test could be done deep underground or on the moon.)
How Do Selection Theorems Relate To Interpretability?
by johnswentworth — 2022-06-09T19:39:59.988Z

One pretty major problem with today’s interpretability methods (e.g. work by

Chris Olah & co

) is that we have to redo a bunch of work whenever a new net comes out, and even more work when a new architecture or data modality comes along (like e.g. transformers and language models).

This breaks one of the core load-pillars of scientific progress: scientific knowledge is cumulative (approximately, in the long run). Newton saw far by standing on the shoulders of giants, etc. We work out the laws of mechanics/geometry/electricity/chemistry/light/gravity/computation/etc, and then reuse those laws over and over again in millions of different contexts without having to reinvent the core principles every time. Economically, that’s why it makes sense to pump lots of resources into scientific research: it’s a capital investment. The knowledge gained will be used over and over again far into the future.

But for today’s interpretability methods, that model fails. Some principles and tools do stick around, but most of the knowledge gained is reset when the next hot thing comes out. Economically, that means there isn’t much direct profit incentive to invest in interpretability; the upside of the research is mostly the hope that people will figure out a better approach.

One way to frame the

Selection Theorems

agenda is as a strategy to make interpretability transferable and reusable, even across major shifts in architecture. Basic idea: a Selection Theorem tells us what structure is selected for in some broad class of environments, so operating such theorems “in reverse” should tell us what environmental features produce observed structure in a trained net. Then, we run the theorems “forward” to look for the analogous structure produced by the same environmental features in a different net.

How would Selection Theorems allow reuse?

An example story for how I expect this sort of thing might work, once all the supporting pieces are in place:

While probing a neural net, we find a circuit implementing a certain filter.
We’ve (in this hypothetical) previously shown that a broad class of neural nets learn

natural abstractions

, so we work backward through those theorems and find that the filter indeed corresponds to a natural abstraction in the dataset/environment.
We can then work forward through the theorems to identify circuits corresponding to the same natural abstr

...
Progress links and tweets, 2022-06-08
by jasoncrawford — 2022-06-09T19:13:37.229Z
Link: https://rootsofprogress.org/links-and-tweets-2022-06-08

Best of my Twitter, May 31–June 6 (a few days behind, still ironing out the process… For the latest, follow me on Twitter: @jasoncrawford.)

After last time someone suggested embedding the tweets instead of linking to them, but I don't think I can post embeds here, and for my main blog I don't like the format as much. I’m sticking with links for now, but let me know if you’d rather scroll through a list of embeds (like this).

Links

A Reddit thread appreciating the engineering of everyday items
Jim Pethokoukis’s “Up Wing” agenda
Arcadia Science is experimenting with a new way of publishing scientific work
Michael Nielsen on vision papers
Ryan Petersen of Flexport talks to Tamara Winter about supply chains
Californians Against Pandemics is running a ballot initiative to invest in pandemic preparedness (h/t @NathanpmYoung)
High housing prices are driving low-skilled workers from high-productivity to low-productivity areas (Alex Tabarrok)
The B612 Foundation discovered more than 100 asteroids by analyzing years-old images (NYT)
“Subsidiarity is a principle of social organization that holds that social and political issues should be dealt with at the most immediate (or local) level that is consistent with their resolution.” (h/t Stewart Brand)

Tweets

The original NYC subway was completed faster than the average environmental impact statement
How many of WIRED’s “boom spoiler” scenarios from 1997 came true? (thread)
Some reasons I am skeptical of “harmony with nature” (thread)
But for the US Army Corps of Engineers, the Mississippi would no longer flow through New Orleans
Why we can’t have electric planes (yet)
One standard deviation in rule of law is associated with a 6.4-fold difference in per capita income

McCloskey on “the idea that ideas about ideas are unscientific”
19th-century medicine would unabashedly rely on anecdotal evidence
Vannevar Bush on the proper lab atmosphere
Who are the real revolutionaries?
In 1901 the market cap of the largest company was greater than the US national debt
Bacon on how to do a proper case-control study
Pixar founder Ed Catmull on how to see randomness as part of the beauty of life

Retweets

Suraj Patel has an abundance agenda (@SurajPatel)
“It seemed to be more a criticism of everything bad phrased as a prediction of doom” (@anderssandberg)
Productivity is the path to equity (@DKThomp)
To learn from mistakes, you have to build at the right pace (@stewartbr

...
If no near-term alignment strategy, research should aim for the long-term 
by harsimony — 2022-06-09T19:10:34.423Z

This is a small point about alignment strategy that gets mentioned occasionally but hasn't been stated explicitly as far as I can tell [1].

If there are no paths to alignment that can be implemented in the near-term, research should focus on building for the long-term. This is independent of AI timelines or existential risk.

In other words, if a researcher was convinced that:

AI will lead to high chance of existential catastrophe in the near-term

There are no approaches that can reduce the chance of existential catastrophe in the near-term

Then they are better off focusing on bigger projects that may pan out over a longer time period, even if they are unlikely to complete those projects due to an existential catastrophe. This is because, by assumption, work on near-term approaches is useless [2].

I think this implies that establishing a research community, recruiting researchers, building infrastructure for the field, and foundational work all have value even if we are pessimistic about AI risk.

That being said, it's uncertain how promising different approaches are, and some may plausibly be implemented in the near-term, so it makes sense to fund a diversity of projects [3]. It's also better to attempt near-term projects even if they are not promising rather than do nothing to try to avert a catastrophe.

Notes:

For an example of this point being made in passing, consider the end of rvnnt's comment here.

I don't actually hold this view.

Assuming these projects don't interfere with each other, but that's a problem for another day.
Operationalizing two tasks in Gary Marcus’s AGI challenge
by Bill Benzon — 2022-06-09T18:31:25.922Z

I’ve been thinking about the Gary Marcus/Elon Musk AGI challenge. As you may know, Elon Musk casually tweeted that he expected to see AGI in 2029. Gary Musk decided to take him seriously and bet him $100,000 that it wouldn’t happen. He specified five criteria in an open letter, stating that if some AI instance satisfied three of the five, Musk wins, otherwise Marcus wins. Others joined Marcus in the bet, bringing his side to $500,000, and Kevin Kelley offered to host the bet at Long Bets. Musk has yet to respond. But Metaculus has established betting markets for the tasks Marcus has specified.

The first two tasks

I want to consider the first two of the five criteria Marcus has established:

In 2029, AI will not be able to watch a movie and tell you accurately what is going on (what I called the comprehension challenge in The New Yorker, in 2014). Who are the characters? What are their conflicts and motivations? etc.
In 2029, AI will not be able to read a novel and reliably answer questions about plot, character, conflicts, motivations, etc. Key will be going beyond the literal text, as Davis and I explain in Rebooting AI.

Here’s what Marcus said about the first one in his New Yorker article:

... allow me to propose a Turing Test for the twenty-first century: build a computer program that can watch any arbitrary TV program or YouTube video and answer questions about its content—“Why did Russia invade Crimea?” or “Why did Walter White consider taking a hit out on Jessie?”

What would be considered acceptable answers to those questions? Though I watched Breaking Bad, it was so long ago that I haven’t the foggiest idea about that second question. But, when you consider the range of discussion about Russia’s reasons for the current invasion of Ukraine, I think we need consider a range of answers as being valid. Moreover, I think that issue needs to be carefully explored prior to the holding the contest.

I note that I have read movie reviews written by professional critics where the critic makes a mistake or mistakes about what happened in the film. I don’t know how reviewing is done these days when DVDs and streaming media can easily be made available to critics, but there was a time when the critic saw the film once and had to write the review based on that single viewing. Simply producing and accurate report of what happened based on a single viewing is not easy. Further...
Why it's bad to kill Grandma
by dynomight — 2022-06-09T18:12:01.131Z
Link: https://dynomight.substack.com/p/grandma

1.

In college, I had a friend who was into debate competitions. One weekend, the debate club funded him to go to a nearby city for a tournament. When I asked him how it went, he said:

Oh, I didn’t have enough time to prepare, so I just skipped the tournament and partied with some friends who live there.

I probably looked a little shocked—I was very idealistic in those days—so he reminded me of our earlier conversations about consequentialism:

This was the right thing to do. There was no way I could participate in the tournament, and they’d already paid for everything. This way I’m happy and they’re happy. Everyone wins!

I got a bit squinty-eyed, but I couldn’t see a flaw. We both agreed it was wrong that he didn’t prepare. But once that was done, was there nothing wrong with what he did, if consequences are all that matter?

2.

Recently, Arjun Panickssery wrote Just Say No to Utilitarianism, channeling ideas from Brian Caplan, Dan Moller, and Fake Nous. The basis of these articles are thought experiments like this one from Moller:

Grandma is a kindly soul who has saved up tens of thousands of dollars in cash over the years. One fine day you see her stashing it away under her mattress, and come to think that with just a little nudge you could cause her to fall and most probably die. You could then take her money, which others don’t know about, and redistribute it to those more worthy, saving many lives in the process. No one will ever know. Left to her own devices, Grandma would probably live a few more years, and her money would be discovered by her unworthy heirs who would blow it on fancy cars and vacations. Liberated from primitive deontic impulses by a recent college philosophy course, you silently say your goodbyes and prepare to send Grandma into the beyond.

Or here’s a classic, as phrased by Fake Nous:

Say you’re a surgeon. You have 5 patients who need organ transplants, plus 1 healthy patient who is compatible with the other 5. Should you murder the healthy patient so you can distribute his organs, thus saving 5 lives?

The argument is that in some situations, utilitarianism leads to crazy conclusions. Since no decent person would kill grandma, utilitarianism must be wrong.

I’d like to explain why I think this critique of utilitarianism is mistaken.

3.

To step back, what’s the point of ethics?

As far as I know, we live in a universe that’s indifferent to us. Fairness...
Modeling humanity's robustness to GCRs?
by T431 — 2022-06-09T17:34:50.722Z

Cross-posted to the EA-Forum

Epistemic Status: I have spent less than 15 minutes thinking about / querying the Internet for answers on what I am going to ask here. As such, it might be the case that this idea has already been discarded due to implementation or theoretical difficulties, or due to it being a mostly useless exercise. The following idea constitutes an attempt to get more concrete results on how humans might react and adapt to a catastrophic event where anywhere between 1-99.9% of the population perishes. I believe the expected value of this line of work is high, even if the probability of success is low. With more searching, I might be able to produce a preliminary experimental design, but whether I do this depends somewhat on the reaction of those in this forum.

The roughly 8 billion humans on Earth make up around 0.01% of Earth's life. Despite making up such a small proportion of life globally, humans have a strong presence across all continents (barring Antarctica), and roughly ~5% of the Earth's surface is habitable by humans. There has been increasing variability in Earth's climate over the last several million years, yet humans have historically adopted well to these changes. I imagine the past, current, and future climate-regions of Earth could be modeled (perhaps using some form of unsupervised learning). There are natural and anthropogenic global catastrophic risks that humanity faces. Many of these risks might leave regions of Earth uninhabitable to humans for some time. For example, nuclear war or super volcanic eruptions could leave many regions agriculturally unsustainable, engineered pandemics might result in large areas being quarantined indefinitely, and a large solar flare might result in some of Earth's climates becoming uninhabitable. These risks might also substantially shift the current climate-regions of Earth.

If a line of bacteria or virus could be bred to reflect human migration and habitation patterns (e.g., "Two-thirds of the world’s population is located within four major clusters: East Asia (China), South Asia (India and Indonesia), Southeast Asia, and Europe, with the majority in East and South Asia." would mean that roughly 2/3rds of the bacteria are located in 4 regions), then it might be possible to determine a rough estimate of humanity's robustness to the direct effects and subsequent environmental eff...
If there was a millennium equivalent prize for AI alignment, what would the problems be?
by Yair Halberstadt — 2022-06-09T16:56:10.788Z

If someone were to create one or a number of huge (e.g 1 to 10 million dollar) prizes for solving important problems related to AI alignment, it might have a decent chance of focusing a lot of researchers on the topic, which might help advance progress towards a solution/s, as well as make the field more respectable.

In order to avoid bad feelings, attract serious academics, and get meaningful results, the problems have to be clearly defined and have unambiguous solutions - something like "solve AI alignment" won't cut it.

The problems wouldn't have to be complete solutions to AI alignment. Instead they should be initial explorations on which further research and new branches of mathematics/computer science can be built. We want people to win those prizes, and quickly, so we can iterate with further prizes and attract more momentum.

So what problems can you think of that are concrete, achievable, and would represent useful milestones towards AI alignment?