trentmkelly/LessWrong-43k huggingface dataset:
List #3: 
Why not to assume on prior that AGI-alignment workarounds are available
by Remmelt — 2022-12-24T09:54:17.375Z

A friend in technical AI Safety shared a list of cruxes for their next career step.
One premise implicit in their writing was that a long-term safe AGI can be build in the first place (or that we should resolve to build safe AGI, since if AGI is inevitable then there is almost nothing else we can do but to resolve to make it safe).

Copy-pasting a list I wrote in response (with light edits):

Why long-term AGI safety might be an unsolvable problem:

Most specifiable problems are unsolvable:
On prior, most engineering problems we can specify (as combinations of desiderata) are impossible to solve.

Survivorship bias of solvable problems:
There are of course higher-level engineering principles for picking what’s solvable, but in practice a lot of engineering looks like tinkering and failing (most inventions seem to have come out such a process).
The engineering problems that we hear about later are mostly ones that turned out possible (or are ones that people have kept on working/trying to solve). So there is a survivorship bias in that most of the problems engineers hear about appear to be solvable (on top of that, there is a motivated focus on remembering the problems that turned out possible after all; try googling "impossible problems" – I was surprised).

Outside the reference class:
Would we presume that one of the most complex, intractable and dangerous engineering problems we could think of – long-term AGI safety – would in fact be solvable?

Neglected research/portfolio diversification:
Few AI safety researchers though raised the question whether any dynamic described by an AGI threat model falls outside a theoretical limit of controllability, ie. whether given unsafe dynamic is uncontrollable. A sequence in which new research is often done: specify a novel threat model, jump to trying to solve it, and then refute solutions that turned out unsound (done either by multiple people, or all in one person all in one person).
So given that the majority of the (short, non-reasoned-through) claims I've at least read from AIS researchers on this crucial consideration are that technical AGI alignment/safety is possible, what direction should we expect researchers' beliefs here to move on average if they open-mindedly and rigorously researched this consideration?

Unreliable intuitions:
Nor can we rely on the confidently voiced intuitio

...
List #2: 
Why coordinating to align as humans to not develop AGI is a lot easier than, well... coordinating as humans with AGI coordinating to be aligned with humans
by Remmelt — 2022-12-24T09:53:19.926Z

A friend in technical AI Safety shared a list of cruxes for their next career step.
A sub-crux for non-stoppable AI progress was that they could not see coordination happen without a world government and mass surveillance.
I asked:

Are the next beliefs and courses of action in your list heavily reliant upon this first premise/belief? ie. If you gradually found out that there are effective methods for groups around the world to govern themselves to not build dangerous auto-scaling/catalysing technology (such as AGI) where those methods do not rely upon centralised world governance or mass surveillance, would that change your mind as to what you / the AIS community needs to focus efforts on?

Copy-pasting a list I then wrote in response (with light edits):

Why coordinating to align as humans to not develop AGI is a lot easier than, well coordinating as humans with AGI coordinating to be aligned with humans

Adding to principal-agent problem complexity:

AI-alignment is roughly an instance of the principal-agent problem (principal – all of many humans; agent – AI).
Human-human conflict, human-human arms races, or humans selling or releasing tech that harms other humans are also instances of principal agent problems.
How are you going to solve all that distributed complexity – all those principal-agent problems expressing themselves in many different ways for many different humans living in many different circumstances – with one general solution?
You cannot just say – ah but AGI has godly powers and can/will solve all of that for us (as in 'Artificial Godly Intelligence'). To me that does not clarify how adding yet another but much more powerful and much more alien agent in the mix is going to help solve all those other principal-agent problems in the mix. ‘Beneficial AGI that takes over the world’ raises many more new questions to me than it answers.

Coordinating with aliens:

"Well that's just not going to happen, the capitalist forces driving R&D cannot be overcome, so the best we can do is work tirelessly for the small but nonzero chance that we can successfully align AGI."
^– A researcher wrote me that they kept getting this reaction when arguing that AGI safety might be impossible.
I also got this reaction a few times.
It makes me wonder how these people mentally represent "AGI" to be like.

Some common representational a

...
List #1: 
Why stopping the development of AGI is hard but doable
by Remmelt — 2022-12-24T09:52:57.266Z

A friend in technical AI Safety shared a list of cruxes for their next career step.
The first crux was that they did not believe that progress on AI can be expected to stop.

Copy-pasting a list that I compiled in response (with light edits):

Why stopping the development of AGI is hard but doable

History of technological restraint by institutions:

From Matthijs Maas' essay:

“In short: some expect that we could not meaningfully slow or halt the development of A(G)I even if we expect extreme risks.
“Yet there is a surprisingly diverse historical track record of technological delay and restraint, even for strategically promising technologies that were seen as 'obvious' and near-inevitable in their time.
“Epistemic hurdles around studying 'undeployed' technologies make it likely that we underestimate the frequency of restraint decisions, or misinterpret their causes. From an outside view, this should lead us to be slightly more optimistic about the viability of restraint for future technologies.

A lot of tech does not get developed:

From Katja Grace's Slow Down AI post:

“There seems to be a common thought that technology is a kind of inevitable path along which the world must tread, and that trying to slow down or avoid any part of it would be both futile and extreme.
“But empirically, the world doesn’t pursue every technology—it barely pursues any technologies.

Sucky technologies

For a start, there are many machines that there is no pressure to make, because they have no value. Consider a machine that sprays shit in your eyes. We can technologically do that, but probably nobody has ever built that machine.

Extremely valuable technologies

It doesn’t look like it to me. Here are a few technologies which I’d guess have substantial economic value, where research progress or uptake appears to be drastically slower than it could be, for reasons of concern about safety or ethics: ...

Coordination is not miraculous world government, usually

The common image of coordination seems to be explicit, centralized, involving of every party in the world, and something like cooperating on a prisoners’ dilemma: incentives push every rational party toward defection at all times, yet maybe through deontologic

...
The case against AI alignment
by andrew sauer — 2022-12-24T06:57:53.405Z

Trigger warning: Discussion of seriously horrific shit. Honestly, everything is on the table here so if you're on the lookout for trigger warnings you should probably stay away from this conversation.

Any community of people which gains notability will attract criticism. Those who advocate for the importance of AI alignment are no exception. It is undoubtable that you have all heard plenty of arguments against the worth of AI alignment by those who disagree with you on the nature and potential of AI technology. Many have said that AI will never outstrip humans in intellectual capability. Others have said that any sufficiently intelligent AI will “align” themselves automatically, because they will be able to better figure out what is right. Others say that strong AI is far enough in the future that the alignment problem will inevitably be solved by the time true strong AI becomes viable, and the only reason we can’t solve it now is because we don’t sufficiently understand AI.

I am not here to level criticisms of this type at the AI alignment community. I accept most of the descriptive positions endorsed by this community: I believe that AGI is possible and will inevitably be achieved within the next few decades, I believe that the alignment problem is not trivial and that unaligned AGI will likely act against human interests to such an extent as to lead to the extinction of the human race and probably all life as well. My criticism is rather on a moral level: do these facts mean that we should attempt to develop AI alignment techniques?

I say we should not, because although the risks and downsides of unaligned strong AI are great, I do not believe that they even remotely compare in scope to the risks from strong AI alignment techniques in the wrong hands. And I believe that the vast majority of hands this technology could end up in are the wrong hands.

You may reasonably ask: How can I say this, when I have already said that unaligned strong AI will lead to the extinction of humanity? What can be worse than the extinction of humanity? The answer to that question can be found very quickly by examining many possible nightmare scenarios that AI could bring about. And the common thread running through all of these nightmare scenarios is that the AI in question is almost certainly aligned, or partially aligned, to some interest of human origin.

Unaligned AI will kill y...
Content and Takeaways from SERI MATS Training Program with John Wentworth
by RohanS — 2022-12-24T04:17:21.094Z

Introduction

I participated in the training program for John Wentworth’s stream of

SERI MATS

, and overall, it was a very good experience! This post is meant to convey the content of the workshops, so that others can carry out the exercises on their own if they would like, along with my thoughts and takeaways from them. Most of these are in the day-by-day breakdown below, but here is a high-level summary of my experience:

The workshops gave me a sense of some skills that John thinks are useful for doing AI alignment research. (My guess is that John may emphasize some of these skills more than other alignment researchers would, but many of them are uncontroversial.) The exercises helped me acquire and practice some of those skills, but they also gave me ideas about how to further develop them going forward.
Some of the prompts for the exercises were very effective at getting me to generate interesting ideas. I wasn’t able to do all the exercises as thoroughly as I would have liked, so I may want to revisit the ones that seem most valuable. These include (but are not limited to):

Learning and distilling a math topic slightly beyond your grasp (Week 3)
Turning an intuitive concept into a formal mathematical object (Week 4, Day 1)
Translating an intuitive high-level claim into a formal mathematical statement, then proving it (Week 4, Day 4)
Making good plans to contribute to alignment (Week 6)

I learned that at this stage, I prefer theory and writing to experiment.

Day-by-Day Breakdown

Participants in the training program met 4 days each week on Gathertown, usually for about 1 hour per workshop (but sometimes up to about 2), for 6 weeks. Below is a day-by-day breakdown of what the workshops involved, with comments about my experiences with them. Naturally, not everyone’s experiences were the same as mine, so take what I say with a grain of salt. Many of the exercises can be carried out individually or in small groups, so some readers of this post may want to try them out! I’m happy to try to provide more details if any of the workshop descriptions are confusing.

(Italicized text describes the content of the workshop (usually John gave us these instructions at the start), regular text describes my thoughts and takeaways.)

Week 1 - Intro Week

Week 1, Day 1 - Intros and Alignment Disagreements

Find a partner, introduce yourselves to each other, and find something you d

...
Löb's Lemma: an easier approach to Löb's Theorem
by Andrew_Critch — 2022-12-24T02:02:42.137Z

Related to: Löb's Theorem for implicit reasoning in natural language: Löbian party invitations

tl;dr: Löb's Theorem is much easier to grok if you separate the parts of the proof that use the assumption □p→p from the parts that don't. The parts that don't use □p→p can be extracted as a stand-alone result, which I hereby dub "Löb's Lemma".

Here's how it works.

Key properties of ⊢ and □

Here is some standard notation. In short, ⊢ is our symbol for talking about what PA can prove, and □ is shorthand for PA's symbols for talking about what (a copy of) PA can prove.

"⊢ 1+1=2" means "Peano Arithmetic (PA) can prove that 1+1=2". No parentheses are needed; the "⊢" applies to the whole line that follows it. Also, ⊢ does not stand for an expression in PA; it's a symbol we use to talk about what PA can prove.
"□(1+1=2)" basically means the same thing. More precisely, it stands for a numerical expression within PA that can be translated as saying "⊢ 1+1=2". This translation is possible because of something called a Gödel numbering which allows PA to talk about a (numerically encoded version of) itself.
"□X " is short for "□(X)" in cases where "X" is just a single character of text.
"X⊢Y" means "PA, along with X as an additional axiom/assumption, can prove Y". In this post we don't have any analogous notation for □.

The proofs here will use the following standard properties of ⊢ and □ (source: Wikipedia), which respectively stand for provability and provability encoded within arithmetic.

(necessitation) From ⊢A, conclude ⊢□A. Informally, this says that if A can be proven, then it can be proven that it can be proven (by just writing out and checking the proof within arithmetic). Note that from X⊢A we cannot conclude X⊢□A, because □ still means "PA can prove", and not "PA+X can prove".
(internal necessitation) ⊢□A→□□A . If A is provable, then it is provable that it is provable (basically the same as the previous point).
(box distributivity) ⊢□(A→B)→(□A→□B). This rule allows one to apply modus ponens inside the provability operator. If it is provable that A implies B, and A is provable, then B is provable.
(deduction theorem) From A⊢B, conclude ⊢A→B: if assu

...
Durkon, an open-source tool for Inherently Interpretable Modelling
by abstractapplic — 2022-12-24T01:49:58.684Z

or, what abstractapplic has been tinkering with for the past two years, and how it can be applied to fictional card games

Motivation

Interpretability of ML models is important. If people don’t understand how a model makes its predictions, they will be (justifiably!) unwilling to use those predictions for anything that matters, especially when the costs of failure are great[1].

LessWrong talks a lot about this in the context of Neural Nets and AIs, but the problem remains present on a much lower level. If you want to model a system with interactions between features[2] and/or nonlinearities in feature effects[3] (i.e. any real-world system), linear regression won’t be sufficient, but more complex models like Random Forests can’t explain themselves in ways humans can understand.

There are some tools and tricks which aim to get only-slightly-worse model performance vs black-box models while remaining inherently human-interpretable: Segmented Linear Regression, Microsoft’s InterpretML package, Dr. Cynthia Rudin’s entire career, and “just build it by hand in Excel lol”. To this list, I add my own creation, Durkon[4].

It’s time to Duel!

To give you a sense of how Durkon works, I’m going to use one of aphyer’s D&D.Sci games, and demonstrate how my library works by (re)assuming the role of someone trying to win this challenge. To give a brief summary (spoilers for that game from this point on, btw): you need to pick a deck for a card game where you don’t know the rules, but do have a record of which decks won and lost a list of previous games.

For simplicity, let’s say I already figured out the game is mostly about synergy within your own deck instead of counters to your opponents’ cards, but haven’t progressed further than that. So I’m going to train a model which predicts probability of success from the count of each card type in the deck.

Durkon is mostly built for regression problems, but can do binary classification tasks as well. I run the “combined ratio” model-building functions, and get back a bunch of Partial Dependency Plots that look like this:

There are also interactive versions of these graphs but I couldn't make LessWrong embed them.

Let’s unpack this. Every deck has X:1 odds of success. When making predictions, the model starts with a default/average value for X (in this situation it's ~1 since the games are fair), then multiplies it (or do...
Issues with uneven AI resource distribution
by User_Luke — 2022-12-24T01:18:00.295Z
Link: https://temporal.substack.com/p/issues-with-uneven-ai-resource-distribution

Uneven resource distribution:

The uneven distribution of the resources needed to produce and use AI in a state-based system is a long-term challenge to developing international AI policy and raises international security risks.

Resources include factors such as the skills, knowledge, compute, industry, people, and education and other factors of production used to build or develop AI systems, as well as access and ability to use AI systems themselves.

This argument is based on a pathway toward AGI. That is, while it will focus on the endpoint, where an AGI is created, it is likely that issues around resource distribution and relative power shifts within the international system caused by AI will come well before the development of AGI. The reason for focussing on the end point is the assumption that it would create an event horizon where the state that develops AGI archives runaway power over its rivals economically, culturally and militarily. But many points before this could be equally as valid depending on circumstances within the international system.

The lack of resource distribution has a twofold problem:

There is a need for agreement on the distribution of AI resources. However, a wider diffusion of AI resources could increase the risk of misuse or AGI ruin leading to a possible reduction in diffusion.
A lack of diffusion could increase conventional security risks. For example, it would make sense if some nations create a powerful first strike capability to guard against or dissuade anyone who achieves AGI like capabilities in the aim of preventing them getting a runaway advantage.

The desire to access the economic and military benefits of AI will drive competition between states. Even if the benefits of AI development were evenly distributed, the places holding the greater number of AI resources will accrue disproportionate power over other geographies, particularly as AI moves toward the level of general intelligence.

The resources needed to develop and use AI cannot be evenly distributed in a zero-sum international system with layers of economic and security competition mixed with technical and research disparities. Attempting to evenly distribute resources may even have a negative impact on the development of AI if it makes research and innovation less effective.

International security and political economy competition:

Th...
Loose Threads on Intelligence
by Shoshannah Tekofsky — 2022-12-24T00:38:41.689Z

Epistemic Status: Unfinished deep-dive into the nature of intelligence[1]. I committed to writing down my research path, but three weeks in I don't have a coherent answer to what intelligence is, and I do have a next question I want to dig into instead. Thus, here are the rough and rambly threads on intelligence that I've gathered. This piece is lower polish than I like cause of trade-off on writing-vs-research. Skimming might be more productive than a full read!

Thread 1: Intelligence as path finding through reality

Intelligence is path finding through world states, where 'path finding' is a poetic term for optimization. Taking a closer look at optimization, it turns out that bad optimizers are still optimizers. Essentially, optimizers do not need to be optimal.

There exist three categories of optimization techniques:

optimization algorithms (finitely terminating)
iterative methods (convergent)
heuristics (approximate solutions, but no guarantee)

Genetic algorithms and evolutionary algorithms are optimization heuristics. Thus we can trace our past from the primordial soup through simpler and simpler optimization techniques, and we can project our future to the singularity through the creation of better and better optimization techniques. Humans are a point on this scale of increasingly sophisticated and optimally performing optimization techniques instantiated in reality.

Each of the optimization techniques can in turn be instantiated in three different ways:

Mechanical
Computational
Collective

I made these up -- There must be an existing framework that outlines something like this. Or maybe I'm misunderstanding the concept of optimization or how one can categorize the types of instantiations. Either way, here is what I mean by each:

Mechanical optimization cannot learn. It's a tree growing toward the light or a water wheel generating power.

Computational optimization can learn but cannot be divided. It can compute all computable functions (Turing machine or a human with pen/paper). However, if you break up the cognitive processing parts, no computation will take place.

Collective optimization can be divided. Every unit can implement mechanical or computational optimization in itself, and the units work together emergently or coordinately to a greater result than the individual pieces. For instance, a fungus can be split in two such that both halves will ...
If you factor out next token prediction, what are the remaining salient features of human cognition?
by Shmi — 2022-12-24T00:38:04.801Z

LLMs showed how far one can get with a good next token prediction engine and a huge amount of training data. This ought to tell us a fair bit about how the human mind works. Specifically, since the next token prediction appears to be so pervasive in human communication, it is bound to shroud other vital parts of it, if any. I wonder what would be left if we managed to factor the prediction engine + training data out of our models of human cognition? Presumably something related to "interpretability" or "qualia" or... something else. What might it be?