trentmkelly/LessWrong-43k huggingface dataset:
Strange Loops - Self-Reference from Number Theory to AI
by ojorgensen — 2022-09-28T14:10:00.106Z

Summary

I see this post as essentially trying to make clear the link between self-reference as discussed in Hofstadter’s “Gödel, Escher, Bach” and self-reference as considered in some of MIRI’s work on Embedded Agency. I start by considering instances of self-reference in mathematics through Gödel's Incompleteness Theorems, before moving on to instances of self-reference in Löb's Theorem and Embedded Agency.

The aims of this post are to provide a different perspective on the issue of self-reference, as well as to give some intuition behind the proof of Gödel's Incompleteness Theorem. I found this helpful when trying to understand some elements of Embedded Agency, and I hope others might too. I doubt this will be particularly novel for anybody who already understands the Löb's Theorem stuff

here

,

or has a good grasp of Embedded Agency. Thanks to Joe Hasson for comments and suggestions.

Introduction

The perfect Artificial Intelligence is one which is perfectly logical. This seems like a pretty reasonable claim: we associate a being that is perfectly logical with one that is able to perfectly plan and execute its goals.

Strangely enough, this is not true in the most naive sense. I claim the reason for this is the idea of Strange Loops, a piece of terminology shamelessly stolen from Hofstadter, author of Gödel, Escher, Bach (which I'll refer to as GEB). To poorly paraphrase Hofstadter, at its core a Strange Loop is an abstract loop, whereby something references itself in what feels like a higher level, yet somehow returns to the same level.

I will introduce increasingly powerful Strange Loops from logic, before showing how we might create a Strange Loop in the context of Artificial Intelligence. The key question to keep in mind as you go through this is: how is the Strange Loop referencing itself, and what issues does this pose? If you can do this successfully, you will get to the heart of not only some of the most interesting areas of logic, but also understand some of the subtle issues that can arise when we are trying to design AI systems that do what we want them to do.

Context

If you’re already familiar with some basic ideas from logic such as formal systems, Peano Arithmetic, and reasoning about a formal system from inside / outside then feel free to skip to Strange Loop 1. If not, here is an attempt at a quick crash course.

A super high level view ...
Why I think strong general AI is coming soon
by porby — 2022-09-28T05:40:38.395Z

I think there is little time left before someone builds AGI (median ~2030). Once upon a time, I didn't think this.

This post attempts to walk through some of the observations and insights that collapsed my estimates.

The core ideas are as follows:

We've already captured way too much of intelligence with way too little effort.
Everything points towards us capturing way more of intelligence with very little additional effort.
Trying to create a self-consistent worldview that handles all available evidence seems to force very weird conclusions.

Some notes up front

I wrote this post in response to the

Future Fund's AI Worldview Prize

[1]. Financial incentives work, apparently! I wrote it with a slightly wider audience in mind and supply some background for people who aren't quite as familiar with the standard arguments.
I make a few predictions in this post. Unless otherwise noted, the predictions and their associated probabilities should be assumed to be conditioned on "the world remains at least remotely normal for the term of the prediction; the gameboard remains unflipped."
For the purposes of this post, when I use the term AGI, I mean the kind of AI with sufficient capability to make it a genuine threat to humanity's future or survival if it is misused or misaligned. This is slightly more strict than the definition in the Future Fund post, but I expect the difference between the two definitions to be small chronologically.
For the purposes of this post, when I refer to "intelligence," I mean stuff like complex problem solving that's useful for achieving goals. Consciousness, emotions, and qualia are not required for me to call a system "intelligent" here; I am defining it only in terms of capability.

Is the algorithm of intelligence easy?

A single invocation of GPT-3, or any large transformer, cannot run any algorithm internally that does not run in constant time complexity, because the model itself runs in constant time. It's a very large constant, but it is still a constant.

They don't have any learnable memory about their internal state from previous invocations. They just have the input stream. Despite all their capability, transformers are

fundamentally limited

.[2]

This is part of the reason why asking GPT-3 to do integer division on large numbers in one shot doesn't work. GPT-3 is big enough to memorize a number of results, so adding small numbers isn't too har...
About Q Home
by Q Home — 2022-09-28T04:56:44.586Z

(I didn't know how to name the post.)

I have an internet friend in Ukraine. My love is in Belarus. I have a non-binary friend in Russia who is a former conscript. I'm a young male in Russia. I'm at risk, but might be relatively safe.

Writing this post to highlight my main ideas (I would like to discuss them while I have the time). The ideas are about Alignment, subjective experience, and informal semantics.

About Alignemt: LW draft, google doc (A single principle related to many Alignment subproblems?).
Ideas which got popular on Reddit: a holistic way to analyze fictional characters, chess positions and synesthesia, very short and weird music loops which evoke contradictory emotions (version 1, version 2), analyzing language with "possibilities".

If the opportunity cost is not too high for you, please help me to explain some of those concepts to you and other people.

...

Since 24.02.22 I desperately tried to share my ideas on Reddit (and later here) while simultaneously trying to advance them in case my posts won't be successful.

I can't imagine what other people feel in this tragedy. I have a goal and it fills my mind and lets me remember the good things. I can't imagine how others live.

Edited, shortened this post. If this date is not changed after 27.07.25, then something happened to me.

...

Whatever I'm writing about now, here's my main latest draft: here on LW (A single principle related to many Alignment subproblems?). A secondary draft: properties of reasoning (LW draft). Third: Substack (using a weird informal concept to explore semantics).
[Linkpost] "Intensity and frequency of extreme novel epidemics" by Mariani et al. (2021)
by T431 — 2022-09-28T03:31:01.783Z
Link: https://pubmed.ncbi.nlm.nih.gov/34426498/

This is a useful paper for those wanting to better understand the risks posed by pandemics and, more generally, global catastrophic biological risk.

I have not seen this paper mentioned on LW or EAF and it does not appear on LW or EAF when entered in the search box, so I thought I'd contribute it to the wider conversation.

In this linkpost, I include, in the following order, the structure of the paper, the paper's abstract, some commentary and quotes, and the 2 most important figures from the paper.

To see the papers that have cited Intensity and frequency of extreme novel epidemics, click here. To see papers that Intensity and frequency of extreme novel epidemics cites, click here.

Possible ways to cite the paper:

Marani, Marco, Gabriel G. Katul, William K. Pan, and Anthony J. Parolari. "Intensity and frequency of extreme novel epidemics." Proceedings of the National Academy of Sciences 118, no. 35 (2021): e2105482118.

@article{marani2021intensity,
title={Intensity and frequency of extreme novel epidemics},
author={Marani, Marco and Katul, Gabriel G and Pan, William K and Parolari, Anthony J},
journal={Proceedings of the National Academy of Sciences},
volume={118},
number={35},
pages={e2105482118},
year={2021},
publisher={National Acad Sciences}
}

Structure

Introduction
Results

The Probability Distribution of Epidemic Intensity
The Probability of Occurrence of Extreme Epidemics

Discussion
Materials and Methods

Abstract

Observational knowledge of the epidemic intensity, defined as the number of deaths divided by global population and epidemic duration, and of the rate of emergence of infectious disease outbreaks is necessary to test theory and models and to inform public health risk assessment by quantifying the probability of extreme pandemics such as COVID-19. Despite its significance, assembling and analyzing a comprehensive global historical record spanning a variety of diseases remains an unexplored task. A global dataset of historical epidemics from 1600 to present is here compiled and examined using novel statistical methods to estimate the yearly probability of occurrence of extreme epidemics. Historical observations covering four orders of magnitude of epidemic intensity follow a common probability distribution with a slowly decaying power-law tail (generalized Pareto distribution, asymptoti

...
Threat-Resistant Bargaining Megapost: Introducing the ROSE Value
by Diffractor — 2022-09-28T01:20:11.605Z

Post Status: Original research, mathy, high-context. Make sure you've read the last three posts in the sequence first, this is going to be a very rough one to start on.

As a recap of the last three posts, bargaining has an elegant notion of what a neutral point is: the Nash bargaining solution. And, games where the two players can pay each other money (transferrable-utility games) have an elegant notion of what a neutral point is: the CoCo value. And, as it turns out, games in general have an elegant notion of what a neutral point is that subsumes the previous two concepts as special cases, which can be motivated by either trying to generalize the Nash bargaining solution to games without a clear disagreement point (as John Harsanyi did), or by trying to generalize the CoCo value to games without transferrable utility (as I did).

Sadly, there's an issue with all of these where a player can get a lot more utility by inventing increasingly effective ways of destroying the utility of everyone else. Basically, someone can go "I have a torturizer, favor me or I'll use it on you", and the threat will work. In fact, such reasoning is the default for the CoCo value. That issue is detailed in the second and third posts in the sequence.

And so, there's a question of "is there a distinguished point in transferrable-utility games which is generated by a more threat-resistant line of reasoning?" As per the third post in the sequence, using a correlated equilibrium as the fallback point can serve that role, but they're not unique.

There is! (kinda)

I have a lot of uncertainty whether the equation I wrote down in this post is The Threat-Resistant Bargaining Solution, but if it isn't, it's at least pretty close to The Solution. My thinking about it is still in flux; here's a rundown of the current situation, which also parallels the structure of the rest of the post.

Following the example set by CoCo values in the first two posts, there's a highly general pathway to amplify a solution concept for 2-player transferrable-utility games into a solution concept for n-player transferrable-utility games, and from there into a notion of equilibrium for games in general. Thus, in the quest for a more threat-resistant solution concept, we can restrict our attention to the special case of 2-player transferrable-utility games. These are vastly simpler because th...
7 traps that (we think) new alignment researchers often fall into
by Orpheus16 — 2022-09-27T23:13:46.697Z

We've noticed that new alignment researchers (and sometimes experienced alignment researchers) often fall into similar traps.

Here are 7 traps that we often see in new alignment researchers:

1. They think they need to be up-to-date on the literature before they can start contributing their own ideas.

Suggestion: Staying up-to-to-date on the literature is useful. But you don’t need to read everything before you contribute your own ideas. You can write

naive hypotheses

and try to

generate new ideas

. For many people, it’s easier to come up with (certain types of) new ideas before reading all of the literature. Once you’ve read the literature, it can be harder to ignore the ideas and frames of others. (see also

Cached Thoughts

)

2. They end up pursuing proxy goals.

A. They end up starting projects (often projects that take 3+ months) without a clear theory of change. They forget the terminal goal (e.g., reducing x-risk) and end up pursuing a proxy goal that looks like it’s helping (e.g., skill-up in ML). To be clear, many proxy goals (like skilling up in ML) are not inherently bad. But they don’t have a sense of why they’re learning ML, which subproblems they’re hoping to solve with ML, and which specific subskills in ML (and other fields) might be helpful. They don’t have a sense of how long they should spend skilling up, what else they should be learning, or what evidence they should be looking for to tell them to stop (or keep going).

B. They lose sight of the terminal goal. The real goal is not to skill-up in ML. The real goal is not to replicate the results of a paper. The real goal is not even to “solve inner alignment.” The real goal is to not die & not lose the value of the far-future.

Suggestion: Keep the terminal goal in mind. Try to have a clear idea of how your actions are getting you closer to the terminal goal. Sometimes, you won’t have clear answers (and indeed if you rely on having a clear end-to-end impact story before doing anything, you may fall into the trap of doing nothing). But notice when you’re doing things that don’t have a clear path toward the terminal goal. Occasionally ask yourself if there are projects that seem useful but don’t actually matter. Be cautious about spending many months on projects unless they have a justifiable theory of change.

3. They assume that if they don’t understand something, it’s be

...
Failure modes in a shard theory alignment plan
by Thomas Kwa — 2022-09-27T22:34:06.834Z

Thanks to David Udell, Alex Turner, and others for conversations that led to this

Recently in a conversation with David Udell, Nate Soares said:

[David's summary of shard theory] doesn't yet convince me that you know something i don't about the hopefullness of such a plan. the sort of summary that might have that effect is a summary of what needs to be true about the world (in this case, probably about RL models, optimization, and/or human values) for this idea to have hope. in particular, the point where i start to be interested in engaging is the point where it seems to me like you perceive the difficulties and have a plan that you expect to overcome those difficulties."

I'm not as pessimistic as Nate, but I also think shard theory needs more concreteness and exploration of failure modes, so as an initial step I spent an hour brainstorming with David Udell, with each of us trying to think of difficulties, and then another couple of hours writing this up. Our methodology was to write down a plan for the sake of concreteness (even if we think it's unlikely to work), then try to identify as many potential difficulties as possible.

Definitions for this document

shard theory is a new alignment research program based on observations about the human reward system; the canonical reference is

here

.
shard means a contextually-activated circuit responsible for some behaviors. Shards can perform powerful optimization that steers the world in certain directions; rather than suggestively calling these directions values, I just call them behavioral tendencies.
corrigibility means a system that will allow itself to be modified by humans, and perhaps be actively helpful in

various ways

reward means something that applies selection to a model, and in particular reinforces and attenuates shards.
values means behavior patterns that are stable under reflection, e.g. a utility function. The utility function

need not be explicitly represented or precise

, nor be over

world-states

.
value formation is the process by which humans or AIs construct values from their existing behavioral tendencies.[1]

A possible shard theory alignment plan

Play around with modern RL models, and extract quantitative relationships between reward schedule and learned behaviors.
Instill corrigibility shards inside powerful RL models to the greatest extent possible.
Scale those aligned

...
Is a PhD necessary to contribute meaningfully to a field?
by TrudosKudos — 2022-09-27T21:27:02.491Z

For context, I have happened to find myself in the highly fortunate situation of being relatively young (late-twenties) and having met my financial retirement goals. I have always held a fascination with cognitive science and want to now orient my life to participation within said field in alignment with some of my other interests, particularly the cognitive processes underlying altered states of consciousness (psychedelics, flow, religious ecstasy). After much thought, I've been able to see that a personal skill I've developed is in being able to synthesize existing information and communicate it to others in a way that is easy to understand and I would like to approach my academic pursuit from this angle instead of being involved in the actual academic lab work. I am still relatively naive to the workings and structure of academia but am trying to find a place within the field where I can combine my interests and skills to meaningfully contribute to the field of study as a whole. My fear is that this would limit me to largely commentary and synthesis of existing information and that it would label me as less credible because I am not actually involved in the work of producing scientific literature itself.

My question is less oriented around Cognitive Science as a field of study specifically but in regards to academic fields of study as a whole. Do I need to have a Ph.D. in order to write books and articles that summarize existing literature, postulate potential avenues of research, and theorize within the field of cog sci? I currently have my bachelors in cloud computing but would any other degree of education work such as an MSc.? I have the resources and time to acquire these if necessary but I am just trying to see the best return on my time investment if I don't necessarily desire to publish academic research.

Thank you.
Why we're not founding a human-data-for-alignment org
by L Rudolf L — 2022-09-27T20:14:45.393Z
Link: https://forum.effectivealtruism.org/posts/iBeWbfQLA9EKfsdhu/why-we-re-not-founding-a-human-data-for-alignment-org

TL;DR

One-paragraph summary: we (two recent graduates) spent about half of the summer exploring the idea of starting an organisation producing custom human-generated datasets for AI alignment research. Most of our time was spent on customer interviews with alignment researchers to determine if they have a pressing need for such a service. We decided not to continue with this idea, because there doesn’t seem to be a human-generated data niche (unfilled by existing services like Surge) that alignment teams would want outsourced.

In more detail: The idea of a human datasets organisation was

one of the winners of the Future Fund project ideas competition

, still figures on their

list

of project ideas, and had been advocated before then by some people, including Beth Barnes. Even though we ended up deciding against, we think this was a reasonable and high-expected-value idea for these groups to advocate at the time.

Human-generated data is often needed for ML projects or benchmarks if a suitable dataset cannot be e.g. scraped from the web, or if human feedback is required. Alignment researchers conduct such ML experiments, but sometimes have different data requirements than standard capabilities researchers. As a result, it seemed plausible that there was some niche unfilled by the market to help alignment researchers solve problems related to human-generated datasets. In particular, we thought - and to some extent confirmed - that the most likely such niche is human data generation that requires particularly competent or high-skill humans. We will refer to this as high-skill (human) data.

We (Matt & Rudolf) went through an informal co-founder matching process along with four other people and were chosen as the co-founder pair to explore this idea. In line with standard startup advice, our first step was to explore whether or not there is a concrete current need for this product by conducting interviews with potential customers. We talked to about 15 alignment researchers, most of them selected on the basis of doing work that requires human data. A secondary goal of these interviews was to build better models for the future importance and role of human feedback in alignment.

Getting human-generated data does indeed cost many of these researchers significant time and effort. However, we think to a large extent this is because dealing w...
A Poorly Planned Loft Bed
by jefftk — 2022-09-27T17:50:01.660Z

Planning

Lily's loft bed reminded
me of a much more poorly constructed one I built in 2004, around when
I

started writing here. I had just
started college and was sorting out a room layout with my roommate.
The beds were three pieces: two vertical "H" pieces, and a metal frame
to connect them and support the mattress. All of this predates my
having a camera that's always in my pocket, so we'll have to make do
with drawings:

The verticals had holes drilled in them which allowed you to stack a
pair of beds to turn them into bunk beds:

We didn't want bunk beds, though, we wanted a loft over a desk:

We found an extra set of verticals, and put them together without the
metal frame to support a lower mattress:

This is a terrible idea: there's very little to keep the top section
from wandering off left or right ("shearing"). It swayed dramatically
as soon as I tried climbing on it for the first time. You'd think we'd
have recognized it as unsafe and taken it down, but I slept on top the
whole year. There were, luckily, no collapses.

There's actually a close contender, however, for the worst way to
arrange these bed components. The room across the hall was a triple,
so they had three of these beds. The ceiling was very high, just over
three times the height of one of the verticals. I helped one of them
triple-bunk the beds:

My memory is that we thought this was so clearly the best arrangement
for the room that there was no "ask the other two roommates" step.
Which meant the only one I'm still
in touch with spent the year sleeping with his nose about a foot
from the ceiling. More things I wouldn't do now...

Comment via: facebook