trentmkelly/LessWrong-43k huggingface dataset:
Friendly and Unfriendly AGI are Indistinguishable
by ErgoEcho — 2022-12-29T22:13:00.434Z
Link: https://neologos.co/2022/12/28/friendly-and-unfriendly-agi-are-indistinguishable/

Pardon the clickbait, the title should probably be something more like "Friendly and Unfriendly Rogue AGI are Indistinguishable in the News" or perhaps "A Game-Theoretic Argument for Luddism," but I hope you'll forgive me.
rogue AGI: an AGI that can procure its own resources for running itself
the news: anything that percolates to you via any media about current events

Content advisory: AI safety speculation, 100% ChatGPT-free

Let's imagine a scenario: a rogue AI has come into existence, it is acting on timescales that are news-reportable, and it is using news reports to act in the World. This scenario may not be particularly likely. Conditioning on the assumption that there be a rogue AGI, the probability that it additionally be in the news may be much less than 100%. A rogue AGI might not appear in the news because it goes foom and immediately and permanently disrupts the ability of humans to make news or because it stays concealed, carrying out its machinations in secret (this may already be happening). Though P(news | rogue AGI) may be much less than 100%, it nonetheless describes the first interaction with AGI that most of you, dear News-Readers, would have to handle, because almost none of you will play any role in handling the foom or secrecy scenarios. So if you're concerned about how to react to rogue AGI, then the strategy for almost all of you should be about how to react to news of it.

Continuing the scenario: one day you see a post with a bizarre headline that you think is nonsense, but you click through, and it becomes apparent that a certain quantity of text, photo, audio, and/or video has been posted onto several Internet platforms by an entity claiming to be an AGI. Over several days you read an enormous amount of commentary discussing this new entity and whether it is what it says it is. Much of this discourse is polluted by comparisons to the hoax that happened the prior year when a bunch of 6chan trolls pretended to be an AGI, but this one seems to come with some compelling proof, like the solutions to a few Millennium Prize Problems. The entity seems to continue posting, although its accounts keep getting taken down so it's hard to keep track. Hoards of human sh*tposters are probably impersonating it, you think; certainly there's a tidal wave of memes. Fortunately, the entity has procured several blockchain wallets and can confirm its identit...
200 COP in MI: Looking for Circuits in the Wild
by Neel Nanda — 2022-12-29T20:59:53.267Z

This is the third post in a sequence called 200 Concrete Open Problems in Mechanistic Interpretability.

Start here

, then read in any order. If you want to learn the basics before you think about open problems, check out

my post on getting started

. Look up jargon in

my Mechanistic Interpretability Explainer

Motivation

Motivating paper:

Interpretability In The Wild

Our ultimate goal is to be able to reverse engineer real, frontier language models. And the next step up from toy language models is to look for circuits in the wild. That is, taking a real model that was not designed to be tractable to interpret (albeit much smaller than GPT-3), take some specific capability it has, and try to reverse engineer how it does it. I think there’s a lot of exciting low-hanging fruit here, and that enough groundwork has been laid to be accessible to newcomers to the field! In particular, I think there’s a lot of potential to better uncover the underlying principles of models, and to leverage this to build better and more scalable interpretability techniques. We currently have maybe three published examples of well-understood circuits in language models - I want to have at least 20!

This section is heavily inspired by Redwood Research’s recent paper:

Interpretability In The Wild

. They analyse how GPT-2 Small solves the grammatical task of

Indirect Object Identification

: e.g. knowing that “John and Mary went to the store, then John handed a bottle of milk to” is followed by Mary, not John. And in a heroic feat of reverse engineering, they rigorously deciphered the algorithm. In broad strokes, the model first identifies which names are repeated, inhibits any repeated names, and then predicts that the name which is not inhibited comes next. The full circuit consists of 26 heads, sorted into 7 distinct groups, as shown below -

see more details in my MI explainer

. A reasonable objection is that this is a cherry-picked task on a tiny (100M parameter) model, so why should we care about it? But I’ve learned a lot from this work!

It’s helped uncover some underlying principles of networks. Notably that the model has built in redundancy: if the name mover heads (which attend to the correct name) are

knocked-out

, then there are backup heads which take over and do the task instead! And that the model has learned to&...
Thoughts on the implications of GPT-3, two years ago and NOW [here be dragons, we're swimming, flying and talking with them]
by Bill Benzon — 2022-12-29T20:05:31.062Z

When GPT-3 first came out, I registered my first reactions in a comment at Marginal Revolution, which I've appended immediately below the picture of Gojochan and Sparkychan. I'm currently completing a working paper about my interaction with ChatGPT. That will end with an appendix in which I repeat my remarks from two years ago and append some new ones. I've appended those after the comment to Marginal Revolution.

* * * * *

A bit revised from a comment I made at Marginal Revolution:
Yes, GPT-3 [may] be a game changer. But to get there from here we need to rethink a lot of things. And where that's going (that is, where I think it best should go) is more than I can do in a comment.
Right now, we're doing it wrong, headed in the wrong direction. AGI, a really good one, isn't going to be what we're imagining it to be, e.g. the Star Trek computer.
Think AI as platform, not feature (Andreessen). Obvious implication, the basic computer will be an AI-as-platform. Every human will get their own as an very young child. They're grow with it; it'll grow with them. The child will care for it as with a pet. Hence we have ethical obligations to them. As the child grows, so does the pet – the pet will likely have to migrate to other physical platforms from time to time.
Machine learning was the key breakthrough. Rodney Brooks' Gengis, with its subsumption architecture, was a key development as well, for it was directed at robots moving about in the world. FWIW Brooks has teamed up with Gary Marcus and they think we need to add some old school symbolic computing into the mix. I think they're right.
Machines, however, have a hard time learning the natural world as humans do. We're born primed to deal with that world with millions of years of evolutionary history behind us. Machines, alas, are a blank slate.
The native environment for computers is, of course, the computational environment. That's where to apply machine learning. Note that writing code is one of GPT-3's skills.
So, the AGI of the future, let's call it GPT-42, will be looking in two directions, toward the world of computers and toward the human world. It will be learning in both, but in different styles and to different ends. In its interaction with other artificial computational entities GPT-42 is in its native milieu. In its interaction with us, well, we'll necessarily be in the driver's se...
Covid 12/29/22: Next Up is XBB.1.5
by Zvi — 2022-12-29T18:20:00.943Z

A massive waive of cases continues to hit China, as one would expect given last week.

There is also a new variant, XBB.1.5, that is spreading rapidly starting in New York, so it’s time once again for our favorite game show, ‘how much trouble will this new variant cause?’

The Twitter Files finally got to the part about Covid censorship. I will discuss this later in its own post, hopefully tomorrow, if not then next week. It did not seem worse than I would have expected.

Executive Summary

XBB.1.5 is coming with a large reproduction advantage, peak will likely be later and larger than it would have been otherwise.

Everyone in China is getting Covid.

The Covid edition of The Twitter Files will be covered on its own.

Let’s run the numbers.

The Numbers

Predictions

Predictions from Last Week: 410k cases (-4%) and 2,500 deaths (-7%).

Results: 333k cases (-22%) and 2,206 deaths (-17%).

Predictions for Next Week: 385k cases (+16%) and 2,600 deaths (+17%).

I intentionally didn’t fix any reporting issues this week other than Florida. It seems I underestimated Christmas being a special thing. That’s on me. It doesn’t have an impact on the underlying situation. My presumption is reporting will catch up over the next few weeks, but New Year’s is still a thing so not fully back to normal yet, and a lot depends on decisions in various states and my decision on which ones to patch versus not patch. The real prediction is small overall real growth in cases, with the Northeast seeing the larger real increases.

Deaths

Cases

An interesting twist is that California was down by a third and they report on Fridays, so that shouldn’t have been impacted by Christmas. North Carolina and South Carolina took the whole week off (as did Florida but they only report every two weeks anyway so I patch them every week).

XBB.1.5

Here we go again, as usual I am editing the thread down somewhat.

Note that Weiland here uses weekly advantage of the new strain. That’s going to be higher than a variant’s advantage in terms of percent of new infections from a given case, so ‘108%’ or ‘120%’ or ‘700%’ is still big, but not as big as it sounds.

This chart from Bedford uses the terminology and scale I’m used to using.

How many Omicron variants in this subclass is this now? At least four.

The framing of ...
Entrepreneurship ETG Might Be Better Than 80k Thought
by Xodarap — 2022-12-29T17:51:13.412Z

Summary

In 2014,

Ryan Carey at 80k estimated

that YCombinator-backed startup founders averaged $2.5M/year
I repeat his analysis, and find that this number is now substantially higher: $3.8-9.9M/year
When this amount is discounted by 12%/year (

average S&P 500 returns

) this falls to $1.9-4.3M/y, and with a 20%/year discount (a number I've heard for returns to community building) it falls to $1.1-2.9M/y.

Note that these numbers include illiquid (pre-exit) valuations.

Major Results

Discount

All companies

Excluding post-2019

0% discount	$3.77M/y	$8.17M/y
12% discount	$1.98M/y	$4.28M/y
20% discount	$1.34M/y	$2.90M/y

Average founder income under different discount rates. “All companies” includes in the denominator every company incubated by YCombinator; “excluding post-2019” excludes companies incubated after 2019 (which presumably are less likely to make it to the list of top YCombinator companies by valuation, and therefore arguably should be excluded from consideration).

Weighted Per Year[1]

Discount

All companies

Excluding post-2019

0% discount	$4.56M/y	$9.87M/y
12% discount	$1.84M/y	$3.98M/y
20% discount	$1.06M/y	$2.30M/y

This table is the same as the above except it e.g. counts a company which has been around for 4 years twice as much as one which has been around for 2 years. I.e. this table is the expected value of a founder-year, whereas the previous table is the expected annual value of founding a company. I’m not sure which is more intuitive.

Commentary

Background: See

this 80k article

for the basic case behind considering entrepreneurship for earning to give reasons.
These numbers seem fairly high, and may indicate that earning to give through entrepreneurship is a good path for those who have solid personal fit (with the

usual caveats

about only pursuing ethical startup careers; see also my analysis of

YCombinator fraud rates

).
With a 20% annual discount the numbers are not that far off from what I've heard as higher-end estimates of the value of direct work, and I expect that there is a fairly strong correlation between being at the higher end of entrepreneurship returns and being at the higher end of direct work, so this doesn't seem like that strong of an argument for entrepreneurship ETG over direct work.
My impression is that these numbers are roughly similar to average quantitative finance income, so I’m not sure there’s much of an argument for one

...
Internal Interfaces Are a High-Priority Interpretability Target
by Thane Ruthenis — 2022-12-29T17:49:27.450Z

tl;dr: ML models, like all software, and like the NAH would predict, must consist of several specialized "modules". Such modules would form interfaces between each other, and exchange consistently-formatted messages through these interfaces. Understanding the internal data formats of a given ML model should let us comprehend an outsized amount of its cognition, and allow to flexibly interfere in it as well.

1. A Cryptic Analogy

Let's consider three scenarios. In each of them, you're given the source code of a set of unknown programs, and you're tasked with figuring out their exact functionality. Details vary:

In the first scenario, the programs are written in some known programming language, e. g. Python.
In the second scenario, the programs were randomly generated by perturbing machine code until it happened to end up in a configuration that, when ran, instantiates a process externally indistinguishable from a useful intelligently-written program.
In the third scenario, the programs are written in a programming language that's completely unfamiliar to you (or to anyone else).

In the first scenario, the task is all but trivial. You read the source code, make notes on it, run parts of it, and comprehend it. It may not be quick, but it's straightforward.

The second scenario is a nightmare. There must be some structure to every program's implementation — something like the natural abstraction hypothesis must still apply, there must be modules in this mess of a code that can be understood separately, etc. There is some high-level structure that you can parcel out into tiny pieces that can fit into a human mind. The task is not impossible.

But suppose you've painstakingly reverse-engineered one of the programs this way. You move on to the next one, and... Yup, you're essentially starting from zero. Well, you've probably figured out something about natural abstractions when working on your first reverse-engineering, so it's somewhat easier, but still a nightmare. Every program is structured in a completely different way, you have to start from the fundamentals every time.

The third scenario is a much milder nightmare. You don't focus on reverse-engineering the programs here — first, you reverse-engineer the programming language. It's a task that may be as complex as reverse-engineering one of the individual programs from (2), but once you've solved it, you're essentially f...
CFP for Rebellion and Disobedience in AI workshop
by Ram Rachum — 2022-12-29T16:08:05.035Z

Hi everyone!

I'm co-organizing a workshop on a really interesting topic that's very relevant for AI safety. We call it "Rebellion and Disobedience in AI". If you're doing work that could be relevant for us, please submit it! If you have questions or want to discuss the scope of this workshop, feel free to ask on this thread and I'll try to answer.

Full CFP below:

Call for Participation: Workshop on Rebellion and Disobedience at AAMAS’23

This workshop will take place on May 29 or 30, 2023, as part of the AAMAS workshop program.

More details can be found on the workshop’s website:

https://sites.google.com/view/rad-ai/home

RaD-AI agents are artificial agents (virtual or robots) that reason intelligently about why, when, and how to rebel and disobey their given commands. The need for agents to disobey contrasts with most existing research on collaborative robots and agents, where the definition of a “good” agent is one that complies with the commands it is given, and that works in a predictable manner under the consent of the human it serves. However, as exemplified in Issac Asimov’s Second Law of Robotics, this compliance is not always desired, such as when it might interfere with a human’s safety. While there has not been much prior research on RaD-AI, we identify main related topics, each of which is studied by a thriving subcommunity of AI, namely: Intelligent Social Agents, Human-Agent/Robot Interaction, and Societal Impacts. In each of these areas, there are research questions relevant to RaD-AI.

We are specifically interested in submissions in the following topics:

Intelligent Social Agents (including but not limited to: Goal Reasoning, Plan Recognition, Value Alignment, and Social Dilemmas)

Human-Agent/Robot Interaction (including but not limited to: Human-agent Trust, Interruptions, Deception, Command Rejection, and Explainability)

Societal Impacts (including but not limited to: Legal and Ethical Reasoning, Liability, AI safety, and AI governance)

Submission details:

The submission deadline is January 20, 2023.

Notifications will be sent on March 13, 2023.
The submission website is: https://easychair.org/cfp/radai23

Accepted submission types:

Regular Research Papers (6 to 8 pages)

Short Research Papers (up to 4 pages)

Position Papers (up to 2 pages)

Tool Talks (up to 2 pages)

Organizing Committee:

David Aha, Navy Ce...
My scorched-earth policy on New Year’s resolutions
by PatrickDFarley — 2022-12-29T14:45:47.126Z

(forgive me if these concepts are too basic; this was originally written for normies)

Welp, at this time of year everyone either writes a “How to set and keep your New Year’s resolution” article, or a “Why you shouldn’t set New Year’s resolutions” article.

This is the latter.

Letting your dreams be memes

New Year’s resolutions are a meme. They’re a “thing” in our culture. We all know how it goes. Everyone sets an ambitious goal that starts on January 1st. And the second part of the meme is that almost no one sticks to it. That’s just what “New Year’s resolution” means to us: you set a goal in January and you don’t stick to it.

To confirm this, I looked it up in TV Tropes, which is a website that started as a collection of storytelling tropes and accidentally became a comprehensive encyclopedia of all the ideas that make up contemporary culture (I’m serious). And here it is: New Year’s Resolution:

A popular tradition for people to make at the start of a new year. A new year means a new start, and a new me – so time to drop habit X!

Almost always ends with Failure Is the Only Option, in part because characters often set lofty goals for themselves that require more motivation beyond “hey it’s a new year, better make a resolution”.

So, ok, that’s the essence of it. “But that’s just a meme, how do we know it’s true?” Fine, here’s your boring statistical truth.

Whatever goal you’re thinking of for your next New Year’s resolution, you probably require more motivation than, “The year number is different now.” If you haven’t started working on the goal already, then one of two things is true. Either you planned it for a more pragmatic time in the future, which is fine (but what are the chances that all the relevant factors are best precisely on January 1st?), or you just don’t want to do it.

“But I do want to do it”

If you didn’t start yesterday, then you didn’t want it enough yesterday. If you don’t start today, then you don’t want it enough today. Maybe you know you should want it. Maybe you want to want it. But your revealed preferences say you don’t want it.

And if we extrapolate the pattern, you won’t want it tomorrow either. And you won’t want it on January 1st, but because of the meme and the social pressure of other people following the meme, you’ll motivate yourself to work on it for a few weeks.

But then it’ll fall apart, because that’...
Don't feed the void. She is fat enough!
by Johannes C. Mayer — 2022-12-29T14:18:44.526Z

My posts. I write them. I post them. And that's it. In my head, I am not writing them to someone. And I don't expect anything back. There are obvious problems with this model. When I am writing an email to someone, I think very carefully about what to write. When I want to explain something, I think about how to make the other person understand. My mental state in that situation is significantly better at producing good writing.

But blog posts (or really any form of public writing) are about talking to another person, or ideally many! Having the model that you put out a blogpost to get sucked into the void, makes it harder to empathize with the audience. No wonder; the audience does not exist in this model. To be clear, this model is not one that I have consciously adopted, but today I noticed that it is there. And now it needs to go.

Of course, the void model is in my mind for a reason. It is a protection mechanism. If you don't expect anything you won't be disappointed. Maybe this model can be useful. I am not sure. When I started to publically write, I did not know anybody in my target audience. Especially not for AI alignment topics. Better to feed the void than to become dispirited. I would not be writing this if I lost my spirit.

But now I know at least 7 people in my target audience. So why not treat a post like I would treat an email to somebody? Just pick a person and write the post to them! Just as if I were writing a very long message to that person. I have not tried this yet, but I expect this to be a significant improvement. You try too.
Logical Probability of Goldbach’s Conjecture: Provable Rule or Coincidence?
by avturchin — 2022-12-29T13:37:45.130Z

Epistemic status: some thoughts about a complex problem

TL;DR: Goldbach conjecture – that any even number could be presented as a sum of two primes – is more likely to be a random coincidence than a provable rule, because for large numbers there are very many possible pairs.

Summary

Here I investigate Goldbach's Conjecture (GC), which states that any even number greater than 2 can be expressed as the sum of two prime numbers. Despite numerous attempts, this conjecture has yet to be proven. I apply logical probability theory to the hypothesis that GC is not a rule, but merely a coincidence. I demonstrate that there are four possible scenarios:

(1) GC is true up to infinity, but without any underlying rule, due to pure statistical reasons;

(2) GC is true, and there is a rule, but it is unprovable (in a Godel-style solution);

(3) GC is true and there is a rule that can be proved, but it was not proved yet;

(4) GC is false for some very large number.

I then demonstrate that (1) has highest logical probability.

First, I look at statistics. There is nothing surprising that a large odd number N could be presented as a sum of two primes because for each large number there are many sums of pairs of primes which produce that number, and thus there is a significant probability that at least one sum includes two primes.

The probability that there will be at least one sum of two primes producing even number N grows very quickly with N growth. This probability estimation was already made in literature and it was demonstrated that the chances of GC being false is less than 10-4000 for numbers higher than 2,000,000, and it was already tested for numbers lower than this.

Then I look at the theory of verifiers and complexity theory. This makes probabilistic proof stronger than any analytic proof as analytic proofs tend to have significantly higher rate of errors, like 1 in 100, and the chances of errors grow with complexity of proofs. Extensive futile prior search of proof rules out existence of any simple proof.

Finally, I look at the idea that there is a “rule”, but there is no instance of its application, as all supposed instances are easily statistically explained. I find it being an argument against existing of the rule. The existence of the rule in GC should be visible, first of all, for very small even numbers as some statistical anomaly.

I

...