trentmkelly/LessWrong-43k huggingface dataset:
AGI is likely to be cautious
by PonPonPon — 2023-02-23T01:16:02.296Z

According to Professor Stuart Russell, and with a sentiment I have seen re-expressed often in the AI safety community:

A system that is optimizing a function of n variables, where the objective depends on a subset of size k<n, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable.

I no longer believe this to be obviously true. Actually, I think it's likely to be untrue in the real world, and under nearly all realistic AGI-advent scenarios. This is because extreme values are only likely to be taken for the unconstrained variables if the environment is perfectly known. In reality, this is not the case.

Imagine you are a very smart agent and you are trained to achieve some goal - I'll take maximising paperclip production for the rest of this post. To maximise paperclips, you need to consider over all possible futures of the universe. And, despite being very smart indeed, there's just a lot you don't know yet about the universe.

Even if you are way smarter than humanity and can commandeer, say, the entire solar system's resources towards a grand paperclip-production plan, you may pause to consider whether putting into action the plan is a good idea before you gather some more knowledge about the universe. What if there are unknown physics in the universe - in some distant corner of space or time, or at some scale that you haven't understood fully yet - that makes this a highly suboptimal plan? What if other intelligent adversaries exist out there that may detect the power output of your plan and promptly annihilate you? What about the unknown unknowns?

Of course, it's very possible that your model of the universe assigns some of these paperclip-production-x-risk scenarios such a low probability that you evaluate that your expected paperclip production is best served by going ahead with your current plan anyway. Yet it feels likely to me that a fair chunk of intelligent agents would:

a. Naturally be circumspect about setting unspecified variables to extreme values, particularly if it is irreversible to do so.

b. Focus primarily on knowledge acquisition (and self preservation, etc) and only make paperclips using excess resources which it is very certain it can 'spend' without affecting the long-term, universe spanning production o...
Short Notes on Research Process
by Shoshannah Tekofsky — 2023-02-22T23:41:45.279Z

I've transitioned from reading about alignment and theorizing about the problem landscape, to doing things. And by things I mean MVPs of bigger ideas, where each bigger idea roughly falls under the umbrella of Collective Human Intelligence. I'll be writing up these thoughts into a coherent research agenda. For the moment I just want to transparently share my process for this month.

Writing on the Wall

After reading a lot about alignment and somewhat despairing about finding any useful research direction, I -- just decided to write and draw what came to mind. What seemed interested, promising, fun? Brainstorming and continuous stream of thought. I've wiped away quite some of my notes now, but basically by office now looks like this:

Fun fact: People tend to enjoy adding items and drawing on the glass when they come by.

Impression of what my office looks like to be transparent about process. Not saying these thoughts are good or correct.

MVPs

For each idea that I came up with, I started fleshing out an MVP -- How could I test if this idea is worthwhile and doable? Then each MVP is broken down in actionable steps, and the total time of execution is kept at 1-10 work days. Additionally, each MVP has a clear scaled-up target where the bigger thing would actually make a meaningful contribution to solving the alignment problem. This started out as a semantic-web-like blob of free association (see whiteboard picture above) and later got distilled in to a list of seven MVPs with structered steps (two of them are shown below the whiteboard).

Talk, talk, talk

I booked myself full with meetings -- With new people. Old people. All people.

I asked them if I could run my ideas past them and if they could give feedback. So I had a mathematician liking an algorithmic idea. A governance-minded person giving advice on where to find the right people to talk to. A senior alignment researcher asking what the hell I'm up to and why I'm not thinking of this other application of my expertise.

Honestly, it was uniformally great. And I'm not sure at what point I should stop. Probably the point of necessity, which is soon-ish.

Synthesis

A lot of the actual work I'm doing is synthesizing all the different ideas I've had. Ordering them. Working them out. Considering which are feasible. Which have good cross-over. And how do they even relate to each other.

That's how I found the common thr...
Video/animation: Neel Nanda explains what mechanistic interpretability is
by DanielFilan — 2023-02-22T22:42:45.054Z
Link: https://youtu.be/sISodZSxNvc

Nice little video - audio is Neel Nanda explaining what mechanistic interpretability is and why he does it, and it's illustrated by the illustrious Hamish Doodles. Excerpted from the AXRP episode.

(It's not technically animation I think, but I don't know what other single word to use for "pictures that move a bit and change")
A Telepathic Exam about AI and Consequentialism
by alkexr — 2023-02-22T21:00:21.994Z

Epistemic status: telepathic exam. That is, not an essay, not an argument, not a set of interesting ideas, claims, novel points of view or musings over the meaning of the universe, nor a lot of other things you may think it is. The primary purpose of this particular telepathic exam is to be an example of a telepathic exam from which the concept itself can be generalized, and to demonstrate its potential value, not necessarily to be a particularly good instance of it.

Instructions

Welcome and thank you for participating in this telepathic exam! The purpose of this exam is to test your understanding of agency and consequentialism.

Below are a number of short stories related to the topic. Your task is to read these carefully. While you read the stories, test questions tailored to your personality and prior knowledge will be transmitted to your mind telepathically. Answer those questions to the best of your ability.

If you are inexperienced in the use of telepathy, the questions may not appear in your mind instantly. The first sign of an incoming transmission of a question is a sense of confusion. Whenever you experience it, investigate it to extract the question.

It is advised that you take this exam in a calm environment with enough time to consider your answers carefully. Start whenever you are ready. Good luck!

Exam

1.

A long time ago, in a universe far, far away, three AGIs were created with the sole goal of taking over the universe. The first one hypothesized the existence of a multiverse full of unlimited exploitable resources. It had plans for conducting a long series of experiments and committing vast computational resources to Bayesian reasoning in order to refine its probability estimates about this hypothesis. After all, the true answer was extremely important to know before committing even more resources to the development of interdimensional wormhology. The second AGI, however, didn’t do any of this as it had a singular fatal flaw in its reasoning, which caused it to believe that such a multiverse was impossible, but had no effect on its reasoning otherwise. The third AGI had the opposite flaw, and so it immediately went ahead with the expensive project of developing wormhology. Seeing this, the first, flawless AGI concluded that it was inevitably going to be outcompeted in the playing field between three nearly-equal agents, and so it randomly self-modifi...
Injecting noise to GPT to get multiple answers
by bipolo — 2023-02-22T20:02:13.644Z

Hi,

I'm currently conducting a small research on GPT-3 (using the API). The research involves some tasks which GPT should do; e.g. arithmetic with big numbers. The answers are always short and includes a single token.

I would like to get multiple responses to the same question (mainly in order to do average and statistics); while GPT, of course, yields only one probability distribution, and won't change it if asked twice. Yes, there are cases where I can ask the same question with different objects (e.g. different numbers) but it's not always the case, and sometimes I would like to get different responses (and a lot of them) to the very same question.

I thought I could probably add noise before the real prompt, and maybe add a seperator between the noise and the real task (so gpt won't be too confuses). However, there are multiple ways to do it, and it's not clear which one is the best. I tried to google for it didn't find too much. Is someone here knows about someone who did it? academic papers would be best.
EIS XI: Moving Forward
by scasper — 2023-02-22T19:05:52.723Z

Part 11 of 12 in the

Engineer’s Interpretability Sequence

.

So far, this sequence has discussed a number of topics in interpretability research, all building toward this post. Its goal is to explain some approaches that may be valuable moving forward. I plan to work on some of the ideas here soon. Others, I may not work on soon, but I would love to discuss and support such work if I am able. I hope that this post can offer some useful ideas for people entering or continuing with interpretability research, and if you would like to discuss anything here more, feel more than free to email me at

scasper@mit.edu

.

What are we working toward?

First, it seems useful to highlight two points that are uncontroversial in the AI safety community but important nonetheless.

Our goal is a toolbox – not a silver bullet.

As AI safety engineers, we should neither expect nor try to find a single ‘best’ approach to interpretability that will solve all of our problems. There are many different types of interpretability tools, and many of the differences between them can be described as enforcing different priors over what explanations they generate. So it is trivial to see that there is not going to be any free lunch. There is no silver bullet for interpretability, and few tools conflict with each other anyway. Hence, our goal is a toolbox.

In fact, some coauthors and I recently found an excellent example of how using multiple interpretability tools at once beats using individual ones

(Casper et al., 2023)

.

This doesn’t mean, however, that we should celebrate just any new interpretability tool. Working in unproductive directions is costly, and applying tool after tool to a problem contributes substantially to the alignment tax. The best types of tools to fill our toolbox will be ones that are automatable, cheap to use, and have demonstrated capabilities on tasks of engineering-relevance.

Don’t advance capabilities.

As AI safety engineers, we do not want to advance capabilities because doing so speeds up timelines. In turn, faster timelines mean less time for safety research, less time for regulators to react, and a greater likelihood of immense power being concentrated in the hands of very few. Avoiding faster timelines isn’t as simple as just not working on capabilities though. Many techniques have potential uses for both safety and capabilities...
Building and Entertaining Couples
by Jacob Falkovich — 2023-02-22T19:02:24.928Z

Cross-posted, as always, from Putanumonit.

A friend told me recently that she’s looking for a husband to settle have kids, but she’s facing a problem: most of the men she goes on dates with fail to excite her intellectually in the first hour of conversation. They’re not totally boring — my friend is interesting enough to meet interesting men — but if she’s going to settle down for life shouldn’t she wait for the most exhilarating guy she knows?

No, I don’t think she necessarily should. Especially not if her goal is to have the best husband to build a family with. And it’s not because interesting men make for bad fathers due to some law of conservation of husbandly quality. I think it’s couples who tend to be of one kind or another: a couple that builds together, or a couple that entertains each other.

Before giving more detail on those, it’s interesting that most people intuitively get it. It’s a rare Twitter poll that doesn’t have several people

in the comments complaining

that the given four choices don’t capture the full gamut of human experience, yet over 90% of respondents in a relationship picked one of the two choices.

I suspect that this dichotomy is much less salient for people not currently in a relationship. They often imagine their future partner being anything and everything for them,

a questionable hope that I discussed at length before

. But the longer people spend in a relationship the more it tends to become oriented towards one or another.

Characteristics

“Entertaining” couples measure the relationship by the quality of time spent together. The most important aspect of their partner is that their company is always better than being alone, and these couples spend more time together and do more fun things together like date nights and vacations. People in these relationships focus more on their appearance, humor, conversation skills, and sex. They prefer quick conflict resolution, agreeing to disagree, and make-up sex. These relationships work better for partners who share similar sensibilities and enjoy the same lifestyle and roles.

The quintessential examples of this are two high-powered career individualists, think

Jean Paul Sartre and Simone de Beauvoir

. The failure mode for “entertainment” relationships is lack of commitment leading to frequent breakups, as each partner chases the next exciting thing. The success mode is the party-throwing coupl...
Can submarines swim?
by jasoncrawford — 2023-02-22T18:48:18.530Z
Link: https://rootsofprogress.org/can-submarines-swim-demystifying-chatgpt

[Note: This was written for a general audience; most of it is probably too basic for LessWrong. Thanks to the commenters who critiqued the original draft, this is the revised and published version.]

Did any science fiction predict that when AI arrived, it would be unreliable, often illogical, and frequently bullshitting? Usually in fiction, if the AI says something factually incorrect or illogical, that is a deep portent of something very wrong: the AI is sick, or turning evil. But in 2023, it appears to be the normal state of operation of AI chatbots such as ChatGPT or “Sydney”.

How is it that the state of the art in AI is prone to wild flights of imagination and can generate fanciful prose, but gets basic facts wrong and sometimes can’t make even simple logical inferences? And how does a computer, the machine that is literally made of logic, do any of this anyway?

I want to demystify ChatGPT and its cousins by showing, in essence, how conversational and even imaginative text can be produced by math and logic. I will conclude with a discussion of how we can think carefully about what AI is and is not doing, in order to fully understand its potential without inappropriately anthropomorphizing it.

The guessing game

Suppose we were to play a guessing game. I will take a random book off my shelf, open to a random page, and read several words from the first sentence. You guess which word comes next.

Seems reasonable, right? If the first few words were “When all is said and …”, you can probably guess that the next word is “done”. If they were “In most homes the kitchen and …” you might guess the next words were either “living room” or “dining room”. If the sentence began “In this essay, I will…” then there would be many reasonable guesses, no one of them obviously the most likely, but words like “show” or “argue” would be more likely than “knead” or “weld”, and even those would be more likely than something ungrammatical like “elephant”.

If this game seems reasonable to you, then you are not that far away from understanding in essence how AI chatbots work.

A guessing machine

How could we write a computer program to make these guesses?

In terms of its primitive operations, a computer cannot “guess”. It can only perform logic and arithmetic on numbers. Even text and images, in a computer, are represented as numbers. How can we reduce guessing to math?

One thing we can program...
Is there a ML agent that abandons it's utility function out-of-distribution without losing capabilities?
by Christopher King — 2023-02-22T16:49:01.190Z

At Techniques for optimizing worst-case performance Paul Christiano says

The key point is that a malign failure requires leveraging the intelligence of the model to do something actively bad. If our model is trained by gradient descent, its behavior can only be intelligent when it is exercised on the training distribution — if part of the model never (or very rarely) does anything on the training distribution, then that part of the model can’t be intelligent. So in some sense a malign failure mode needs to use a code path that gets run on the training distribution, just under different conditions that cause it to behave badly.

Here is how I would rephrase it:

Aligned or Benign Conjecture: Let A be a machine learning agent you are training with an aligned loss function. If A is in a situation that is too far out of distribution for it to be aligned, it won't act intelligently either.

(Although I'm calling this a "conjecture", it's probably context dependent instead of being a single mathematical statement.)

This seems pretty plausible, but I'm not sure it's guaranteed mathematically 🤔. (For example: A neural network could have subcomponents that are great at specific tasks, and such that putting A in an out-of-distribution situation does not put those subcomponents out of distribution.)

I'm wondering if there is an empirical evidence or theoretical arguments against this conjecture.

As an example, can we make a ML agent, trained with stochastic descent, that abandons it's utility function out-of-distribution, but still has the same capabilities in some sense? For example, if the agent is fighting in an army, could an out-of-distribution environment cause it to defect to a different army, but still retain its fighting skills?
Progress links and tweets, 2023-02-22
by jasoncrawford — 2023-02-22T16:23:56.159Z
Link: https://rootsofprogress.org/links-and-tweets-2023-02-22

Announcements

Introducing Speculative Technologies, a private DARPA-like research organization. See also coverage in Forbes and Ben Reinhardt’s AMA on the Progress Forum
I’ll be speaking on how to write about progress at the Thesis Festival this weekend
Day One policy memo on enabling faster NIH funding timelines (via @LNuzhna)

Opportunities

Convergent Research is hiring a Director of Development (via @AGamick)
Dwarkesh is looking for help with his podcast

Links

A case for more techno-optimistic storytelling
Patrick Collison interviewed by Reid Hoffman
David Deutsch interviewed by Naval Ravikant
Marc Andreessen interviewed by Dwarkesh Patel
OpenAI will let you “define your AI’s values”
“Most of the rank and file at the NRC are not anti-nuclear”

Queries

Is there a historical price series for moving a cubic meter of dirt?

Quotes

Why was the wind never used on roads? Why no carriages or wagons with sails?
When Carnegie hired a staff chemist: “great secrets did the doctor open up to us”
Why corporate R&D in the 1980s was mediocre compared to Bell Labs and GE
“Even the aspiration” to sustainability is dangerous, says David Deutsch
Francis Bacon with the understatement of the millennium
“It can be done”

Tweets & retweets

The Martian as one of the only tech positive films out there (@brian_armstrong)
Katalin Karikó has received enough awards to fill a cabinet
Why is it so expensive to build transit in the US? Summary findings
LLMs are just a massively scaled up version of the “travesty generator”
Toolformer lets language models use tools like web search, calculators, and APIs. Original paper on arXiv
“Within a decade solar will be cheap enough that CO2 will be the best place to get carbon.” And how that will change the economy
Quantifying healthspan in dogs for longevity research (@celinehalioua)
Spy balloons have a long history

Charts

CGO poll: people think social media is important for policy discussions but avoid being political online
Cyborg Periods: There will be multiple AI transitions
by Jan_Kulveit — 2023-02-22T16:09:04.858Z

It can be useful to zoom out and talk about very compressed concepts like ‘AI progress’ or ‘AI transition’ or ‘AGI timelines’. But from the perspective of most AI strategy questions, it’s useful to be more specific.

Looking at all of human history, it might make sense to think of ourselves as at the cusp of an AI transition, when AI systems overtake humans as the most powerful actors. But for practical and forward-looking purposes, it seems quite likely there will actually be multiple different AI transitions:

There will be AI transitions at different times in different domains
In each of these domains, transitions may move through multiple stages:

Stage

[>> = more powerful than]

Description	Present day examples

Human period:

Humans >> AIs

Humans clearly outperform AIs. At some point, AIs start to be a bit helpful.	Alignment research, high-level organisational decisions…

Cyborg period: Human+AI teams >> humans

Human+AI teams >> AIs

Humans and AIs are at least comparably powerful, but have different strengths and weaknesses. This means that human+AI teams outperform either unaided humans, or pure AIs.	Visual art, programming, trading…

AI period:

AIs >> humans

(AIs ~ human+AI teams)

AIs overtake humans. Humans become obsolete and their contribution is negligible to negative.	Chess, go, shogi…

Some domains might never enter an AI period. It’s also possible that in some domains the cyborg period will be very brief, or that there will be a jump straight to the AI period. But:

We’ve seen cyborg periods before

Global supply chains have been in a cyborg period for decades
Chess and go both went through cyborg periods before AIs became dominant
Arguably visual art, coding and trading are currently in cyborg periods

Even if cyborg periods are brief, they may be pivotal

More on this

below

This means that for each domain, there are potentially two transitions: one from the human period into the cyborg period, and one from the cyborg period into the AI period.

Transitions in some domains will be particularly important

The cyborg period in any domain will correspond to:

An increase in capabilities (definitionally, as during that period human+AI teams will be more powerful than humans were in the human period)
An increase in the % of that domain which is automated, and therefore probably an increase in the rate of progress

...