trentmkelly/LessWrong-43k huggingface dataset:
AGI in sight: our look at the game board
by Andrea_Miotti — 2023-02-18T22:17:44.364Z
Link: https://andreamiotti.substack.com/p/agi-in-sight-our-look-at-the-game

From our point of view, we are now in the end-game for AGI, and we (humans) are losing. When we share this with other people, they reliably get surprised. That’s why we believe it is worth writing down our beliefs on this.

1. AGI is happening soon. Significant probability of it happening in less than 5 years.

Five years ago, there were many obstacles on what we considered to be the path to AGI.

But in the last few years, we’ve gotten:

Powerful Agents (

Agent57

,

GATO

,

Dreamer V3

)
Reliably good Multimodal Models (

StableDiffusion

,

Whisper

,

Clip

)
Just about every language tasks (

GPT3

,

ChatGPT

,

Bing Chat

)

Human and Social Manipulation

Robots (

Boston Dynamics

,

Day Dreamer

,

VideoDex

,

RT-1: Robotics Transformer

[1])
AIs that are superhuman at just about any task we can (or simply bother to) define a benchmark, for

We don’t have any obstacle left in mind that we don’t expect to get overcome in more than 6 months after efforts are invested to take it down.

Forget about what the social consensus is. If you have technical understanding of current AIs, do you truly believe there are any major obstacles left? The kind of problems that AGI companies could reliably not tear down with their resources? If you do, state so in the comments, but please do not state what those obstacles are.

2. We haven’t solved AI Safety, and we don’t have much time left.

We are very close to AGI. But how good are we at safety right now? Well.

No one knows how to get LLMs to be truthful. LLMs make things up, constantly. It is really hard to get them not to do this, and we don’t know how to do this at scale.

Optimizers quite often break their setup in unexpected ways. There have been

quite a few examples

of this. But in brief, the lessons we have learned are:

Optimizers can yield unexpected results
Those results can be very weird (like breaking the simulation environment)
Yet very few extrapolate from this and find these as worrying signs

No one understands how large models make their decisions.

Interpretability

is extremely nascent, and mostly empirical. In practice, we are still completely in the dark about nearly all decisions taken by large models.

RLHF and Fine-Tuning

have

not worked

well

so far. Models are often unhelpful, untruthful, inconsistent, in many ways that had been theorized in the past. We also witness goal misspecif...
We should be signal-boosting anti Bing chat content
by mbrooks — 2023-02-18T18:52:18.818Z

The more we can get the public (and investors) to realize the failures of the Bing chatbot launch, the more the market will turn against Microsoft and make further launches fiscally riskier.

It seems like Microsoft has

rushed the launch to pwn Google

. If we make this a big enough corporate mistake, then Microsoft (and likely Google) will be a bit more hesitant in the future.

Less Wrong (and EA) has many thousands of people deeply concerned about AI progress. Why aren't we collectively signal-boosting content that is critical of the Bing chat bot and Microsoft's strategy?

What do I mean by signal-boosting?

A simple example:

The most read anti-Bing content is likely this

NYT article

(described in this

Less Wrong post

), yet it only has 2.7k comments. Less Wrong + EA could probably drive those numbers up to 20k if we actually cared/tried/wanted to. How much more would NYT signal-boost this article to their readers if it breaks the record for most comments? (One of the most-read articles in 2022 was

Will Smith Apologizes to Chris Rock After Academy Condemns His Slap

and it has 6.3k comments.)

A slightly less obvious example:

https://www.tiktok.com/@benthamite

is the most popular EA-aligned TikTok I know of. Why aren't we helping Lacey and Ben (and any other EA/Rat/AI safety-aligned TikTokers like

https://www.tiktok.com/@robertmilesai

) make dozens of videos a day about Bing chat and boosting them in any way possible (paying them to be shared as ads as well?) to reach hundreds of thousands of people.

Eliezer apparently agrees with the general idea and has signal-boosted some content on Twitter.

This tweet

has 65 retweets and 569 likes yet it has 100k views.

What would the views look like if there were 2k retweets and 2k likes 10 minutes after he posted it?

How many other pieces of content could we easily signal-boost? Other articles, podcasts, YouTube channels, tweets, TikToks, etc.

I honestly believe that thousands of dedicated and smart individuals all collectively trying to signal-boost a story across all media channels could significantly shape the public's perception of the launch.

So this leaves us with a simple question. Why aren't we?

Some guesses off the top of my head:

People are lazy and don't think their single comment on an NYT will do anything (cause it won't) so they don't do anything.
People would rather test out the

...
Can talk, can think, can suffer.
by Ilio — 2023-02-18T18:43:04.215Z

Executive summary: heavy on Westworld, SF, AI, Cognitive Sciences, metaphysical sprouting and other cracked poteries.

//

Supplementary tag: lazystory, which means a story starting short and dirty, then each iteration is refined and evolved with the help of muses, LLMs and other occasional writers.

//

Rules: if we need a safe space to discuss please shoot a private message and I’ll set the first comment for your thread with our agreed local policy. You can also use the comment system as usual if you’d like to suggest the next iteration for the main text, or provide advices on potential improvements.

//

Violence is the last refuge of the incompetent. You’re with me on this, Bernaaard?

//

What I regret most about Permutation City is my uncritical treatment of the idea of allowing intelligent life to evolve in the Autoverse. Sure, this is a common science-fictional idea, but when I thought about it properly (some years after the book was published), I realised that anyone who actually did this would have to be utterly morally bankrupt. To get from micro-organisms to intelligent life this way would involve an immense amount of suffering, with billions of sentient creatures living, struggling and dying along the way. Yes, this happened to our own ancestors, but that doesn’t give us the right to inflict the same kind of suffering on anyone else.

This is potentially an important issue in the real world. It might not be long before people are seriously trying to “evolve” artificial intelligence in their computers. Now, it’s one thing to use genetic algorithms to come up with various specialised programs that perform simple tasks, but to “breed”, assess, and kill millions of sentient programs would be an abomination. If the first AI was created that way, it would have every right to despise its creators.

Greg Egan

https://www.gregegan.net/PERMUTATION/FAQ/FAQ.html

//

Epistemic Status: It is now. You’re in my dream.

//

Dear lords of the AIs,

Congrats!

You have now successfully answered one of the most amazingly deep scientific question, what it means to have meaning. A few noticed it from as far as 2010 (not me!). By 2023 it’s basically common knowledge that it’s data compression, at least for our current AIs.

I love to know that. I’ve been wondering about these questions for about fourthy years, so let me tell you that was the best advance in cognitive science since Split-Brain phenom...
Parametrically retargetable decision-makers tend to seek power
by TurnTrout — 2023-02-18T18:41:38.740Z
Link: https://arxiv.org/abs/2206.13477

This paper—accepted as a poster to NeurIPS 2022— is the sequel to Optimal Policies Tend to Seek Power. The new theoretical results are extremely broad, discarding the requirements of full observability, optimal policies, or even requiring a finite number of options.

Abstract:

If capable AI agents are generally incentivized to seek power in service of the objectives we specify for them, then these systems will pose enormous risks, in addition to enormous benefits. In fully observable environments, most reward functions have an optimal policy which seeks power by keeping options open and staying alive. However, the real world is neither fully observable, nor must trained agents be even approximately reward-optimal.

We consider a range of models of AI decision-making, from optimal, to random, to choices informed by learning and interacting with an environment. We discover that many decision-making functions are retargetable, and that retargetability is sufficient to cause power-seeking tendencies. Our functional criterion is simple and broad.

We show that a range of qualitatively dissimilar decision-making procedures incentivize agents to seek power. We demonstrate the flexibility of our results by reasoning about learned policy incentives in Montezuma's Revenge. These results suggest a safety risk: Eventually, retargetable training procedures may train real-world agents which seek power over humans.

Examples of agent designs the power-seeking theorems now apply to:

Boltzmann-rational agents,
Expected utility maximizers and minimizers,

Even if they uniformly randomly sample a few plans and then choose the best sampled

Satisficers (as I formalized them),
Quantilizing with a uniform prior over plans, and
RL-trained agents under certain modeling assumptions.

The key insight is that the original results hinge not on optimality per se, but on the retargetability of the policy-generation process via a reward or utility function or some other parameter. See Satisficers Tend To Seek Power: Instrumental Convergence Via Retargetability for intuitions and illustrations.

Why am I only now posting this?

First, I've been way more excited about shard theory. I still think these theorems are really cool, though.

Second, I think the results in this paper are informative about the default incentives for decision-makers which "care about things." IE, make decisions on the basi...
Near-Term Risks of an Obedient Artificial Intelligence
by ymeskhout — 2023-02-18T18:30:50.745Z

[Hi everyone, Yassine here: long-time orbiter first-time poster. I figured this piece I published on 1/23/23 would be as good as any of an introduction.]

I’ll be honest: I used to think talk of AI risk was so boring that I literally banned the topic at every party I hosted. The discourse generally focused on existential risks so hopelessly detached from any semblance of human scale that I couldn’t be bothered to give a shit. I played the Universal Paperclips game and understood what a cataclysmic extinction scenario would sort of look like, but what the fuck was I supposed to do about it now? It was either too far into the future for me to worry about it, or the singularity was already imminent and inevitable. Moreover, the solution usually bandied about was to ensure AI is obedient (“aligned”) to human commands. It’s a quaint idea, but given how awful humans can be, this is just switching one problem for another.

So if we set aside the grimdark sci-fi scenarios for the moment, what are some near-term risks of humans using AI for evil? I can think of three possibilities where AI can be leveraged as a force multiplier by bad (human) actors: hacking, misinformation, and scamming.

(I initially was under the deluded impression that I chanced upon a novel insight, but in researching this topic, I realized that famed security researcher Bruce Schneier already wrote about basically the same subject way back in fucking April 2021 [what a jerk!] with his paper The Coming AI Hackers. Also note that I’m roaming outside my usual realm of expertise and hella speculating. Definitely do point out anything I may have gotten wrong, and definitely don’t do anything as idiotic as make investment decisions based on what I’ve written here. That would be so fucking dumb.)

Computers are given instructions through the very simple language of binary: on and off, ones and zeroes. The original method of “talking” to computers was a punch card, which had (at least in theory) an unambiguous precision to its instructions: punch or nah, on or off, one or zero. Punch cards were intimate, artisanal, and extremely tedious to work with. In a fantastic 2017 Atlantic article titled The Coming Software Apocalypse, James Somers charts how computer programming changed over time. As early as the 1960s, software engineers were objecting to the introduction of this new-fangled “assembly language” as a re...
EIS VII: A Challenge for Mechanists
by scasper — 2023-02-18T18:27:24.932Z

Part 7 of 12 in the

Engineer’s Interpretability Sequence

.

Thanks to Neel Nanda. I used some very nicely-written code of his from

here

. And thanks to both Chris Olah and Neel Nanda for briefly discussing this challenge with me.

MI = “mechanistic interpretability”

Given a network, recover its labeling function's program.

In the last post, I argued that existing works in MI focus on solving problems that are too easy. Here, I am posing a challenge for mechanists that is still a toy problem but one that is quite a bit less convenient than studying a simple model or circuit implementing a trivial, known task. To the best of my knowledge:

Unlike prior work on MI from the AI safety interpretability community, beating this challenge would be the first example of mechanistically explaining a network’s solution to a task that was not cherrypicked by the researcher(s) doing so.

Gaining a mechanistic understanding of the models in this challenge may be difficult, but it will probably be much less difficult than mechanistically interpreting highly intelligent systems in high stakes settings in the real world. So if an approach can’t solve the type of challenge posed here, it may not be very promising for doing much heavy lifting with AI safety work.

This post comes with a GitHub repository.

Check it out here.

The challenge is actually two challenges in one, and the basic idea is similar to some ideas presented in

Lindner et al. (2023)

.

Challenge 1, MNIST CNN

I made up a nonlinear labeling function that labels approximately half of all MNIST images as 0’s and the other half as 1’s. Then I trained a small CNN on these labels, and it got 96% testing accuracy. The challenge is to use MI tools on the network to recover that labeling function.

Hint 1: The labels are binary.

Hint 2: The network gets 95.58% accuracy on the test set.

Hint 3: The labeling function can be described in words in one sentence.

Hint 4: This image may be helpful.

Challenge 2, Transformer

I made up a labeling function that takes in two integers from 0 to 113 and outputs either a 0 or 1. Then, using a lot of code from Neel Nanda’s

grokking work

, I trained a 1-layer transformer on half of the data. It then got 97% accuracy on the test half. As before, the challenge is to use MI tools to recover the labeling function.

Hint 1: The labels are binary.

Hint 2: The network is trained on 50% o...
Reading Speed Exists!
by Johannes C. Mayer — 2023-02-18T15:30:52.681Z

Today, I talked with somebody about reading speed. I asked him how fast he can read. He didn't answer, instead he said that the concept is abused by people. He said, it's more complicated than to say that you read at a certain reading speed. It depends on if you're reading a novel, a history textbook, or a poem.

I feel like he was falling into a kind of fallacy. He observed that a concept isn't entirely coherent, rejected the concept. However, the concept of reading speed seems real. It seems to capture something about reality.

This becomes obvious once you think about an experiment where we have two people that read the same material and time them. I read "The adventures of the Lightcone team" (or whatever it is called) together with Chu. We made the bed that I can get more than halfway through the book, before she finishes. I bet $5 on that. When she was finished, I almost managed to get halfway through the book. I was trying to read really fast, at the edge of comprehensibility.

Clearly there are latent causes, in each of our brains, that determine how fast we can read while still comprehending the text.

Trying to operationalize the concept that you're talking about, and imagining what sorts of experiments you would try to run to measure it, might be a good general way to avoid the fallacy of dropping a concept and losing its true kernel. Often you don't even need to run the experiment. Imagining it is sufficient.

Edit: see also this follow up comment.
The Practitioner's Path 2.0: the Meditative Archetype
by  — 2023-02-18T15:23:06.622Z
Link: https://guildoftherose.org/articles/path-2-0-the-meditative-archetype

In our last two announcements, we introduced the Practitioner's Path 2.0, our new framework for structured self improvement. The heart of the Path is the Skilltree, which is divided into three archetypes: Pragmatist, Meditative, and Empiricist.

Today, we'll go over the Meditative archetype and their associated Skills.

More than anything, the Meditative wants to live a good life. What counts as a 'good life' varies from one Meditative to the next. One Meditative might focus on their artistic skills, another might become a beloved community leader, and a third might strive for perfection in mind and body.

What unites Meditatives is the way they approach the world. Rather than seeking to exploit or codify the world in rigid data, the Meditative wants to experience it. They see the world through narratives, and they understand their place in it. They face their fears and live their passions.

In the Meditative tree, you'll find Skills focused on:

Mental and physical health

Artistic endeavors

Growing the garden of community

Understanding and using narratives

Some people might call the Meditatives aimless or excessively open-minded, but the Meditative knows that a relentless focus on one part of life leads to dissatisfaction and burnout. A Meditative's first priority is themself — not in a selfish, hedonistic sense, but in a setting-good-boundaries sense.

Introductory Skill Examples

The Archetypal Meditative

Morgan Vesper always loved drawing — and their parents always said art wasn't a proper career. That shadow whispered in Morgan's ear when they went to college, and again when they graduated into a job in a field they hated.

It took another three years for Morgan to realize that their life was on a dead-end course. Now, a decade later, Morgan is a max-level Meditative with a underground hit webcomic, Inkwitch.

During the day, Morgan works on their webcomic. Sometimes they look back and marvel at how much their art has improved since their early posts. Probably from the daily practice, though it never feels like work. Drawing is just too much fun.

In the evening, Morgan works out three times a week with an eclectic routine that varies every time. (It helps stave off boredom.) They also hold regular parties at their house. Most of Morgan's friends aren't anyone 'of note' — but they don't care. Status games have no place in their life.

The Practitioner's Path 2.0 laun...
Should we cry "wolf"?
by Tapatakt — 2023-02-18T11:24:17.799Z

Epistemic status: I am confused and trying to become less confused.

So, our shepherd community noticed the wolf. It's a small wolf, about the size of a chihuahua, and is unlikely to cause any serious damage. Should we cry "wolf"?

Possible arguments against crying "wolf"

When someone cries "wolf", villagers think that what is really meant is "a big wolf that is going to eat us all". Therefore, villagers will decide that we are lying, and will not listen to us next time.
Maybe, publicity is useless, and can even harm us, we need to develop anti-wolf weapons ourselves.
In fact, we're not sure if it's really a wolf and not a duck with a strange combination of mutations that makes it look like a wolf, run like a wolf, and howl like a wolf.
If this miniature wolf bites someone's fingers, villagers may finally recognize that wolves are dangerous.

Possible arguments for crying "wolf"

If we now ignore a chihuahua-sized wolf, it makes it more likely that we will ignore a corgi-sized wolf, then a sheepdog-sized wolf, and then a normal-sized wolf.
If we now cry "wolf" and no one comes to the rescue, then later we can use this precedent to argue for the creation of adequate wolf prevention systems.
If we now cry "wolf" and help comes to us, then next time villagers will already have the experience of responding to the cries "wolf" and it will be easier for them to come to the rescue again.
We have serious concerns that sufficiently advanced normal-sized wolves will have stealth technology that allows them to pretend to be the size of a chihuahua.

What is the relative weights of these arguments?

What arguments did I miss?
Name of the fallacy of assuming an extreme value (e.g. 0) with the illusion of 'avoiding to have to make an assumption'?
by FlorianH — 2023-02-18T08:11:07.506Z

In a model, often people have the intuition that removing a transformation is 'removing the need for assuming a particular value', while in reality, it simply means they are imposing a trivial/extreme value.

Example, Joe: "Let's take the unweighted sum of the next 20 years' expenditures, instead of their present-value discounted sum, so we don't have to decide on a discount rate". In reality, Joe has, once again, sneaked-in a 0%p.a. discount rate, a value neither Joe nor most others would want to defend as a palatable rate. Joe would not defend that, per se, not discounting gives a clearer picture; and we would be readily able to agree with Joe on some substantially non-zero rate to be at least more realistic than 0%. Joe really only wants the 'not discount' such as to, seemingly, 'avoid having to pick a value'.

A common fallacy that I think might have a name but I'm not sure anymore whether I've seen it anywhere, and don't know how to search for it. Help anyone?

(Let me know if this an unworthily trivial question spamming the forum, happy to remove it in this case)