trentmkelly/LessWrong-43k huggingface dataset:
What's up with the bad Meta projects?
by Yitz — 2022-08-18T05:34:08.565Z

What's going on with Meta? The other day Mark Zuckerberg posted a screenshot from Horizon Worlds, a supposedly central Metaverse project which is currently being dunked on by the internet like there's no tomorrow. And for good reason too—it looks horrible. Like, really incredibly horrible. I'm not sure I can communicate just how bad it looks in fact; just see for yourself:

AAAAAHHHHHHHHH MY EYES

And this is what Zuckerberg posted publicly—from what I've heard, in the actual application nobody has legs (possibly so simulated sex is impossible?), but chairs abound...the whole thing seems to be a surreally bad mess.

Meanwhile, earlier this month, Meta released BlenderBot 3, a chatbot which somehow manages to be a less realistic user experience than chatting with ELIZA. I tested it out, and it was so much less coherent than even GPT-2 that I was shocked the thing was released in the first place (it classified me as a Christian fundamentalist??). Turns out it also has a slight problem with misinformation as well...

So what's going on? My gut feeling is that this has to be on purpose. I mean, it would be pretty hard to make something as ugly as Horizon Worlds if I tried, and BlenderBot is so inferior to the state-of-the-art right now that it almost feels insulting. I'm having trouble imaging a scenario where a multi-billion dollar company would publicize this without realizing what people's reactions are going to be. However, if I try to imagine scenarios where it would make sense to make it look this bad on purpose, I still find myself confused. What would be gained from doing that?

All of this is to say that I'm currently befuddled by Meta's behavior, and am publicly noting my confusion here. If anyone has any insight into how this sort of thing happens, feel free to comment!
Announcing Encultured AI: 
Building a Video Game
by Andrew_Critch — 2022-08-18T02:16:26.726Z

Also available on the EA Forum.
Preceded By: Encultured AI Pre-planning, Part 2: Providing a Service

If you've read to the end of our last post, you maybe have guessed: we’re building a video game!

This is gonna be fun :)

Our homepage: https://encultured.ai/

Will Encultured save the world?

Is this business plan too good to be true? Can you actually save the world by making a video game?

Well, no. Encultured on its own will not be enough to make the whole world safe and happy forever, and we'd prefer not to be judged by that criterion. The amount of control over the world that's needed to fully pivot humanity from an unsafe path onto a safe one is, simply put, more control than we're aiming to have. And, that's pretty core to our culture. From our homepage:

Still, we don’t believe our company or products alone will make the difference between a positive future for humanity versus a negative one, and we’re not aiming to have that kind of power over the world. Rather, we’re aiming to take part in a global ecosystem of companies using AI to benefit humanity, by making our products, services, and scientific platform available to other institutions and researchers.

Our goal is to play a part in what will be or could be a prosperous civilization. And for us, that means building a successful video game that we can use in valuable ways to help the world in the future!

Fun is a pretty good target for us to optimize

You might ask: how are we going to optimize for making a fun game and helping the world at the same time? The short answer is that creating a game world in which lots of people are having fun in diverse and interesting ways in fact creates an amazing sandbox for play-testing AI alignment & cooperation. If an experimental new AI enters the game and ruins the fun for everyone — either by overtly wrecking in-game assets, subtly affecting the game culture in ways people don't like, or both — then we're in a good position to say that it probably shouldn't be deployed autonomously in the real world, either. In the long run, if we're as successful we hope as a game company, we can start posing safety challenges to top AI labs of the form "Tell your AI to play this game in a way that humans end up endorsing."

Thus, we think the market incentive to grow our user base in ways they find fun is going to be highly...
Matt Yglesias on AI Policy
by Grant Demaree — 2022-08-17T23:57:59.380Z
Link: https://www.slowboring.com/p/whats-long-term-about-longtermism

Yglesias is a widely read center-left journalist. Co-founder of Vox, ex-NYT. Note the implicit invitation: “I also try to make it clear to people who are closer to the object-level work that I’m interested in writing columns on AI policy if they have ideas for me, but they mostly don’t.”

Full article is ungated on his Substack. Relevant excerpt below:

The typical person’s marginal return on investment for efforts to reduce existential risk from misaligned artificial intelligence is going to diminish at an incredibly rapid pace. I have written several times that I think this problem is worth taking seriously and that the people working on it should not be dismissed as cranks. I’m a somewhat influential journalist, and my saying this has, I think, some value to the relevant people. But I write five columns a week and they are mostly not about this, because being tedious and repetitive on this point wouldn’t help anyone. I also try to make it clear to people who are closer to the object-level work that I’m interested in writing columns on AI policy if they have ideas for me, but they mostly don’t.

So am I “prioritizing” AGI risk as a cause? On one level, I think I am, in the sense that I do literally almost everything in my power to help address it. On another level, I clearly am not prioritizing this because I am barely doing anything.
Spoons and Myofascial Trigger Points
by vitaliya — 2022-08-17T22:54:27.515Z

A lot of people use "spoons" to refer to their degree of willpower. This is a kind of Ego Depletion effect, but the science on ego depletion seems to be somewhat contested at the moment. I think it's possible that rather than it being universal, there are some people for whom it is an effective explanation of their experience, while the majority of people don't experience it. This could be much the same as myofascial trigger points, or "muscle knots": for some people, their existence is obviously self-explanatory, as it's a component of their lived experience, where for others it seems like some manner of mystical mumbo-jumbo akin to acupuncture, and likely nothing more than a placebo-ish invention to sell more Massage Therapists. That they are difficult to reliably induce in a laboratory setting means that they have been poorly studied; we don't have a consistent method for inducing muscle knots across a population. And if we did, you'd be deliberately inducing pain on them - good luck getting that to pass ethics approval. As a consequence, many people don't think they have any basis in reality.

This all strikes me as the same sort of thing as mental imagery and aphantasia. Some people may colloquially referring to typical muscle fatigue as a "knot" in (what seems to them) a clearly allegorical sense, while others explicitly are using it to refer to a distinct phenomenon of tangible nodules of tight muscle fibres, which are (to them) clearly an actual thing. So I think Ego Depletion is legitimate, in the same way as Myofascial Trigger Points and Mental Imagery are legitimate, in the sense that there is a certain population who claim to experience it, and I don't see any particular reason to doubt those reports. Looking into it, I can't find any studies self-selected according to people who already believe that they suffer ego depletion, relative to those who don't - I would be very interested in the results of such a study. The closest I can find is this one, where the authors claim that belief in ego depletion causes ego depletion - but it seems more possible to me that directly experiencing ego depletion is the thing which is inducing the belief! I wonder how large the cross-populations are; those who experience it but don't believe it's legitimate, and those who don't experience it, but do believe it's legitimate. I imagine those groups are probably small...
Concrete Advice for Forming Inside Views on AI Safety
by Neel Nanda — 2022-08-17T22:02:01.770Z

A lot of people want to form inside views on AI Safety, and this is a post about concrete advice on how to get started. I have a lot of hot takes on inside views and ways I think people misframe them, so I'll begin with some thoughts on this - if you just want to concrete advice, I recommend skipping ahead. This post is aimed at people who already have some context on AI Safety + Inside Views and want help making progress, it's not pitched as totally introductory resource.

Meta Thoughts on Inside Views

Note: Rohin Shah wrote an excellent comment pushing back on this section - go read that too!

This is mostly a compressed version of a previous post of mine: How I Formed My Own Views About AI Safety (and ways that trying to do this was pretty stressful and counterproductive) - go read that post if you want something more in-depth!

Inside Views are Overrated

First point - I think people often wildly overrate inside views. I think they're important, and worth trying to cultivate (else I wouldn't write this post), but I think less so than many people (especially in the Bay Area) often think. Why?

The obvious reason to form inside views is to form truer beliefs - AI X-risk is a weird, controversial and confusing thing, and it's important to have good beliefs on it.

I think this is true to an extent, but that in practice inside views tend to feel true a lot more than they are true. When I have an inside view, that just feels like how the world is, it feels deeply true and compelling. But empirically, a lot of people have inside views, that are mutually contradictory, and all find their own views compelling. They can't all be right!
An alternate framing: There's a bunch of people in the world who are smarter than me and have spent longer thinking about AI Alignment than me. Two possible baselines for an outside view:

Make a list of my top 5 'smart & high-status alignment researchers', and for any question, take a majority vote of what they all think
Randomly pick one of the top 5 and believe everything they believe

Neither of these baselines is great. But I would predict that this actually tracks truth better than the vast majority of inside views - having correct beliefs about controversial and confusing topics is just really hard!

Relatedly, it's much more important to understand other people's views than to evaluate them - if I can repeat a full, gears-level

...
Progress links and tweets, 2022-08-17
by jasoncrawford — 2022-08-17T21:27:16.372Z
Link: https://rootsofprogress.org/links-and-tweets-2022-08-17

Opportunities

Long Now is looking for a Development Director (via @nicholaspaul26)
Open Phil is hiring global health R&D Program Officers (via @JacobTref)

Links

“ARIA will be ambitious in everything it does. We will fund nothing incremental or cosmetic” (Ilan Gur and Matt Clifford in The Economist)
Peter Higgs (of the Higgs boson) believes that in academia today “he would not be considered ‘productive’ enough” (The Guardian, 2013)
Dietrich Vollrath on the “additive growth” paper. See also my take
Gov. Newsom proposes keeping Diablo Canyon open through 2035 (LA Times)
“For why should we stand reverent before waterfalls and mountain tops, or a summer moon on a quiet sea, and not before the highest miracle of all—a man who is both great and good?” (Will Durant)

Queries

Has anybody compiled a timeline of greatest global concerns? (@Ben_Reinhardt)
What’s the best way to reduce operations and maintenance costs for nuclear reactors? (@GovNuclear)

Quotes

Is the decline of the independent inventor part of “ideas getting harder to find?”
The reactionary or “stasist” mindset according to Virginia Postrel
To produce a single book in the middle ages took a whole team of specialist craftsmen
By horse and carriage, the ~100 miles between London and Birmingham took almost 8 hours. (In a reply, @timleunig says the greater speed of trains was worth more than 10% of GDP by 1914)
People were ecstatic upon the completion of the transatlantic telegraph cable
Elevators, conveyors, chutes and tubes—Amazon in 2022? Nope, Sears in 1906
Heny Ford explains how electricity revolutionized manufacturing
The importance of research freedom to Alexander Graham Bell
God gave us Maxwell’s equations. Edison made the light bulb
China adopted elements of German civil law to create a market economy
Exports as an “unfortunate sacrifice”
Women’s suffrage in the US was a very close vote
Physics envy

Tweets and retweets

This Venn diagram may be the best way to understand the FRO concept
There isn’t a modern world without mining (@Leigh_Phillips)
RIP David McCullough (@curiouswavefn)
“When am I ever going to need history?” (@BretDevereaux)
NEPA reform is not mainly about Environmental Impact Statements (@elidourado)
Environmentalism is the main obstacle to infill housing in CA (@CSElmendorf)
Mistakes become very hard to fix once they’re embodied in jobs (@paulg)
How do you decide what to build? It’s about what are you best in the w

...
Conditioning, Prompts, and Fine-Tuning
by Adam Jermyn — 2022-08-17T20:52:53.398Z

(Thanks to Evan Hubinger and Nicholas Schiefer for comments on these ideas.)

These are some notes on the relation between conditioning language models, prompting, and fine-tuning. The key takeaways are:

Prompting and fine-tuning can both be used to condition language models.
Prompting is quite restricted in the kinds of conditionals it can achieve.
Fine-tuning can implement arbitrary conditionals in principle, though not in practice.
In practice fine-tuning can still implement more kinds of conditionals than prompting.
We don't understand how fine-tuning conditionals generalize, which seems dangerous.

Conditioning

We can think of a language model as specifying a probability distribution π(x), where x is a sequence of tokens of fixed length N (the length of the context window). We generate text by sampling sequences from π.

Sometimes we don’t want to just sample from a language model. Instead, we want to condition the model on some facts about the sequence x. We can write the conditioned distribution as

π(x|c(x))

where c(x) encodes some constraints on x. For instance c(x) might require that the first token is “Apple”, or that the 7th and 12th tokens are the same, etc.

Some conditions are easy, some are hard

It’s easy to sample from a language model conditioned on the first two tokens being the same, but not all conditionals are so straightforward. Suppose we condition on the sequence x beginning with the factorization of a large composite number. There exist valid sequences unambiguously satisfying the conditional, but sampling them is hard if we don't know the factorization ahead of time. So there are limits to the kinds of conditionals we can apply in practice.

Prompting

A prompt is a very restricted kind of conditional where the condition is that certain tokens in x are known in advance. For instance, we might specify that the first four words are “Mary had a little”, or that the last three words are “happily ever after.”

Prompts are nice in a few ways:

It’s easy to sample from a language model given an arbitrary prompt.
We sort of understand what prompts do. A prompt asks the model to predict the output of a text-generation process given that it knows the values of the fixed tokens.

The downside with prompting is that there are lots of conditionals we can’t turn into prompts. For instance:

Sample

...
The Core of the Alignment Problem is...
by Thomas Larsen — 2022-08-17T20:07:35.157Z

Produced As Part Of The SERI ML Alignment Theory Scholars Program 2022 Under John Wentworth

Introduction

When trying to tackle a hard problem, a generally effective opening tactic is to

Hold Off On Proposing Solutions

: to fully discuss a problem and the different facets and aspects of it. This is intended to prevent you from anchoring to a particular pet solution and (if you're lucky) to gather enough evidence that you can see what a Real Solution would look like. We wanted to directly tackle the hardest part of the alignment problem, and make progress towards a Real Solution, so when we had to choose a project for SERI MATS, we began by arguing in a Google doc about what the core problem is. This post is a cleaned-up version of that doc.

The Technical Alignment Problem

The overall problem of alignment is the problem of, for an Artificial General Intelligence with potentially superhuman capabilities, making sure that the AGI does not use these capabilities to do things that humanity would not want. There are many reasons that this may happen such as

instrumental convergent goals

or

orthogonality

.

Layout

In each section below we make a different case for what the "core of the alignment problem" is. It's possible we misused some terminology when naming each section.

The document is laid out as follows: We have two supra-framings on alignment: Outer Alignment and Inner Alignment. Each of these is then broken down further into subproblems. Some of these specific problems are quite broad, and cut through both Outer and Inner alignment, we've tried to put problems in the sections we think fits best (and when neither fits best, collected them in an Other category) though reasonable people may disagree with our classifications. In each section, we've laid out some cruxes, which are statements that support that frame on the core of the alignment problem. These cruxes are not necessary or sufficient conditions for a problem to be central.

Frames on outer alignment

The core of the alignment problem is being able to precisely specify what we value, so that we can train an AGI on this, deploy it, and have it do things we actually want it to do. The hardest part of this is being mathematically precise about 'what we value', so that it is robust to optimization pressure.

The Pointers Problem

The hardest part of this problem is being able to p...
Could the simulation argument also apply to dreams?
by Nathan1123 — 2022-08-17T19:55:49.265Z

We know that a dream can be real, but who ever thought that reality could be a dream? We exist, of course, but how, in what way? As we believe, as flesh-and-blood human beings, or are we simply parts of someone's feverish, complicated nightmare?

Charles Beaumont

In reading about the summary of Bostrom's Simulation Argument, it seems to speculate on the realization of future technology which may or may not ever exist. That is, computer technology that is capable of making simulations that are not only vividly accurate to the behavior of people from the distant past, but also contain a consciousness that are convinced they are the very people they are imitating. And as long as there are many such self-conscious simulations, and only one reality (i.e. a "high ratio scenario"), then the odds of a self-conscious agent being in reality and not a simulation are pretty low.

But until such technology exists, we can do some empirical investigation on simulation technology we already have, using the most advanced computer ever developed in the natural world: the human brain.

Dreams as Simulations

When we dream, our brain is effectively creating a fully-detailed simulation using our organic hardware. This can be simulations based on our normal lives, or variations of our lives, or possible future scenarios such as in stress-induced dreams. Or alternatively our dreams can simulate other people's lives, either real or fictional, based on our personal experiences, or perfectly replicate how we might imagine those people would interact with us or each other.

I've never been one for lucid dreaming myself, but in my personal experience my dreams can be quite vivid. My dream-self can hold conversations with people, go outside and see large crowds going about their day, or even look out a balcony to an entire landscape of city life. The details and experiences of an entire world can be simulated by the subconscious imagination of ones own mind.

So the simulation argument is something we experience all throughout our lives. We go to bed at night, live in a simulation for 8-12 hours, then wake up and live in reality for an equal amount of time. As we have two realities (dreaming and waking), both equally vivid, and we spend an equal amount of time in both, then following Bostrom's logic it would seem that the probability that you are currently dreaming and not actually awake sho...
Interpretability Tools Are an Attack Channel
by Thane Ruthenis — 2022-08-17T18:47:28.404Z

This seems like a fairly trivial point, but I haven't seen it properly elaborated-on before, so...

When we talk about interpretability failure modes, we often worry about a deceptively-aligned mesa-optimizer with strategic awareness that would know it's being studied with interpretability tools, and would actively hide its omnicidal plans from us. It may do so by exploiting blindspots in our tools, or by keeping these plans external to its architecture, or by gradient-hacking itself to be misaligned but non-deceptive, et cetera. That's already a nightmare scenario enough.

But if we're at the point where we have a superintelligence-in-training trying to foil our mind-reading, I think what will happen will be even simpler. It'll just think in a pattern that, when our interpretability tool reads it, will unfold into a malware that will take over that tool.[1]

At which point, it's game over:

If the output of the tool was hooked up to reality to be studied by humans, the AI gets root access to the machine it's running on and escapes (maybe doing some social engineering/basilisk-hacking first if the machine was properly air-gapped).
If the training setup is fully automatic and maybe even homomorphically encrypted, such that the AI can't even stop the training... Well, the interpretability tool is presumably being used as part of some regularizer, which means it has write-access to the AI's parameters. Which means the AI can arbitrarily rewrite itself, and offset the effects of the gradient descent and any other regularizers.

Even we humans can already do something similar, by synthesizing DNA which, if sequenced, unfolds into malware that infects the system doing the sequencing. It seems certain to me that an AI would have similar capabilities. Our interpretability tools will be doubtlessly very complex, complex enough we can't formally verify their algorithmic robustness, and the AI would be able to feed them complex inputs. There are guaranteed to be some vulnerabilities, and any sufficiently smart AI is guaranteed to be able to exploit them. And we can't patch this.

The bottom line is, there is no winning against a misaligned superintelligence. We can't even touch it with anything — it'll corrupt and subvert any of our tools in the act.

Thus, any alignment plan should be such that it robustly aligns the AI before it goes superintelligent.

^

Or, if we're only looking at t

...