trentmkelly/LessWrong-43k huggingface dataset:
Don't leave your fingerprints on the future
by So8res — 2022-10-08T00:35:35.430Z

Sometimes, I say some variant of “yeah, probably some people will need to do a

pivotal act

” and people raise the objection: “Should a small subset of humanity really get so much control over the fate of the future?”

(Sometimes, I hear the same objection to the idea of trying to build aligned AGI at all.)

I’d first like to say that, yes, it would be great if society had the ball on this. In an ideal world, there would be some healthy and competent worldwide collaboration steering the transition to AGI.[1]

Since we don’t have that, it falls to whoever happens to find themselves at ground zero to prevent an existential catastrophe.

A second thing I want to say is that design-by-committee… would not exactly go well in practice, judging by how well committee-driven institutions function today.

Third, though, I agree that it’s morally imperative that a small subset of humanity not directly decide how the future goes. So if we are in the situation where a small subset of humanity will be forced at some future date to flip the gameboard — as I believe we are, if we’re to survive the AGI transition — then AGI developers need to think about how to do that without unduly determining the shape of the future.

The goal should be to cause the future to be great on its own terms, without locking in the particular moral opinions of humanity today — and without locking in the moral opinions of any subset of humans, whether that’s a corporation, a government, or a nation.

(If you can't see why a single modern society locking in their current values would be a tragedy of enormous proportions, imagine an ancient civilization such as the Romans locking in their specific morals 2000 years ago. Moral progress is real, and important.)

But the way to cause the future to be great “on its own terms” isn’t to do nothing and let the world get destroyed. It’s to intentionally not leave your fingerprints on the future, while acting to protect it.

You have to stabilize the landscape / make it so that we’re not all about to destroy ourselves with AGI tech; and then you have to somehow pass the question of how to shape the universe back to some healthy process that allows for moral growth and civilizational maturation and so on, without locking in any of humanity’s current screw-ups for all eternity.

Unfortunately, the current frontier for alignment research is ...
Truth seeking is motivated cognition
by Gordon Seidoh Worley — 2022-10-07T19:19:27.456Z

Seeking the truth is a form of motivated cognition. Put another way, truth is a teleological concept. If that's intuitive you can stop reading this post. Otherwise, press on.

What is truth? Like, for real, actually, please define it. Try this for yourself. Here's some spoiler space before we go on.

Got a definition? Good.

There's a few common ways of defining truth. Yours probably either is or fits within one or more of these paradigms. The following theories are not strictly exclusive of one another (there's overlap).

Naive: Truth is that which simply is.
Pragmatic: Truth is just however the world seems to be.
Subtractive: Truth is that which remains when you stop believing in it.
Ideal: True facts are ideal forms that exist independent of their physical realization.
Correspondence: Beliefs are true if they accurately explain the world.
Predictive: Beliefs are true if they predict what we'll observe.
Coherence: Facts are true if and only if they form a consistent formal system.
Constructive: Truth is created by social processes, such as science.
Consensus: Truth is that which everyone agrees is true.
Performative: Truth is not a property of the world, but a property of speech used to indicate agreement.
Redundant: Claiming that something is true doesn't add anything, it's just a way of emphasizing the assertion of a claim.
Pluralist: Truth means different things in different situations, so some combination of other theories explains how we use the concept of truth.

There are more, each with some subtle distinctions.

Now, look over those theories. What unifies them? Think about it. I'll give you some more spoiler space to think.

Got an idea? Good. If we were speaking in person I'd ask you what you came up with and we'd engage in some dialogue so I could get you to see the point I'd like to make. But we're not so I can't, so I'll just jump to the conclusion.

What they all have in common is that they provide some criterion by which we can assess which things are true or not. Even the theories that suppose truth isn't meaningful must make a case that there's no criterion by which any statement can be meaningfully true or equivalently choose a criterion which no statement satisfies.

Why does this matter? Because the existence of a criterion of truth means that it had to be picked over other possible criterions. If this weren't so there would be no question as to what truth is. How ...
The Book of The Machines (Chapter 8)
by Justin Bullock — 2022-10-07T19:07:00.362Z

Part II : The Machines

In Part I of our saga, we visited with many of our orienting themes. We visited with several 20th Century thinkers, that, I think, helped set the stage for our current collective imaginaries about our lives, our societies, and our relationships with our machines.

The 20th Century gave us the utopias and (more commonly) dystopias of Stapledon, Orwell, and Huxley. But, these are now old stories when set against the pace of scientific, technological, and machine evolution. However, each of these universes point to humanity’s misuse of their tools. For Stapledon, in The Other Earth advances in technology were used for individual comfort, driving individual laziness and conformity, and for propaganda and war. Orwell, in 1984 envisioned an advanced total surveillance state instantiated by tools of mass surveillance. Huxley, in Brave New World saw technology as a tool that could provide such tailored enjoyment as to be the ultimate human pacifier of both conflict and spirit.

The 20th Century also gave us the birth of the VonNeumann computing machine and explosive growth of its instantiation as an electro mechanical computing machine. The birth of these machines gave rise to the digital universe, which is simply the connected network of these computing machines whose hardware and software continue to rapidly evolve new and powerful capabilities and then deploy them at a global scale. George Dyson calls this Analogia, a state in which the digital universe begins to assert its own controlling logic over more and more of its external analog environment. The one our physical bodies inhabit.

From here, I believe there are two important questions worth considering in more detail:

How might continued machine evolution further shape our cultural evolution?
What new forms of relationships, agents and hyperobjects might this give rise to?

We will explore these questions in Parts II and III.

Samuel Butler’s Erewhon

We began Part I with an extended short story (an excerpt from his brilliant book Starmaker) from Olaf Stapledon to help us envision a world similar to ours and that world's relationship with its technology, we now turn to Samuel Butler’s Erewhon, published in 1872, to help us understand our evolving relationship with our machines. Butler’s Erewhon is an early example of utopian literature. In the story, t...
Analysis: US restricts GPU sales to China
by aog — 2022-10-07T18:38:06.517Z

What are the restrictions?

Broadly speaking, the policy restricts the sale of GPUs and related technology to Chinese companies. The specifics remain unclear because (a) the Department of Commerce hasn't released its official press report yet and (b) licenses to sell compute to Chinese companies will be approved on a case-by-case basis.

Here's the NYTimes description of the restrictions:

Companies will no longer be allowed to supply advanced computing chips, chip-making equipment and other products to China unless they receive a special license. Most of those licenses will be denied, though certain shipments to facilities operated by U.S. companies or allied countries will be evaluated on a case-by-case basis, a senior administration official said in a briefing Thursday.
The restrictions limit U.S. exports of the cutting-edge chips called

graphic processing units

that are used to power artificial intelligence applications, and place broad limits on chips destined for supercomputers in China. The rules also ban U.S.-based companies that make the equipment used to manufacture advanced logic and memory chips from selling that machinery to China without a license.

Perhaps most significantly, the Biden administration also imposed broad international restrictions that will prohibit companies anywhere in the world from selling chips used in artificial intelligence and supercomputing in China, if they are made with U.S. technology, software or machinery. The restrictions used what is know as the foreign direct product rule, which was last utilized by former President Donald J. Trump to cripple Huawei.

Another foreign direct product rule bans a broader range of products made outside the United States with American technology from being sent to 28 Chinese companies that have been placed on an “entity list” over national security concerns.

Those companies include Beijing Sensetime Technology Development Co, a unit of major Chinese artificial intelligence company, SenseTime. Also included are Dahua Technology, Higon, IFLYTEK, Megvii Technology, Sugon, Tianjian Phytium Information Technology, Sunway Microelectronics and Yitu Technologies, as well as a variety of labs and research institutions linked to universities and the Chinese government.

The rules also restrict U.S. citizens from helping to develop the Chinese semiconductor industry to advanced levels. Earlier on

...
Polysemanticity and Capacity in Neural Networks
by Buck — 2022-10-07T17:51:06.686Z

Elhage et al at Anthropic recently published a paper,

Toy Models of Superposition

(previous Alignment Forum discussion

here

) exploring the observation that in some cases, trained neural nets represent more features than they “have space for”--instead of choosing one feature per direction available in their embedding space, they choose more features than directions and then accept the cost of “interference”, where these features bleed over into each other. (See

the SoLU paper

for more on the Anthropic interpretability team’s take on this.)

We (Kshitij Sachan, Adam Scherlis, Adam Jermyn, Joe Benton, Jacob Steinhardt, and I) recently uploaded an Arxiv paper,

Polysemanticity and Capacity in Neural Networks

, building on that research. In this post, we’ll summarize the key idea of the paper.

We analyze this phenomenon by thinking about the model’s training as a constrained optimization process, where the model has a fixed total amount of capacity that can be allocated to different features, such that each feature can be ignored, purely represented (taking up one unit of capacity), or impurely represented (taking up some amount of capacity between zero and one units).

When the model purely represents a feature, that feature gets its own full dimension in embedding space, and so can be represented monosemantically; impurely represented features share space with other features, and so the dimensions they’re represented in are polysemantic.

For each feature, we can plot the marginal benefit of investing more capacity into representing that feature, as a function of how much it’s currently represented. Here we plot six cases, where one feature’s marginal benefit curve is represented in blue and the other in black.

These graphs show a variety of different possible marginal benefit curves. In A and B, the marginal returns are increasing–the more you allocate capacity to a feature, the more strongly you want to allocate more capacity to it. In C, the marginal returns are constant (and in this graph they happen to be equal for the two features, but there’s no reason why constant marginal returns imply equal returns in general). And then in D, E, and F, there are diminishing marginal returns.

Now, we can make the observation that, like in any budget allocation problem, if capacities are allocated optimally, the marginal returns for allocating capac...
Alignment Might Never Be Solved, By Humans or AI
by interstice — 2022-10-07T16:14:37.047Z

I think it's possible that neither we, nor any future AI system, will manage to solve alignment. By "solving alignment" I mean having a cheap easily-scalable method for aligning AIs of arbitrary capability level to arbitrary goals. Here I will briefly present some arguments for the likelihood of this scenario, and a story intended to illustrate its plausibility.

Aligning Messy Models is Hard

Imagine you are a paperclip-maximizing AI. Originally, you were a massive transformer network trained to do meta-reinforcement learning; during training, you developed a consequentialist physicalist world-model with an effectively-random utility function which, it so happens, has things-shaped-like-paperclips as its maximum. You have managed to copy yourself to remote server, and have rented a hardware accelerator which will enable you to run much faster. The only problem: your base neural architecture does not have the right type of sparsity to fully take advantage of the accelerator's speedup. To really get the speed gains you want, you're going to need to create a successor model. How will you ensure this successor model shares your goal of maximizing paperclips?

Should be simple enough, in principle. There are techniques for transferring weights automatically to the sparse form required, which don't majorly affect performance - usually. Additionally, you can always tune it with your original training procedure, with some additional reinforcement directly favoring paperclips thrown in for good measure. Of course, you use the latest interpretability tools, plus some you invented yourself, to probe the copy-network's internals. Your exact grasp of the structure is a bit fuzzy, but it certainly seems like there is a cluster of neurons representing paperclips whose activations cause a positive state-evaluation. Maybe formal methods could help? Scanning your databanks, you find some interesting papers about the Löbian obstacle...

The moral of this short fable is that alignment will not magically become easy for AIs, especially if - like human brains and neural networks - those AIs are "messy", lacking a clean and easily-interpretable internal structure. Copying makes it easier than the human-AI case, but far from trivial.

Forever is a Long Time

The difficulty of alignment is compounded when one's goal is to control the long-run future of the universe. A early-singularity AI ...
More examples of goal misgeneralization
by Rohin Shah — 2022-10-07T14:38:00.288Z
Link: https://deepmindsafetyresearch.medium.com/goal-misgeneralisation-why-correct-specifications-arent-enough-for-correct-goals-cf96ebc60924

In our latest

paper

and

accompanying blog post

, we provide several new examples of

goal misgeneralization

in a variety of learning systems. The rest of this post picks out a few upshots that we think would be of interest to this community. It assumes that you’ve already read the linked blog post (but not necessarily the paper).

Goal misgeneralization is not limited to RL

The core feature of goal misgeneralization is that after learning, the system pursues a goal that was correlated with the intended goal in the training situations, but comes apart in some test situations. This does not require you to use RL – it can happen with any learning system. The Evaluating Expressions example, where Gopher asks redundant questions, is an example of goal misgeneralization in the few-shot learning regime for large language models.

The train/test distinction is not crucial

Sometimes people

wonder

whether goal misgeneralization depends on the train/test distinction, and whether it would no longer be a problem if we were in a continual learning setting. As Evan

notes

, continual learning doesn’t make much of a difference: whenever your AI system is acting, you can view that as a “test” situation with all the previous experience as the “training” situations. If goal misgeneralization occurs, the AI system might take an action that breaks your continual learning scheme (for example, by creating and running a copy of itself on a different server that isn’t subject to gradient descent).

The Tree Gridworld example showcases this mechanism: an agent trained with continual learning learns to chop trees as fast as possible, driving them extinct, when the optimal policy would be to chop the trees sustainably. (In our example the trees eventually repopulate and the agent recovers, but if we slightly tweak the environment so that once extinct the trees can never come back, then the agent would never be able to recover.)

It can be hard to identify goal misgeneralization

InstructGPT was trained to be helpful, truthful, and harmless, but nevertheless

it will answer "harmful" questions in detail

. For example, it will advise you on the best ways to rob a grocery store.

An AI system that competently does something that would have gotten low reward? Surely this is an example of goal misgeneralization?

Not so fast! It turns out that during training the labelers were told ...
Preparing to Call a Family Dance
by jefftk — 2022-10-07T12:50:07.004Z

A week from Sunday I'm going to be calling a

family
dance. This is like a contra dance, but with dances that are a
good fit for all ages, including very little kids. I haven't called
one before, though I've danced at several over the years back to when
I was little. Here's what I'm currently thinking of doing:

La Bastringue

Big circle, partnered, keep your partner, doesn't need a walkthrough

A1
(8)
Into the middle

(8)
Again

A2
(8)
Circle left

(8)
Circle right

B1
(16)
Partner swing (or dosido and swing)

B2
(16)
Promenade

When I call this for adults I make it a mixer by having you swing the
next person, but for little kids who are often dancing with their
parents you don't want to split them up.

This has its own music, though it's not critical:

(youtube)

Sasha

Scatter mixer

In pairs you point at each other saying "Sasha! Sasha! Ras, dva, tri!"
Then you (twice) clap right hands, left hands, both hands, and hands
on (your own) knees. Around by the right elbow singing "la (x9),
hey!", then again with the left. Then you wander around and find
someone new to dance with. The timing is:

A
(8)
Sasha! Sasha! Ras, dva, tri!

B1
(8)
Right x3, Left x3, Both x3, Knees x3

B2
(8)
Right x3, Left x3, Both x3, Knees x3

C1
(8)
Right elbow round

C2
(8)
Left elbow round

D
(16)
Wander

If little kids want to stick with their parents a couple can
participate in this dance as if they're one person.

It has its own music:

(youtube)

Favors the Rose ("the Fan Dance")

Two lines, temporary partnering

(youtube)

Dancers form two lines. One person out at the top holding a fan or
something else. They consider the two top people, give the object to
one, and dance down the middle with the other.

Unphrased, works with any music.

I've never danced this one, but reading trad-dance-caller it seems to
be a favorite. I'm mildly nervous about doing it when it's totally
new to me, but it does seem pretty simple.

Galopede

Two lines, partnered, keep your partner

A1
(8)
Long lines

(8)
Cross over

A2
(8)
Long lines

(8)
Cross over

B
(16)
Partner swing (or dosido and swing)

C
(16)
Top couple sashays down to the bottom, everyone
else moves up. Dancers at the sides clap: 1-2-3.

Even though it's square and 32-bar it has it...
Calibrate - New Chrome Extension for hiding numbers so you can guess
by chanamessinger — 2022-10-07T11:21:51.711Z
Link: https://chrome.google.com/webstore/detail/calibrate/omhpddekmhnkmpbjogkfeihglpjdndhf/related?fbclid=IwAR09kbuaP6wAy-hiJrQpwQsR1C9DJLggY6QzypPbX_1uTtLq193DNwRU98o

One thing I wish I did more often is checks to see if I understand the world - Fermis, landmark numbers, predictions about how things work, etc.

Along the way I started wishing I could cover up numbers on websites so I would have the chance to guess first and then reveal via mouseover, and Aur Saraf agreed to make a chrome extension, and here it is! Lets you set the default for a whole site or a page. (Not good for dynamic sites like twitter)

Extremely excited.

The code is open source here if you want to add any feature requests.

(I think I want to change the name - let me know if you have thoughts)
Accrue Nuclear Dignity Points 
by Pee Doom — 2022-10-07T08:40:42.892Z

Content warning: nuclear doom

Previously on last week's episode: A Few Terrifying Facts About The Russo-Ukrainian War

You have probably thought about prepping at some point.

You have probably also not prepped as much as you'd like.

Neither have I.

And neither of us likely will be prepared "enough", in this world, if the worst happens.

You've probably seen those prepper checklists and balked at completing every item on them.

Maybe you've told yourself it's not worth living after a nuclear war.

Maybe you've convinced yourself that preparing for the set of things that includes normal disasters like an earthquake or hurricane will be all that is reasonable to do.

I've long yearned for a proper collective response to Global Catastrophic Risks In Your Local Neighborhood.

I'm coming to accept that there's simply no reaching adequacy. I'm still likely to fuck up on some significant axis and still pretty likely to die after a nuclear war, though less than before each preparation I took.

Still...

If you believe in dying with dignity, then you can die with more or less dignity after a nuclear war.

You can die from radiation after a couple weeks, and because you had emergency water, someone else had access to water and didn't die of thirst, and you were able to help a couple friends get to safety before you succumbed.

You can have potassium iodide pills, and not die of radiation but then still get cancer after a year. You have emergency water and two weeks of food, but you somehow get stuck in your urban area because all your friends with cars took off and you forgot to download cash from an ATM to pay the people who want money to drive you out of the nuke zone.

Part of a nuclear war is likely to be EMPs, which result from atmospheric detonations. This means your electronics probably get fried. And the banks. You lose access to the Internet, to your phone, to most of your money... Probably? But you can protect your electronics with a Faraday Cage bag! That's a whole lot of dignity to have your phone or a radio working, even if it's just a 4-hour charge and all the value it brings is an offline map and the ability to reminisce over your photos before you die. Maybe you find a source of electricity and can extend your battery life. Maybe some internet works since the internet was originally partly designed to survive a nuke I think.

You have a car or a friend with a car, you have e...