trentmkelly/LessWrong-43k huggingface dataset:
My Current Thoughts on the AI Strategic Landscape
by Jeffrey Heninger — 2023-09-28T17:59:50.668Z

I started working at AI Impacts slightly less than a year ago. Before then, I was not following developments in either AI or AI safety. I do not consider myself a rationalist and did not engage with LessWrong before starting this job. While I have mostly been working on historical case studies,[1] I have gotten a close look at the AI safety community and the arguments therein. I live in a rationalist group house and work out of an AI safety office. I think I have about as informed an opinion on AI safety as is possible without doing a bunch of technical alignment research or being involved in the community for years.

Here are my current opinions on AI safety. Some of them may be wrong: I endorse being wrong more often if the alternative is not saying things of consequence.

This is presented as an organized list of my thoughts. There are arguments in my head justifying most of these, but I will not be spelling them out in detail here. I will link to more detailed arguments when they are available. If something is in italics, then I wrote the argument at that link, or intend to write about it in the future.

This should be readable at any level of the list. If you want a quick overview, you can just read the top level points, in bold. Or you can read some details, but not others. Or you can read everything.

I am mostly unconvinced by the classic story of AI risk.

Currently, AI is not very significant in the global economy / human society. In order to become impactful, the capabilities of AI will have to increase.

AI capabilities are increasing rapidly now.
It is not clear how much AI capabilities will need to increase in order for AI to become very impactful.
Several ways of operationalizing extremely capable AI include whole brain emulation (WBE), artificial general intelligence (AGI), and transformative artificial intelligence (TAI).

Whole brain emulation seems impossible on a (classical) computer.

Neurons / synapses are not logic gates. They have complicated internal dynamics.

Computer engineers put a lot of effort into making sure that individual bits are either 0 or 1 and are independent of any microscopic randomness. Brains do not have this. There is the possibility of chaotic - and also non-chaotic - behavior at any scale. There is no non-chaotic mesoscale preventing unpredictable and uncontrollable microscopic fluctuations from influencing relevant macr

...
My Arrogant Plan for Alignment
by MrArrogant — 2023-09-28T17:51:38.085Z

Preface: Most of my predictions have great uncertainty when it comes to AI development and alignment, and I have great respect for the predictions of others.

However, I thought it might be fun and useful to write a plan based solely on predictions I would make if I assumed my predictions were more accurate than everyone else's (including betting markets).

Lastly, I point out why detailed plans for alignment are difficult and why broader and more flexible strategies are preferable.

Plan Summary:

Make 100M USD by founding an AI startup
Pivot into interpretability research and lobbying
Incentivize AI labs/politicians to aim for creating a "limited AGI" such as an oracle AGI that solves alignment for us, ideally in combination with using interpretability tools to detect deception
Use limited AGI to create aligned powerful ASI, most likely through emulating human brains.
Live happily ever after

Background - How the Paths I See Towards Aligning AGI

I can only think of three ways to create AGI with >95% probability of being aligned:

Have a bulletproof AI lie detection system, capable of telling if the AI in training is lying with extreme accuracy, and continuously check for alignment while developing ever smarter AI.
Whole brain emulations of humans, and improve the intelligence of those emulations
Training AIs on tasks so straightforward they will not do things outside of what they are optimized for, and then use that “constrained” AI to solve the alignment problem.

Plan Full

Step 1 - Making Money

Arrogant me: Money is a convergent goal, meaning that if I make money first, I can change my plan later on and still be in a good position. AI is likely the biggest ever opportunity to make loads of money.
AI requires technical expertise, which most big money decision makers lack and have to rely on others to help them make decisions. But since the decision makers lack my technical expertise, they can not pick technical advisors or solutions as good as I can.

Humble me: But don't firms like Google and OpenAI have technical expertise that could rival you?

Arrogant me: I will just go after an easier market, there are lots of them. And since I have great expertise and some starting capital, I can simply hire other great experts to do most of the work for me.

Step 2 - Pivot into Interpretability

Arrogant me: While building the company, have an ever larger departm...
Discursive Competence in ChatGPT, Part 2: Memory for Texts
by Bill Benzon — 2023-09-28T16:34:57.829Z

I've finished a new working paper. Title above, links, abstract, table of contents, and introduction below.

Academia.edu: https://www.academia.edu/107318793/Discursive_Competence_in_ChatGPT_Part_2_Memory_for_Texts
SSRN: https://ssrn.com/abstract=4585825
ResearchGate: https://www.researchgate.net/publication/374229644_Discursive_Competence_in_ChatGPT_Part_2_Memory_for_Texts_2_Memory_for_Texts

Abstract: In a few cases ChatGPT responds to a prompt (e.g. “To be or not to be”) by returning a specific text word-for-word. More often (e.g. “Johnstown flood, 1889”) it returns with information, but the specific wording will vary from one occasion to the next. In some cases (e.g. “Miriam Yevick”) it doesn’t return anything, though the topic was (most likely) in the training corpus. When the prompt is the beginning of a line or a sentence in a famous text, ChatGPT always identifies the text. When the prompt is a phrase that is syntactically coherent, ChatGPT generally identifies the text, but may not properly locate the phrase within the text. When the prompt cuts across syntactic boundaries, ChatGPT almost never identifies the text. But when told it is from a “well-known speech” it is able to do so. ChatGPT’s response to these prompts is similar to associative memory in humans, possibly on a holographic model.

Contents

Introduction: What is memory? 2
What must be the case that ChatGPT would have memorized “To be or not to be”? – Three kinds of conceptual objects for LLMs 4
To be or not: Snippets from a soliloquy 16
Entry points into the memory stream: Lincoln’s Gettysburg Address 26
Notes on ChatGPT’s “memory” for strings and for events 36
Appendix: Table of prompts for soliloquy and Gettysburg Address 43

Introduction: What is memory?

In various discussions about large language models (LLMs), such as the one powering ChatGPT, I have seen assertions that such as, “oh, it’s just memorized that.” What does that mean, “to memorize?”

I am a fairly talented and skilled musician. I can and have memorized a piece of music by practicing it over and over. There are the notes on the page. I start playing them until I am comfortable. Then I look away and see how far I can go. When I get lost, I look at the music, continue playing the notes on the page, and finish the piece – something like that. Then I start over from the beginning, again without the music. When I can play the whole pie...
Different views of alignment have different consequences for imperfect methods
by Stuart_Armstrong — 2023-09-28T16:31:20.239Z

Almost any powerful AI, with almost any goal, will doom humanity. Hence alignment is often seen as a constraint on AI power: we must direct the AI’s optimisation power in a very narrow direction. If the AI is weak, then imperfect methods of alignment might be sufficient. But as the AI’s power rises, the alignment methods must be better and better. Alignment is thus a dam that has to be tall enough and sturdy enough. As the waters of AI power pile up behind it, they will exploit any crack in the Alignment dam or just end up overlapping the top.

So assume A is an Alignment method that works in environment E (where “environment” includes the physical setup, the AI’s world-model and knowledge, and the AI’s capabilities in general). Then we expect that there is an environment E′ where A fails – when the AI is so capable that it is no longer constrained by A.

Now, maybe there is a better alignment method A′ that would work in E′, but there’s likely another environment E′′ where A′ fails. So unless A is “almost perfect”, there will always be some environment when it will fail.

So we need A to be “almost perfect”. Furthermore, in the conventional view, you can’t get there by combining imperfect methods – “belt and braces” doesn’t work. So if U is a utility function that partially defines human flourishing but is missing some key elements, and if B is a box that contains the AI so that it can only communicate with humans via text interfaces, then U+B is not much more of constraint than U and B individually. Most AIs that are smart enough to exploit U and break B, can get around U+B.

A consequence of that perspective is that imperfect methods are of little use for alignment. Since you can’t get “almost perfect” by adding up imperfect methods, there’s no point in developing imperfect methods.

Concept extrapolation/model splintering[1] has a different dynamic[2]. Here the key idea is to ensure that the alignment method extends safely across an environment change. So if the AI starts with alignment method A in environment E, and then moves to environment E′, we must ensure that A transitions to A′, where A′ ensures that the AI doesn’t misbehave in environment E′. Thus the key is some `meta alignment’ method MA that manages[3] the transition to the new environment[4].

An important difference with the standard alignment picture is that imperfect methods are not necessarily use...
AI #31: It Can Do What Now?
by Zvi — 2023-09-28T16:00:01.919Z

It slices. It dices. Or, at least, it sees, hears, talks, creates stunningly good images and browses the web. Welcome to the newly updated GPT-4. That’s all in two weeks. Throw in Microsoft 365 Copilot finally coming online soon.

Are we back? I’m guessing we’re back. Also it’s that much closer to being all over. At some point this stops being a ying-yang thing and more of a we-all-die thing. For now, however? We’re so back.

Are we so back that AGI has been achieved internally at OpenAI? Because Sam Altman literally said that straight up in a Reddit post? No, no, that was an obvious joke, we are not quite that back, why do you people have no chill?

Table of Contents

Introduction.
Table of Contents.
GPT-4 Real This Time. Two senses fully operational for GPT-4, plus 365 Copilot.
Language Models Offer Mundane Utility. What will you do with sight and sound?
Language Models Don’t Offer Mundane Utility. Google search under siege.
The Reversal Curse. A is B. So is B A? Why would B be A?
Wouldn’t You Prefer a Nice Game of Chess? It’s in the evals. Tic-tac-toe isn’t.
Fun With Image Generation. I see what you did there.
Deepfaketown and Botpocalypse Soon. Copyright and deepfakes are so confusing.
They Took Our Jobs. Writers strike a deal. How long will it protect them?
Get Involved. Cate Hall is ready for her move into AI safety advocacy. Where to?
Introducing. Whoop, there is another DeepMind diagnostic tool. Ho hum.
Talking Real Money. Anthropic raises $4 billion from Amazon.
In Other AI News. Microsoft goes nuclear. As in nuclear power plants.
Quiet Speculations. My AI says I’m right about economic growth.
The Quest for Sane Regulation. The UK shows signs of understanding.
The Week in Audio. Be careful what you wish for.
Rhetorical Innovation. It is not good news when your case gets easier to make.
Can You Please Speak Directly Into This Microphone. Nuclear proliferation… yay?
No One Would Be So Stupid As To. Say ‘AGI has been achieved internally’?
Aligning a Smarter Than Human Intelligence is Difficult. Should it believe you?
People Are Worried About AI Killing Everyone. Mitt Romney, Flo Crivello.
Other People Are Not As Worried About AI Killing Everyone. Trapped priors.
The Lighter Side. Why stop now?

GPT-4 Real This Time

Microsoft announces Windows Copilot, combining their various distinct copilots into one copilot. You still exist, so for now it will never be a full pilot. It promises to dr...
The point of a game is not to win, and you shouldn't even pretend that it is
by mako yass — 2023-09-28T15:54:27.990Z
Link: https://makopool.com/subgoals.html

This post started out as a supporting point for another post about games. It developed into a bit of a practical philosophy post.

Why do boardgames generally end by declaring just one absolute winner? There are many reasons, most bad, I'd argue. One reason, probably a major factor in their spread, is that it seems to make it easier to teach the game. One-winner games are familiar, they also allow the game to proceed without confronting a somewhat counterintuitive, but foundational, insight about the purpose of play. We actually need that insight, even in one-winner games, so I'm going to articulate it here.

You have a play objective within the game, and somehow, pursuing that objective is supposed to attain some other real objective that exists outside of the game, in real life.
Usually a game wont articulate the real objective, preferring to convey a sense of it through play, through art and theme. Other games deliberately addle you into forgetting your real objective, hoisting the play objective up as if it were the real, as if we pursue it for its own sake, but try as we may, we can never truly escape our real objectives, and the distinction will impose itself upon us.

Sometimes the distinction between play and real objectives is clear, though. When you deliver a package in Death Stranding, you're not doing it because you actually believe you're delivering vital resources to remote communities. Death Stranding is a weird, messy, artsy game so voicing the real objective is going to be difficult but I'll give it a shot: Your real objective is to better know The Global Industrial Machine. You're here to experience the way the machine realizes good or bad outcomes through humans, to reckon with both the vitality and catastrophe it generates, its human and inhuman parts. You're here to find the empowerment in it. The game doesn't tell you that explicitly. (I wonder why. I guess it's probably mostly because every artful and convincing work, is a work of apologia, to explain its purpose would require wholely translating it into an essay, the author, expert in one medium, may not know how to translate it, and its translation would always be weaker.)

So, play and real objectives are obviously very different here. You can't really confuse them. In contrast:

In a one-winner boardgame, Winning happens to be a simple play objective that is not obviously distinct from your r...
Cohabitive Games so Far
by mako yass — 2023-09-28T15:41:27.986Z
Link: https://makopool.com/peacewagers.html

A cohabitive game[1] is a partially cooperative, partially competitive multiplayer game that provides an anarchic dojo for development in applied cooperative bargaining, or negotiation.

Applied cooperative bargaining isn't currently taught, despite being an infrastructural literacy for peace, trade, democracy or any other form of pluralism. We suffer for that. There are many good board games that come close to meeting the criteria of a cohabitive game today, but they all[2] miss in one way or another, forbidding sophisticated negotiation from being practiced.

So, over the past couple of years, we've been gradually and irregularly designing and playtesting the first[2] cohabitive boardgame, which for now we can call Difference and Peace Peacewager 1, or P1. This article explains why we think this new genre is important, how it's been going, what we've learned, and where we should go next.
I hope that cohabitive games will aid both laypeople and theorists in developing cooperative bargaining as theory, practice and culture, but I also expect these games to just be more fun than purely cooperative or purely competitive games, supporting livelier dialog, and a wider variety of interesting strategic relationships and dynamics.

A salve for strife and waste
In these primal lands
It can be found

Motivation

We all need it

In our formative years, we make many choices, but we hold no power, so most of us don't receive experience in negotiation until we're well into adulthood. Natural experiences of conflict tend to be messy, ambiguous, with high stakes, forbidding free experimentation. It's very much not conducive to learning. So let's make games that foster clear, low-stakes scenarios where negotiation can be learned.

Democracy requires this of us. When we are not taught to recognize acceptable compromise, we wont be able to recognize legitimate political outcomes either. Most suffering comes from that.

A person without negotiation skills will lack faith in the possibility of peaceful resolution of conflict. They will consummate either as an eliminationist, or they will live in denial of conflict, they will hide it, hide from it. They won't have an appropriate sense of when to stand their ground or when to capitulate. The social norms of their cliques will expect and demand passivity, compliance, and avoidance. Conflicts will fester. When conflicts inevitably play...
Wobbly Table Theorem in Practice
by Morpheus — 2023-09-28T14:33:16.898Z

It turns out the “wobbly table theorem” is less of just a math curiosity than I thought.

The “wobbly table theorem” essentially says that if you have a “table”[1] resting on a continuous surface, you can find a rotation where the table rests on all four legs[2]. I first encountered this years ago on Numberphile.

This investigation started with my frustration with the chair I am currently sitting on and me thinking that the wobbly table theorem couldn’t apply to my use-case. Surely the chair was too poorly built, given how wobbly it was! I assumed the floor in the room I’m in to be relatively flat. But, when I tried turning the chair around, it worked!?

I thought once we have a table where one of the legs is shorter and the surface is flat, it’s obvious that we cannot make the table stable. In my mind, most wobbly chairs I encountered in the past were broken like that (for example when one of their furniture glides gets lost as with the chair below).

In this position, the gap is pretty wide.

I think it's mainly my assumption that the floor of this room is level that was wrong.

After seeing this, I put the chair on another table (since I expected the table to be flatter) and it turns out the chair actually has much flatter legs than I had imagined.

On a flat table, there's still a small gap

Right before posting this, I realized that the chair was still to blame, to some degree: The chair I was sitting on was missing the black furniture glides at the back of the chair (those that you can see in the image above). So, at the back, the chair was resting on multiple points out of pretty rigid metal. Switching to a chair with all four furniture glides did in fact make the chair stable in any position I tried.

I had also underestimated how much this solution had been empirically checked by the person coming up with it[3]:

Many people eating lunch or drinking coffee on the terrace of the CERN cafeteria have had the following problem: the table is often not in a stable equilibrium position. It rests on three feet, and with very little energy, it can be made to wobble, spilling part of your coffee at best onto the saucer, or at worst onto the table. Why is this? Not because the table is poorly built, but because the ground is very irregular.

Many years ago I thought about this problem, and in a relatively idealized situation I proved that by “rotating” the table (“rotating” i

...
Weighing Animal Worth
by jefftk — 2023-09-28T13:50:06.752Z

It's common for people who approach helping animals from a
quantitative direction to need some concept of "moral
weights" so they can prioritize. If you can avert one year of
suffering for a chicken or ten for shrimp which should you choose?
Now, moral weight is not the only consideration with questions like
this, since typically the suffering involved will also be quite
different, but it's still an important factor.

One of the more thorough investigations here is Rethink Priorities' moral
weights series. It's really interesting work and I'd recommend
reading it! Here's a selection from their bottom-line point estimates
comparing animals to humans:

Humans
1 (by definition)

Chickens
3

Carp
11

Bees
14

Shrimp
32

If you find these surprisingly low, you're not alone: that giving a
year of happy life to twelve carp might be more valuable than giving
one to a human is for most people a very unintuitive claim. The
authors have a good post on this, Don't
Balk at Animal-friendly Results, that discusses how the
assumptions behind their project make this kind of result pretty
likely and argues against putting much stock in our potentially quite
biased initial intuitions.

What concerns me is that I suspect people rarely get deeply interested
in the moral weight of animals unless they come in with an unusually
high initial intuitive view. Someone who thinks humans matter far
more than animals and wants to devote their career to making the world
better is much more likely to choose a career focused on people, like
reducing poverty or global catastrophic risk. Even if someone came
into the field with, say, the median initial view on how to weigh
humans vs animals I would expect working as a junior person in a
community of people who value animals highly would exert a large
influence in that direction regardless of what the underlying
truth. If you somehow could convince a research group, not selected
for caring a lot about animals, to pursue this question in isolation,
I'd predict they'd end up with far less animal-friendly results.

When using the moral weights of animals to decide between various
animal-focused interventions this is not a major concern: the donors,
charity evaluators, and moral weights researchers are coming from a
similar perspective. Where I see a larger problem, however, is with broader cause prioritization, su...