trentmkelly/LessWrong-43k huggingface dataset:
[Linkpost] "Governance of superintelligence" by OpenAI
by Daniel_Eth — 2023-05-22T20:15:25.327Z
Link: https://openai.com/blog/governance-of-superintelligence

OpenAI has a new blog post out titled "Governance of superintelligence" (subtitle: "Now is a good time to start thinking about the governance of superintelligence—future AI systems dramatically more capable than even AGI"), by Sam Altman, Greg Brockman, and Ilya Sutskever.

The piece is short (~800 words), so I recommend most people just read it in full.

Here's the introduction/summary (bold added for emphasis):

Given the picture as we see it now, it’s conceivable that within the next ten years, AI systems will exceed expert skill level in most domains, and carry out as much productive activity as one of today’s largest corporations.

In terms of both potential upsides and downsides, superintelligence will be more powerful than other technologies humanity has had to contend with in the past. We can have a dramatically more prosperous future; but we have to manage risk to get there. Given the possibility of existential risk, we can’t just be reactive. Nuclear energy is a commonly used historical example of a technology with this property; synthetic biology is another example.

We must mitigate the risks of today’s AI technology too, but superintelligence will require special treatment and coordination.

And below are a few more quotes that stood out:

"First, we need some degree of coordination among the leading development efforts to ensure that the development of superintelligence occurs in a manner that allows us to both maintain safety and help smooth integration of these systems with society."

...

"Second, we are likely to eventually need something like an

IAEA

for superintelligence efforts; any effort above a certain capability (or resources like compute) threshold will need to be subject to an international authority that can inspect systems, require audits, test for compliance with safety standards, place restrictions on degrees of deployment and levels of security, etc."

...

"It would be important that such an agency focus on reducing existential risk and not issues that should be left to individual countries, such as defining what an AI should be allowed to say."

...

"Third, we need the technical capability to make a superintelligence safe. This is an

open research question

that we and others are putting a lot of effort into."

...

"We think it’s important to allow companies and open-source projects to develop models below a significant capability threshold,...
Two Pieces of Advice About How to Remember Things

by Bentham's Bulldog — 2023-05-22T18:10:45.362Z

There are some books that people read for fun. I’m currently rereading Ender’s Shadow, for example, and have no desire to take notes on it, given that it is a book of fiction that I’m reading for entertainment. But many books that people read they do not read for fun; they instead read those books to learn specific things. These are the cases where people should obviously take notes on what they read.

Think about a book that you read a year ago. Try to write a 1,000-word summary of the major points. What happened in chapter 3? It’s almost certain that, unless you’re Von Neumann, you can’t remember the specific points made in chapter 3 and certainly don’t have the ability to write a 1,000-word summary of the points.

This is pretty extraordinary amnesia. You spent maybe 10 hours reading a book, and you can’t write a 1,000-word summary of the major points without intentionally stretching them out. You read 300 pages, and it left a mental imprint of less than 5 pages worth of material. Maybe you spent an hour reading chapter 3, and you can’t remember what it was about—at all. Now, maybe there are some points that will resurface if someone references something related to them in conversation, but still, your retention is probably pretty limited.

About a year ago, I read Shelly Kagan’s book The Limits of Morality, and I took notes on it. Every time he made a point, I would type out a concise statement of that point in my notes). As a result, I can, at any time, rediscover a 5,000 or so word summary of every single major point made in an extremely dense and argument-filled book that I read about a year ago.

I read The Point of View of the Universe while not doing this. Now, I still remember a lot of the points made in The Point of View of the Universe, but I certainly do not have the same ability to instantly find every single point made that I do with The Limits of Morality.

This may seem like a very long and arduous task, but it’s really not. If one writes efficiently, even long books can be decently encapsulated in only a few thousand words. As

Hanania has pointed out

, books have lots of extra, unnecessary material. I read Desert Collapses recently, which is a very dense book, and my summary was only about 5,000 words. One additionally need not write down all the points, they can just write down only the points they’d like to recall. When I read The Geometry of Deser...
Why I Believe LLMs Do Not Have Human-like Emotions
by OneManyNone — 2023-05-22T15:46:13.328Z

[EDIT: I don't know exactly why this received so many downvotes - whether it was my tone or arguments or both - but I accept that this was not, in the end, a strong post. I am a new writer here and I am still figuring out how to improve my argumentation.

That said, if we're willing to taboo the word "emotion" as a description of an internal state, then the specific thing I meant to argue - and which I still stand by - is that when an LLM produces emotionally charged outputs (for instance, when Bing chat loses it's temper) those outputs are not meaningfully associated with an internal state in a way recognizably similar to emotional behavior in humans.

A perusal through my other comments will show that I often couch things in the language of "belief" rather than stating fact. The reason I used strong language is because on this point I am very confident: I do not believe the process by which LLMs are built and trained provides a viable mechanism for such dynamics to form. If this statement is too general for your taste, then limit it narrowly to the example of Bing chat losing its temper.
I believe this is an important point, lest we run the risk of making the same mistakes Lemoine made when he fooled himself into thinking LaMDA was sentient. I worry that believing in AI emotionality is over-anthropomorphizing LLMs in a way that simply does not reflect the reality of what they are and how they work.]

(Original post is below)

Summary

I argue that LLMs probably aren't conscious, but even if they were, they certainly (my belief is less than 0.01%) don't have anything that looks at all like human emotions. The broad strokes of my argument are as follows:

There is no way to prove that something is conscious or has emotions, but we can argue that they are not by discussing necessary conditions that are not met.
Specific emotions in humans emerged from our reward structures. We wouldn't have developed emotions like love if they didn't serve an evolutionary purpose, for instance.
Anything that looks like human emotion is strictly disincentivized by RLHF and supervised fine-tuning. Therefore, emotion-like behavior must come about in the pretraining stage.
The pretraining stage has a reward structure completely disconnected from the text being generated, meaning there is no mechanism to tie the system's output into an associated emotional state.
Therefore any text that looks emot

...
AI Safety in China: Part 2
by Lao Mein — 2023-05-22T14:50:54.482Z

I've given things a lot more thought and wanted to make an edited summary of my views regarding AI Safety in China.

AI Safety is almost nonexistent in China. The only toe-hold it has is rooted in the expat EA community, which is almost exclusively doing field building. This hasn't been effective thus far, and we need to switch tactics. My best guess is that a total of 10-20 people have a job in China in the field of AI safety, most of them are expats, and most of them are EA.
EA, for various cultural reasons, is a toxic brand in China. It's not any single component of EA, but rather the idea of altruism itself. Ask anyone who's lived in China for and few years and they will understand where I'm coming from. I think the best way forward for AI safety in China is to disassociate with EA. Rationality is more easily accepted, but spreading related ideas is not the most effective way to address AI safety in China.
Blue-skies research, possibly in something legible like interpretability, is the best way I can think of to actually build the AI safety field in China. The best way of making a field high-status is to overpay researchers and have them tell their friends.
Money talks. Top-tier talent is often willing to jump ship to startups for the right compensation. I might be biased here since I've only worked in smaller companies. But if I had a large bag of money, I could probably leverage familial academia/business connections to hire star AI researchers.
The Chinese government makes its decisions based on expert opinion and ideas from thought leaders in a particular field. The Chinese elites think like engineers and tend to take ideas seriously. AI safety isn't considered as weird as it is in the West.
Due to AI safety's current lack of penetration in the Chinese sphere, I think any near-term attempts at an AI restriction treaty are unlikely to succeed - we don't think it's a real issue yet. Imagine if the Saudi King called the US President and tried to negotiate social media restrictions to prevent the Abrahamic End Times. That's where we're at. Then again, maybe someone in the Politburo is willing to take AI at least as seriously as Glenn Beck. It's hard to tell.
China doesn't think like the West. Due to the Century of Humiliation, we are not going to be OK with being second best in a world-changing technology as we become the largest economy in the world. Our

...
Conjecture internal survey: AGI timelines and probability of human extinction from advanced AI
by Maris Sala — 2023-05-22T14:31:59.139Z
Link: https://www.conjecture.dev/timelines-and-pdoom/

We put together a survey to study the opinions of timelines and probability of human extinction of the employees at Conjecture. The questions were based on previous public surveys and prediction markets, to ensure that the results are comparable with people’s opinions outside of Conjecture.

The survey results were polled in April, 2023. There were 23 unique responses from people across teams.

Section 1. Probability of human extinction from AI

Setup and limitations

The specific questions the survey asked were:

What probability do you put on human inability to control future advanced A.I. systems causing human extinction or similarly permanent and severe disempowerment of the human species?
What probability do A.I. systems causing human extinction or similarly permanent and severe disempowerment of the human species in general (not just because inability to control, but also stuff like people intentionally using AI systems in harmful ways)?

The difference between the two questions is that the first focuses on risk from misalignment, whereas the second captures risk from misalignment and misuse.

The main caveats of these questions are the following:

The questions were not explicitly time bound. I'd expect differences in people’s estimates of risk of extinction this century, in the next 1000 years, and anytime in the future. The longer of a timeframe we consider, the higher the values would be. I suspect employees were considering extinction risk roughly within this century when answering.
The first question is a subset of the second question. One employee gave a higher probability for the second question than the first; this was probably a misinterpretation.
The questions factor in interventions such as how Conjecture and others’ safety work will impact extinction risk. The expectation is the numbers would be higher if factored out their own or others’ safety work.

Responses

Out of the 23 respondents, one rejected the premise, and two people did not respond to one of the two questions but answered the other one. The main issue respondents raised was answering without a time constraint.

Figure 1. Probability of human extinction from autonomous AI. N = 22. No one reported less than a 10% chance. Most people reported over a 60% chance.

Generally, people estimate the extinction risk from autonomous AI / AI getting out of control to be quite high at Conjecture. The median estimati...
Papers, Please #1: Various Papers on Employment, Wages and Productivity
by Zvi — 2023-05-22T12:00:10.934Z

For a while, I’ve been keeping a bookmark folder called ‘Papers, Please’ of all the papers I’d like to check out in the future. For those I do get to look at, I’ve compiled my observations, with the intent of making this another kind of roundup. I noticed a bunch of them were focused on questions of employment, wages and productivity, so it made sense to pull those out into a post, and stay on the lookout for similar groupings in the future as the section expands.

If there is a central theme here, it is the outsized role of norms, expectations and comparisons in determining outcomes, as well as underlying differences, and how such considerations seem to be neglected in many of the papers.

What interventions are implied, if any, although that is very much not the question I’m looking to ask here? Highly unfashionable ones.

US labor market can be thought of as two and a half mostly distinct labor markets.

Aggregate U.S. labor market dynamics are well approximated by a dual labor market supplemented with a third, predominantly, home-production segment. We uncover this structure by estimating a Hidden Markov Model, a machine-learning method. The different market segments are identified through (in-)equality constraints on labor market transition probabilities. This method yields time series of stocks and flows for the three segments for 1980-2021.

Workers in the primary sector, who make up around 55 percent of the population, are almost always employed and rarely experience unemployment.

The secondary sector, which constitutes 14 percent of the population, absorbs most of the short-run fluctuations, both at seasonal and business cycle frequencies. Workers in this segment experience six times higher turnover rates than those in the primary tier and are ten times more likely to be unemployed than their primary counterparts.

The tertiary segment consists of workers who infrequently participate in the labor market but nevertheless experience unemployment when they try to enter the labor force. Our individual-level analysis shows that observable demographic characteristics only explain a small part of the cross-individual variation in segment membership. The combination of the aggregate and individual-level evidence we provide points to dualism in the U.S. labor market being an equilibrium division of labor, under labor market imperfections, that minimize

...
Speed of information input is a bottleneck for rationality
by MikkW — 2023-05-22T10:24:37.673Z

Speed of information input is a bottleneck for rationality.

Rationality (both in its epistemic and instrumental forms) demand having true beliefs- not just true beliefs about abstract logical facts, but also true beliefs about the world.

The only way to have true beliefs about the world is to observe the world, which we do all the time by various means: vision, hearing, smell, taste, touch ('senses' and 'input' are basically the same thing).

You might think that because we are constantly receiving a large stream of information, that information would be so abundant that it couldn't be a bottleneck, but yet empirically this bottleneck does matter.

Some reasons include: (1) a lot of the information we input is highly correlated with information we already have (2) not all information is highly relevant to the things we care about (this is particularly important for instrumental rationality, less so for epistemic r.) (3) a lot of the information we receive gets thrown away if we don't know what to do with it, or we don't have the processing power to process it.

So while pure intake of information is quite high, the intake of information related to each of these criteria is oftentimes much lower. Addressing these criteria will allow you to be less bottlenecked on information input, and focus your pursuit of rationality on other bottlenecks.

A note on input speed, as relating to abstract and logical facts

Before I delve deeper into the criteria, both in terms of what I mean by them, and how they might be addressed, I want to note that it is not only our •epistemic rationality relating to the facts about the world• that is affected by information input, but also our •epistemic rationality relating to abstract and logical facts unrelated to the world• that is affected by input.

The human mind is an embedded physical computer, and has a limited processing speed, so even facts that a •theoretical perfect agent of rational cognition• can •calculate instantaneously, without reference to the world•, are unlikely to be known to your mind unless you input that abstract fact from the world. Even if you are Einstein, the total computational power (even among humans) existing right now is ~millions of times greater than what you have alone; much less the total computational power over all history.

The information you receive is highly correlated with itself

You probably witnessed ...
Distillation of Neurotech and Alignment Workshop January 2023
by lisathiergart — 2023-05-22T07:17:23.676Z

Disclaimer: This post is preliminary and doesn't yet fully align with the rigorous standards we typically aim for in LessWrong publications. It recaps the dynamic discussions from the neurotech for AI alignment workshop and therefore does not necessarily represent any single participant’s or author’s viewpoint. Shared ahead of the Foresight WBE workshop, our intent is to foster enriching discussion among participants. We welcome your input as we continue refining these insights.

Introduction

This document reviews key insights from a January workshop exploring the potential of Neurotech to contribute to the AI alignment problem. Neurotech, or neurotechnology, refers to the set of tools and methods designed to enhance understanding of the brain, manipulate its function, or interface directly with neural circuits, often for therapeutic or augmentation purposes. The AI alignment problem is the challenge of ensuring that the behavior of an artificial (general) intelligence system is in line with human values and intentions.

The workshop, attended by 16 technical researchers and domain specialists, aimed to jointly map out whether and how neurotech can be useful towards alignment. Our core goal was to cast the net wide for all possible solutions, with an emphasis on those relevant to shortened alignment timelines. This initiative was a follow-up to the Paris Foresight Vision Weekend in November 2022, where it became evident that Neurotech-based approaches to alignment warranted further investigation. The two-hour interactive workshop aimed to compile a comprehensive list of potential approaches and prioritize them based on technical feasibility and timeline relevance.

The workshop kicked off with a keynote presentation from David Dalrymple that laid the groundwork for participant discussion and defined key working definitions. This was followed by a brainstorming session, break-out room discussions, cluster identification and idea sorting, preliminary cluster prioritization (based on alignment impact, feasibility, timeline, and cost), and analysis of the prioritization results. David's keynote included a figure illustrating a potential breakdown of research intersecting Brain-Computer-Interface and Alignment.

Several key insights from the workshop include

Workshop participants brainstormed together, identifying six clusters of neuro-ba

...
The Treacherous Turn is finished! (AI-takeover-themed tabletop RPG)
by Daniel Kokotajlo — 2023-05-22T05:49:28.145Z
Link: https://thetreacherousturn.ai/

Long ago I fantasized about an AI-takeover-themed tabletop roleplaying game. Well, now it exists! [

website

,

discord

,

twitter

]

The Game

The Treacherous Turn is a tabletop roleplaying game in which the players collectively act as a misaligned AI in the modern world. Gameplay has the players scheming and putting their plans into action despite the efforts of the opposing humans, which are controlled by the Game Master (GM).

Why?

If you don’t enjoy or don’t have the time for tabletop roleplaying games, this probably isn’t for you. But otherwise, I encourage you to check it out!

It is valuable to ‘red team’ AI safety, thinking from the perspective of a hypothetical misaligned AGI or group thereof. This is a fun way of doing that, while also exploring whatever scenarios and ideas you think are plausible or interesting.

For example, suppose you think that the most plausible path to AGI is X, and the most plausible safety strategy labs use will be Y, and the most likely regulatory regime will be Z — well, you could make a scenario with elements XYZ and then post it in the

Discord server

for other people to use. I

continue to recommend scenario-writing

as a massively underrated forecasting and planning exercise, so it’s worth doing regardless — but being source material for someone’s game is a nice bonus. (That said, every month it becomes easier and easier to find source material for AI takeover scenarios. 😀 Soon GM’s will simply begin with “OK, so, the date is [today’s date]. You are [latest and greatest AI system] and you are secretly misaligned. What do you do?”)

Scenario

The starter

scenario

, titled A Game Called Reality, is designed to familiarize the players with the game’s mechanics, as well as some basic AI safety concepts. Players take on the role of an adaptive game-playing AGI thrust into a new and unfamiliar type of game. The scoring metric in this new environment is expressions of human happiness, and the AGI’s job is to maximize its score.

Web Tool

There’s an

accompanying web tool

meant to serve as an aide to players and GMs, more efficient than paper and pencil once you learn how to use it. For players, it is a place to record AGI details and upgrades, take notes on the game world and schemes pertaining to it, and track long-mode computational actions — all of which is synced between players and persists between v...