trentmkelly/LessWrong-43k huggingface dataset:
Consider applying to a 2-week alignment project with former GitHub CEO
by Bird Concept — 2023-04-04T06:20:49.532Z
Link: https://twitter.com/natfriedman/status/1643031087676022784

Signal-boosting this tweet from Nat Friedman, former github CEO, who says:

I'm seeking a researcher/journalist with an interest in AI/rationalism/xrisk for a 1-2 week project. To apply please email job@nat.org with evidence that you (a) read scifi and (b) understand this tweet:
https://twitter.com/JimDMiller/status/1642978110894391296?s=20

Linked tweet says:

What's you response to Orthogonality Thesis+Instrumental Convergence+AI arms race+ likely Mesa Optimization+ rapid recent progress in AI via ML = we are likely doomed? I have a PhD in econ so I do understand risk.

I don't know much about this nor I do have any affiliation with it. But it seems pretty cool to me and at least worth considering for people in the target reference class!
On how it feels generating art with DALL-E
by  — 2023-04-04T04:13:54.317Z
Link: https://cortrinkau.bearblog.dev/on-how-it-feels-generating-art-with-dall-e/

[This is a linkpost to my blog, cortrinkau.bearblog.dev. I regularly make art with DALL-E and have been making art with traditional media like pencil, ink, and watercolor for fifteen years. This article is a meditation on what makes AI art feel so different, after so long making art exclusively by hand.]

When I was a little kid, I loved to draw, and sometimes an idea for a drawing would come to me as I was falling asleep. At that moment, I felt inspired to make art, but spending the time to make the idea I had in my head come through on paper would have meant getting much less sleep, so I would write a description of the image in my head in order to still remember it the next morning. I still have notebooks and notebooks filled with art prompts for myself, and it’s fun to look through them and see what child-me wanted to draw---like “red boots walking on a telephone wire.”

Years later, that habit turned out to have been great practice at writing prompts for DALL-E. However, there’s a difference in what makes generating art with DALL-E feel satisfying versus the satisfaction of drawing it myself. When I have a specific image in my head, I want that exact image to be transferred to the paper, as faithfully as I can possibly get it. If my art skills aren’t quite up to what I want to draw, I’m often willing to spend time investing in my own drawing skills so I can get that image to look the way I want it. I feel frustrated when I can’t get the image to look the same as the image in my head, even if it still looks beautiful on its own.

Recently, I had a conversation with a friend recently about this, where I brought up the boots-on-telephone-wire example. As I was describing the image, I was generating an image in my own mind, but the image that those words generated in my friend’s mind was slightly different. When I try to generate that image with DALL-E, it wouldn’t get the specific details right – whether there’s a person wearing the boots, the angle the boots might be swaying at, the off-white texture of the paper, or the pencil-and-ink medium of the drawing, the Banksy-esque contrast of the black-and-white drawing and the hue of the girl’s boots. I have to draw the image myself, and I have to invest in my own drawing skills, if I want the exact image in my head to be out there in the world.

When I go to an art museum and look at oil paintings, I don’t think to m...
AI Summer Harvest
by Cleo Nardo — 2023-04-04T03:35:58.473Z

Trending metaphor

I've noticed that AI safety advocates have recently been appealing to a new metaphor: the AI summer harvest.

(1) Here's the FLI open letter Pause Giant AI Experiments:

Humanity can enjoy a flourishing future with AI. Having succeeded in creating powerful AI systems, we can now enjoy an "AI summer" in which we reap the rewards, engineer these systems for the clear benefit of all, and give society a chance to adapt. Society has hit pause on other technologies with potentially catastrophic effects on society.[5] We can do so here. Let's enjoy a long AI summer, not rush unprepared into a fall.

(source)

(2) And here's Yudkowsky talking on the Lex Friedman podcast:

If it were up to me I would be like — okay, like this far and no further.

Time for the Summer of AI, where we have planted our seeds and now we wait and reap the rewards of the technology we've already developed, and don't do any larger training runs that that."

(source, time-stamped)

Maybe I'm out-of-the-loop, but I haven't seen this metaphor until recently.

Brief thoughts

I like the metaphor and I wrote this article to signal-boost it. It's concise, accessible, and intuitive. It represents a compromise (Hegelian synthesis?) between AI decelerationism and AI accelerationism.
The metaphor points at the reason why slowing down AI progress won't (imo) impose significant economic costs — i.e. just as a farmer gains nothing by sowing seeds faster than they can harvest crops, so too will a lab gain nothing by scaling AI models faster than the economy can productise them.
Instead of "AI summer", we should call it an "AI summer harvest" because "AI summer" is an existing phrase referring to fast AI development; trying to capture an existing phrase is difficult, confusing, and impolite. Moreover, "AI summer harvest" does a better job at expressing the key idea.
I think that buying time is probably more important than anything else at the moment. (Personally, I'd endorse a 0.2 OOMs/year target.) And I think that "AI summer harvest" should be the primary angle by which we communicate buying-time to policy-makers and the general public.

Origins

The metaphor was accredited to @gwillen on March 16th 2023.
How to respond to the recent condemnations of the rationalist community
by Christopher King — 2023-04-04T01:42:49.225Z

On Twitter, there has been some concerning discourse about rationalism and AI. I'd rather not link to it directly, but just look up the term "TESCREAL" on Twitter. The R stands for rationalism. Here is a particularly strong tweet:

We need a support group for those of us who've been around the #TESCREAL bundle for years, watching them amass more power, start their various institutes, get all sorts of $$ & mainstream media megaphones, see what their various cults have been doing & just watching them expand.

From what I can tell, the "contra-TESCREAL" (as I am coining it) crowd do not seem interested in object level arguments about AI existential safety at this time. They are analyzing the situation purely from a social perspective.

So, what do we do? Do we get mad at them? What is the rational thing to do?

I think the answer is understanding and kindness.

AI safetyists and AI ethicists are aligned

The trigger seems to have been the difference between AI safety and AI ethics. However, I'd like to argue we are quite aligned each other, as the vast majority of human groups ultimately are.

I think most AI safety people agree that social issues are important. Although not specifically about AI, Elizer Yudkowsky published a post titled A Comprehensive Reboot of Law Enforcement that advocated such points as "Nationwide zero-tolerance for death of unarmed persons caused by law enforcement", "Comprehensively decriminalize most victimless offenses", and "Restore disparate enforcement as a judicial reason to challenge and strike down a law". These are things Eliezer Yudkowsky actually wants in real life right now, not things that he is just hypothetically exploring the consequences of.
I think most AI ethicists agree that, hypothetically, if a machine turned their and everyone else's bodies into paperclips, that would be at least somewhat worse than current social issues.

So I think it's fair to say that we have a similar utility function; AI safetyists and AI ethicists just come into conflict because they have different beliefs about how likely the singularity is. In the specific case of contra-TESCREAL, they also don't understand rationalist conversational norms. But I fear most of us don't understand their conversational norms either.

This is why I am proposing the prefix "contra-" instead of "anti-"; we are not enemies, we are in disagreement.[1]

How to respond?

I feel l...
Steering systems
by Max H — 2023-04-04T00:56:55.407Z

Preface: who this post is for

This post is mainly targeted at people who are already familiar with the concepts and terminology in alignment, AI x-risk, and particularly the case for high probabilities of doom, but disagree with the arguments or have bounced off of them for one reason or another.

If you've read the 2021 MIRI conversations and remain confused about or disagree with some of the things Nate or Eliezer were saying, this post is for you.

If you understand and agree with everything Nate and Eliezer say in those transcripts already, you probably won't get many new insights from this post, but you may find it useful as a new way of explaining one aspect which (I think) is important to their case.

If you're not familiar with those discussions at all, you may still get value out of this piece, but there is a background assumption of familiarity with some of the concepts discussed there.

Introduction

This post is my own attempt at conveying some intuitions and concepts which various others have already written about at length. I doubt I'll do a better job than those before me, but it seems worth a shot anyway, because:

It explains why I am not optimistic about AI x-risk, even if current approaches to alignment research and AI governance succeed far more swiftly and totally than they appear to be on track for.

It appears to be a persistent source of confusion and disagreement among alignment researchers and the LW commentariat. (This is my attempt at focusing on a place where I think others are dropping the ball).

In short, I'm going to unpack the concept in this tweet:

into some thought experiments that build an intuition for what "smarter than human" systems might look like, and explain why that intuition makes me think that humanity is on track to build and run powerful systems in ways likely to result in bad outcomes, even if some or all aspects of alignment and governance go unexpectedly (to me) well.

I'm going to introduce a new term, define it loosely, provide some examples, and apply it in the context of current systems and a hypothetical future system that performs tree search over world states.

I'll conclude with some implications about the different potential impacts of current alignment research approaches, depending on how this intuition differs from reality.

That which has many names

It has been called "powerful optimization process", "tra...
ChatGPT Suggests Listening To Russell & Yudkowsky
by JenniferRM — 2023-04-04T00:30:08.951Z

Methods: The bold text is me. The unbolded text is Chat. The headers were added at the end, after the conversation was done, to help people refer to specific parts of the discussion. More details at the end.

The Conversation

Getting Inside Reinforcement Learning

Can you explain reinforcement learning to me as if I was the agent getting signals and trying to change my internal structure in response to them?

Certainly! Reinforcement learning is a type of learning where an agent, like you, learns to take actions based on the feedback it receives from the environment. In this case, the feedback comes in the form of rewards or punishments, which are signals that tell you whether the action you just took was good or bad.

Imagine that you are playing a game where you have to navigate a maze to find a reward, like a piece of cheese. At first, you don't know the correct path to take, so you might take random actions, like turning left or right, moving forward, or moving backward. As you move through the maze, you receive feedback from the environment in the form of rewards or punishments. For example, if you move in the right direction, you might receive a reward, like a point. If you move in the wrong direction, you might receive a punishment, like losing a point.

As you receive feedback, you use it to update your internal structure or policy, which tells you what action to take in a given situation. For example, if you receive a reward for turning left at a certain point in the maze, you might update your policy to say that turning left in that situation is a good action to take. On the other hand, if you receive a punishment for turning right at a certain point, you might update your policy to say that turning right in that situation is a bad action to take.

Over time, as you receive more feedback and update your policy accordingly, you become better and better at navigating the maze and finding the reward. This is because your policy is now optimized to take actions that lead to the most rewards and the fewest punishments.

In summary, reinforcement learning is a process where you, as the agent, learn to take actions based on feedback in the form of rewards or punishments, and update your internal policy accordingly to optimize your future actions.

So at the beginning of that, I wouldn't even know that "bumping into walls was bad" and then somehow I would change to learn t...
Complex Systems are Hard to Control
by jsteinhardt — 2023-04-04T00:00:13.634Z

The deployment of powerful deep learning systems such as ChatGPT raises the question of how to make these systems safe and consistently aligned with human intent. Since building these systems is an engineering challenge, it is tempting to think of the safety of these systems primarily through a traditional engineering lens, focusing on reliability, modularity, redundancy, and reducing the long tail of failures.

While engineering is a useful lens, it misses an important part of the picture: deep neural networks are complex adaptive systems, which raises new control difficulties that are not addressed by standard engineering methodology. I’ve discussed some particular examples of this before, but here I want to focus on the broader underlying intuition that generated them.

A complex adaptive system is a system with many interacting components that adapt to their environment and co-evolve over time (in our case, the weights / layers of the neural network). Beyond neural networks, other examples of complex adaptive systems include firms, financial markets, political parties, culture, traffic flows, pathogens, ecosystems, human brains, the Earth’s climate, and the Internet.

A common thread in all these systems is that straightforward attempts to control their behavior lead to unintended consequences. I’ll demonstrate this through concrete examples, then step back and consider the broader properties that make these systems difficult to control, including emergent goals. Finally, I’ll propose safety measures that account for the complex adaptive nature of deep learning systems.

Many of the ideas in this post have been discussed before, and my thinking owes significantly to Dan Hendrycks, who was an early proponent of the complex systems perspective as a PhD student in my lab (see e.g. Unsolved Problems in ML Safety, the lecture on accident models from Dan’s course, or this blog post).

Control Difficulties in Complex Systems

Let’s examine several examples of complex systems, and see why each is difficult to control, in the sense that they either resist or respond unpredictably to external feedback.

Traffic. A city builds new highways to reduce traffic congestion. The newly increased road capacity attracts new drivers, leading to worse levels of congestion than before. The adaptive behavior leads to unintended consequences.

Ecosystems. A park introduces a predator to reduc...
Apply to the Cavendish Labs Fellowship (by 4/15)
by agg — 2023-04-03T23:09:07.309Z
Link: https://forum.effectivealtruism.org/posts/S8xfeJGER74xwqLta/apply-to-the-cavendish-labs-fellowship-by-4-15

Cavendish Labs is a new research organization in Vermont focused on technical work on existential risks. We'd like to invite you to apply to our fellowships in AI safety and biosecurity!

Positions are open for any time between June 1 and December 10, 2023. We pay a stipend of $1,500/month, plus food and housing are provided. Anyone with a technical background is encouraged to apply, even if you lack specific expertise in these fields.

Applications for summer research fellows are closing April 15th. Apply here!

Research lab on a river in Vermont, AI artist's conception

(Note: we likely cannot accept people who need visa sponsorship to work in the U.S.)
Communicating effectively under Knightian norms
by Richard_Ngo — 2023-04-03T22:39:58.350Z

tl;dr: rationalists concerned about AI risk often make claims that others consider not just unjustified, but unjustifiable using their current methodology, because of high-level disagreements about epistemology. If you actually want to productively discuss AI risk, make claims that can be engaged with by others who have a wide range of opinions about the appropriate level of Knightian uncertainty.

I think that many miscommunications about AI risk are caused by a difference between two types of norms for how to talk about the likelihoods of unprecedented events. I'll call these "inside view norms" versus "Knightian norms", and describe them as follows:

Inside view norms: when talking to others, you report your beliefs directly, without adjusting for "Knightian uncertainty" (i.e. possible flaws or gaps in your model of the world that you can't account for directly).
Knightian norms: you report beliefs adjusted for your best estimate of the Knightian uncertainty. For example, if you can't imagine any plausible future in which humanity and aliens end up cooperating with each other, but you think this is a domain which faces heavy Knightian uncertainty, then you might report your credence that we'll ever cooperate with aliens as 20%, or 30%, or 10%, but definitely nowhere near 0.

I'll give a brief justification of why Knightian norms seem reasonable to me, since I expect they're counterintuitive for most people on LW. On a principled level: when reasoning about complex domains like the future, the hardest part is often "knowing the right questions to ask", or narrowing down on useful categories at all. Some different ways in which a question might be the wrong one to ask:

The question might have important ambiguities. For example, consider someone from 100 years ago asking "will humans be extinct in 1000 years?" Even for a concept like extinction that seems very black-and-white, there are many possible futures which are very non-central examples of either "extinct" or "not extinct" in the questioner's mind (e.g. all humans are digital; all humans are dramatically genetically engineered; all humans are merged with AIs; etc). And so it'd be appropriate to give an answer like "X% yes, Y% no, Z% this is the wrong question to ask".
The question might be confused or ill-posed. For example, "how heavy is phlogiston?"
You might be unable to conceptualize the actual answer. For

...
If interpretability research goes well, it may get dangerous
by So8res — 2023-04-03T21:48:18.752Z

I've

historically

been pretty publicly

supportive

of interpretability research. I'm still supportive of interpretability research. However, I do not necessarily think that all of it should be done in the open indefinitely. Indeed, insofar as interpretability researchers gain understanding of AIs that could significantly advance the capabilities frontier, I encourage interpretability researchers to keep their research

closed

.

I acknowledge that spreading research insights less widely comes with real research

costs

. I'd endorse building a cross-organization network of people who are committed to not using their understanding to push the capabilities frontier, and sharing freely within that.

I acknowledge that public sharing of research insights could, in principle, both shorten timelines and improve our odds of success. I suspect that

isn’t the case in real life

.

It's much more important that blatant and direct capabilities research be made private. Anyone fighting for people to keep their AI insights close to the chest, should be focusing on the capabilities work that's happening out in the open, long before they focus on interpretability research.

Interpretability research is, I think, some of the best research that can be approached incrementally and by a large number of people, when it comes to improving our odds. (Which is not to say it doesn't require vision and genius; I expect it requires that too.) I simultaneously think it's entirely plausible that a better understanding of the workings of modern AI systems will help capabilities researchers significantly improve capabilities. I acknowledge that this sucks, and puts us in a bind. I don't have good solutions.

Reality doesn't have to provide you any outs.

There's a tradeoff here. And it's not my tradeoff to make; researchers will have to figure out what they think of the costs and benefits. My guess is that the current field is not close to insights that would significantly improve capabilities, and that growing the field is important (and would be hindered by closure), and also that if the field succeeds to the degree required to move the strategic needle then it's going to start stumbling across serious capabilities improvements before it saves us, and will need to start doing research privately before then.

I reiterate that I'd feel ~pure enthusiasm about a cross-or...
Towards empathy in RL agents and beyond: Insights from cognitive science for AI Alignment 
by Marc Carauleanu — 2023-04-03T19:59:00.057Z
Link: https://clipchamp.com/watch/6c0kTETRqBc

This is a talk I gave at the recent AI Safety Europe Retreat (AISER) on my research on obtaining insights from the cognitive science of empathy and applying them to RL agents and LLMs.

Talk link: https://clipchamp.com/watch/6c0kTETRqBc

Slides link: https://bit.ly/3ZFmjN8

Talk description: I begin by presenting a short review on the cognitive science of empathy as a Perception-Action Mechanism (PAM) which relies on self-other overlap at the neuronal level. I continue by presenting the theory of change of this research direction by arguing that inducing self-other overlap as empathy is model agnostic and that it has the potential to avert AI x-risk and be sub-agent stable in the limit. Then I present experimental evidence of the emergence of PAM in RL agents and present a way of inducing PAM in RL agents. I end the talk by discussing how this paradigm could be extended to LLMs.

Acknowledgements: I am thankful for Dr. Bogdan Ionut-Cirstea for inspiring me to look into this neglected research direction, the Long-Term Future Fund for funding the initial deep dive into the literature, and for Center for AI Safety for funding half of this research as part of their Student Researcher programme. Last but not least, I want to thank Dr. Matthias Rolf for supervising me and providing good structure and guidance. The review on the cognitive science of empathy is adapted from a talk given by Christian Keysers from the Netherlands Institute for Neuroscience.