trentmkelly/LessWrong-43k huggingface dataset:
"Brain enthusiasts" in AI Safety
by Jan — 2022-06-18T09:59:04.149Z
Link: https://universalprior.substack.com/p/brain-enthusiasts-in-ai-safety

TL;DR: If you're a student of cognitive science or neuroscience and are wondering whether it can make sense to work in AI Safety, this guide is for you! (Spoiler alert: the answer is "mostly yes").

Motivation

AI Safety is a rapidly growing field of research that is singular in its goal: to avoid or mitigate negative outcomes from advanced AI. At the same time, AI Safety research touches on many aspects of human lives: from the philosophy of human values, via the neuroscience of human cognition, to the intricacies of human politics and coordination. This interplay between a singular goal with multiple facets makes the problem intrinsically interdisciplinary and warrants the application of various tools by researchers with diverse backgrounds.

Both of us (Sam & Jan) have backgrounds in cognitive science and neuroscience (we'll use the blanket term "brain enthusiasts" from here on). Important characteristics of brain enthusiasts are (see if these apply to you, dear reader):

A propensity for empirical data and experiments.
Regarding coding and mathematics as tools rather than as ends in themselves.
Having accumulated semi-random facts about some biological systems.

While being brain enthusiasts undoubtedly makes us biased in favor of their importance, it also gives us a (hopefully) useful inside view of how brain enthusiasts might meaningfully contribute to AI Safety[1]. In this post, we attempt to shine a light on the current representation of brain enthusiasts in AI Safety and provide some advice on how brain enthusiasts might enter the field.

A long tail of number-lovers

When you hear terms like "AI Alignment", "AI Safety," or "AGI", you probably think of people with a strong technical background, e.g. in computer science, mathematics, or physics. This is an instance where stereotypes are correct:

Using a previously published dataset, we determined the number of Alignment Forum posts per researcher and sorted the resulting table. We then added a "Background" column, where we noted the field in which the researcher obtained their most recent degree[2]. As observed previously, the number of published posts is a long-tailed distribution[3], with Stuart Armstrong dominating over everyone.

Almost everyone on the list has a computer science, physics, or mathematics background. Notable exceptions are three researchers with an "officially different" formal education (...
To what extent have ideas and scientific discoveries gotten harder to find?
by lsusr — 2022-06-18T07:15:44.193Z

This post was funded by a grant from Ben Pace.

Ben Pace asks:

To what extent have ideas and scientific discoveries gotten harder to find?

Related: Why are there no gentlemen scientists any more — i.e. rich people who make novel scientific discoveries? Like Fermat and Pascal.

Some theories have been put forward by Scott Alexander and Holden Karnofsky. Maybe they’re right, maybe not.

Scott Alexander treats scientific ideas as nonrenewable.

Imagine scientists venturing off in some research direction. At the dawn of history, they don’t need to venture very far before discovering a new truth. As time goes on, they need to go further and further.

Holden Karnofsky believes the pattern applies to both art and science.

The broad theme is that across a variety of areas in both art and science, we see a form of "innovation stagnation": the best-regarded figures are disproportionately from long ago, and our era seems to "punch below its weight" when considering the rise in population, education, etc. Since the patterns look fairly similar for art and science, and both are forms of innovation, I think it's worth thinking about potential common factors.

I have a different perspective.

Science

Physics

The most important scientific discoveries are those which are the most general and the most useful. Physics is the most general of sciences. Physics is basically solved.

There are unsolved problems in physics. Dark matter remains a mystery. Quantum mechanics has yet to be unified with general relativity. But the holes in physics don't matter. You don't need a Grand Unified Theory of quantum relativity to build a Mars base or a fusion reactor. All you need is today's physics plus a whole lot of engineering.

The recent discoveries in physics like quantum computing and the photograph of a black hole aren't really discoveries about the fundamental Laws of Physics. They're technological achievements.

All of the rest of science is just applied physics too. One could argue that biology, chemistry and so on are just footnotes to Einstein. The fundamental laws of the universe are (for all practical purposes) known. The remaining questions are:

Astrophysics i.e. the study of places that don't matter because we lack the technology to go there.
What has biology built out of matter?
What can we build out of matter?

No physicist will ever again make a discovery (in physics) as impactful as the great 20th cen...
What's the goal in life?
by Konstantin Weitz — 2022-06-18T06:09:25.356Z

I've learned a lot of ways to make smarter decisions (e.g. by being aware of cognitive biases) by reading LessWrong, but there are a ton of important questions where I don't even know what the goal is, so all those techniques don't really help. For example:

Should I spend more time with my family, or at work?
Should I quit my job and travel the world?
How many kids should I have?
Should I use my money to help others, or use it to help me and my family?
Should I aim to maximize happiness, or wealth?
Should I save my money and give it to my kids, or spend it all in my lifetime?

There is a lot of advice out there, e.g. just do what makes you happy, work hard and play hard, family always comes first, treat others the way you would like to be treated, what would Jesus do?, the meaning of life is to find the meaning of life, etc. While these sound good, the rationale for why these are good goals is usually pretty hand wavy (or maybe I just don't understand it).

Are you aware of any good answers, or research into what our goals in life should be? I'm not entirely sure what a good answer would look like, but I could see it being based on biology/evolution (they have good explanations for how we got where we are, maybe they have good explanations for where we should go?), or game theory (they often provide good recommendations on what to do, e.g. in auctions, and nuclear war), or AI research (if you're researching goals for computers, maybe you've got something smart to say about human goals as well), or maybe it's something completely different.

I'm very curious to hear your thoughts!
Can DALL-E understand simple geometry?
by Isaac King — 2022-06-18T04:37:46.471Z

I recently got access to DALL-E and wanted to experiment with what it could and couldn't understand. (I'm a little behind the times, I know. Us non-famous people have to spend a while longer on the waitlist.) Most of the other investigations into DALL-E seemed to be focusing on highly complicated real-world concepts, so I went the other way and tried very simple, abstract things.

DALL-E is incredibly bad at this. It doesn't seem to be able to generalize at all, and it refuses to draw anything simple without adding a bunch of extra details I didn't ask for.

Full post here.
Scott Aaronson is joining OpenAI to work on AI safety
by peterbarnett — 2022-06-18T04:06:55.465Z
Link: https://scottaaronson.blog/?p=6484

Scott Aaronson is a computer scientist at the University of Texas in Austin, whose research mainly focuses on quantum computing and complexity theory. He's at least very adjacent to the Rationalist/LessWrong community. After some comments on his blog and then coversations with Jan Leike, he's decided work for one year on AI safety at OpenAI.

To me this is a reasonable update that people who are sympathetic to AI safety can be convinced to actually do direct work.

Aaronson might be one of the easier people to induce to do AI safety work, but I imagine there are also other people who are worth talking to about doing direct work on AI safety.
Why don't we think we're in the simplest universe with intelligent life?
by ADifferentAnonymous — 2022-06-18T03:05:57.194Z

My gut instinct on metacosmology is that if were a simpler computation than our universe that produced intelligent life, we'd probably be there instead of here. I'm not sure that's valid anthropics, but it still surprises me to see apparent assumption (in posts like this of the opposite conclusion—that there are many universes simpler than ours capable of producing intelligence. (EDIT: that post doesn't actually make that assumption, and I don't have another example ready. Still turned out to be a fruitful question, though)

(Yes, we know cellular automata can implement intelligence, but AFAIK we don't know that they can do it more simply than by implementing a Turing machine stimulating our universe).

Is there an argument I've missed?
Do yourself a FAVAR: security mindset
by lemonhope — 2022-06-18T02:08:47.415Z

(We're always in search of a concise and actionable way to communicate security mindset. Here's my attempt.)

Just as a sprocket you don't use never fails and code you don't write has no bugs, an input constraint you don't require is never violated.

Want a secure system? Then DO yourself a FAVAR:

Describe Objectives: How is the system supposed to function? What constraints does its operation and output have?

F

ind

A

ssumptions

: What is the system assuming about its inputs, its environment, or itself?
Validate Assumptions: Are the assumptions we're making always correct? Even if they all are, then ↓
Remove the ones you can: Of course any incorrect assumptions have to be removed, but it is important to remove as many assumptions as possible. Helicopters land even if the engine dies.

Let's take this transferMoney procedure in a token program or an online bank as a case study:

transferMoney(fromId, fromSig, amount, destId):
if amount < 0: return
if not validate(fromId, fromSig): return
curBal := db.get(fromId)
if curBal < amount: return
db.set(fromId, curBal - amount)
db.set(destId, db.get(destId) + amount)

Our objective is for total balance to be conserved, and for transactions to only be initiated by the sender.
Finding assumptions is the hard part. This code actually assumes at least this much:

Addition and subtraction do not overflow the numbers
(Signature validation is trusted)
Nothing else updates the db during transfer's execution
The second-to-last line never throws an error

Validating: We scour the codebase. It seems, hopefully, the numbers come from a trusted source, the function caller locks the records, the database is very reliable, etc. But we notice that if we're wrong about any of these, even once, then a user could probably create or destroy unlimited money. Simply double-clicking the transfer button might actually send 2x the money but only remove 1x!!

Note, we could add more checks, try/catches, do more static analysis, etc. Instead, we'll try this:

Remove assumptions: Instead of being so careful in everything about the function's environment, we can eliminate most of these assumption by writing the code differently!

transferMoneyBetter(fromId, fromSig, amount, destId):
if amount < 0: return
if not validate(fromId, fromSig): return
with db.transaction as tx:
curBal := tx.get(fromId)
if (curBal < amount): return
db.set(fromId, safeSub(cur

...
Forecasting Fusion Power
by Daniel Kokotajlo — 2022-06-18T00:04:34.334Z
Link: https://astralcodexten.substack.com/p/your-book-review-the-future-of-fusion

This is a linkpost for "The future of fusion" book review.
IMO it's not really a book review, it's a well-written and data-driven argument that commercial fusion power is about 10-15 years away, no really for real this time. (!!!) Good discussion in the comments too. I'm interested to see what people here think. [ETA: the argument is for fusion power in less than 10-15 years; I mentally added a few years and slapped "commercial" on the front, though yeah I admit it could plausibly take longer idk.]
The author's credences/timelines:

I'm going to update my post on what 2040 looks like assuming no singularity accordingly.
Pivotal outcomes and pivotal processes
by Andrew_Critch — 2022-06-17T23:43:19.230Z

tl;dr: If you think humanity is on a dangerous path, and needs to "pivot" toward a different future in order to achieve safety, consider how such a pivot could be achieved by multiple acts across multiple persons and institutions, rather than a single act. Engaging more actors in the process is more costly in terms of coordination, but in the end may be a more practicable social process involving less extreme risk-taking than a single "pivotal act".

Preceded by: “Pivotal Act” Intentions: Negative Consequences and Fallacious Arguments

[This post is also available on the EA Forum.]

In the preceding post, I argued for the negative consequences of the intention to carry out a pivotal act, i.e., a single, large world-changing act sufficient to 'pivot' humanity off of a dangerous path onto a safer one. In short, there are negative side effects of being the sort of institution aiming or willing to carry out a pivotal act, and those negative side effects alone might outweigh the benefit of the act, or prevent the act from even happening.

In this post, I argue that it's still a good idea for humanity-as-a-whole to make a large / pivotal change in its developmental trajectory in order to become safer. In other words, my main concern is not with the "pivot", but with trying to get the whole "pivot" from a single "act", i.e., from a single agent-like entity, such a single human person, institution, or AI system.

Pivotal outcomes and processes

To contrast with pivotal acts, here's a simplified example of a pivotal outcome that one could imagine making a big positive difference to humanity's future, which in principle could be brought about by a multiplicity of actors:

(the "AI immune system") The whole internet — including space satellites and the internet-of-things — becomes way more secure, and includes a distributed network of non-nuclear electromagnetic pulse emitters that will physically shut down any tech infrastructure appearing to be running rogue AI agents.

(For now, let's set aside debate about whether this outcome on its own would be pivotal, in the sense of pivoting humanity onto a safe developmental trajectory... it needs a lot more details and improvements to be adequate for that! My goal in this post is to focus on how the outcome comes about. So for the sake of argument I'm asking to take the "pivotality" of the outcome ...
Quantifying General Intelligence
by JasonBrown — 2022-06-17T21:57:13.592Z

Introduction

This piece seeks to explore an interesting way of defining intelligent systems such that we can theoretically quantify their general intelligence.
From this, further tools and ideas for comparing these entities could be developed.
The definitions are not meant to be philosophical truths, rather they are meant to be useful tools that will allow us to analyse and gain insight into these systems and how they relate to one another.
At least that's the hope, failing that they can perhaps at least provide some food for thought.

This post is meant to be accessible to non-technical readers so some terms may be explained to a level of detail unnecessary for people familiar with machine learning.

Desirable Properties

We begin by identifying several desired properties that would increase the utility and robustness of our framework, giving us something to aim at.

Sufficient: If our definitions relied upon, or referenced, things that are poorly defined themselves, we would just be moving the problem back a step and not actually gaining any insight.

Measurable: Intelligence is a broad spectrum, this especially visible in the natural world. A good definition would reflect this and give us a continuous measure of intelligence that allows sensible comparisons.

Implementation Independent: It's easy to compare somethings capabilities to humans in order to ascertain their intelligence. We want our definitions to be free from bias towards any particular implementation or version of intelligence, so that it can recognise intelligence which operates in a way unfamiliar to us, or in a way we don't understand.

Minimal Grey Areas: Many definitions could leave large grey areas on boundaries between classifications, or not make sense when applied to domains they were not designed with in mind. This should be avoided.

Useable: Sometimes a seemingly 'perfect' definition is infeasible to actually apply, and so is of no practical use. A definition which is infeasible to theoretically calculate, could have a method to estimate it. Quantifying how reliable or accurate the estimates are would also be useful.

Agents

A classic definition of an agent is something that interacts with an environment, choosing an action (or actions), to achieve some desired goal. To make this rigorous, we will define an agent in our framework as something that produces an action based on the state of its env...
Apply for Productivity Coaching and AI Alignment Mentorship
by Nick — 2022-06-17T21:36:10.177Z

Hello. I am an AI alignment researcher who has a background in mental health and teaching. I originally trained to do therapy before deciding I could best help the world by working on AI alignment directly. I want to use my experience in mental health and productivity coaching to help young, aspiring alignment researchers be their best selves.

Rationale:

I want to emphasize mental wellness over mental health. Mental wellness is the effectiveness by which one can achieve their goals in relation to one's innate cognitive abilities. I think there is a lot of low-hanging fruit for each individual to better themselves; however, mental health services are designed to treat chronic problems. This service I will be providing aims to apply some of the techniques found in therapy to help already successful individuals be more successful, happier, and

more effective

.

Who this is for:

Effective altruists and aspiring AI alignment researchers.

Who this is not for:

People who want help with specific mental health diagnoses. For help with a mental health diagnosis (or suspected one), call your insurance company and ask what services they cover.

My experience:

Therapy, 2 years

Tutoring math, physics, and computer science, 5 years

AI research, 3 years

What this productivity coaching/mentorship will cost:

Nothing! For now, this service is free.

To apply, fill out this form.
Things That Make Me Enjoy Giving Career Advice
by Neel Nanda — 2022-06-17T20:49:43.172Z
Link: www.neelnanda.io/blog/49-mentoring

Introduction

I spend a bunch of my time having various kinds of mentoring conversations with people. I generally enjoy doing this, but some things make these conversations much more fun for me than others! This post is an attempt to compile some of these.

Obviously, writing this post is somewhat self-serving, since I hope future people I mentor might read it! But I expect there’s decent chance that at least some of this will generalise to other people. And, hopefully this is also useful to mentees! Based on my experiences trying this, approaching people for mentorship can be pretty anxiety inducing. And having a clearer model of how to make this more fun for them and more respectful of their time helps make reaching out feel easier. But note that there’s a bunch of things in this post - please take them as some tips and nice-to-haves, rather than as eg things where I’ll judge you if you miss out some of them!

Context: The central example of a mentoring chat I imagine in this post is a career coaching chat, where I talk to someone who wants to work on AI Safety, who I don’t already know, and don’t necessarily expect to talk to again. But many of these generalise to other kinds of mentoring chats, eg where I’m orienting more towards giving life or productivity advice, trying to help out a friend/someone I know better, etc.

Tips

Context Setting

Useful things to happen at the start of a call, to give me an idea of what’s going on and how I can best help:

Making it clear what you want out of the call, and how I can be most useful

An easy way to do this is to explicitly state your goals and key uncertainties at the start

A common mistake here is trying to only state key uncertainties that sound good/rational. If your key uncertainty is a less tangible emotional crux, like “I basically buy the arguments that AI Safety matters, but feel emotionally uncomfortable with the idea of working on something with a small chance of mattering”, say! I can’t be useful if I’m focusing on stuff that isn’t relevant.
A good litmus test - if you changed your mind on the question we’re discussing, how would this change your actions?

Another angle is setting out the kind of call you want. Do you have a few concrete options you want help deciding between? Is this more of an exploratory conversation to figure out what’s out there? Do you want to dive deep into a speci

...
The Unified Theory of Normative Ethics
by Thane Ruthenis — 2022-06-17T19:55:19.588Z

Epistemic status: Reframing of a trivial result, followed by a more controversial assertion.

Summary: For bounded agents acting in complex environments, using raw consequentialist reasoning for every moment-to-moment decision is prohibitively costly/time-inefficient. A rich suite of heuristics/approximations may be developed to supplement it, recording the correspondences between certain classes of actions and the consequences actions from that class robustly tend to cause. Instincts, traditions, and moral inclinations/deontology are well-explained by this: an agent that started out as a pure environment-based consequentialist would end up deriving them ex nihilo.

I use this to argue that humans are pure consequentialists over environment-states: we don't have real terminal preferences for some actions over others.

Bootstrapping

Suppose you're an agent with an utility function over world-states. You exist in a complex environment populated by other agents, whose exact utility functions are varied and unknown. You would like to maneuver the environment into a high-utility state, and prevent adversarial agents from moving it into low-utility states. How do you do that?

Consequentialism is the natural, fundamental answer. At every time-step, you evaluate the entire array of actions available to you, then pick the action that's part of the sequence most likely to move the world to the highest-value state.

That, however, is a lot of computation. You have to consider what actions you can in fact take, their long-term terminal effects, their instrumental effects (i. e., how much it expands or constraints your future action-space), their acausal effects, and so on.

Every time-step.

And you have to run that computation every time-step, because you're an embedded agent: at any given time your model of the world is incomplete, and every time-step you get new information you must update on. You can't pre-compute the optimal path through the environment, you have to continuously re-plot it.

If you're a bounded agent with limited computational capabilities, all of this would take time. Agents who can think quicker would out-compete those who think slower. (As a trivial example, they'd win in a knife fight.) Therefore, it'd be beneficial to find faster algorithms for computing the consequences, to replace or supplement "raw" consequentialism.

Is there a way to do that?

Suppose the en...