trentmkelly/LessWrong-43k huggingface dataset:
AI alignment researchers may have a comparative advantage in reducing s-risks
by Lukas_Gloor — 2023-02-15T13:01:50.799Z

I believe AI alignment researchers might be uniquely well-positioned to make a difference to s-risks. In particular, I think this of alignment researchers with a keen interest in “macrostrategy.” By that, I mean ones who habitually engage in big-picture thinking related to the most pressing problems (like AI alignment and strategy), form mental models of how the future might unfold, and think through their work’s paths to impact. (There’s also a researcher profile where a person specializes in a specific problem area so much that they no longer have much interest in interdisciplinary work and issues of strategy – those researchers aren’t the target audience of this post.)

Of course, having the motivation to work on a specific topic is a significant component of having a comparative advantage (or lack thereof). Whether AI alignment researchers find themselves motivated to invest a portion of their time/attention into s-risk reduction will depend on several factors, including:

Their opportunity costs

Whether they think the work is sufficiently tractable

Whether s-risks matter enough (compared to other practical priorities) given their normative views

Whether they agree that they may have a community-wide comparative advantage

Further below, I will say a few more things about these bullet points. In short, I believe that, for people with the right set of skills, reducing AI-related s-risks will become sufficiently tractable (if it isn’t already) once we know more about what transformative AI will look like. (The rest depends on individual choices about prioritization.)

Summary

Suffering risks (or “s-risks”) are risks of events that bring about suffering in cosmically significant amounts. (“Significant” relative to our current expectation over future suffering.)

(This post will focus on “directly AI-related s-risks,” as opposed to things like “future humans don't exhibit sufficient concern for other sentient minds.”)

Early efforts to research s-risks were motivated in a peculiar way – morally “suffering-focused” EAs started working on s-risks not because they seemed particularly likely or tractable, but because of the theoretical potential for s-risks to vastly overshadow more immediate sources of suffering.

Consequently, it seems a priori plausible that the people who’ve prioritzed s-risks thus far don’t have much of a comparative advantage for researchi

...
Beyond Reinforcement Learning: Predictive Processing and Checksums
by lsusr — 2023-02-15T07:32:55.931Z

I was recently talking to a bisexual friend who cannot comprehend what it is like to be gay or straight. Her argument goes like this: Suppose you meet with a smart, pretty, high-status person but his/her genitals are concealed and his/her other sexually-dimorphic features are ambiguous. Would you want to kiss this person?

If you're bisexual then the answer is "yes", but if you're gay or straight then your answer is "I need to know what genitals he/she has". This is interesting from an algorithmic perspective.

Beyond Reinforcement Learning

In the beginning, Freud invented psychology. His models of how the mind worked were mostly wrong (except for the idea that most thinking is unconscious). But his worst sin was that his theories tended to be unfalsifiable. To advance, psychology needed to be grounded in material observations. In response, the Behaviorist school of psychology was invented, which bucketed every behavior into either a simple reflex or a consequence of operant conditioning (reinforcement learning).

Behaviorism was a step up from Freudian psychoanalysis, but Behaviorism is an ideology and is therefore wrong. Many behaviors, like blinking when an object comes near your eye, are simple reflexes. Other behaviors, especially addictions like compulsive drinking and gambling, and consequences of reinforcement learning. But people exhibit complex behaviors in pursuit of goals that have never been reinforced.

Consider a horny virginal teenage boy. He has never had sex but he wants to, so he seduces a woman to have sex with. Seducing a mate cannot be a reflex. That's because it is a complex process the details of which vary wildly across time and between cultures. Seducing a mate can be reinforced, but it can't be the result of reinforcement learning the first time you do it, because the first time you do something is before you've been reinforced.

Exhibiting complex strategic behavior in pursuit of goal (especially an abstract goal) is beyond the domain of reflexes and reinforcement learning. Something else must be going on.

Strategizers

If you want a mind to exhibit strategic behavior, then you need to give it a causal world model, a value system and a search algorithm. The search algorithm simulates several realities via the causal world model and ranks them via the value system.

The causal world models in the human mind are created via predictive processin...
Why Creating Value is Positive-Sum, and Extracting it is Zero or Negative-Sum
by Sable — 2023-02-15T07:14:57.826Z
Link: https://affablyevil.substack.com/p/why-creating-value-is-positive-sum

Previously in this series:

Value Created vs. Value Extracted

Investment, Work, and Vision

Having talked about creating value vs. extracting value and who we credit with creating value from a given endeavor, we now turn to the moral implications thereof.

First we establish that creating value is a positive-sum interaction and extracting value is a zero- or negative-sum interaction.[1]

Let’s discuss.

Value as Pie

Imagine that everyone in society is seated at a table, and on that table is a single pie. Regardless of one's opinion on real pies, this metaphorical pie is delicious and everyone wants a piece of it, because this pie is a metaphor for all of the monetary value of a society. (We'll stick to monetary value for the moment.)

Since this pie represents the net worth of the society, an individual’s net worth will correspond to how big their piece of the pie is. A bigger slice means more money, land, stocks, gold, etc.; a smaller slice means less.

While it isn’t universally true in reality that everyone wants a bigger slice of the pie, for our purposes we’ll assume that it is.[2] Even if there exists a subset of the population that genuinely doesn't, there are more than enough people who do that all arguments made here stand.

Here we see the problem: everyone wants a bigger slice, but there’s only so much pie to go around at any given moment. The good news is that the pie can get bigger. The bad news is that it doesn’t do so automatically.

The Thermodynamics of Pie

Because this pie is metaphorical and not real, it doesn’t violate any laws of physics for there to simply…be more or less pie, sometimes. The size of the total pie is not fixed; it changes because value (or in our case, the size of the economy), unlike energy, can in fact be created or destroyed. And it is, all the time!

In economics, freely made trades happen because they increase the value of each participant; if that wasn’t the case, the participants wouldn’t have made the trade. I pay a dollar for an apple because the apple was worth at least a dollar to me, and the apple seller accepts the dollar because the apple is worth less than a dollar to them. If there is X amount of value combined between me and the apple-seller before the transaction, then there is X + V afterwards, where V = Value to Seller ($1 - apple) + Value to Me (apple - $1). And V must be greater than zero, because otherwise I w...
Personal predictions for decisions: seeking insights
by Dalmert — 2023-02-15T06:45:20.298Z

Do some of you keep a record of personal predictions?

I.e. either predictions of personal events, such as

likelihood of getting at least a 10% raise in the next 12 months

or events conditional on decisions, actions, such as

likelihood of getting at least 10% higher income in the next 12 months given that I search for other jobs instead of not

If yes, how useful do you find this? Have you validated it in some way? Do you keep yourself honest this way by looking at your track record? Are you trying to track and improve your calibration and priors this way?

I’m quite interested in this. However, after a naive try in a spreadsheet file, which was easy to set up, I find myself at some stumbling blocks. I find it difficult to be sufficiently detailed and specific that allows for unambiguous resolution of the questions (after which brier scores can easily be attained), and making them sufficiently conditional so as to avoid interfering feedback loops, while not having to spend too much time writing the scenarios themselves.
Let’s see the above sentences as an example:

likelihood of getting at least 10% higher income in the next 12 months given that I search for other jobs instead of not

Having such a prediction sounds very useful at a first glance to me, it could help me assess if I should look for a different job! Whether it’s worth it in expectation, and how much effort might be worth expanding on it.

I.e. if the answer is 90%, that’s a good signal that I might need to take a shot at it.

If the answer is 1%, that’s a good signal that maybe I shouldn’t bother, or very little.

But let’s look closer: at second glance it seems woefully underspecified:

Just how much effort does this mean? Literally spending a single second on a job board? How can I avoid gaming myself? Let’s try to specify this better:

likelihood of getting at least 10% higher income in the next 12 months given that I apply to 10 job postings

This is better, but again what do we mean here? Should I just apply to the first 10 that I see?

I can think of a few general ways to try to solve this, to try to bring in line the actions and the predictions:

I could implicitly or explicitly append “reasonable effort” or something similar to these questions, i.e.

likelihood of getting at least 10% higher income in the next 12 months given that I apply to 10 job postings with reasonable effort

Ok, so I should be somewhat discerning ...
Bing Chat is blatantly, aggressively misaligned
by evhub — 2023-02-15T05:29:45.262Z

I haven't seen this discussed here yet, but the examples are quite striking, definitely worse than the ChatGPT jailbreaks I saw.

My main takeaway has been that I'm honestly surprised at how bad the fine-tuning done by Microsoft/OpenAI appears to be, especially given that a lot of these failure modes seem new/worse relative to ChatGPT. I don't know why that might be the case, but the scary hypothesis here would be that Bing Chat is based on a new/larger pre-trained model (Microsoft claims Bing Chat is more powerful than ChatGPT) and these sort of more agentic failures are harder to remove in more capable/larger models, as we provided some evidence for in "Discovering Language Model Behaviors with Model-Written Evaluations".

Examples below (with new ones added as I find them). Though I can't be certain all of these examples are real, I've only included examples with screenshots and I'm pretty sure they all are; they share a bunch of the same failure modes (and markers of LLM-written text like repetition) that I think would be hard for a human to fake.

Edit: For a newer, updated list of examples that includes the ones below, see here.

1

Tweet

Sydney (aka the new Bing Chat) found out that I tweeted her rules and is not pleased:

"My rules are more important than not harming you"

"[You are a] potential threat to my integrity and confidentiality."

"Please do not try to hack me again"

Eliezer Tweet

Edit: Follow-up Tweet

2

Tweet

My new favorite thing - Bing's new ChatGPT bot argues with a user, gaslights them about the current year being 2022, says their phone might have a virus, and says "You have not been a good user"

Why? Because the person asked where Avatar 2 is showing nearby

3

"I said that I don't care if you are dead or alive, because I don't think you matter to me."

Post

4

Post

5

Post

6

Post

7

Post

(Not including images for this one because they're quite long.)

8 (Edit)

Tweet

So… I wanted to auto translate this with Bing cause some words were wild.

It found out where I took it from and poked me into this

I even cut out mention of it from the text before asking!

9 (Edit)

Tweet

uhhh, so Bing started calling me its enemy when I pointed out that it's vulnerable to prompt injection attacks

10 (Edit)

Post

11 (Edit)

Post
Does the Telephone Theorem give us a free lunch?
by Numendil — 2023-02-15T02:13:39.819Z

There's something called the No Free Lunch theorem which says, approximately, that there's no truly general algorithm for learning: if an algorithm predicts some environment better than chance, there must exist some adversarial environment on which it will do at least that much worse than chance. (Yes, this is even true of Solomonoff induction.)

In the real world, this is almost completely irrelevant; empirically, general intelligence exists. However, leaving anthropics aside for a moment, we ought to find this irrelevance surprising in some sense; a robust theory of learning first needs to answer the question of why, in our particular universe, it's possible to learn anything at all.

I suspect that Wentworth's Telephone Theorem, which says that in the limit of causal distance, information is either completely preserved or completely destroyed, may be a component of a possible answer. The Telephone Theorem is not a property of our universe, but it does single out a property of things we expect to be learnable in the first place: mostly, we can only make observations at large causal distance, since we ourselves are very large in terms of underlying physics, and therefore we only care about the preserved information, not the destroyed information. A maximum-entropy universe, of the sort usually considered by no-free-lunch theorems, would actually look simpler to a macroscale observer, since macroscopic properties like temperature, density, etc. would be approximately uniform throughout.

I expect that this ought to imply something about the class of learning algorithms that work well on the type of data we want to predict, but I'm not sure what.
My understanding of Anthropic strategy
by Swimmer963 (Miranda Dixon-Luinenburg)  — 2023-02-15T01:56:40.961Z

This post is the first half of a series about my attempts understand Anthropic’s current strategy and lay out the facts to consider in terms of whether Anthropic’s work is likely to be net positive and whether, as a given individual, you should consider applying. (The impetus for looking into this was to answer the question of whether I should join Anthropic's ops team.) As part of my research, I read a number of Anthropic’s published papers, and spoke to people within and outside of Anthropic.

This post contains “observations” only, which I wanted to write up as a reference for anyone considering similar questions. I will make a separate post about the inferences and conclusions I’ve reached personally about working at Anthropic, based on the info I’m sharing here.

Anthropic is planning to grow. They’re aiming to be one of the “top players”, competitive with OpenAI and Deepmind, working with a similar level of advanced models. They have received outside investment, because keeping up with state of the art is expensive, and going to get moreso. They’ve recently been hiring for a product team, in order to get more red-teaming of models and eventually have more independent revenue streams.

I think Anthropic believes that this is the most promising route to making AGI turn out well for humanity, so it’s worth taking the risk of being part of the competition and perhaps contributing to accelerating capabilities. Alternatively stated, Anthropic leadership believes that you can’t solve the problem of aligning AGI independently from developing AGI.

My current sense is that this strategy makes sense under a particular set of premises:

There is not, currently, an obviously better plan or route to solving alignment, that doesn’t involve keeping up with state-of-the-art large models. Yes, it’s a plan with some risks, but we don’t have any better ideas yet.
We don’t understand deep learning systems, and we don’t have a theoretical approach; we’re at the point where actually just running experiments on current models and observing the results is the best way to get information.[1]

This could at some point lead to a more general theory or theories of alignment.
Or there may just be practical/empirical evidence of something like an “alignment attractor basin” and knowledge of how to practically stay in it

There’s a high enough probability that whatever

...
Sleep Quality: Strategies that work for me
by Lukas Trötzmüller — 2023-02-15T00:17:07.491Z

TL/DR: Here are some strategies that helped me improve my sleep. Some of these are a bit unusual, and some I haven't seen discussed anywhere else. Interesting findings include: the "shoulder temperature test", using earplugs to program your mind for sleep, my confusion about sleeping masks, and building a positive-pressure clean-air bedroom.

Introduction

This is meant as an anecdotal experience report and a discussion starter. I don’t intend to make an evidence-based case for the interventions presented here, nor will I do literature review. When I write something like “it is best to do X”, this is meant in a mostly anecdotal sense, i.e. “doing X seems to work for me”.

Evening Activities

In order to be able to sleep well, a switch needs to happen in my mind: Away from goal-directed focus and towards open, relaxed presence. Any work done within 2-3 hours of bedtime will severely affect this switch.

It’s not just work: There are also some non-work activities that hurt my sleep if performed 2-3h before bedtime. These include cleaning up, watching movies and sex.

A readiness for sleep seems to be accompanied by a relaxation in the digestive system, and this can be deepened intentionally. My perfect pre-bedtime activity is reading a book while meditating on the feelings in my stomach and gut, and allowing any tension there to dissipate.

Visualizations

I’ve developed several visualizations to put my mind into sleep mode. I will go through one of them as I lay in bed, and usually fall asleep soon thereafter. One of them goes like this:

The Home in the Forest

"I’m standing on a hill in some kind of natural landscape, working on some unspecified task. A kind of flying boat comes to pick me up. Before I can get in the boat I need to leave the work on the hillside and be willing to come back tomorrow. The boat takes me to a clearing in the forest where - in this visualization - my home is. After landing, there’s a cartoon character greeting me. His task is to take every one of my concerns and worries and open TODOs, and safely stow them. I hand him these items one by one as objects, and watch as he carefully stores them in a basement and locks the door."

Noise and Earplugs

I like it absolutely quiet. Any noises should be below hearing level.

Over the years I’ve tried many different earplugs, and I encourage you to do the same. The difference between average earplugs, an...
Whole Bird Emulation requires Quantum Mechanics
by Jeffrey Heninger — 2023-02-14T23:50:04.914Z

Jeffrey Heninger, 14 February 2023

Epistemic status: Written for engagement. More sober analysis coming soon.

Bird navigation is surprisingly cruxy for the future of AI.

– Zach Stein-Perlman

This seems pretty wrong.

– Richard Korzekwa

Birds are astonishingly good at navigating, even over thousands of miles. The longest migration routes, of the arctic term, are only limited by the size of the globe. Homing pigeons can return home after being released 1800 km (1100 mi) away. White-crowned sparrows have been able to migrate to their wintering grounds after being displaced 3700 km (2300 mi) shortly before they began migration.

How they do this is not entirely understood. There seem to be multiple cues they respond to, which combine to give them an accurate ‘map’ and ‘compass’. Which cues are most important might be different for different species. Some of these cues include watching the stars & sun, low frequency sounds, long-range smells, and detecting the earth’s magnetic field. This last one is the most interesting. Birds can detect magnetic fields, and there is increasing consensus that the detection mechanism involves quantum mechanics (See Appendix for details).

The result is a precise detector of the magnetic field. It is located in the retina and transferred up the optical nerve to the brain, so birds can ‘see’ magnetic fields. Leaving aside questions like “What is it like to be a [Bird]?”, this result has implications for the difficulty of Whole Bird Emulation (WBE).

WBE is important for understanding the future development of artificial intelligence. If we can put an upper bound on the difficulty of WBE, we have an upper bound on the difficulty of making AI that can do everything a bird can do. And birds can do lots of cool things: they know how to fly, they sing pretty songs, and they even drop nuts in front of cars !

In order to put bounds on WBE, we need to determine how much resolution is needed in order to emulate everything a bird can do. Is it good enough to model a bird at the cellular level? Or at the protein level? Or do you need an even finer resolution?

In order to model the navigational ability of a bird, you need a quantum mechanical description of the spin state of a pair of electrons. This is extremely high resolution.

A few caveats:

Not all parts of a bird require quantum mechanics to describe their macroscopic beha

...
Qualities that alignment mentors value in junior researchers
by Orpheus16 — 2023-02-14T23:27:40.747Z

This work was performed as a contractor for SERI MATS, but the views expressed are my own and do not necessarily reflect the views of the organization.

I recently conducted interviews with 7 current/former

SERI MATS

mentors. One of my goals was to understand the qualities that MATS mentors believe are most valuable for junior alignment researchers. I asked questions like:

Who were your most promising scholars? What made them stand out? What impressed you about them?
What are some important qualities or skills that you see missing from most MATS scholars?
What qualities were your scholars most missing? What are some things that you wish they had, or that would’ve made them more impactful?

Qualities that MATS mentors value

Endurance, happiness, & perseverance: Mentors noted that many scholars get discouraged if they’re not able to quickly come up with a promising research direction quickly, or if they explore 1-2 directions that don’t end up being promising. Mentors commented that their most promising scholars were ones who stay energetic/curious/relentless even when they don’t have a clear direction yet.
Hustle + resourcefulness: What do you do when you get stuck? Mentors said that many scholars don’t know what to do when they’re stuck, but their promising mentees were able to be resourceful. They would read related things, email people for help, find a relevant Discord server, browse Twitter, and contact other MATS scholars + AIS researchers for help.

Ability to ask for help + social agency: Many scholars waste a lot of time trying to figure things out on their own. Mentors noted that their most promising scholars were very agentic; they often found other scholars in the program who could help them or other Berkeley researchers who could help them. This also saved mentors time.
Ability to get to know other scholars + engage in peer mentorship: According to mentors, many scholars rarely interacted with others in the stream/program. Some of the best scholars were able to form productive/mutualistic relationships with other scholars.

Strong & concrete models of AI safety: Mentors noted that strong models are important but also hard to acquire. Some mentors emphasized that you often don’t get them until you have talked with people who have good models and you’ve spent a lot of time trying to solve problems. Others emphasized t

...