trentmkelly/LessWrong-43k huggingface dataset:
How To Get Into Independent Research On Alignment/Agency
by johnswentworth — 2021-11-19T00:00:21.600Z

I’m an independent researcher working on AI alignment and the theory of agency. I’m 29 years old, will make about $90k this year, and set my own research agenda. I deal with basically zero academic bullshit - my grant applications each take about one day’s attention to write (and decisions typically come back in ~1 month), and I publish the bulk of my work right here on

LessWrong

/

AF

. Best of all, I work on some

really

cool

technical

problems

which I expect are central to the future of humanity.

If your reaction to that is “Where can I sign up?”, then this post is for you.

Background Models

Independence

First things first: the “independent” part of “independent research” means self-employment, and everything that goes with it. It means the onus is on you to figure out what to do, how to provide value, what to prioritize, and what to aim for. In practice, it also usually means “independent” in a broader sense: you won’t have a standard template or agenda to follow. If you go down this path, assume that you will need to chart your own course - in particular, your own research agenda.

For the sort of person this post is aimed at, that will be a very big upside, not a downside.

Disclaimer: there are ways to get into alignment research which don’t involve quite so much figuring-it-all-out-on-your-own. Some people receive mentorship from existing researchers. Some people go work for alignment research organizations. Either of those paths can involve “independent research” in the sense that you are technically self-employed, but those paths aren’t “independent” in the broader sense of the word, and they’re not the main topic of this post.

Preparadigmicity

As a field, the study of alignment and agency is especially well-suited to independent research, because they center around

problems we don’t understand

. It’s not just that we don’t have the answers; we don’t even have the right frames for thinking about the problems. Agency is an area where

we are fundamentally confused

. AI alignment is largely

a problem which hasn’t happened yet, on technology which hasn’t been invented yet, which we nonetheless want to solve in advance

. Figuring out the right frames - the right paradigm - is itself a central part of the job.

The field needs people who are going to come up with new frames/approaches/models/paradigms/etc, because we’re pretty sure the current frames/approaches/models/parad...
Ngo and Yudkowsky on AI capability gains
by Eliezer Yudkowsky — 2021-11-18T22:19:05.913Z

This is the second post in a series of transcribed conversations about AGI forecasting and alignment. See the first post for prefaces and more information about the format.

Color key:

Chat by Richard Ngo and Eliezer Yudkowsky	Other chat	Inline comments

5. September 14 conversation

5.1. Recursive self-improvement, abstractions, and miracles

[Yudkowsky][11:00]

Good morning / good evening.

So it seems like the obvious thread to pull today is your sense that I'm wrong about recursive self-improvement and consequentialism in a related way?

[Ngo][11:04]

Right. And then another potential thread (probably of secondary importance) is the question of what you mean by utility functions, and digging more into the intuitions surrounding those.

But let me start by fleshing out this RSI/consequentialism claim.

I claim that your early writings about RSI focused too much on a very powerful abstraction, of recursively applied optimisation; and too little on the ways in which even powerful abstractions like this one become a bit... let's say messier, when they interact with the real world.

In particular, I think that Paul's arguments that there will be substantial progress in AI in the leadup to a RSI-driven takeoff are pretty strong ones.

(Just so we're on the same page: to what extent did those arguments end up shifting your credences?)

[Yudkowsky][11:09]

I don't remember being shifted by Paul on this at all. I sure shifted a lot over events like Alpha Zero and the entire deep learning revolution. What does Paul say that isn't encapsulated in that update - does he furthermore claim that we're going to get fully smarter-than-human in all regards AI which doesn't cognitively scale much further either through more compute or through RSI?

[Ngo][11:10]

Ah, I see. In that case, let's just focus on the update from the deep learning revolution.

[Yudkowsky][11:12][11:13]

I'll also remark that I see my foreseeable mistake there as having little to do with "abstractions becoming messier when they interact with the real world" - this truism tells you very little of itself, unless you can predict directional shifts in other variables just by contemplating the unknown messiness relative to the abstraction.

Rather, I'd see it as a neighboring error to what I've called the Law of Earlier Failure, whe

...
Covid 11/18: Paxlovid Remains Illegal
by Zvi — 2021-11-18T15:50:00.635Z

I had already written most of this week’s post, and also spun out a way-too-long way-too-deep dive on what happened with a retracted FLCCC paper that that I was planning to publish today but I can’t yet because the rabbit hole is still getting deeper, when Scott Alexander posted Ivermectin: Much More Than You Wanted To Know.

My reaction to that was a deep sense of gratitude, as I was worried I’d have to write something because someone had to and no one else would. Then Scott Alexander wrote it. So now I didn’t have to. I don’t agree with every decision he made, I definitely ‘have thoughts’ and I’ll probably end up writing about that, but it can wait for another week.

There’s a variety of other stuff going on as well, including a rise in cases that is unlikely to be going away any time soon, none of it terribly surprising. Paxlovid remains illegal and somehow this may not be easy, various mandates were imposed, children noticed it was nice not to wear masks, and those against vaccination have their latest crazy nonsense where they think they can ‘detox’ and undo the vaccination shot. What a time to be alive.

Executive Summary

Ivermectin: Much More Than You Wanted To Know.
Rise in cases is accelerating. Winter is coming.
Paxlovid remains illegal.

Let’s run the numbers.

The Numbers

Predictions

Prediction from last week (accidentally made on Monday 11/15, so it doesn’t count for prediction scoring purposes, but I didn’t get meaningful new data before then and it still attempts to represent my expectations at the time): 507k cases (+8%) and 7,690 deaths (-5%)

Results: 554k cases (+18%) and 7,427 deaths (-8%).

Prediction for next week: 654k cases (+18%) and 7,500 deaths (+1%).

Prediction for two weeks from now: We’ll see less deaths than normal due to the Thanksgiving holiday messing up reporting, and the counterfactual Zvi that didn’t put in this note likely would have forgotten to adjust for it.

Winter is coming. Boosters and child vaccinations will help a nonzero amount but they’re not sufficiently powerful to put a stop to this, and there’s very little appetite or tolerance for trying to shut things down again. There is no reason to think cases are going to stop rising any time soon, and having seen the acceleration this past week there’s a chance that the rise will be even larger in percentage terms over the next month. That means cases are likely to more than double befor...
Satisficers Tend To Seek Power: Instrumental Convergence Via Retargetability
by TurnTrout — 2021-11-18T01:54:33.589Z
Link: https://www.overleaf.com/read/kmjjqwdfhkvy

Summary: Why exactly should smart agents tend to usurp their creators? Previous results only apply to optimal agents tending to stay alive and preserve their future options. I extend the power-seeking theorems to apply to many kinds of policy-selection procedures, ranging from planning agents which choose plans with expected utility closest to a randomly generated number, to satisficers, to policies trained by some reinforcement learning algorithms. The key property is not agent optimality—as previously supposed—but is instead the retargetability of the policy-selection procedure. These results hint at which kinds of agent cognition and of agent-producing processes are dangerous by default.

I mean "retargetability" in a sense similar to Alex Flint's definition:

Retargetability. Is it possible, using only a microscopic perturbation to the system, to change the system such that it is still an optimizing system but with a different target configuration set?

A system containing a robot with the goal of moving a vase to a certain location can be modified by making just a small number of microscopic perturbations to key memory registers such that the robot holds the goal of moving the vase to a different location and the whole vase/robot system now exhibits a tendency to evolve towards a different target configuration.

In contrast, a system containing a ball rolling towards the bottom of a valley cannot generally be modified by any microscopic perturbation such that the ball will roll to a different target location.

(I don't think that "microscopic" is important for my purposes; the constraint is not physical size, but changes in a single parameter to the policy-selection procedure.)

I'm going to start from the naive view on power-seeking arguments requiring optimality (i.e. what I thought early this summer) and explain the importance of retargetable policy-selection functions. I'll illustrate this notion via satisficers, which randomly select a plan that exceeds some goodness threshold. Satisficers are retargetable, and so they have orbit-level instrumental convergence: for most variations of every utility function, satisficers incentivize power-seeking in the situations covered by my theorems.

Many procedures are retargetable, including

every

procedure which only depends on the expected utility of different plans. I think that alignment is hard ...
Forecasting: Zeroth and First Order
by jsteinhardt — 2021-11-18T01:30:19.127Z
Link: https://bounded-regret.ghost.io/forecasting-zeroth-and-first-order/

Part of lecture notes for the upcoming Stat157 class on Forecasting.

Let's say you are trying to predict how long it will take to finish your homework assignment. You think about all the problems and how long each will take. Problem 4 looks a little hard but you're sure if you try real hard you can solve it in 2 hours. Okay, so maybe 7 hours total? Plenty of time to finish if you start the afternoon before it's due!

Of course, the last 3 homework assignments all took between 12 and 16 hours...

You're getting ready to go to class in the morning. Your first class is at 9:10 am. When should you leave? Well, google maps says 8 minutes from your dorm, but if you bike real fast you can do it in 6. So it's probably fine to leave at 9:04...

...you say to yourself, and then get to class at 9:20 (just like the last 3 times). Because of course, you forgot to account for the time it takes to tie your shoes, find your keys, put on your backpack, unlock your bike, lock your bike, and walk to the classroom.

These are both instances of the planning fallacy, where humans regularly underestimate the time it takes to finish a task. In the real world, employees and teams that can avoid the planning fallacy have a huge advantage over those that don't. The planning fallacy is an example of a cognitive bias, which we'll talk about in more detail in a later lecture. For now, though, I want to talk about a simple but very effective technique for avoiding the planning fallacy, and more generally for making accurate predictions, called reference class forecasting.

Reference Class Forecasting (Zeroth-order Approximation)

The basic idea behind reference class forecasting is simple: rather than thinking in detail how about how long a task will take (or about any other prediction), just think of the last 3-5 instances where something similar happened, and give the average answer. So for homework, a good predictor of how long it takes to complete is the average of other problem sets for that class. For commute time, take the average commute time over the past week.

A more mathematical way to think of reference class forecasting is as a zeroth-order approximation: we are assuming that today will look like yesterday, i.e. that the world is roughly constant.

This works for more than just avoiding the planning fallacy. For instance, it's also useful when making a budget. In the policy world, carbon emi...
Experience on Methotrexate
by jefftk — 2021-11-17T22:40:06.965Z

Several months ago

I wrote about how my
wrists/hands were still not doing well, and about how I was going to
be trying Methotrexate. I have since gone off it, initially for a
covid shot, but I'm now considering staying off it.

After meeting again with my rheumatologist on 2021-06-28 I started on
10mg/wk (four tablets) taken all at once, plus 1mg/d folic acid (one
tablet). My first dose was 2021-06-30. This was just over two weeks
after Nora was born and my
third day back at work, so it's not a good example of "change only one
thing at a time". Initially I noticed a small increase in mouth
sores, but not anything else. It's possible that my wrists and hands
were feeling somewhat better, but it's already very variable, with
good and bad days.

I met with the rheumatologist again on 2021-08-04, and we increased my
dose to 15mg/wk (six tablets). I still didn't notice much of a
change.

In October I noticed that my food preferences had changed. It happened
very slowly, over many weeks. I became less interested in strongly
flavored savory foods (normally something I enjoy a lot), and would
have more days when I forgot to eat lunch (usually only happens
rarely, when I'm super interested in what I'm working on). It became
much more common that I would notice that it was early afternoon, feel
like one ought to should eat lunch, go down to the kitchen, and end up
having a bowl of cereal. I often just wasn't hungry, though I would
still eat out of feeling like that's what I should do, and I stayed in
my normal weight range. If I did actually eat
some savory food, I often enjoyed it once I was doing it, but before I
started the smell and thought were pretty negative. I thought it was
likely that this was related to the medication, but didn't know for
sure.

We met again on 2021-10-29, and we decided that since I was
immunocompromised I should get a third covid shot. This meant going
off the medication, so my immune system would be able to react to the
vaccination, and my last dose was Wednesday 2021-10-27. The plan was
that when I started again, I would go up to 20mg/wk (8 tablets) still
once a week but ~10hr apart (evening and following morning).

I didn't feel any change to my wrists/hands as I went off the
medication, but on 2021-11-06 (10d) I noticed that I was hungry
again. The next day, (11d) it was very clear that I wanted savory food
agai...
Applications for AI Safety Camp 2022 Now Open!
by adamShimi — 2021-11-17T21:42:31.672Z

If you’ve read about alignment research and you want to start contributing, the new iteration of the AI Safety Camp is a great opportunity!

It’s a virtual camp from January to May 2022, where you collaborate with other applicants to work (1h / normal workday, 7h / weekend sprint day) on open problems proposed and supervised by mentors like John Wentworth, Beth Barnes, Stuart Armstrong, Daniel Kokotajlo… Around this core of research, the camp also includes talks and discussions about fundamental ideas in the field, how alignment research works, and how and where to get a job/funding.

All in all, the AI Safety Camp is a great opportunity if:

You have read enough about alignment that you’re convinced of the importance of the problem
You want to do alignment research (whether conceptual or applied), or to collaborate with alignment researchers (doing policy for example)
You don’t feel yet like you have enough research taste and grasp of the field to choose your research problems yourself yet

Note that you don’t need advanced maths skills to participate in the camp, as some of the projects don’t require any specific skillset or very unusual ones (evolutionary genetics, history...). If you care about alignment and are in this situation, I encourage you to apply for a project without required skillsets and learn what you need as you go along.

All the details on how to apply are available on

the website

(including the list of open problems).
Did EcoHealth create SARS-CoV-2?
by jamal — 2021-11-17T20:42:36.745Z

Science writer Nicholas Wade does a detailed write up here: https://thebulletin.org/2021/05/the-origin-of-covid-did-people-or-nature-open-pandoras-box-at-wuhan/

I'd like to hear the LessWrong evaluation of these claims.
On Raising Awareness
by Tomás B. — 2021-11-17T17:12:36.843Z

Those who can, do. Those who can't, teach. And those who can't teach, "raise awareness".

I have been spending some time thinking about ideas for "raising awareness" of the AI alignment problem, focusing on those that sound crazy enough that perhaps smarter people have glossed over them, or avoided thinking about what literally implementing them might look like. I have made some comments about these ideas here and on Discord, and the purpose of this post is to centralize them, and hopefully get some more crazy ideas in the comments.

Become a Human Catalyst

A well-known alignment researcher and a very famous engineer had a conversation about alignment recently. I will not get into specifics, but the obvious inference is obvious.

My impression is the conversation did not go well (which may be putting it mildly), but what interests me more is I am acquainted with the person who was responsible for this meeting occurring. He knew someone who knew them both, and suggested that it might be interesting if they had a conversation.

This struck me as an example of someone being a "human catalyst", which I think is a really high-leverage social role.

I have been trying to write a post about this role for awhile now. But it is very illegible, and it is hard to find historical examples - Oppenheimer, Wheller, Harry Nyquist and various enlightenment saloniers might fit. Though I am not involved in the IRL rationalist community, I get the impression that Alicorn and Anna Salamon may also be human catalysts.

If you are a person with interesting contacts, it is always worth thinking hard about whom it would be productive to introduce to whom.

And if you are a human catalyst, please write a post about it! I think this is a social role that should be more legible as something one can aspire to be.

Message Targeting Using Low-Status Signals of Cognitive Ability

Psychometrics

is both correct and incredibly unpopular - so this means there is possibly an arbitrage opportunity for anyone willing to act on it.

Very high IQ people are rare, and sometimes have hobbies that are considered low-status in the general population. Searching for low-status signals that are predictive of cognitive ability looks to be an efficient means of message targeting.

It is interesting to note that Demis Hassabis’s prodigious ability was obvious...
Sasha Chapin on bad social norms in rationality/EA
by Kaj_Sotala — 2021-11-17T09:43:35.177Z
Link: https://sashachapin.substack.com/p/your-intelligent-conscientious-in

So, I’ve noticed that a significant number of my friends in the Rationalist and Effective Altruist communities seem to stumble into pits of despair, generally when they structure their lives too rigidly around the in-group’s principles. Like, the Rationalists become miserable by trying to govern their entire lives through nothing but rationality, and the EAs feel bad by holding themselves to an impossible standard of ethics. [...]

I’ve tried to figure out why this happens, and I’ve tried to write about it several times, batting around complex notions that, as I examine them, reveal themselves to be fake models that make me sound smart but don’t explain anything.

But today I realized that it’s generally much simpler than I thought previously. Most of it is just toxic social norms. These groups develop toxic social norms. In the Rationalist community, one toxic norm is something like, “you must reject beliefs that you can’t justify, sentiments that don’t seem rational, and woo things.” In the EA community, one toxic norm is something like, “don’t ever indulge in Epicurean style, and never, ever stop thinking about your impact on the world.”

Generally, toxic social norms don’t develop intentionally, nobody wants them to happen, they’re not written down, and nobody enforces them explicitly. (The intentional development of toxic social norms is otherwise known as founding a cult.) What happens is that there are positive social norms, like, “talking about epistemics and being curious about beliefs is cool,” or “try to be intentional about the positive impact you can have on the world.” These norms are great! But then, the group acts like a group, which is to say, people confer status depending on level of apparent adherence to values. This leads insecure people who completely depend on the group to over-identify with the set of values, to the extent that even slightly contrary actions become forbidden. Not forbidden in the like “we’ll arrest you” way, but in the like “everyone in the room immediately looks at you like you’re being rude if you talk about spirituality” way.

And then the second, more sinister stage occurs—the point at which these toxic norms are internalized such that they apply to you when you’re in a room alone. As Wittgenstein noted, it’s hard to tell where aesthetics end and ethics begin; it can start to feel unethical, like, dirty, to perform

...
What are the mutual benefits of AGI-human collaboration that would otherwise be unobtainable?
by M. Y. Zuo — 2021-11-17T03:09:40.733Z

By mutual benefits I am thinking of what activities, endeavours, and designs may both future AGI(s) and future human groups, or society at large, invest in that would yield net returns.

The most commonly mentioned benefits are the ones where AGI would be enhancing existing human activities in statistical analysis, stock markets, forecasting, optimization, etc… that could be carried on regardless of AGI involvement or not. Where AGI would be a supplement.

However what can collaboration exclusively bring about?

It would be nice to hear some thoughts on what novel developments may be possible. (Also by what ways AGI(s) could be benefiting, other than more compute, since there likely will be synergistic effects.)