trentmkelly/LessWrong-43k huggingface dataset:
Refine: An Incubator for Conceptual Alignment Research Bets
by adamShimi — 2022-04-15T08:57:35.502Z

I’m opening an incubator called Refine for

conceptual alignment

research in London, which will be hosted by

Conjecture

. The program is a three-month fully-paid fellowship for helping aspiring independent researchers find, formulate, and get funding for new conceptual alignment research bets, ideas that are promising enough to try out for a few months to see if they have more potential.

If this sounds like something you’d be interested in, you can apply here!

Why?

I see a gaping hole in the alignment training ecosystem: there are no programs dedicated specifically to creating new independent conceptual researchers and helping them build original research agendas.

The programs that do exist (AI Safety Camp, SERI MATS) tend to focus on an apprenticeship (or “accelerated PhD”) model in which participants work under researchers on already-established research directions. And while there are avenues for independent alignment researchers to

get started on their own

, it is fraught with many risks, slowing down progress considerably.

So I feel the need for a program geared specifically towards conceptual alignment researchers that are interested in doing their own research and making their own research bets.

Who?

This program is for self-motivated and curious people who want to become independent conceptual alignment researchers and expand the portfolio of alignment bets and research ideas available.

When I look at great conceptual researchers like John Wentworth, Paul Christiano, Evan Hubinger, Steve Byrnes, Vanessa Kosoy, and others, as well as at the good (famous and not) researchers I know from my PhD, they all have the same thing in common: they ask a question and keep looking for the answer. They tolerate confusion, not in the sense that they accept it, but in that they are able to work with it and not hide away behind premature formalization. They don’t give up on the problem; they search for different angles and approaches until it yields. Paul Graham calls this being

relentlessly resourceful

.

(

Relentlessly Resourceful

, Paul Graham, 2009)

I was writing a talk for investors, and I had to explain what to look for in founders. What would someone who was the opposite of hapless be like? They'd be relentlessly resourceful. Not merely relentless. That's not enough to make things go your way except in a few mostly uninteresti

...
Pivotal acts from Math AIs
by azsantosk — 2022-04-15T00:25:21.286Z

Eliezer Yudkowsky on Math AIs

Here are some interesting quotes from the alignment debate with Richard Ngo.

If it were possible to perform some pivotal act that saved the world with an AI that just made progress on proving mathematical theorems, without, eg, needing to explain those theorems to humans, I'd be extremely interested in that as a potential pivotal act. We wouldn't be out of the woods, and I wouldn't actually know how to build an AI like that without killing everybody, but it would immediately trump everything else as the obvious line of research to pursue.

(...)

If there were one theorem only mildly far out of human reach, like proving the ABC Conjecture (if you think it hasn't already been proven), and providing a machine-readable proof of this theorem would immediately save the world - say, aliens will give us an aligned superintelligence, as soon as we provide them with this machine-readable proof - then there would exist a plausible though not certain road to saving the world, which would be to try to build a shallow mind that proved the ABC Conjecture by memorizing tons of relatively shallow patterns for mathematical proofs learned through self-play; without that system ever abstracting math as deeply as humans do, but the sheer width of memory and sheer depth of search sufficing to do the job. I am not sure, to be clear, that this would work. But my model of intelligence does not rule it out.

Here is a caveat and further explanation:

If you knew about the things that humans are using to reuse their reasoning about chipped handaxes and other humans, to prove math theorems, you would see it as more plausible that proving math theorems would generalize to chipping handaxes and manipulating humans.

(...)

Human math is very much about goals. People want to prove subtheorems on the way to proving theorems. We might be able to make a different kind of mathematician that works more like GPT-3 in the dangerously inscrutable parts that are all noninspectable vectors of floating-point numbers, but even there you'd need some Alpha-Zero-like outer framework to supply the direction of search.

That outer framework might be able to be powerful enough without being reflective, though. So it would plausibly be much easier to build a mathematician that was capable of superhuman formal theorem-proving but not agentic. The reality of the world might tell us "lolnope" bu

...
My least favorite thing
by sudo — 2022-04-14T22:33:20.408Z

Epistemic status: Anger. Not edited.

TL;DR The hamster wheel is bad for you. Rationalists often see participation in the hamster wheel as instrumentally good. I don't think that is true.

Meet Alice. She has had the opportunity to learn many skills in her school years. Alice is a bright high school student with a mediocre GPA and a very high SAT score. She doesn't particularly enjoy school, and has no real interest in engaging in the notoriously soul-crushing college admissions treadmill.

Meet Bob. Bob understands that AGI is an imminent existential threat. Bob thinks AI alignment is not only urgent and pressing but also tractable. Bob is a second-year student at Ivy League U studying computer science.

Meet Charlie. Charlie is an L4 engineer at Google. He works on applied machine learning for the Maps team. He is very good at what he does.

Each of our characters has approached you for advice. Their terminal goals might be murky, but they all empathize deeply with the AI alignment problem. They'd like to do their part in decreasing X-risk.

You give Alice the following advice:

It's statistically unlikely that you're the sort of genius who'd be highly productive without at least undergraduate training. At a better college, you will not only receive better training and have better peers; you will also have access to opportunities and signalling advantages that will make you much more useful.

I understand your desire to change the world, and it's a wonderful thing. If you'd just endure the boredom of school for a few more years, you'll have much more impact.

Right now, MIRI wouldn't even hire you. I mean, look at the credentials most AI researchers have!

Statistically, you are not Eliezer.

You give Bob the following advice:

Graduating is a very good signal. A IvyLeagueU degree carries a lot of signalling value! Have you gotten an internship yet? It's great that you are looking into alignment work, but it's also important that you take care of yourself.

It's only your second year. If the college environment does not seem optimal to you, you can certainly change that. Do you want study tips?

Listen to me. Do not drop out. All those stories you hear about billionaires who dropped out of college might be somewhat relevant if you actually wanted to be a billionaire. If you're optimizing for social impact, you do not do capricious things like that.

Rememb

...
Early 2022 Paper Round-up
by jsteinhardt — 2022-04-14T20:50:29.365Z

My students and collaborators have been doing some particularly awesome work over the past several months, and to highlight that I wanted to summarize their papers here, and explain why I’m excited about them. There’s six papers in three categories.

Human-Aligned AI

The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models (w/ Alex Pan, Kush Bhatia)

Summarizing Differences between Text Distributions with Natural Language (w/ Ruiqi Zhong, Charlie Snell, Dan Klein)

Robustness

Predicting Out-of-Distribution Error with the Projection Norm (w/ Yaodong Yu, Zitong Yang, Alex Wei, Yi Ma)

Capturing Failures of Large Language Models via Human Cognitive Biases (w/ Erik Jones)

PixMix: Dreamlike Pictures Comprehensively Improve Safety Measures (w/ Dan Hendrycks, Andy Zou, Mantas Mazeika, Leonard Tang, Bo Li, Dawn Song)

Science of ML

More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalize (w/ Alex Wei, Wei Hu)

I'll go over the first category (human-aligned AI) today, and save the other two for next week. As always, we love getting feedback on our work, so let us know what you think!

Human-Aligned AI

While AI alignment is a somewhat subtle and complex problem, two basic issues are that (1) ML systems often hack their reward functions, and (2) human supervision doesn’t necessarily solve this, because humans can’t easily understand the consequences of intervening on complex systems. Alex and Ruiqi’s papers help address each of these questions in turn.

Mapping and Mitigating Misaligned Models. What Alex Pan and Kush Bhatia did was construct a wide variety of reinforcement learning environments where reward hacking is possible, and measured the extent to which it occurred. They do this by defining both a “proxy” and “true” reward, and look at what happens to the true reward as we optimize the proxy reward. Two key insights are that:

Optimizing the proxy reward for longer, or with larger policy models, often leads to lower true reward.

When this happens, it sometimes occurs suddenly, via a phase transition (in both the quantitative reward and the qualitative behavior).

A simple illustration of both is a traffic simulator, where the RL agent is trying to shape traffic flow to be more efficient. Small neural net models help cars merge efficiently onto the highway, but large models instead block cars from merging ...
Can someone explain to me why MIRI is so pessimistic of our chances of survival?
by  — 2022-04-14T20:28:31.423Z

To be clear, when I reference MIRI as being pessimistic, I’m mostly referring to the broad caricature of them that exists in my mind. I’m assuming that their collective arguments can be broken down into:

1.) There is no plan or workable strategy which helps us survive, and that this state of ignorance will persist until the world ends.

2.) The time we have left is insufficient to save the world, judging by the rate of progress in AI capabilities and lack of progress in AI alignment.

3.) Resources allocated to boosting AI capabilities far outweigh the resources allocated to solving alignment.

4.) The alignment problem is incredibly difficult. (Why this is true is somewhat unclear to me, and I would appreciate some help elaboration in the comments).

5.) Any prosaic alignment strategies that could possibly exist are necessarily doomed. (Why this is true is also very unclear to me, and I would greatly appreciate some elaboration in the comments).

6.) We have to get alignment right on the first try.

Put all of that together and I’ll be the first to admit that the situation doesn’t look good at all. I’d just like to know, is there some more implicit, even more damning piece of evidence which lends toward hopelessness which I’m missing here?
Redwood Research is hiring for several roles (Operations and Technical)
by Jessica W — 2022-04-14T16:57:20.915Z

Redwood Research

is a research organization with the mission of aligning superhuman AI. In addition to our primary work of applied AI alignment research, we host and operate Constellation, a shared office space we created to foster knowledge-sharing and collaboration between longtermist organizations. People from the Alignment Research Center, MIRI, and Open Philanthropy often work from the space.

Redwood and Constellation are expanding. Redwood Research’s staff includes Nate Thomas (CEO), Buck Shlegeris (CTO), and Bill Zito (COO), and a board of Nate Thomas, Holden Karnofsky, and Paul Christiano.

We’re running a hiring round for a number of Operations roles, across a range of experience levels. The roles we are most excited to hire are:

Constellation Operations Manager

Constellation: Head of Community

Constellation IT Analyst

Recruiting & MLAB Lead

We also have a number of other roles, including technical roles, which you can also apply to

here

.

Our office is in downtown Berkeley, thirteen seconds from the nearest BART stop. We offer catered lunches and dinners, and cover PPO health, dental, and vision insurance. We will consider international applicants who need visas.

Below is a picture taken from the office -- we have beautiful views in all directions!

If you’re interested in any of the roles listed above, fill out this

application form

. If you would like to express interest in this role but don’t want to be considered for the current hiring round there is the option to specify this on the form. Any questions, please email ops-recruiting@rdwrs.com, if you’re on the fence about applying, please email us. Deadline for applications is 13th May 2022.
Covid 4/14/22: China Stays the Course
by Zvi — 2022-04-14T16:10:00.742Z

Will China abandon its zero Covid policy? For now it seems they are determined to stay the course and hope for the best. They continue to claim there have been zero Covid deaths. There are increasing signs the people are not happy about the situation, especially in Shanghai. At some point something has to give, but after one types similar sentences enough times perhaps things can be kept going for quite a while before that happens. Or it could happen on Tuesday.

Outside of China, the news has been extremely quiet on the Covid front other than the numbers themselves. Cases are up substantially, and deaths are up as well, neither of which fits my models of what should be happening here. I am unsure what to make of it all but I do not expect it to be fully sustained in the next week or two.

Executive Summary

China continues its Zero Covid policy and lockdowns.
CDC extends travel mask mandate.
Cases rose more than expected.

Let’s run the numbers.

The Numbers

Predictions

Prediction from last week: 190k cases (+6%) and 2,950 deaths (-23%)

Results: 222,300 cases (+25%) and 4,304 deaths (+12%)

Prediction for next week: 250,00 cases (+13%) and 3,900 deaths (-10%).

I talk a bit more below but both these movements seem very out of step with previous weeks and I doubt they will be sustained.

Deaths

The death number in particular does not make sense to me. There are still a lot of with-Covid deaths rotating out at the top, and three-week-old cases are roughly static. There is no reason to think deaths should be rising here, yet they did. Given the large unexpected rise in cases alongside it, I am going to expect a modest fall again, and chalk this up in part to the previous week being artificially low given there was a very large drop there. We are still far below the rate we were at two weeks ago.

Cases

Cases jumped substantially more than expected, which is not a good sign, but I still remain not all that worried. My expectations for how high case counts will get has gone up, but I still expect things to stabilize before too long. There is not much more upward pressure to come from increased share of BA.2, so it is about how high cases need to rise and for how long before adjustments and cumulative infections send us going back down again.

Physical World Modeling

Interesting hypothesis from Gottlieb, that those ...
Only Asking Real Questions
by jefftk — 2022-04-14T15:50:02.970Z

It's easy to accidentally ask children questions that aren't real
questions. Perhaps the right answer is so obvious I don't realize in
time that they might decide otherwise: "do you want to wait to eat
your messy candy until we're out of the car?" Or I don't figure out
that there was only one answer that would work until after I finish
asking the question: "do you want to do your bath now, or before
dinner? ... Oh, wait, we have guests coming over in the afternoon and
we can't do the bath then, now's the only time that works."

I find that these interactions tend to go poorly, much worse than if
I'd never asked a question. Something like, they thought they had a
choice, try to make a choice, and now they're feeling me take that
away.

I've tried pretty hard to get myself to stop doing this, and one
strategy that has worked for me was deciding to stand by my mistakes.
If I accidentally offer them a choice when I shouldn't have, I won't
withdraw it if they choose (to my mind) badly. This isn't a
completely strict rule: I can imagine foolish-enough things that I
could accidentally ask that it wouldn't make sense to stand by. On
the other hand, though, I've found I can recover from most things if
I'm willing to put some extra effort. Going back to the messy candy
example above, and imagining some further discussion:

Parent: Do you want to wait to eat your messy candy until we're
out of the car?

Child: I want to eat it now!

Parent: That's gonna make a mess everywhere. I don't think
that's a good idea.

Child: But I want the candy now!

Parent: If it makes a mess, will you clean it up?

Child: No.

Parent: How will it get cleaned up?

Child: You can do it.

Parent: I don't want to clean it up.

Child: I don't want to either.

Parent: If you make a mess, it's your job to clean it up.

Child: Ok, I'll do it.

Parent: Will you put up a $1 bond? Where I get to
keep the dollar if you don't clean up?

Child: Ok. [eats messy candy]

[time passes, arrival, getting out of the car]

Parent: It looks like there's a mess on the seat.

Child: I don't want to clean it up.

Parent: If you don't, I will keep your dollar.

Child: [cleans up, slowly, with parental help depending on age]

This definitely requires a lot more time and back-and-forth than just
saying "sorry, I shouldn't have offered that, we shouldn't eat messy
candy in ...
Lies Told To Children
by Eliezer Yudkowsky — 2022-04-14T11:25:10.282Z

Growing up, as a kid, I was always told that every sapient life is precious, everything that thinks and knows itself -

Yes, this is a tale about lies-told-to-children. You'll probably figure it out yourself before too long. For now, just listen.

Where was I? Right. As children, we were always told that every sapient life is precious. It was told to us by the teachers, and shown to us in children's television - though I saw less children's television than most children in our age cohort - children's TV was censored where I grew up, though, of course, I didn't find that out until much later -

I see you're starting to guess under what sort of circumstances I grew up. Go ahead, write down the prediction if you want. Maybe you already see where this entire thing is headed. But you asked me for a story about the lies I was told as a child, and that's what you're getting. It's not my fault, if a lot of stories like that are predictable; people who lie to children have other things to optimize for than unpredictability.

So where was I? Right. I grew up in a remote village of about three thousand people, the sort that's more hills than houses. Charming travel-pathways that cut through forests. Not everyone knows everyone, but you sure know somebody who knows anybody.

Children's television in my region was censored, though of course they didn't tell us that as children. But the children's television that we saw had aliens and monsters and creatures of fantasy, with four legs or fourteen legs, three faces or no face at all, and all of them were treated by the television show as having lives that meant something. Sometimes in the children's show there were alien monsters who only thought their own kind of life was valuable, and then maybe you couldn't trade with them as friends. Maybe they'd already lied to you once and you couldn't trust them enough to bargain with them, maybe you couldn't talk to them at all. But their lives still had meaning to the story's human protagonists, even some aliens whose lives had no meaning to themselves. You didn't cause them pain if there was any way to avoid it; you didn't kill them unless their biology was sufficiently similar to human that you were confident in your ability to cryopreserve them afterwards.

The shows never spelle...
Unchangeable Code possible ?
by AntonTimmer — 2022-04-14T11:17:52.559Z

I was thinking about AI going FOOM and one main argument I found is that it would just rewrite its own code.

Is there any research going into the direction of code that is not changeable ? Could their be code that is unchangeable ?

Would their be some obvious disadvantages other than that we could also not fix a misaligned AI instantly ?

Would something like this stop an AI going FOOM ?
Emacs Recentering With Context
by jefftk — 2022-04-14T01:20:02.366Z

In emacs, when you press Ctrl-L, by default it runs

recenter-top-bottom.
This scrolls the document ("buffer") so the line the cursor is
currently on is centered in the screen. If you press it again, it
scrolls so the cursor is at the very top of the screen, and again puts
the cursor at the very bottom. I've always been mildly irritated by
this: the middle of the screen is lower than I want for the initial
state; I generally want more context below the cursor than above.
Additionally, scrolling to the top or bottom is nearly useless because
I almost always want at least a little bit of context on both sides of
the cursor.

Everything in emacs can be configured, but sometimes it's quite
a pain, so I kept putting off looking into this. With Nora napping
and the older two playing with a neighbor friend, however, I finally
looked into this, and it turns out to be very easy. There's a
recenter-positions variable that defaults to
'(middle top bottom), but you can set it to anything you
want. The default is equivalent to '(0.0, 0.5, 1.0), and
I'm now using:

(setq recenter-positions '(0.25 0.5 0.9))

This means that the first Ctrl-L puts my cursor 25% of the way down
the page, which is where I'm most likely to want it. If I find I want
more context above I can press it again, to put it at 50%, and again
to put it at 90%.

These are good for my usage, but I run tall terminals and if I ran shorter
ones I might want different values.

Comment via: facebook