trentmkelly/LessWrong-43k huggingface dataset:
2020 Review: The Discussion Phase
by Vaniver — 2021-12-15T01:12:44.746Z

LessWrong is currently doing a major review of 2020 — looking back at old posts and considering which of them have stood the test of time.

There are three phases:

Preliminary Voting (completed)
Discussion (ends Jan 11th)
Final Voting (ends January 25th)

We're now in the Discussion Phase, and there are 402 posts that got at least one upvote in the Preliminary Voting Phase. The full list is here. Now is the time to dig into those posts, and for each one ask questions like “What did it add to the conversation?”, “Was it epistemically sound?” and “How do I know these things?”.

The Less Wrong team will award at least $2000 in prizes to the reviews that are most helpful. (We might or might not make a Best of 2020 book.)

If you're an author and for whatever reason don't want one or more of your posts in the Review or considered for the book, contact any member of the team (e.g. email Ray Arnold at raemon777@gmail.com ).

Why Review?

The overall goal of the annual review is to revisit old posts with additional context and distance. Did an idea that seemed promising pan out? Was something that didn't seem that exciting come up over and over again? Was something that you strong-upvoted the day it came out as good as you thought, and the sort of thing everyone should be sure not to miss?

But all that will show up in the votes. Why write text about the posts instead of just voting? Five reasons:

Reward authors. Writing a review puts a name to the number, and helps motivate authors through reading the impact they had on people. (Note that nominations used to fill this role.)
Share experiences. You might have changed how you thought or acted because of a post, but did other people? Talking about our reactions helps ground our impressions of how other people reacted.
Summarize and analyze. Coming back to a post a year later, you might have a better sense of how it fits into the bigger picture, and recording that in a comment will make the post more intelligible to future readers. It's also quite useful for authors to see what people took away from their posts; a summary that seems like it missed the point might prompt another post that better approaches the target.
Test and verify. As something moves from "something we talked about for a week" to "something that people will remember about 2020", it helps to do some digging and share what you find. Did the speculation check out? Did someo

...
The Natural Abstraction Hypothesis: Implications and Evidence
by CallumMcDougall — 2021-12-14T23:14:24.825Z

This post was written under Evan Hubinger’s direct guidance and mentorship, as a part of the

Stanford Existential Risks Institute ML Alignment Theory Scholars (MATS) program

.

Additional thanks to Jamie Bernardi, Oly Sourbut and Shawn Hu, for their thoughts and feedback on this post.

TL;DR

The Natural Abstraction Hypothesis, proposed by John Wentworth, states that there exist abstractions (relatively low-dimensional summaries which capture information relevant for prediction) which are "natural" in the sense that we should expect a wide variety of cognitive systems to converge on using them.

If this hypothesis (which I will refer to as NAH) is true in a very strong sense, then it might be the case that we get "alignment by default", where an unsupervised learner finds a simple embedding of human values, and a supervised learner can be trained with access to this learned model. However, even if the NAH is true in a weaker sense, this has important implications, for areas such as:

Deceptive alignment - since a natural abstraction will be easier for an objective to point at
Instrumental convergence - since as I will discuss below, the formalisation for an abstraction resembles the idea of instrumental goals
Interpretability - since the extent to which human-interpretable abstractions are also "natural" has direct implications for how much of a trade-off incentivising interpretability will be

Several different fields can provide evidence for & against different forms of the NAH, for instance:

Studying the abstractions formed by different cultures can help us understand whether humans generally seem to converge on the same abstractions
Current empirical study of neural networks, primarily in the context of interpretability research, can help us understand whether they are forming human-interpretable abstractions

Since the NAH is primarily an empirical claim about the kinds of abstractions we expect to see forming in the real world, I will conclude by discussing some promising ways the hypothesis could be tested. This will include John Wentworth's own research agenda (which involves a feedback loop of training cognitive models on systems that contain abstractions, and examining then proving theorems about which abstractions are learned by which agents in which environments), as well as two other potentially promising directions: further interpretability work, and neuros...
Robin Hanson's "Humans are Early"
by Raemon — 2021-12-14T22:07:00.000Z
Link: https://www.overcomingbias.com/2021/02/humans-are-early.html

This post was nominated for the 2021 Review by Algon. The post is short, so I'm including the full text:

Imagine that advanced life like us is terribly rare in the universe. So damn rare that if we had not shown up, then our region of the universe would almost surely have forever remained dead, for eons and eons. In this case, we should still be able to predict when we humans showed up, which happens to be now at 13.8 billion years after the universe began. Because we showed up on a planet near a star, and we know the rate at which our universe has and will make stars, how long those stars will last, and which stars where lived far enough away from frequent sterilizing explosions to have at least a chance at birthing advanced life.

However, this chart (taken from our new paper) calculates the percentile rank of our current date within this larger distribution. And it finds that we are surprisingly early, unless you assume both that there are very few hard steps in the evolution of advanced life (the “power n”), and also that the cutoff in lifetime above which planets simply cannot birth advanced life is very low. While most stars have much longer lives, none of those have any chance whatsoever to birth advanced life. (The x-axis shown extends from Earth’s lifetime up to the max known star lifetime.)

In the paper (in figures 2,17), we also show how this percentile varies with three other parameters: the timescale on which star formation decays, the peak date for habitable star formation, and a “mass favoring power” which says bu how much more are larger mass stars favored in habitability. We find that these parameters mostly make only modest differences; the key puzzle of humans earliness remains.

Yes, whether a planet gives rise to advanced life might depend on a great many other parameters not included in our calculations. But as we are only trying to estimate the date of arrival, not many other details, we only need to include factors that correlate greatly with arrival date.

Why have others not reported the puzzle previously? Because they neglected to include the key hard-steps power law effect in how chances vary with time. This effect is not at all controversial, though it often seems counter-intuitive to those who have not worked through its derivation (and who are unwilling to accept a well-established literature they have not worked out for themselve

...
Ngo's view on alignment difficulty
by Richard_Ngo — 2021-12-14T21:34:50.593Z

This post features a write-up by Richard Ngo on his views, with inline comments.

Color key:

Chat	Google Doc content	Inline comments

13. Follow-ups to the Ngo/Yudkowsky conversation

13.1. Alignment difficulty debate: Richard Ngo's case

[Ngo][9:31] (Sep. 25)

As promised, here's a write-up of some thoughts from my end. In particular, since I've spent a lot of the debate poking Eliezer about his views, I've tried here to put forward more positive beliefs of my own in this doc (along with some more specific claims): [GDocs link]

[Soares: ✨]

[Ngo] (Sep. 25 Google Doc)

We take as a starting observation that a number of “grand challenges” in AI have been solved by AIs that are very far from the level of generality which people expected would be needed. Chess, once considered to be the pinnacle of human reasoning, was solved by an algorithm that’s essentially useless for real-world tasks. Go required more flexible learning algorithms, but policies which beat human performance are still nowhere near generalising to anything else; the same for StarCraft, DOTA, and the protein folding problem. Now it seems very plausible that AIs will even be able to pass (many versions of) the Turing Test while still being a long way from AGI.

[Yudkowsky][11:26] (Sep. 25 comment)

Now it seems very plausible that AIs will even be able to pass (many versions of) the Turing Test while still being a long way from AGI.

I remark: Restricted versions of the Turing Test. Unrestricted passing of the Turing Test happens after the world ends. Consider how smart you'd have to be to pose as an AGI to an AGI; you'd need all the cognitive powers of an AGI as well as all of your human powers.

[Ngo][11:24] (Sep. 29 comment)

Perhaps we can quantify the Turing test by asking something like:

What percentile of competence is the judge?
What percentile of competence are the humans who the AI is meant to pass as?
How much effort does the judge put in (measured in, say, hours of strategic preparation)?

Does this framing seem reasonable to you? And if so, what are the highest numbers for each of these metrics that correspond to a Turing test which an AI could plausibly pass before the world ends?

[Ngo] (Sep. 25 Google Doc)

I expect this trend to continue until after we have AIs

...
A proposed system for ideas jumpstart
by Valentin2026 — 2021-12-14T21:01:00.506Z

What do you need to come up with an idea for an interdisciplinary project? Maybe deep expertise in all the relevant disciplines? No - for just the initial idea, you shouldn’t need this level of knowledge. For example, if you have a very general knowledge of what machine learning or genetic engineering can do, you can have an idea of how combining both approaches can be applied to the problem you are interested in. Maybe your idea will be wrong, and actually not work at all, and maybe it will produce results that are not worth the effort. But you can’t know that until you either gain the necessary expertise, or discuss your idea with the specialists in each field.

If a project requires a lot of different skills (and big funding), you might just give up. Well, yeah, you have an idea that sounds cool to you. But if you want to see it implemented, you need to talk with many different specialists. You need to be really passionate about it to overcome the social awkwardness of discussing your potentially crazy idea with a person whom you barely know (or don’t know at all). Finally, even if you succeed at this step, and are assured that there are no obvious flaws and your idea actually may work, you need somehow to persuade people to work on it (or get money somewhere to fund it). Of course, you must do all of this with no guarantee of success. Start-up accelerators, funding applications, etc. can significantly help, but still, you will need to do lots and lots of work. If you see all this in front of you when you think about implementing your idea, and you are not possessed by it, most likely you will just mention it in occasional conversations with friends without going any further.

How many wonderful projects are never born because of this barrier between idea and implementation? I don’t know. Imagine the world where, as soon as a seemingly good idea comes to your mind, you just put it to the black box (let’s call it Ideas Implementer, or shortly I.I.), and, if the idea is indeed promising, it gets implemented. Would such a world have more projects with a positive impact? The naïve answer seems yes. How many more – I don’t know, and I even don’t know how to know it. Can we create I.I.? In some sense – yes.

The mechanism besides I.I. can be quite straightforward. You come up with the idea. You submit it to a website and tag areas of expertise required to evalu...
Should we rely on the speed prior for safety?
by Marc Carauleanu — 2021-12-14T20:45:02.478Z

This post was written under Evan Hubinger’s direct guidance and mentorship, as a part of the

Stanford Existential Risks Institute ML Alignment Theory Scholars (MATS) program

.

TL;DR: There is some hope that by penalizing time complexity we could strongly select for non-deceptive models as they require time to figure out if they are out of training. Evan Hubinger argued in his post

‘Are minimal circuits deceptive?’

that we cannot fully rely on the speed prior to ensure non-deceptiveness by attempting to show that even minimal circuits, which we can intuitively understand as “fastest programs”, can be deceptive. This is because of the risk of spontaneous meta-learning which doesn’t inherit the safety guarantees that we would hope to get from the speed prior. This argument could be wrong if meta-learning is never the fastest way to solve a problem or if spontaneous meta-learning is constrained by the speed prior.

Introduction

A possible approach to tackling inner alignment failures is to shape the model’s inductive biases to make them unlikely. The two most natural inductive biases are the simplicity prior and the time complexity/speed prior. Increasing the number of parameters of the model is conducive to the simplicity prior as you are increasing the possible number of models from which SGD can optimize for the simplest model whereas decreasing the number of parameters is promoting the speed prior.[

1

] [

2

]

In ‘Risks from Learned Optimization’[

3

] the concept of deceptive alignment is presented. That is, a model is said to be deceptively aligned if during training it pretends to be aligned in order to defect later on once it’s deployed. Paul Christiano

previously argued

that in the limit of perfect capabilities the simplicity prior can result in deceptive models. That is because, between the models that are capable of deception, the ones with less complex (unaligned) objectives are simpler than the aligned ones which have a

more complex

objective. He then left the open question of whether penalizing time complexity can lead to a deceptive model or not.

Deceptive models have a particular property: they need to spend time figuring out if they are out of the training or not. The hope would be that by penalizing time complexity the model wouldn’t have time to figure out if it is out of training and therefore it would not be capable of deception.

Minimal c

...
ARC's first technical report: Eliciting Latent Knowledge
by paulfchristiano — 2021-12-14T20:09:50.209Z
Link: https://docs.google.com/document/d/1WwsnJQstPq91_Yh-Ch2XRL8H_EpsnjrC1dwZXR37PC8/edit?usp=sharing

ARC has published a report on

Eliciting Latent Knowledge

, an open problem which we believe is central to alignment. We think reading this report is the clearest way to understand what problems we are working on, how they fit into our plan for solving alignment in the worst case, and our research methodology.

The core difficulty we discuss is learning how to map between an AI’s model of the world and a human’s model. This is closely related to

ontology identification

(and

other

similar

statements

). Our main contribution is to present many possible approaches to the problem and a more precise discussion of why it seems to be difficult and important.

The report is available

here

as a google document. If you're excited about this research, we're hiring!

Q&A

We're particularly excited about answering questions posted here throughout December. We welcome any questions no matter how basic or confused; we would love to help people understand what research we’re doing and how we evaluate progress in enough detail that they could start to do it themselves.

Thanks to María Gutiérrez-Rojas for the illustrations in this piece (the good ones, blame us for the ugly diagrams). Thanks to Buck Shlegeris, Jon Uesato, Carl Shulman, and especially Holden Karnofsky for helpful discussions and comments.
ARC is hiring!
by paulfchristiano — 2021-12-14T20:09:33.977Z

The Alignment Research Center is hiring researchers; if you are interested, please apply!

(Update: We have wrapped up our hiring "round" for early 2022, but are still accepting researcher applications on a rolling basis

here

, though it may take us longer to get back to you.)

What is ARC?

ARC is a non-profit organization focused on theoretical research to align future machine learning systems with human interests. We are aiming to develop alignment strategies that would continue to work regardless of how far we scaled up ML or how ML models end up working internally.

Probably the best way to understand our work is to read

Eliciting Latent Knowledge

, a report describing some recent and upcoming research, which illustrates our general methodology.

We currently have 2 research staff (Paul Christiano and Mark Xu). We’re aiming to hire another 1-2 researchers in early 2022. ARC is a new organization and is hoping to grow significantly over the next few years, so early hires will play a key role in helping define and scale up our research.

Who should apply?

Most of all, you should send in an application if you feel excited about proposing the kinds of algorithms and counterexamples described in our report on ELK.

We’re open to anyone who is excited about working on alignment even if you don't yet have any research background (or your research is in another field). You may be an especially good fit if you:

Are creative and generative (e.g. you may already have some ideas for potential strategies or counterexamples for ELK, even if they don't work).
Have experience designing algorithms, proving theorems, or formalizing concepts.
Have a broad base of knowledge in mathematics and computer science (we often draw test cases and counterexamples from these fields).
Have thought a lot about the AI alignment problem, especially in the limit of very powerful AI systems.

Hiring will be a priority for us in early 2022 and we don't mind reading a lot of applications, so feel free to err on the side of sending in an application.

Hiring process and details

You can apply by filling out

this short form

. We will begin reviewing applications and interviewing candidates on January 3, 2022. Our hiring process involves a series of 1-2 hours interviews followed by a paid day-long work sample. Where possible we also prefer to do a longer trial although we understand that’s not practical for everyone.

We a...
Interlude: Agents as Automobiles
by Daniel Kokotajlo — 2021-12-14T18:49:20.884Z

I ended up writing a long rant about agency in my review of Joe Carlsmith’s report on x-risk from power-seeking AI. I’ve polished the rant a bit and posted it into this sequence. The central analogy between APS-AI and self-propelled machines (“Auto-mobiles”) is a fun one, and I suspect the analogy runs much deeper than I’ve explored so far.

For context, the question being discussed is whether we should “expect incentives to push relevant actors to build agentic planning and strategically aware systems [APS systems] in particular, once doing is possible and financially feasible.”

Joe says 80% yes, 20% no:

“The 20% on false, here, comes centrally from the possibility that the combination of agentic planning and strategic awareness isn’t actually that useful or necessary for many tasks -- including tasks that intuitively seem like they would require it (I’m wary, here, of relying too heavily on my “of course task X requires Y” intuitions). For example, perhaps such tasks will mostly be performed using collections of modular/highly specialized systems that don’t together constitute an APS system; and/or using neural networks that aren’t, in the predictively relevant sense sketched in 2.1.2-3, agentic planning and strategically aware. (To be clear: I expect non-APS systems to play a key role in the economy regardless; in the scenarios where (2) is false, though, they’re basically the only game in town.)”

I agree that “of course X requires Y” intuitions have been wrong in the past and also that evidence from how nature solved the problem in humans and nonhuman animals will not necessarily generalize to artificial intelligence. However:

Beware

isolated demands for rigor.

Imagine someone in 1950 saying “Some people thought battleships would beat carriers. Others thought that the entire war would be won from the air. Predicting the future is hard; we shouldn’t be confident. Therefore, we shouldn’t assign more than 90% credence to the claim that powerful, portable computers (assuming we figure out how to build them) will be militarily useful, e.g. in weapon guidance systems or submarine sensor suites. Maybe it’ll turn out that it’s cheaper and more effective to just use humans, or bio-engineered dogs, or whatever. Or maybe there’ll be anti-computer weapons that render them useless. Who knows. The future is hard to predict.” This is what Joe-with-skeptic-hat-on sounds like

...
Zvi’s Thoughts on the Survival and Flourishing Fund (SFF)
by Zvi — 2021-12-14T14:30:01.096Z

Epistemic Status: The views expressed are mine alone. They are various degrees of strong and weak, and are various degrees of weakly or strongly held. Due to time constraints, I’m not confident I was careful to mark every claim here with the proper ‘I think that’ style caveats. If something seems to you like it has a huge ‘citation needed’ sign on it, I’m probably not claiming to have proven anything.

I was one of the recommenders for the most recent round of grants for the Survival and Flourishing Fund. In accordance with our recommendations, $9.6 million dollars was allocated for distribution to various charities. The process involved four hours-long meetings where we discussed various questions, several additional discussions with other recommenders individually, many hours spent reading applications, doing research and thinking about what recommendations to make, and a number of meetings with various applicants.

This felt simultaneously like quite a lot of time, and like far less time, and the utilization of far fewer resources than the scope justified. $9.6 million dollars is a lot of money, and the share of that directed by my recommendations was also quite a lot of money.

Getting both the process right and the answers right are rather big deals.

The Survival and Flourishing Fund is related to the Effective Altruism (EA) movement, in that some of the people involved in SFF have also been involved in various EA activities, like EA Global, and SFF is trying to answer a question (how to do good with money) that is historically core to EA thinking and discourse.

It is also related to the EA movement in that, despite no official relationship between SFF and EA, despite the person who runs SFF not considering himself an Effective Altruist (Although he definitely believes, as I do, in being effective when being an altruist, and also in being effective when not being an altruist), despite SFF not being an EA organization, despite the words ‘altruist’ or ‘effective’ not appearing on the webpage, at least this round of the SFF process and its funds were largely captured by the EA ecosystem. EA reputations, relationships and framings had a large influence on the decisions made. A majority of the money given away was given to organizations with explicit EA branding in their application titles (I am including Lightcone@CFAR in this category).

Before going fur...