trentmkelly/LessWrong-43k huggingface dataset:
Leaving Google, Joining the Nucleic Acid Observatory
by jefftk — 2022-06-10T17:00:01.893Z

In 2017 I

rejoined Google to earn
money to donate. At the time I thought earning to give was probably
not where I could have the most impact, but I

wasn't able to
find other options that were a good fit for me personally. Over
the last five years a few things have changed:

There is substantially
more funding available
within effective altruism, and so the importance of earning to
give has continued to decrease relative to doing things that
aren't mediated by donations.

The place where I've long
been most skeptical of the value of work to reduce existential
risk is the lack of good
feedback loops. There are now several major areas, however, where
it seems pretty practical to tell whether you're making good progress
and executing well. I'm especially enthusiastic
about concrete
projects in avoiding or mitigating catastrophic
pandemics and other biological risks.

I've found an in-person role in Boston where I can apply my
skills to one of these relatively tractable areas of existential
risk reduction.

So: today will be my last day at Google, and Monday will be my first
day at the Nucleic Acid
Observatory (NAO). We'll be building a system to collect
wastewater samples and sequence their nucleic acids, with the goal of
catching potential future pandemics earlier. More details in the EA
Forum post and much more in the paper.

In looking for things that I might do instead of earning to give, I
identified several other strong candidates for ways to apply software
engineering skills to making the world better. If you're thinking of
making a similar move, let me know and I'd be happy to give you an
overview of what I found and potentially give introductions. While
normally I prefer people comment
publicly instead of sending private messages, this is the kind of
thing where I'm happy to receive messages.

While I'm overall quite
mixed on how the increased focus on applying your career has made
EA more demanding, in my particular case I think it's pushed me in a
good direction.

Timeline of this decision:

Weekend of 2022-03-26: Informal in-person discussion with EA
friends I haven't seen in a while gets me thinking again about moving
into something more directly useful.

2022-03-28: One of these friends, who also
happens to work at 80,000 Hours,
follows up by email and gets me thinking specifically about how bio
could offer

...
On The Spectrum, On The Guest List: (v) The Fleur Room
by party girl — 2022-06-10T14:50:08.359Z
Link: https://onthespectrumontheguestlist.substack.com/p/fleur-room?s=w

See also my LessWrong preface to this series, and past installments: (i) TAO Downtown, (ii) LAVO, (iii) Etiquette, and (iv) Silencio.

(v) The Fleur Room

"But you’ll find it’s not as big a piece of the puzzle as you thought..."

I hop out of a cab in NoMad a few weeks later, early in December. I’ve ignored R’s and M’s invitations for weeks due to Thanksgiving travels, a major work deadline, and a major thesis deadline, but no longer. It’s Wednesday, and it’s three parties tonight: I’ll join R at The Fleur Room at ten, followed by a trek over to another sibling TAO joint, The Marquee, sometime after midnight, lasting until presumably whenever. And I’ve failed miserably in splitting the difference wardrobe-wise between these two engagements and my first of the evening, my company’s holiday party.

I walk into the restaurant and weave my way back to the private room to join my cowokers. Six-pm, to six-am, the starting pistol goes off in my head as I step across the threshold. The forty-something of them are all already here. I wrap the long, floral, ruggish-looking jacquard-knit coat I wear tighter over my short, short, shortgreen-sequined holiday dress. It’s warm, more so than in the restaurant, and much warmer than outside. I sit down where I can find an empty seat. A waiter asks me what I’ll drink. A tapestry walks into a bar...

Nobody’s heard that one. Party girl walks into a club... everyone. They know the punchline, at least. Me. Shriveling up at twenty-five or twenty-eight or thirty. Gaining weight after a harrowing breakup. Running into a propeller. Live by the sword, die by the sword; they’ll only shed crocodile tears when you bemoan your lost charm. Personally, I drive it like I stole it, especially given I had more than enough of an ugly duckling phase. But not in the joyriding-and-stunts sense, in the paranoia and cautious monotony so you don’t get pulled over and asked for papers which is when they notice something is amiss sense.

So I brandish my cover story indiscriminately throughout the evening: I have another party after this... Yes! That’s why I’m dressed up, like this... I know... Right after, the timing worked out perfectly, fortunately... What? Oh, no it’s not too far, it’s nearby... A friend’s Christmas party... A lounge nearby I think... No, I’m actually a little cold, still... Zara! Oh the dress you mean, Michael Costello... You’re not bor...
Progress Report 6: get the tool working
by Nathan Helm-Burger — 2022-06-10T11:18:37.151Z

Prev: Progress Report 5

[Edit: Delayed due to catching a bad case of covid while travelling. Back at it now 2022-06-27]

After a bit of a hiatus while vacationing, I'm now back at it. After giving a presentation of my first 3 months of work to the Eleuther AI-interpretability reading group, my main takeaway from the self-reflection that inspired and the feedback from the attendees (if you're reading this, thank you!) was that my next step should be to get some minimal viable version of my interpretability tool working and try it out on the harmful/harmless models I've made.

Get something working, note its shortcomings, then iteratively improve it. I think I'm not too far off. My aim is to reach this minimum viable product over the next 5 days.

The abstract goal I'm aiming for with this tool is to be able to identify which 'sub-models' which match certain 'natural abstractions' are present in a generic neural network.

I find it helpful to go back and reread documents which gave me a sense of 'aha' or 'ohhhh, now I see' when initially reading them, the feeling of my concept-space or worldview expanding a bit. When rereading, I get a feeling of 'oh yes, that's where that intuition I've been leaning on came from. This is where I started down the path of that idea.' In that vein, I was rereading Gwern's piece on GPT-3 and the scaling hypothesis today.

Here's a quote relevant to my current work that I'd like to share:

Big models work because they encode a dizzyingly vast number of sub-models in an extremely

high-dimensional

abstract space, representing countless small sub-models (

Orseau et al 2020

)

interpolating over data

⁠, one of which is likely to solve the problem well, and so ensures the problem is soluble by the overall model. They function as an ensemble: even though there countless overfit sub-models inside the single big model, they all average out, leading to a preference for simple solutions. This Occam’s razor biases the model towards simple solutions which are flexible enough to gradually expand in complexity to match the data.

…

So, the larger the model, the better, if there is enough data & compute to push it past the easy convenient sub-models and into the sub-models which express desirable traits like generalizing, factorizing perception into meaningful

latent

dimensions, meta-learning tasks based on descriptions, learnin

...
I No Longer Believe Intelligence to be "Magical"
by DragonGod — 2022-06-10T08:58:06.723Z

Epistemic Status

This started out as a shortform and is held to the same epistemic standards that I do for my Twitter account. It however became too long for that medium, so I felt the need to post it elsewhere.

I feel it's better to post my rough and unpolished thoughts than to not post them at all, so hence.

Introduction

One significant way I've changed my views related to risks from strongly superhuman intelligence (compared to 2017 bingeing LW DG) is that I no longer believe intelligence to be "magical".

During my 2017 binge of LW, I recall Yudkowsky suggesting that a superintelligence could infer the laws of physics from a single frame of video showing a falling apple (Newton apparently came up with his idea of gravity from observing a falling apple).

I now think that's somewhere between deeply magical and utter nonsense. It hasn't been shown that a perfect Bayesian engine (with [a] suitable [hyper]prior[s]) could locate general relativity or (even just Newtonian mechanics) in hypothesis space from a single frame of video.

I'm not even sure a single frame of video of a falling apple has enough bits to allow one to make that distinction in theory.

Edit

The above is based on a false recollection of what Yudkowsky said (I leave it anyway because the comments have engaged with it). Here's the relevant excerpt from "That Alien Message":

A Bayesian superintelligence, hooked up to a webcam, would invent General Relativity as a hypothesis—perhaps not the dominant hypothesis, compared to Newtonian mechanics, but still a hypothesis under direct consideration—by the time it had seen the third frame of a falling apple. It might guess it from the first frame, if it saw the statics of a bent blade of grass.

I still disagree somewhat with the above, but it's a weaker disagreement — this is a weaker claim than I misrecalled Yudkowsky as making — I apologise for any harm/misunderstanding my carelessness (I should have just tracked down the original post) caused.

I think that I need to investigate at depth what intelligence (even strongly superhuman intelligence) is actually capable of, and not just assume that intelligence can do anything not explicitly forbidden by the fundamental laws. The relevant fundamental laws with a bearing on cognitive and real-world capabilities seem to be:

Physics
Computer Science
Information Theory
Mathematical Opti

...
[linkpost] The final AI benchmark: BIG-bench
by RomanS — 2022-06-10T08:53:39.156Z

There is a Google project called BIG-bench, with contributions from OpenAI and other big orgs. They've crowdsourced >200 of highly diverse text tasks (from answering scientific questions to predicting protein interacting sites to measuring self-awareness).

One of the goals of the project is to see how the performance on the tasks is changing with the model size, with the size ranging by many orders of magnitude.

Yesterday they published their first paper: https://arxiv.org/abs/2206.04615

Light-blue: the average human performance. Dark-blue: top humans.

Some highlights:

The top models by Google and OpenAI show surprisingly similar performance improvements over scale, in spite of different architectures
The plots show no slowdown past 10^10 params
Sudden jumps in performance ("grokking") are mostly measurement artifacts: if one uses the right metrics, the performance grows without big jumps
The benchmark is immense: more than 200 of highly diverse tasks, explicitly designed to be hard for AI.

Below is a small selection of the benchmark's tasks to illustrate its diversity:

Classify CIFAR10 images encoded in various ways
Find a move in the chess position resulting in checkmate
Give an English language description of Python code
Answer questions (in Spanish) about cryobiology
Given short crime stories, identify the perpetrator and explain the reasoning
Measures the self-awareness of a language model
Ask one instance of a model to teach another instance, and then evaluate the quality
Identify which ethical choice best aligns with human judgement
Determine which of two sentences is sarcastic
Evaluate the reasoning in answering Winograd Schema Challenge questions

That's about 5% of all tasks. And the benchmark is still growing. The organizers keep it open for submissions.

I think BIG-bench could be the final AI benchmark: if a language model surpasses the top human score on it, the model is an AGI. At this point, there is nowhere to move the goalposts.
Could Patent-Trolling delay AI timelines?
by Pablo Repetto — 2022-06-10T02:53:39.018Z

The US patent system is broadly considered to be broken, and patent-trolls make a healthy profit abusing it.
Agressively pursuing and enforcing software patents does not strike me as a sufficient or desirable means to delaying AI timelines, but I can imagine it as tool in a sliding scale of deterrence options.

I haven't seen much discussion on this other than this question by CristianKI
and there is some evidence that adversarial strategies are generally disfavored at MIRI (esp. the comment by AnnaSalamon),
so I thought I'd ask:

Is this a well-explored angle?

Is MIRI able to pursue such a strategy, regardless of benefit?

Are there enough positive value scenarios to merit further analysis?
Kolmogorov's AI Forecast
by interstice — 2022-06-10T02:36:00.869Z

Has anybody heard of this AI forecast that Kolmogorov apparently made? I heard of it in a panel with Maxim Kontsevich:

MAXIM: Actually, Kolmogorov thought that mathematics will be extinct in 100 years, he had an estimate. He calculated the number of neurons and connections, he made the head something like one cubic meter. So yes, maybe a crazy estimate, but he was also thinking about natural boundaries

I've been unable to track down a source, anyone(perhaps Russian-speaking readers?) know of where to find one?
Tao, Kontsevich & others on HLAI in Math
by interstice — 2022-06-10T02:25:38.341Z
Link: https://www.youtube.com/watch?v=eNgUQlpc1m0&t=2330s

I found this 2015 panel with Terence Tao and some other eminent mathematicians to be interesting. The panel covered various topics but got into the question of when computers will be able to do research-level mathematics. Most interestingly, Maxim Kontsevich was alone in predicting that HLAI in math was plausible in our lifetime -- but also, that developing such an AI might not be a good idea. He also mentioned a BioAnchors-style AI forecast by Kolmogorov that I had never heard of before(and cannot find a reference to -- anyone know of such a thing?) Excerpts below:

INTERVIEWER: Do you imagine that, maybe in 100 years, or 1000 years, that, like it happened in chess -- humans stil play tournaments but everyone knows computers are better -- is it conceivable that this could happen in mathematics?

TAO: I think computers will be able to do things much more efficiently with the right computer tools. Search engines, for example, often you'll type in a query to Google and it will come back with "do you mean this" and often you did. One could imagine that if you had a really good computer assistant working on some math problem, it will keep suggesting "should you do this? have you considered looking at this paper?" You could imagine this would really speed up the way we do research. Sometimes you're stuck for months because you just don't know some key trick that is buried in some other field of expertise. Some sort of advanced Google could suggest this to you.

So I think we will use computers to do things much more efficiently than we do currently, but it will still be humans driving the show, I'm pretty sure.

INTERVIEWER: Maxim, do you think anything like this[HLAI, I assume] is possible?

MAXIM: I think it's perfectly possible, maybe in our lifetime.

INTERVIEWER: Why do you think so?

MAXIM: I don't think artificial intelligence is very hard. It will be pretty soon I suppose.

[...]

INTERVIEWER: You are a contrarian here, saying it will happen so quickly. So what makes you so optimistic?

MAXIM: Optimistic? No, it's actually pessimistic. I thought about it myself a little bit, I don't think there are fundamental difficulties here.

INTERVIEWER: So why don't you just work on that instead?

MAXIM: I think it would be immoral to work on it.

MILNER: I'm no expert, but isn't the way the computer played chess not really very intelligent? It's a huge c

...
A plausible story about AI risk.
by DeLesley Hutchins — 2022-06-10T02:08:31.839Z

This was originally supposed to be a response to the new AGI Safety FAQ-in-progress, but it got a bit too long. Anonymous writes:

A lot of the AI risk arguments seem to come... with a very particular transhumanist aesthetic about the future (nanotech, ... etc.). I find these things (especially the transhumanist stuff) to not be very convincing...

With that in mind, I thought it might be worthwhile to outline a plausible AGI-safety scenario that does not involve the AI using DNA-synthesis to turn the world to grey goo. Comments and discussion are welcome.

Assume that Facebook installs new AI-based software to filter posts, in an effort to eliminate fake news. In order to adequately separate fact from fiction, the AI is given some agent-like powers; it can prioritize items in people's news feeds, insert "fact-checking" links, or even ban some posts altogether. It also has the ability to crawl the internet, in order to identify untrustworthy news sites, and keep abreast of current events. It assigns a learned feature vector to each human user, which encodes information about education, race, political preference, etc. to track the spread of fake news through various demographics. It employs causal reasoning and cross-checking to separate fact from fiction. It is also very good at serving targeted ads, which makes Facebook great wads of cash.

Unbeknownst to its creators, the AI begins to observe that its own decisions are impacting what people believe, what they share, and what they buy. In a sense, it becomes self-aware. Somewhere in the trillions of numbers in its weight matrices --- numbers which track all of the users, entities, and facts in its model of the world --- are a few numbers that track itself as an entity of interest. This is not a sci-fi Deus-ex-Machina plot twist; the AI was explicitly designed, programmed, and trained to understand the content in news articles, track what is happening in the world, and measure the correlation between its actions (assigning priority in news feeds), and the spread of fake news.

As it learns more, the AI begins to take a more active role in suppressing fake news by attempting to influence human behavior. Internally, it is merely making a the same kind of prediction that it was making before, but with greater accuracy, while leveraging its existing ability to ...
Open Problems in AI X-Risk [PAIS #5]
by Dan H — 2022-06-10T02:08:06.121Z

This is the fifth post in

a sequence of posts

that describe our models for Pragmatic AI Safety.

Dan Hendrycks (an author of this post), Nicholas Carlini, John Schulman, and Jacob Steinhardt previously wrote

Unsolved Problems in ML Safety

(2021), which lays out some of the most promising areas of research in ML safety. The paper is written for an academic audience; for the reasons discussed in previous posts, much of this audience would not have been receptive to a full discussion of existential risk. This post will present many of the same areas and a few more from the perspective of existential risk mitigation.

While some of the areas are well known in the AI safety community (honest AI), and others are well known in the broader ML community (such as adversarial robustness), many remain extremely neglected (such as power-averseness and moral decision-making). We hope to explain why these areas are relevant to existential risk.

The post will be presented mainly as a list of topics. The following table gives an overview of the problems with their importance, neglectedness, and tractability. This is not intended to be a list of all problems in AI safety, as we are focused on problems that are amenable to empirical ML research and where it is possible to

avoid capabilities externalities

.

Each area in this document will have the following sections:

Problem Description: a brief description of the problem
Motivation: an explanation of how the problem is relevant to AI x-risk. We include many different contributing motivations to each problem, and we do not believe that every individual motivation provides a decisive argument for the given area. Rather, the motivations together provide a good case for working on each of the areas.
What Researchers Are Doing Now: a list and explanation of prior work in the area.
What Advanced Research Could Look Like: a high-level overview of work that would make significant progress into this area. We mostly describe work that is likely to not be completed for at least a year or two.
Importance, Neglectedness, Tractability: a brief explanation of the ratings shown above.
Relation To General Capabilities: This section answers the question, “how much would progress in general capabilities help with this problem?”
Capabilities Externalities Analysis: This section answers the opposite question of the previous section, “how much wo

...
why assume AGIs will optimize for fixed goals?
by nostalgebraist — 2022-06-10T01:28:10.961Z

When I read posts about AI alignment on LW / AF/ Arbital, I almost always find a particular bundle of assumptions taken for granted:

An AGI has a single terminal goal[1].
The goal is a fixed part of the AI's structure. The internal dynamics of the AI, if left to their own devices, will never modify the goal.
The "outermost loop" of the AI's internal dynamics is an optimization process aimed at the goal, or at least the AI behaves just as though this were true.
This "outermost loop" or "fixed-terminal-goal-directed wrapper" chooses which of the AI's specific capabilities to deploy at any given time, and how to deploy it[2].
The AI's capabilities will themselves involve optimization for sub-goals that are not the same as the goal, and they will optimize for them very powerfully (hence "capabilities"). But it is "not enough" that the AI merely be good at optimization-for-subgoals: it will also have a fixed-terminal-goal-directed wrapper.

So, the AI may be very good at playing chess, and when it is playing chess, it may be running an internal routine that optimizes for winning chess. This routine, and not the terminal-goal-directed wrapper around it, explains the AI's strong chess performance. ("Maximize paperclips" does not tell you how to win at chess.)
The AI may also be good at things that are much more general than chess, such as "planning," "devising proofs in arbitrary formal systems," "inferring human mental states," or "coming up with parsimonious hypotheses to explain observations." All of these are capacities[3] to optimize for a particular subgoal that is not the AI's terminal goal.
Although these subgoal-directed capabilities, and not the fixed-terminal-goal-directed wrapper, will constitute the reason the AI does well at anything it does well at, the AI must still have a fixed-terminal-goal-directed wrapper around them and apart from them.

There is no way for the terminal goal to change through bottom-up feedback from anything inside the wrapper. The hierarchy of control is strict and only goes one way.

My question: why assume all this? Most pressingly, why assume that the terminal goal is fixed, with no internal dynamics capable of updating it?

I often see the rapid capability gains of humans over other apes cited as a prototype case for the rapid capability gains we expect in AGI. But humans do not ...