trentmkelly/LessWrong-43k huggingface dataset:
How can one rationally have very high or very low probabilities of extinction in a pre-paradigmatic field?
by Shmi — 2023-04-30T21:53:15.843Z

It is generally accepted in the local AI alignment circles that the whole field is pre-paradigmatic, in the Kuhnian sense (phase 1, as summarized here, if longer reading is not your thing). And yet, plenty of people are quite confident in their predictions of either doom or fizzle. A somewhat caricature way of representing their logic is, I think, "there are so many disjunctive ways to die, only one chance to get it right, and we don't have a step-by-step how-to, so we are hooped" vs "this is just one of many disruptive inventions whose real impact can only be understood way down the road, and all of them so far have resulted in net benefit, AI is just another example" (I have low confidence in the accuracy of the latter description, feel free to correct.) I can see the logic in both of those, what I do not see is how one can rationally have very high or very low confidence, given how much inherent uncertainty there is in our understanding of what is going on.

My default is something more cautious, akin to Scott Alexander's https://slatestarcodex.com/2019/06/03/repost-epistemic-learned-helplessness/

where one has to recognize their own reasoning limitations in the absence of hard empirical data, not "The Lens That Sees Its Flaws", but more like "The Lens That Knows It Has Flaws" without necessarily being able to identify them.

So, how can one be very very very sure of something that has neither empirical confirmation, nor sound science behind it? Or am I misrepresenting the whole argument?
A small update to the Sparse Coding interim research report
by Lee Sharkey — 2023-04-30T19:54:38.342Z

This is a linkpost to a set of slides containing an update to a project that was the subject of a previous post ([Interim research report] Taking features out of superposition with sparse autoencoders).

The update is very small and scrappy. We haven't had much time to devote to this project since posting the Interim Research Report.

TL;DR for the slides:

We trained a minuscule language model (LM) (residual size = 16; 6 layers) and then trained sparse autoencoders on MLP activations (dimension = 64) from the third layer of that model.
We found that, when we compared the 'ground truth feature recovery' plots, the plots for the toy data and LM data were much more similar than in the Interim Research Report.
Very, very tentatively, we found the layer had somewhere between 512-1024 features. By labelling a subset of these features, we estimate there are roughly 600 easily labellable (monosemantic) features. For instance, we found a feature that activates for a period immediately after 'Mr', 'Mrs', or 'Dr'.
We suspect that the reason the toy data and LM data plots had previously looked different was due to severely undertrained sparse autoencoders.

We're hopeful that with more time to devote to this project we can confirm the results and apply the method to larger LMs. If it works, it would give us the ability to tell mechanistic stories about what goes on inside large LMs in terms of monosemantic features.
Discussion about AI Safety funding (FB transcript)
by Orpheus16 — 2023-04-30T19:05:34.009Z

Kat Woods recently wrote a Facebook post about Nonlinear's new funding program.

This led to a discussion (in the comments section) about funding norms, the current funding bar, concerns about lowering the bar, and concerns about the current (relatively centralized) funding situation.

I'm posting a few of the comments below. I'm hoping this might promote more discussion about the funding landscape. Such discussion could be especially valuable right now, given that:

Many people are starting to get interested in AI safety (including people who are not from the EA/rationalist communities)
AGI timeline estimates have generally shortened
Investment in overall AI development is increasing quickly
There may be opportunities to spend large amounts of money in the upcoming year (e.g., scalable career transition grant programs, regranting programs, 2024 US elections, AI governance/policy infrastructure, public campaigns for AI safety).
Many ideas with high potential upside also have noteworthy downside risks (phrased less vaguely, I think that among governance/policy/comms projects that have high potential upside, >50% also have non-trivial downside risks).
We might see pretty big changes in the funding landscape over the next 6-24 months

New funders appear to be getting interested in AI safety
Governments are getting interested in AI safety
Major tech companies may decide to invest more resources into AI safety

Selected comments from FB thread

Note: I've made some editorial decisions to keep this post relatively short. Bolding is added by me. See the full thread here. Also, as usual, statements from individuals don't necessarily reflect the views of their employers.

Kat Woods (Nonlinear)

I often talk to dejected people who say they tried to get EA funding and were rejected

And what I want to do is to give them a rousing speech about how being rejected by one funder doesn't mean that their idea is bad or that their personal qualities are bad.

The evaluation process is noisy. Even the best funders make mistakes. They might just have a different world model or value system than you. They might have been hangry while reading your application.

That to succeed, you'll have to ask a ton of people, and get a ton of rejections, but that's OK, because you only need a handful of yeses.

(Kat then describes the new funding program from Nonlinear. TLDR: People submit an applicati...
money ≠ value
by stonefly — 2023-04-30T17:47:42.844Z

The relationship between money and value is not linear or consistent. This means betting odds don't necessarily correspond to the probability of an event.

--

Converting money to value is done by exchanging it for things you want. The value of increasing your wealth by n dollars is bounded by the most valuable thing you might use n more dollars for. The most valuable thing you might do with n more dollars is affected by what you want and what you know how to get.

People's efficiency at converting money to value changes on short-term and long-term scales, and depends on how much money they have in total.

--

We can graph the value to a person of increasing their wealth by n dollars from a certain point. This graph is different for different people because different people want different things and can get different things.

In a simplified example, where a person starts with 0 dollars and will never make more money in the future, their graph might look like this:

The slope of the graph increases in the range where the person can barely afford housing, because increasing how much money they are given within this range makes a bigger difference than increasing how much money they are given within other ranges.

In real life people don't start with 0 dollars and only receive money once. Each increase in money shifts by a little the probability they will buy something they are not sure they can afford. It does not determine with certainty what they can afford.

In real life the function will also be less simple; there will be many places where the rate of value increase per money increase changes, rather than just one that occurs when the person can afford housing.

The function will still be bounded by the most cost-efficient thing they might buy or save to buy, which changes depending on who they are and how much money they are gaining.

People's intuition about how much they value an increase in money is different from how much they actually value that increase in money. However, this intuition is also not linear.

--

The value of money becomes linear when the thing you prefer to buy gives you the same incremental value every time no matter how many you bought already.

For example, when saving up to buy something, each increase in money affects the ability to buy it the same amount. This makes t...
Vaccine Policies Need Updating
by jefftk — 2023-04-30T17:20:04.919Z

It's

reasonably
common for events like social dances to require vaccination. This
usually looks like requiring attendees be "fully vaccinated", "up to
date", or "fully vaccinated and boosted", and cashes out to the people
at the door checking for a primary series plus potentially a booster.

A bit over a week ago, however, the CDC put out updated
guidance, including:

Monovalent (original) mRNA COVID-19 vaccines will no longer be
recommended for use in the United States.

And:

CDC recommends that everyone ages 6 years and older receive an updated
(bivalent) mRNA COVID-19 vaccine, regardless of whether they
previously completed their (monovalent) primary series.

This reads to me that if someone is not yet vaccinated and decides to
become so, they'll be given a single bivalent mRNA dose.

Events that require vaccination need to decide how to handle this, and
it would be good to think about before someone shows up at the dance!
At the dance I help organize we phrase it as
requiring a booster if eligible, but if it's your first shot then it's not
boosting anything? And some dances are even more explicit about
requiring something you can't get anymore:

All students, instructors, and staff who attend in-person
classes and/or events must be fully vaccinated against COVID-19. Fully
vaccinated means that it has been at least two weeks since your second
dose of Pfizer or Moderna vaccine or first dose of Johnson &
Johnson vaccine. Starting January 2022, individuals must have received
a COVID-19 booster shot if they received Johnson & Johnson only or if
their 2nd dose of Pfizer or Moderna was more than 6 months ago.
—Blues
Union

(Not trying to say anything here on whether or when I think events
should require vaccination.)
Fundamental Uncertainty: Chapter 7 - Why is truth useful?
by Gordon Seidoh Worley — 2023-04-30T16:48:58.312Z

N.B. This is a chapter in a book about truth and knowledge. It is the first draft. I have since revised it. You can find the most up-to-date info/version on the book's website.

Last chapter we proved that our knowledge of the truth is fundamentally uncertainty. But this didn't mean there was no truth to be found. Instead, it simply meant that we could not guarantee that our beliefs about what's true are always 100% accurate. We saw that there is always some irreducible error in our models of the world, because even if we manage to justify all of our beliefs upon a handful of assumptions, those assumptions cannot themselves be fully verified.

Despite the impossibility of certainty about the truth, we are able to get on with our lives anyway and accomplish many things. We're even able to do science, engineering, medicine, and more that requires understanding deep truths about the working of the world. We do this via pragmatism, by intuitively basing our knowledge of relative truth upon that which is useful.

But what's so special about usefulness? Why base our knowledge on what's useful rather than something else? To begin our investigation, let's check back in on Dave.

Several more years have passed, and Dave has spent nearly a decade in college. As an undergrad he majored in biology with a specialization in botany and a minor in philosophy. He then went on to earn a PhD in neurobiology and has done research on plant cells that behave, in some limited ways, like neurons. Alas, after much research, he's been forced to conclude that, although plants exhibit many complex behaviors, it's very unlikely that plants can meaningfully be said to think.

Even though plants can't think, Dave learned that they do work towards goals. He was surprised by this notion at first. Since plants don't think, it seemed impossible for them to have goals. But then he realized he had already proved that plants are goal-oriented, he just hadn't realized it at the time.

Recall Dave's science experiment. He showed that plants grow to get into the light, not just along a predefined plan, but by twisting and turning to get access to as much light as possible. They don't think about doing this or make plans, but instead carry out simple behaviors, like growing stems and leaves on the parts of themselves that are receiving the most light, that serve to fulfill the goal of growing towards the light....
Simulators Increase the Likelihood of Alignment by Default
by Wuschel Schulz — 2023-04-30T16:32:43.651Z

Alignment by Default is the idea that achieving alignment in artificial general intelligence (AGI) may be more straightforward than initially anticipated. When an AI possesses a comprehensive and detailed world model, it inherently represents human values within that model. To align the AGI, it's merely necessary to extract these values and direct the AI towards optimizing the abstraction it already comprehends.

In a summary of this concept, John Wentworth estimates a 10% chance of this strategy being successful, a perspective I generally agree with.

However, in light of recent advancements, I have revised my outlook, now believing that Alignment by Default has a higher probability of success, perhaps around 30%. This update was prompted by the accomplishments of ChatGPT, GPT-4, and subsequent developments. I believe these systems are approaching AGI closely enough that it is reasonable to assume the first AGI will be based on the large language model (LLM) paradigm, which in turn, makes Alignment by Default more likely. In this post, I will outline the reasons behind my updated belief.

This post is also an entry in the Open Philanthropy AI Worldviews Contest. It is supposed to address Question 2: How great is our Exiential Risk, if we develop AGI before 2070?

Alignment by Default

The Alignment by Default concept suggests that if we can direct AGI toward human values without any significant breakthroughs in alignment, we could avoid catastrophe. At first glance, this idea might seem implausible, given our limited understanding of AGI and human values, and the relatively small target we're trying to hit. Even if we almost succeed in alignment, it might not be enough.

For example, let's say we nearly succeeded in aligning an AI with human values but misdefined what constitutes a person. The outcome would be a universe populated by unconscious robots expressing happiness and seemingly leading fulfilled lives. Claiming that we can align an AGI to a non-catastrophic level with our current understanding of the alignment problem is like saying we can hit a bulls-eye without really knowing how bows work and only a vague idea of where the target is.

John Wentworth, however, argues in his post that this might still be possible due to natural abstractions. He envisions building an AGI by first training a world model based solely on prediction and then creating an agent ...
Connectomics seems great from an AI x-risk perspective
by Steven Byrnes — 2023-04-30T14:38:39.738Z

Context

Numerous people are in a position to accelerate certain areas within science or technology, whether by directing funds and resources, or by working in the area directly. But which areas are best to accelerate?

One possible consideration (among others) is the question: “Is accelerating this technology going to increase the chance that our future transition to superhuman artificial general intelligence (AGI) goes well? Or decrease it? Or make no difference?” My goal here is to try to answer that question for connectomics (the science & technology of mapping how neurons connect to each other in a brain).

This blog post is an attempt to contribute to

Differential Technology Development

(DTD) (part of the broader field of

Differential Intellectual Progress

). Successful DTD involves trying to predict complicated and deeply uncertain future trajectories and scenarios. I think the best we can hope for is to do better than chance. But I’m optimistic that we can at least exceed that low bar.

My qualifications: I’m probably as qualified as anyone to discuss

AI x-risk and how it relates to neuroscience

. As for connectomics, I’m not too familiar with the techniques, but I’m quite familiar with how the results are used—in the past few years I have scrutinized probably hundreds of journal articles describing neural tracer measurements. (Think of neural tracer measurements as the traditional, “artisanal”, small-scale version of connectomics.) I find such articles extremely useful; I would happily trade away 20 fMRI papers for one neural tracer paper. This post is very much “my opinions” as opposed to consensus, and I’m happy for further discussion.

TL;DR

Improved connectomics technology seems like it would be very helpful for the project of reverse-engineering circuitry in the hypothalamus and brainstem that implement the “innate drives” upstream of human motivations and morality. And that’s a good thing! We may wind up in a situation where future researchers face the problem of

designing “innate drives” for an AI

; knowing how they work in humans would be helpful

for various reasons

.
Improved connectomics technology seems like it would NOT be very helpful for the project of reverse-engineering the learning algorithms implemented by various parts of the brain, particularly the neocortex. And that’s a good thing

...
The voyage of novelty
by TsviBT — 2023-04-30T12:52:16.817Z

[Metadata: crossposted from https://tsvibt.blogspot.com/2023/01/the-voyage-of-novelty.html. First completed January 17, 2023. This essay is more like research notes than exposition, so context may be missing, the use of terms may change across essays, and the text might be revised later; only the versions at tsvibt.blogspot.com are definitely up to date.]

Novelty is understanding that is new to a mind, that doesn't readily correspond or translate to something already in the mind. We want AGI in order to understand stuff that we haven't yet understood. So we want a system that takes a voyage of novelty: a creative search progressively incorporating ideas and ways of thinking that we haven't seen before. A voyage of novelty is fraught: we don't understand the relationship between novelty and control within a mind.

What takes the voyage, from where to where? The voyage is taken by some AI system, from not understanding what we want it to understand, to understanding all that. A schematic:

Novelty

Novelty is structure that is new to a mind. A mind acquires novelty via creativity.

Endosystemic novelty is novelty that fits in with the preexisting mind in ways analogous to how preexisting elements of the mind fit in with the mind; e.g., a new sub-skill such as "opening a car door", or a new word. Diasystemic novelty is novelty (new structure) that involves through-going changes to the mind; e.g. a new heuristic like "investigate boundaries" applied automatically across many contexts. At the extreme end of diasystemic novelty are cognitive realms distinct from the cognitive realm currently occupied by the mind.

Structure hand-holdable by humanity

Where does the voyage of novelty start? It starts wherever the mind is. But where can it safely start?

A lot of structure has the property that it is easy enough for humans to render non-dangerous a mind's exploration (discovery, creation) and exercise (application, expression) of that structure. Humanity can "hold a mind's hand" through its voyage of novelty through such structure.

Some overlapping reasons that structure can be hand-holdable, with examples:

The structure just doesn't contribute much to a mind having large effects on the world. For example, a mind might know about some obscure species of grass that no human knows about; a human can't understand the mind's thoughts in absolutely full detail, at least without zer

...
Call for submissions: Choice of Futures survey questions
by c.trout — 2023-04-30T06:59:11.034Z
Link: https://airtable.com/shrV2dCDwEW6jo6Jz

TL;DR: independent researcher (M.A. Philosophy) crowdsourcing questions for an upcoming survey: submit here some controversial societal goals and fears you think we should get cross-comparison public opinion data on! The driving question of the survey: what do we want the world to look like in ~50 years time? More details below:

The world is changing rapidly and we face a number of challenges: environmental collapse, general-purpose Artificial Intelligence, geopolitical instability, faltering trust in democratic institutions, to name a few. Technical experts can tell us which actions will result in which outcomes, but no expert can tell you which outcome is most desirable: that’s a question of values and terminal goals. Not all values and goals are compatible though. For example, in the degrowth vs green growth debate, the disagreement is not merely empirical but also political: there are competing visions of what future we seek. Broadly, that debate asks us to consider: do we prefer rewilding/reforestation and slow living, or prefer doubling down on rapid technological change and ever more efficient production of abundant consumer goods? Or consider Bostrom's Vulnerable World Hypothesis: if true, then continued technological development, greater privacy/freedom from state oversight, and a high degree of safety from X or S-risk seem all but incompatible. One must go.

In an effort to gain greater strategic clarity, I'll be conducting a survey that seeks to answer the following question: what future do people actually want? As a society, what are our goals/priorities for this century? And what do we fear? What do we most want to avoid? If the public doesn’t make its preferences known, they will effectively be forfeiting their say to corporations, wealthy special interest groups, and technocrats. This is not only unjust, but likely to result in worse outcomes from the perspective of the general public (why should we expect these groups to accurately represent the interests of the general public?).

While many surveys of individual issues exist, I plan to collect many such competing goals and risks into one survey to study how people make tradeoffs between them. To avoid bias, I’m trying to crowdsource the goals and risks I present to survey respondents: that’s where you come in! I’m posting this call for submission among various groups who I believe have controvers...