trentmkelly/LessWrong-43k huggingface dataset:
The value of x-risk reduction
by  — 2022-05-22T07:44:47.829Z

In his interview on the 80000 hours podcast after the publication of the Precipice, Toby Ord claimed that x-risk reduction was worth pursing regardless of weather x-risk was high or low. The claim is that when x-risk is low, it's worth working on x-risk because a small decline in the probability of x-risk has very large returns.

This is because by reducing the value of x-risk you're increasing the value of the future and the value of the future is greatest when x-risk is smallest. On the other hand, when x-risk is large, the return is also high because x-risk is likely to be neglected and therefore the marginal impact of work on x-risk is likely to be very high. This post aims to show the cases when this does and doesn't hold true.

I'll first lay out intuitively the cases where it does and doesn't hold, formalise it these notions and then give some numerical examples.

Intuition

There the are two parts to the claim: the value of the future part and the neglectedness part. The value of the future is the value of humanity per year, divided by the probability of x-risk per year. I'm assuming that the probability of x-risk is constant every year and I'll assume that throughout the post. This has some quite counter-intuitive effects. It means that decreasing the probability of x-risk by 1/10 per year to 1/100 makes the future 10 times more valuable in expectation. However, the much larger percentage point decline in year probability of x-risk from 1/2 to 1/10 only increases the expected value of the future 5 fold.

In general, a decline in x-risk from x to y increases the the value of the future by x/y. This does indeed mean that the value of decreasing x-risk is very large when x-risk is very small, and in fact in increases with square of x-risk. However, if you think that x-risk is very high the value of x-risk reduction is proportionally as small.

There's also a second factor to consider. We can think of probability of an existential risk in a given year as being dependent on all of the inputs going into x-risk reduction. This means that when we change the an input we get two effects - the effect from the change in probability of x-risk in the year, multiplied by the everything the effect of the input is proportional to. For instance, the amount of x-risk per year could be dependent on the product of the amount of research being done and the amount of money bein...
On The Spectrum, On The Guest List
by party girl — 2022-05-22T03:31:36.653Z
Link: https://onthespectrumontheguestlist.substack.com/p/prologue?s=w

This is the prologue from a lengthy piece I'll be serializing over the next few months. Yes, the title is a little is a little clickbaity, and superficially, a collection of stories about going clubbing in New York City isn't quite the standard rationalist fare.

But at its core are questions about the philosophy of (social) science, the value of science being objective/systematic/detached versus embracing the idiosyncratic talent of individuals, and the merits of method versus anarchy in one's work (a later chapter even includes a tounge-in-cheek rant about Against Method), all with a novel framing I think LessWrong might find worthwhile.

This started as a review—it quickly veered off-course and became something else entirely, but that’s how it started—of Very Important People: Status and Beauty on the Global Party Circuit. My favorite book reviews typically waste little time before they crack their knuckles, don their p-hats, and set off spelunking through the stats. Something about the rigor of it all soothes me.

But when it comes to quantitatives, Very Important People cares about just four: height (ideally, north of six feet in heels), weight (conspicuously, if not ludicrously thin), age (certainly less than thirty, preferably less than twenty-five), and facial attractiveness (left as an exercise to the reader).

That’s me. Hello.

So in the name of qualitative research, I found my sample-size-of-one self in a Manhattan restaurant one evening last November, accepting a fruity drink nestled in an inflatable flamingo koozie as I sat down at a long table of like-bodied women, bracing myself for the global, the party, and the circuit.

Tonight, and in the nights that follow, I, unlikeliest of party animals, will venture to explore terra luxuriosus. And if there’s any fault I come to find in the book, it’s this: by nature, nothing in nightlife is amenable to the rigors of sociology. There can be no impedance mismatch between analysis and subject; any true knowledge of this world will be neither objective nor subjective, but interactive:

If you cut things open, you can’t find the meaning inside. It doesn’t ooze out. So maybe meaningfulness is more like a fluid that suffuses objects. If you soak a sponge cake in rum, that invisible essence pervades the dessert, and you can’t specifically locate it—although you can taste it.

Where abstraction fails, enumeration must ...
An Agent Based Consciousness Model (unfortunately it's not computable)
by Logan Zoellner — 2022-05-21T23:00:47.417Z

This post is a response to Scott Aaronson's "Why I Am Not An Integrated Information Theorist (or, The Unconscious Expander)".

He proposes the following "test" for any theory of consciousness:

The test of such a theory is whether it can produce results agreeing with “commonsense intuition”: for example, whether it can affirm, from first principles, that (most) humans are conscious; that dogs and horses are also conscious but less so; that rocks, livers, bacteria colonies, and existing digital computers are not conscious (or are hardly conscious); and that a room full of people has no “mega-consciousness” over and above the consciousnesses of the individuals.

In this post, I will describe an Agent Based Consciousness model, and consider how well it passes Aaronson's "test".

Intuitive Model

The intuition behind my Agent Based Consciousness Model is the following claim. In order to interact with the world, agents build models of the world that allow them to make predictions about the results of their interactions. Consciousness is directly related to the complexity and predictive accuracy of those models. A being with a very simple model of the world (or one that makes poor predictions) is less "conscious" of the world around it than one with a more detailed model (that it uses to make accurate predictions).

Note that this definition explicitly excludes self-consciousness. A being which does not interact with or predict the world around it is defined as unconscious regardless of how rich its inner mental life is. This can be fixed (by partitioning an agent from its observations about itself), or it can be accepted as a weakness of this model.

This model also focuses on the size complexity of the modeled agent, not its computational complexity or processing speed. While we might intuitively think that a human who thinks twice as fast is twice as conscious, this model does not take that effect into account. It may be helpful to think of this as a qualitative measure of consciousness rather than a quantitative once. For example, Ents think much more slowly than human beings, but their thoughts are not any less rich as a result.

Finally, this model is pan-psychic. It attributes a Consciousness Score to all agents and is permissive about what can be defined as an "agent". It may be helpful to think of ...
Wormy the Worm
by River Lewis — 2022-05-21T15:34:12.866Z
Link: https://heytraveler.substack.com/p/wormy-the-worm

I.

Planarian worms are perfect regenerators. That is, they’re immortal. If you cut off a chunk of one, it will grow back, and the worm will continue as before.

This also means that a large enough chunk will grow into a separate worm. And in fact, some species of planaria reproduce this way, instead of sexually. They split in half, and each half becomes a full worm again.

So, a question: if Wormy the planarian splits in half and becomes two new worms, which one is Wormy?

You can’t just pick one, since they share equal amounts of flesh with Wormy. You could argue that Wormy is dead and that these are his children. But what if only one of the worms survived the process? Wouldn’t it seem odd to say he wasn’t Wormy? Or, a similar case: does a salamander who loses and regrows her tail become her own child? No.

If Wormy isn’t simply dead, and neither of the new worms is the true Wormy, we’re left with only one possibility: both worms are Wormy.

II.

This seems reasonable, at first glance. Wormy is the name of a continuous process, which starts as a single worm and becomes two.

But what about Wormy’s predecessor? Let’s call him Wormy Prime; the little guy who split in half to produce Wormy and some other forgotten twin. Is he also part of that process? The answer can only be yes.

Now we’ve created a monster. Because the same applies to Wormy Prime’s predecessor, and the eventual successors of the newly-split Wormy. By our prior logic, Wormy is the name of a continuous process, which once was a single worm and now is many millions.

Okay, fine, let’s bite this bullet: Wormy is the name for this entire species of planaria. He’s a single entity who happens to have many spatially distinct parts. Similar to the way our original one-worm Wormy was composed of many cells.

III.

But what about other species of planaria? At some point, they had a common ancestor with Wormy. For the asexual varieties, whatever distinctions we see as dividing them happened as a result of random mutations during this splitting/cloning process. Are they not Wormy?

This would seem somewhat arbitrary. If one of Wormy’s halves were to be struck by radiation and slightly mutated while regrowing, we wouldn’t say that he suddenly becomes severed from his Wormy-ness.

Similarly, we can return to our salamander friend: if she were mutated by radiation while regrowing her tail, we wouldn’t say she becomes a new salamander...
Philosophical considerations of cessation of brain activity
by clockwork — 2022-05-21T15:18:01.992Z

I'm unfamiliar with the philosophies of personal identity. Which theories would postulate that a total interruption of consciousness/neural activity (e.g., a coma), but where the brain itself is completely undamaged, would be "death", in the sense of the person before the coma wouldn't be able to feel what happens after it?

Reason is I need to make a decision about elective surgery under general anesthesia imminently. I'm concerned about the possibility that from my current perspective I will die as I'm put under even though from everyone else's perspective I'll wake up all the same, as I would be "rebooted" into a new "session" of consciousness and my current session won't be able to access/experience what happens in the new one the same way I can feel what happens to me 5 minutes from now. Of course this may happen every night during sleep. However, the risk is much greater under general anesthesia because of the much more complete loss of activity and information processing much like a coma, e.g. even during the deepest stage of sleep perhaps only 1 brain hemisphere sleeps at a time. Hence a coma being a much better proxy for this question: if sleep is okay anesthesia may not be, but if a coma's okay it definitely is too.

I realize LWers are broadly on board with cryonics and thus unconcerned with this, but I'd still like to know which specific theories are more in line with my intuitive concerns.
How Big Are Covid Particles?
by jefftk — 2022-05-21T14:00:03.931Z

To assess what sort of filtration you might want for reducing covid
risk, it would be useful to know how the different sizes of aerosol
particles contribute to covid spread. Unfortunately, we don't know
that much here.

We know that sars-cov-2 itself is ~0.1µm, but it's probably not
transmitted as bare virus (Azimi and
Stephens 2013):

Some researchers have assumed that the individual virus or bacteria
particles are aerosolized and exist suspended as individual organisms;
however, it is likely more appropriate to consider the particles as
larger expelled droplets that contain aggregates of the smaller
infectious particles within.

We also know that there is significant
transmission from droplets well under
100µm, since larger ones generally quickly settle out of the
air and there's lots of evidence of sars-cov-2 lingering in poorly
ventilated spaces.

If you search [what size are sars-cov-2 particles], Google will give
you Lee
2020, but I don't think it's useful:

In the case of maximum viral-loading derived from experimental data of
COVID-19 patients, 8.97 x 10-5% of a respiratory fluid
particle from a COVID-19 patient is occupied by SARS-CoV-2. Hence, the
minimum size of a respiratory particle that can contain SARS-CoV-2 is
calculated to be approximately 9.3 µm. The minimum size of the
particles can decrease due to the evaporation of water on the particle
surfaces.

Stepping through their calculation, first they looked at what fraction
of an infected person's saliva is sars-cov-2: typically 3e-7% but a maximum
of 9e-5%. Then they ask how big a particle would have to be to
contain at least one sars-cov-2 viron, and get 65µm for 3e-7% and
9.3µm for 9e-5%. If you generate a respiratory particle of size
1µm then yes, that's not big enough to contain both sars-cov-2 and
everything else in your spit in represenative proportions, and they
conclude that this means particles of that size can't contain sars-cov-2.
Instead, all this tells us is that only ~0.1% of those particles would
contain sars-cov-2.

They acknowledge this ("viruses can be distributed nonhomogeneously
inside the respiratory fluid particles; therefore, the minimum size
can vary owing to the clustering of viruses in fluid particles") but
they don't take it seriously enough to conclude that their whole
estimation process doesn't work.

Much better would be to measur...
[Short version] Information Loss --> Basin flatness
by Vivek Hebbar — 2022-05-21T12:59:10.932Z

This is an overview for advanced readers. Main post: Information Loss --> Basin flatness

Summary:

Inductive bias is related to, among other things:

Basin flatness
Which solution manifolds (manifolds of zero loss) are higher dimensional than others. This is closely related to "basin flatness", since each dimension of the manifold is a direction of zero curvature.

In relation to basin flatness and manifold dimension:

It is useful to consider the "behavioral gradients" ∇θf(θ,xi) for each input.
Let G be the matrix of behavioral gradients. (The ith column of G is gi=∇θf(θ,xi)).[1] We can show that dim(manifold)≤N−Rank(G).[2]
Rank(Hessian)=Rank(G).[3][4]
Flat basin ≈ Low-rank Hessian = Low-rank G ≈ High manifold dimension
High manifold dimension ≈ Low-rank G = Linear dependence of behavioral gradients
A case study in a very small neural network shows that "information loss" is a good qualitative interpretation of this linear dependence.
Models that throw away enough information about the input in early layers are guaranteed to live on particularly high-dimensional manifolds. Precise bounds seem easily derivable and might be given in a future post.

See the main post for details.

^

In standard terminology, G is the Jacobian of the concatenation of all outputs, w.r.t. the parameters.

^

N is the number of parameters in the model. See claims 1 and 2 here for a proof sketch.

^

Proof sketch for Rank(Hessian)=Rank(G):

span(g1,..,gk)⊥ is the set of directions in which the output is not first-order sensitive to parameter change. Its dimensionality is N−rank(G).
At a local minimum, first-order sensitivity of behavior translates to second-order sensitivity of loss.
So span(g1,..,gk)⊥ is the null space of the Hessian.
So rank(Hessian)=N−(N−rank(G))=rank(G)

^

There is an alternate proof going through the result Hessian=2GGT. (The constant 2 depends on MSE loss.)
Information Loss --> Basin flatness
by Vivek Hebbar — 2022-05-21T12:58:19.908Z

This work was done under the mentorship of Evan Hubinger through the SERI MATS program. Thanks to Lucius Bushnaq, John Wentworth, Quintin Pope, and Peter Barnett for useful feedback and suggestions.

In this theory, the main proximate cause of flat basins is a type of information loss. Its relationship with circuit complexity and Kolmogorov complexity is currently unknown to me.[1] In this post, I will demonstrate that:

High-dimensional solution manifolds are caused by linear dependence between the "behavioral gradients" for different inputs.
This linear dependence is usually caused when networks throw away information which distinguishes different training inputs. It is more likely to occur when the information is thrown away early or by RELU.

Overview for advanced readers: [Short version] Information Loss --> Basin flatness

Behavior manifolds

Suppose we have a regression task with 1-dimensional labels and k training examples. Let us take an overparameterized network with N parameters. Every model in parameter space is part of a manifold, where every point on that manifold has identical behavior on the training set. These manifolds are usually[2] at least N−k dimensional, but some are higher dimensional than this. I will call these manifolds "behavior manifolds", since points on the same manifold have the same behavior (on the training set, not on all possible inputs).

We can visualize the existence of “behavior manifolds” by starting with a blank parameter space, then adding contour planes for each training example. Before we add any contour planes, the entire parameter space is a single manifold, with “identical behavior” on the null set. First, let us add the contour planes for input 1:

Each plane here is an n-1 dimensional manifold, where every model on that plane has the same output on input 1. They slice parameter space into n-1 dimensional regions. Each of these regions is an equivalence class of functions, which all behave about the same on input 1.

Next, we can add contour planes for input 2:

When we put them together, they look like this:

Together, the contours slice parameter space into n-2 dimensional regions. Each “diamond” in the picture is the cross-section of a tube-like region which extends vertically, in the direc...
The Future of Nuclear War
by avturchin — 2022-05-21T07:52:34.257Z

Here I present a view on the future of nuclear war which takes into account the expected technological progress as well as global political changes.

There are three main directions in which technological progress in nuclear weapons may happen:

1) Many gigaton weapons.

2) Cheaper nuclear bombs which are based on the use of the reactor-grade plutonium, laser isotope separation or are hypothetical pure fusion weapons. Also, advanced nanotechnology will provide the ability to quickly build large nuclear arsenals and AI could be used in designing, manufacturing and nuclear strategy planning.

3) Specialized nuclear weapons like nuclear-powered space lasers, hafnium bombs and nuclear-powered space ships as kinetic weapons.

Meanwhile, the nuclear war strategy also has changed as the bipolar world has ended and as new types of weapons are becoming available.

The first strategy change is that Doomsday weapons for blackmail will become attractive for weaker countries which can’t use ICBM to penetrate the anti-missile defense of the enemies.

Secondly, the cheaper nukes will become available to smaller actors, who may be involved in “worldwide guerilla”. Cheaper nukes and a larger number of actors also encourage regional nuclear wars, nuclear terrorism and anonymous or false-flag warfare. This will result in disruption of social complexity and global food production shortages because of effects on climate.

The third change of the strategy is the use of nukes not against primary military targets but on other objects, which could increase their effects: nuclear power plants, supervolcanos, EMP, tsunamis, taiga fires for global cooling, and even more hypothetical things like asteroid deflection to Earth.

All these unconventional means and strategies of nuclear war could become available somewhere in the 21st century and may cause or contribute to the civilizational collapse and even human extinction.

Permalink: https://philpapers.org/rec/TURTFO-14

Introduction

When we speak about global catastrophic risks connected with new technologies, we assume that in the decades from now these technologies will advance by many orders of magnitude. AI will become much stronger, nanobots will appear and biohackers will get access to better manufacturing capabilities. But the discussions about nuclear war assumed that this war will be the same as it was...
Clarifying what ELK is trying to achieve
by Towards_Keeperhood — 2022-05-21T07:34:13.347Z

(Basic familiarity with ELK required.)

I noticed that some people who haven’t thought that much about ELK yet (including my former self) are confused about (1) why the ELK setup is structured how it is structured and what we want to achieve, and relatedly (2) what's the underlying problem behind the "human simulator".

TL;DR: The problem with the "human simulator" isn't that the AI wants to "look good to a human" instead of "correctly answering questions". The "human simulator" is just one specific example where the reporter learns to reason on its own, instead of looking at the predictor. When a reporter reasons on its own from answering human-level questions, its capabilities won't generalize to harder questions. This is why we need a predictor with a superhuman world model.
The primary goal of ELK isn’t that we want to train a specific objective into the reporter, we want the reporter to have a certain structure, the direct-translator structure.

(EDIT: As I'm using the term "human simulator", I mean the class of AIs that reason very similar to humans. It is possible that other people rather mean the broader class of self-reasoning reporters, even if the reporter reasons with different concepts than humans, but I'm not sure. Let me know if most other people (or just Paul) use the term "human simulator" consistently different than me.)

The confusion: Thinking the problem is what the reporter wants

As said, I think some people mistakenly think the difficulty of getting a "direct translator" vs a "human simulator", is that the AI may learn "look good to humans" rather than "answer questions honestly" as its objective, and that for some reason the former is fundamentally more likely by default than the latter.

That isn't right. It's not that the human simulator wants to look good to humans, it just reasons on its own to find answers, similar to how humans reason.

The problem is that in the ELK setup this kind of reporter reasoning for itself may be much simpler than finding the correct way to extract answers from the predictor.

While making sure that the AI doesn't just learn to "look good to humans" as its objective is a very important and difficult problem in AI safety, and possibly (though not necessarily) a problem we still need to solve for ELK, it isn't the main problem we are trying to solve right now.

But we aren’t primarily trying to train the reporter...
[retracted] A really simplistic experiment for LessWrong and /r/SneerClub
by Nicholas / Heather Kross — 2022-05-21T05:52:28.796Z

In this thread: What's a key thing you would tell /r/SneerClub users, to try and bridge the "worldview gap"?

I am talking about extremely-basic background / shared assumptions, that you really wish the other people had. What would you say to them, to plant the seed of changing their mindset from their current one?

Diagrams, stories, and hokey analogies encouraged. Condescension completely allowed. (The more basic and obvious the worldview difference, the more it needs to be written down).

No jargon.

Top-level comments should be cringe-inducingly earnest.

This question is mirrored and inverted on /r/SneerClub.