trentmkelly/LessWrong-43k huggingface dataset:
A Pin and a Balloon: Anthropic Fragility Increases Chances of Runaway Global Warming 
by avturchin — 2022-09-11T10:25:40.707Z

Alexey Turchin, alexeiturchin@gmail.com

Anonymous coauthor 1

Anonymous coauthor 2

TL;DR: Because of survival bias we underestimate the location of climate tipping points and thus our world is more fragile to small anthropogenic actions. Therefore, human extinction because of runaway global warming is more probable.

Abstract: Humanity may underestimate the rate of natural global catastrophes because of the survival bias (“anthropic shadow”). But the resulting reduction of the Earth’s future habitability duration is not very large in most plausible cases (1-2 orders of magnitude) and thus it looks like we still have at least millions of years.

However, anthropic shadow implies anthropic fragility: we are more likely to live in a world where a sterilizing catastrophe is long overdue and could be triggered by unexpectedly small human actions. In the same way, an over-inflated toy balloon, which will soon burst, is very fragile.

Anthropic fragility can manifest itself in the higher chances of runaway global warming. It has often been suggested that the Earth's atmosphere remained life-supporting for billions of years by sheer chance. Therefore, the survival bias can be strong. It is also known that Earth-like water worlds could experience transitions into deadly moisture greenhouse (mean T = 65C). This means that relatively small anthropogenic actions could put the climate above an unpredictable tipping point, which could lead to the moisture greenhouse. Thus, it is necessary to carry out urgent geoengineering studies and prepare to prevent an unexpected climate catastrophe.

There are three main counterarguments against the existence of the anthropic shadow: self-indication assumption (SIA), past observers and the Gaia hypothesis; we show that they fail. It was proposed that SIA exactly compensates the anthropic shadow as an observer unlikely to find herself in a world with a strong anthropic shadow; however, there is a baseline level of the anthropic shadow for all habitable planets, similar to the rate of evolutionary transitions like abiogenesis. There are no “past observers” as qualified observers appeared only 50 years ago. Gaia hypothesis assumes existence of self-stabilizing feedback in climate, but new types of events like quick CO2 growth could override its coping ability.

We present a list of other catastrophes that may have been u...
Is there an Ultimate text editor? 
by Johannes C. Mayer — 2022-09-11T09:19:51.436Z

Is there a text editor that allows me to

enter normal text,

latex that is rendered in line, but is derendered when my cursor is over the latex,

draw images in line directly in the text editor, e.g. using a drawing tablet,

has headings like emacs org mode, that you can collapse the headings and nest them arbitrarily deep, and you can rearrange the headings. For example if you think of the headings as trees, we can move an entire subtree of headings at a certain level of depth before or after leaf notes in the tree, e.g. that exists a shortcut, such that I can turn this tree

item 1
-item 2

item 3

into this one

item 3

item 1
-item 2

Ideally the program would also have a mobile app, where you can edit your notes from.

vim mode would be nice too

being a zettlekasten like roam research would be nice

Here a program so I already know which don't satisfy all of the requirements. Especially the inline image drawing one.

obsidian

emacs org mode

roam research

one note

So far nothing seems better than org mode.

[Edit: 2024-09-20]

After using obsidian for 2 years I switched back to Emacs. I made it such that I can use typst. I preview documents in the browser, so my documents can in principle contain anything that the browser can display (or run!).

To figure out things I first write things on a whiteboard, before writing down text.
Pascal: The Greatness and Littleness of Man, A Thinking Reed
by NoBadCake — 2022-09-10T20:05:15.377Z

Excerpt from Thoughts(Pensées), Blaise Pascal, c.1662(published later, posthumously)

The greatness of man consists in thought.
A thinking reed. — Not from space must I seek my dignity,
but from the ruling of my thought. I should have no more if I
possessed whole worlds. By space the Universe encompasses
and swallows me as an atom, by thought I encompass it.
Man is but a reed, weakest in nature, but a reed which thinks.
It needs not that the whole Universe should arm to crush him.
A vapour, a drop of water is enough to kill him. But were the
Universe to crush him, man would still be more noble than that
which has slain him, because he knows that he dies, and that
the Universe has the better of him. The Universe knows nothing
of this.
All our dignity therefore consists in thought. By this must we
raise ourselves, not by space or duration which we cannot fill.
Then let us make it our study to think well, for this is the
starting-point of morals.
[Job] Project Manager: Community Health (CEA)
by Xodarap — 2022-09-10T18:40:33.889Z
Link: https://www.centreforeffectivealtruism.org/careers/project-manager-community-health

With recent media attention, increased funding, and growing ambition among community members, this is one of the most exciting times for EA, but also one of the riskiest. We need an ops-minded generalist to help address these risks, through both end-to-end ownership of targeted projects as well as building broader systems and processes to support other team members.

Example projects you might own:

Create and manage a fund for community members’ physical and mental health
Categorize the EA community into useful segments and conduct interviews to discover and understand their challenges, then summarize this feedback for stakeholders
Organize a retreat for team members to discuss and work on key problems in the EA community

If this sounds like you, please apply!
Unbounded utility functions and precommitment
by MichaelStJules — 2022-09-10T16:16:22.254Z

According to orthodox expected utility theory, the boundedness of the utility function follows from standard decision-theoretic assumptions, like Savage's fairly weak axioms or the von Neumann-Morgenstern continuity/the Archimedean property axiom. Unbounded expected utility maximization violates the sure-thing principle, is vulnerable to Dutch books and is vulnerable to money pumps, all plausibly irrational. See, for example, Paul Christiano's comment with St. Petersburg lotteries (and my response). So, it's pretty plausible that unbounded expected utility maximization is just inevitably formally irrational.

However, I'm not totally sure, since there are some parallels to Newcomb's problem and Parfit's hitchhiker: you'd like to precommit to following a rule ahead of time that leads to the best prospects, but once some event happens, you'd like to break the rule and maximize local value greedily instead. But breaking the rule means you'll end up with worse prospects over the whole sequence of events than if you had followed it. The rules are:

Newcomb's problem: taking the one box
Parfit's hitchhiker: paying back the driver
Christiano's St. Petersburg lotteries: sticking with the best St. Petersburg lottery offered

So, rather than necessarily undermining unbounded expected utility maximization, maybe this is just a problem for "local" expected utility maximization, since there are other reasons you want to be able to precommit to rules, even if you expect to want to be able to break them later. Having to make precommitments shouldn't be decisive against a decision theory.

Still, it seems better to avoid precommitments when possible because they're messy, risky and ad hoc. Bounded utility functions seem like a safer and cleaner solution here; we get a formal proof that they work in idealized scenarios. I also don't even know if precommitments generally solve unbounded utility functions' apparent violations of decision-theoretic principles that bounded utility functions don't have; I may be generalizing too much from one case.
What is the "Less Wrong" approved acronym for 1984-risk?
by Logan Zoellner — 2022-09-10T14:38:39.006Z

I suspect the terminology we use shapes the discussions we have.

We have X-Risk for extinction, and S-risk for Roko's basilisk torturing us all.

What is the common way of referring to the thing we were all terrified of in the late 20th century, namely that corporations and government would finally get together and work out a way to oppress everyone for all of time.

I want a universally comprehensible way of writing sentences like "Stable Diffusion increases X-risk by accelerating AI development, but decreases 1984-Risk by making it less likely that AI will be centrally controlled".
Find out how utilitarian you are - a mega thread of philosophy polls
by spencerg — 2022-09-10T14:05:35.600Z
Link: https://twitter.com/SpencrGreenberg/status/1568595511522852871

I created a mega thread of philosophy polls to help you find out how utilitarian you are:

https://twitter.com/SpencrGreenberg/status/1568595511522852871

It includes thought experiments related to:

Equality
Delusion
Preferences
The Repugnant Conclusion
Love
Theft
Shutting up and multiplying
Complexity
Abortion
Heaven lottery
Avoiding hell
Longevity
Animal suffering
Replacing beings
Put Dirty Dishes in the Dishwasher
by jefftk — 2022-09-10T13:10:03.565Z

"It's not worth using the dishwasher: by the time you've
gotten your dishes clean enough to put in, you might as well just
finish washing them yourself."

People have a wide range of ways they deal with dishes before putting
them in the dishwasher:

Rinse well. The dishes should look clean before
they go in.

Rinse lightly. It's fine if the dishes don't look clean, but
they should be close.

Scrape well. There shouldn't be anything on the dishes that
further scraping would remove.

Scrape lightly. Get most of the food off, but it's fine if
some stuff remains.

Do nothing. Just dump the whole thing in there, food and
all. [Not recommended.]

Earlier options are more work, give you cleaner dishes, and mean you
need to manually clean the dishwasher's filter less often. Later are the reverse.

In my experience, people tend to be too far towards the early end for
their particular dishwasher and time-cleanliness preferences. This
makes sense: if you're too far toward the later end you notice and
adjust, while in the other direction there's no signal that you could
be doing better. Try a later option; you could be putting
effort into pre-cleaning for minimal impact on your dishes. It's
a cheap experiment to run: there's a good chance you're happy, and if
not you can just re-run this load and go back to more thorough
pre-cleaning.

Personally, I'm pretty happy with #4. It's very little work, and
usually all the dishes come out clean. When one is still dirty, I
just include it in the next load after scraping a bit. If dishes
come out unusually dirty it's generally because something was loaded
wrong (too tall on the bottom rack, poking through the middle rack)
and prevented the middle arm from spinning.

Comment via: facebook
Join ASAP! (AI Safety Accountability Programme) 🚀
by CallumMcDougall — 2022-09-10T11:15:48.589Z

Edit - this group was an experiment which I consider to have mostly been unsuccessful, and the group is no longer very active. I think the two main reasons for this were: (1) lacking a single thing which people were congregating around (e.g. the TransformerLens library in the case of the Open Source Mech Interp Slack group, or the ARENA course material in the case of the ARENA Slack), and (2) top-down rather than bottom-up design of the Slack group and its features.

TL;DR

I'm creating a Slack group for people who are interested in working in AI safety at some point in the future, but who aren't working on it right now[1], and would like extra accountability and motivation while they pursue their goals.

Join with this link!

Why am I creating this?

I just spent an awesome summer in Berkeley doing MLAB, surrounded by people who are really passionate about AI safety, and this definitely had a positive impact on my level of motivation. I think trying to recreate some (even much weaker) version of that would be really valuable. An accountability system is the most basic version of this, because making commitments to other people is a really nice way of motivating yourself to get shit done!

I've spoken to a few people from MLAB, and several seem to agree (at least five participants have mentioned to me that they'd like to join a group like this).

How will this work?

(Note - this might all be changed depending on how many people join, and their suggestions & preferences. Hopefully by the end of next week, the group will be larger and we will have made many improvements to this basic design!)

The core mechanism of the group will be everyone posting regular short updates (might be Slack message, or filling out a Google Form)[2] (maybe once per 2 weeks) summarising what they've done over that period. For instance:

Books you've read, or courses you've taken, or progress in structured self-study like this
Blog posts you've written
Projects you've done, or are doing, and your progress on them
Companies or other opportunities you've applied for
Dank AI memes you've designed

There will also be optional extra commitment mechanisms like weekly Zoom calls. Also if enough people join (e.g. more than 6) then we'll probably divide people into smaller groups for personal check-ins, since larger groups tend to lead to individuals feeling less accountability. Progress reports will s...
Quintin's alignment papers roundup - week 1
by Quintin Pope — 2022-09-10T06:39:01.773Z

Introduction

I've decided to start a weekly roundup of papers that seem relevant to alignment, focusing on papers or approaches that might be new to safety researchers. Unlike the Alignment Newsletter, I'll be spending relatively little effort on summarizing the papers. I'll just link them, copy their abstracts, and potentially describe some of my thoughts on how the paper relates to alignment. Hopefully, this will let me keep to a weekly schedule.

The purpose of this series isn't so much to share insights directly with the reader, but instead to make them aware of already existing research that may be relevant to the reader's own research.

Papers

Locating and Editing Factual Associations in GPT

We analyze the storage and recall of factual associations in autoregressive transformer language models, finding evidence that these associations correspond to localized, directly-editable computations. We first develop a causal intervention for identifying neuron activations that are decisive in a model's factual predictions. This reveals a distinct set of steps in middle-layer feed-forward modules that mediate factual predictions while processing subject tokens. To test our hypothesis that these computations correspond to factual association recall, we modify feed-forward weights to update specific factual associations using Rank-One Model Editing (ROME). We find that ROME is effective on a standard zero-shot relation extraction (zsRE) model-editing task, comparable to existing methods. To perform a more sensitive evaluation, we also evaluate ROME on a new dataset of counterfactual assertions, on which it simultaneously maintains both specificity and generalization, whereas other methods sacrifice one or another. Our results confirm an important role for mid-layer feed-forward modules in storing factual associations and suggest that direct manipulation of computational mechanisms may be a feasible approach for model editing. The code, dataset, visualizations, and an interactive demo notebook are available at this https URL

My opinion:

Most people I talk to about this paper have heard of it previously, so it's hardly ”new”. However, I think a lot of people underestimate how significant the paper is. The authors use a very cool interpretability method to show that the middle-stage MLP layers are acting as a key-value memory system. They then guess at the specific mathematic...
Path dependence in ML inductive biases
by Vivek Hebbar — 2022-09-10T01:38:22.885Z

In this post, we define path dependence as the sensitivity of a model's behavior to the details of the training process and training dynamics.[1] High path-dependence indicates that small changes to the training process can cause significant changes to how the final model generalizes (such as the details of off-distribution behavior). It implies that inner alignment can be reasoned about by thinking about what the model looks like at various stages of training, and how its structure is affected by the immediate pressures of gradient descent. It implies that early-training interventions can be quite potent in shaping how a model turns out, and that a proper theory of inductive bias must reason about the order in which features are learned (where features learned faster/earlier can “screen off” the need for other implementations of a similar niche, in a way that affects the final model).

In contrast, a world with low path dependence allows us to reason about inductive bias in terms of priors and updates, sparing the details of training dynamics. It is more pessimistic about the ability to steer the model’s ontology through early interventions, believing instead that the final result is overdetermined. As Evan discusses in a previous post, it makes us less worried about variance in alignment outcomes between labs, since small changes to the training procedure don’t strongly affect alignment outcomes.

Possible mechanistic reasons for high path dependence would include the existence of distinct stable ontologies, the ability for early features to kill gradients, and the difficulty of building highly serial features whose components aren't independently useful. Mechanistic reasons for low path dependence would include grokking-like phase transitions which wipe out early circuits, overdetermination of correct ontologies, and an abundance of low loss paths between seemingly dissimilar solutions.[2]

We remain mostly agnostic about which world we are in. The purpose of this post is to elucidate the various strands of path dependence, and their implications for alignment. We hope it will encourage people to run experiments determining where reality falls on these spectra.

Path dependence

Fundamentally, path dependence is about the sensitivity of a model's behavior to the details of the training process and training dynam...
Keeping Time in Epoch Seconds
by Gordon Seidoh Worley — 2022-09-10T00:28:08.137Z

Unix-like computers keep time in seconds since the Unix Epoch—the 0 second, which is set to midnight, January 1st, 1970, Zulu/UTC. We call a particular second since the Unix Epoch a timestamp.

Keeping time this way is pretty handy, but looking at a long string of numbers is hard to parse. What to do?

Right now, as I'm typing, the current Unix timestamp is 16'62'7'68'531. See how I put those ' in there? That's how I try to break up the number into parts to make sense of it.

Starting from the right, the two rightmost digits make up what I think of roughly as a Unix minute, but rather than 60 seconds it's 100, so that's 123 standard minutes. One more unit over and we get the Unix hour, which is about a quarter of a standard hour long at 1000 seconds or 1623 standard minutes.

Calling 1000 seconds an hour might seem confusing, but the purpose of hours is to have a convenient unit for dividing up days, just like minutes are a convenient way of dividing hours. Speaking of which, 100 Unix hours equates to 1 Unix day, or 2779 standard hours. This is a bit longer than a standard day, but since some people seem to operate on 28 hour days anyway, they might find this appealing.

So that takes care of the 5 rightmost digits: 3 to track the seconds and minutes within an hour and 2 to track the hour within a day. The 6th digit from the right tracks the day of the Unix week, which is 10 Unix days, or 113154 standard days. It's a bit longer than our standard 7 day lunar weeks, but 10 day weeks were good enough for the ancient Greeks, so they're good enough for Unix nerds.

Since months are tied to the cycle of the moon there are no real months in Unix time. Instead 100 weeks make up a Unix year, which is approximately 3.17 standard years long. If it helps, though, a tenth of a Unix year is about three and a half months long, so you can think of each ten-week as like the three month quarters or seasons we informally divide our calendar by.

Now we've accounted for all the digits. If we break down my example timestamp of 16'62'7'68'531, we can say that it was the 31st second of the 5th minute of the 68th hour of the 7th day of the 62nd week of the 16th year of the Unix Epoch. Breaking it down in a table:

16	62	7	68	531
year	week	day	hour	minute and seconds

Do I expect this to catch on? Not really. This is all just a bit of fun to make sense of Unix timestamp...
Ought will host a factored cognition “Lab Meeting”
by jungofthewon — 2022-09-09T23:46:08.412Z

Ought will host a factored cognition “Lab Meeting” on Friday September 16 from 9:30AM - 10:30AM PT.

We'll share the progress we've made using language models to decompose reasoning tasks into subtasks that are easier to perform and evaluate. This is part of our work on

supervising process, not outcomes.

It’s easier for us to show you than to tell you about it in a post (though written updates will hopefully follow).

Then, we'll cover outstanding research directions we see and plan to work on, many almost shovel-ready. If the alignment community can parallelize this work across different alignment research teams, we can make progress faster. We'd love to coordinate with other alignment researchers thinking about task decomposition, process supervision, factored cognition, and IDA-like approaches (where efficient to do so). We want to save you time and mistakes if we can!

What is the agenda?

30 min | Updates on Ought’s work decomposing reasoning tasks

The specific alignment problems we’re trying to solve and our vision of a solution that mitigates these risks (see also:

Supervise Process, not Outcomes

)
Early progress on decomposing reasoning about evidence quality in randomized controlled trials
Our tools for building and debugging reasoning traces of language models (

preview

)

15 min | Related research directions we’re excited about and how they fit in, e.g.

Automating evaluation through critique models or verifier models
Distillation
Comparing scaling trends for process-based systems vs. end-to-end systems
Testing process-based systems for adversarial robustness

15 min | Q&A

There will be more to discuss than we can fit into an hour. We’ll get to what we can and consider making this a regular meeting if there’s appetite (likely with more sharing from other researchers)!

Who should attend?

You should attend if:

You are interested in Ought’s research and want updates.
You want to build off of Ought’s learnings from doing this research.
You want to use our tools for running and debugging compositional language models tasks.
You want concrete research ideas in this domain.
You are not a researcher but want to learn how other backgrounds can support this work (engineers can build debugging infrastructure, non-ML researchers can help create datasets, etc.).

How can I attend?

You can register for the Lab Meeting

here

. Emai...