trentmkelly/LessWrong-43k huggingface dataset:
Elementary Infra-Bayesianism
by Jan — 2022-05-08T12:23:00.192Z
Link: https://universalprior.substack.com/p/elementary-infra-bayesianism

TL;DR: I got nerd-sniped into working through some rather technical work in AI Safety. Here's my best guess of what is going on. Imprecise probabilities for handling catastrophic downside risk.

Short summary: I apply the updating equation from Infra-Bayesianism to a concrete example of an Infradistribution and illustrate the process. When we "care" a lot for things that are unlikely given what we've observed before, we get updates that are extremely sensitive to outliers.

I've

written previously

on how to act when confronted with something smarter than yourself. When in such a precarious situation, it is difficult to trust “the other”; they might dispense their wisdom in a way that steers you to their benefit.

In general

, we're screwed.

But there are ideas for a constrained set-up that

forces “the other” to explain itself and point out potential flaws in its arguments

. We might thus leverage “the other”'s ingenuity against itself by slowing down its reasoning to our pace. “The other” would no longer be an oracle with

prophecies that might or might not kill us

but instead a teacher who lets us see things we otherwise couldn't.

While that idea is nice, there is a severe flaw at its core:

obfuscation

. By making the argument sufficiently long and complicated, “the other” can sneak a false conclusion past our defenses. Forcing “the other” to lay out its reasoning, thus, is not a foolproof solution. But (

as some have argued

), it's unclear whether this will be a problem in practice.

Why am I bringing this up? No reason in particular.

Why Infra-Bayesianism?

Engaging with the work of Vanessa Kosoy is a rite of passage in the AI Safety space. Why is that?

The pessimist answer is that alignment is really, really difficult, and if you can't understand complicated math, you can't contribute.
The optimist take is that math is fun, and (a certain type of) person gets nerd sniped by this kind of thing.
The realist take naturally falls somewhere in between. Complicated math can be important and enjoyable. It's okay to have fun with it.

But being complicated is (in itself) not a mark of quality.

If you can't explain it, you don't understand it

. So here goes my attempt at "Elementary Infrabayesianism", where I motivate a portion of Infrabayesianism using pretty pictures and high school mathematics[1].

Uncertain updates

Imagine it's late in the night, the lights are off, and you are trying ...
Video and Transcript of Presentation on Existential Risk from Power-Seeking AI
by Joe Carlsmith — 2022-05-08T03:50:12.758Z

In March 2022, I gave a presentation about existential risk from power-seeking AI, as part a lecture series hosted by Harvard Effective Altruism. The presentation summarized my report on the topic. With permission from the organizers, I'm posting the video here, along with the transcript (lightly edited for clarity/concision) and the slides.

Main Talk

Thanks for having me, nice to be here, and thanks to everyone for coming. I'm Joe Carlsmith, I work at Open Philanthropy, and I'm going to be talking about the basic case, as I see it for, for getting worried about existential risk from artificial intelligence, where existential risk just refers to a risk of an event that could destroy the entire future and all of the potential for what the human species might do.

Plan

I'm going to discuss that basic case in two stages. First, I'm going to talk about what I see as the high-level backdrop picture that informs the more detailed arguments about this topic, and which structures and gives intuition for why one might get worried. And then I'm going to go into a more precise and detailed presentation of the argument as I see it -- and one that hopefully makes it easier to really pin down which claims are doing what work, where might I disagree and where could we make more progress in understanding this issue. And then we'll do some Q&A at the end.

I understand people in the audience might have different levels of exposure and understanding of these issues already. I'm going to be trying to go at a fairly from-scratch level.

And I should say: basically everything I'm going to be saying here is also in a report I wrote last year, in one form or another. I think that's linked on the lecture series website, and it's also on my website, josephcarlsmith.com. So if there's stuff we don't get to, or stuff you want to learn more about, I'd encourage you to check out that report, which has a lot more detail. And in general, there's going to be a decent amount of material here, so some of it I'm going to be mentioning as a thing we could talk more about. I won't always get into the nitty-gritty on everything I touch on.

High-level backdrop

Okay, so the high-level backdrop here, as I see it, consists of two central claims.

The first is just that intelligent agency is an extremely powerful force for transforming the world on purpose.
And because of that, cl

...
Algorithmic formalization of FDT?
by Shmi — 2022-05-08T01:36:10.778Z

I occasionally see a question like "what would FDT recommend in ....?" and I am puzzled that there is no formal algorithm to answer it. Instead humans ask other humans, and the answers are often different and subject to interpretation. This is rather disconcerting. For comparison, you don't ask a human what, say, a chessbot would do in a certain situation, you just run the bot. Similarly, it would be nice to have an "FDTbot" one can feed a decision theory problem to. Does something like that exist? If not, what are the obstacles?
Experience on Meloxicam
by jefftk — 2022-05-08T00:30:01.997Z

I continue to have wrist and hand issues, though they are mostly fine
as long as I minimize how much I use them. Which mostly means a lot
of dictation! Still, it would be good both to understand what is
wrong with them and figure out how I can resume some of the activities
I enjoy that I've stopped.

On 2022-03-04 I started taking 15mg of Meloxicam, daily with dinner.
This is a nonsteroidal anti-inflammatory (NSAID), which often reduces
arthritic swelling. I continued for a month, with my last dose on
2022-04-04. Unlike when I tried Methotrexate, I didn't
notice any changes either when I started or when I went off.

To understand whether there might be a more subtle effect, I also
checked my level of finger swelling each morning on waking by clasping my
hands and seeing how tight it felt. I rated this feeling on a scale
of 1 to 10, and all days were in the range 6 to 9 inclusive:

I take this to mean Meloxicam is very unlikely to reduce my swelling,
and may even slightly increase it.

Still trying to figure out what to try next. One candidate is to stop
restricting activity, let my wrists and hands get really bad, and
then repeat the bloodwork. The hope is that if there are markers that
are only elevated when my hands are agitated we might see them then.

Comment via: facebook
Best open-source textbooks (goal: make them collaborative)?
by shuffled-cantaloupe — 2022-05-07T23:40:03.993Z

I'm looking for online open-source / generously licensed textbooks, papers or tutorials.

Think of stuff like: http://neuralnetworksanddeeplearning.com/

Why? I'm currently running https://chimu.sh --a collaborative learning platform. Quick explanation: Chimu combines an e-reader with a Stack Overflow-like Q&A forum. As people read, they can view others' questions and ask their own. Demo here (desktop works best).

I need to seed the site with initial content, and I figured LW would be a great place to ask.

With this in mind, what are good online tutorials / textbooks that people here have learned from? Is there any book or paper that you wish that you could discuss with your friends?
What does Functional Decision Theory say to do in imperfect Newcomb situations?
by Daniel_Eth — 2022-05-07T22:26:33.469Z

Imagine a variation of Newcomb's problem where the Oracle is correct 99% of the time (say, half of people 1-box, half 2-box, and the Oracle is correct 99% of the time on each). What would FDT imply is the correct course of action then?
The glorious energy boost I've gotten by abstaining from coffee
by Sameerishere — 2022-05-07T20:21:29.816Z

Over the years, I've sometimes heard people rave about how they cut out caffeine (or just coffee) and have way more energy. My reaction has typically been to feel kind of dismissive / not really believe them / assume that means that they need to sleep more now, or have less flexibility in how much they sleep.

I kind of stumbled into cutting out coffee a few months ago, and it's just given me a lot of energy for free (i.e. without having to take more time for sleep or make any other non-trivial tradeoffs).

I wrote this up to remind myself why it's been super worth it for me, and decided to post it here as well in case it is helpful to others.

tl;dr:

Not drinking coffee is SUPER worth it for me.

On the same amount of sleep as before, I generally:

have more energy throughout the day
avoid what was previously a terrible late afternoon crash
feel less tense / anxious
often have energy into the night

Given that energy is one of the top, if not the top, constraint in how good I feel and how much I can get done of the stuff I want to do, this feels magical. It's well worth giving up the awesome jolt that coffee provides.

Previously I was drinking a fair amount of coffee:

(typically:

1 100 mg caffeine pill + 1-2 cups of coffee/day on weekdays, never after 4pm
typically 1 100 mg caffeine pill + 0-1 cups on weekends (when I got 9h of sleep)

Now I:

Drink 0-2 cups of English Breakfast tea a day (typically 1, sometimes 3)
Will drink tea after 4pm sometimes
Haven't had a cup of caffeinated coffee in weeks or months (I think I had half a cup, a month or two ago. I also had a decaf cappuccino a couple weeks ago).

How have things changed?

(Both before and after, I got:

7-7.5h of sleep on weekdays
9h of sleep on Fri and Sat
A 30 second blast of cold shower (typically at the end of the shower) almost always

This seems much more important now than it did before in terms of waking me up

Working out in the AM probably 4-5 mornings a week

The big benefits:

The very big ones:

I used to feel pretty terrible in the mid-late afternoon / early evening - exhausted in my body, such that it felt very difficult to do anything productive for 1-2 hours during that time. I often felt the need to take a nap if I hadn't gotten 7.5-8 hours of sleep at night. And even after the nap / break I felt very sluggish

I now still feel sleepy (and still end up taking a break for an hour much of the time, spanning meditating/napping an

...
Sealed predictions thread
by Zach Stein-Perlman — 2022-05-07T18:00:04.705Z

Want to publicly register predictions or other statements without yet revealing what they say? Do so here. The advantage of commenting here (in addition to, say, your Shortform) is that I'll record all comments here so they can't be deleted or edited later, and I may publicly shame you if you don't reveal your statement when you say you will.

So, comment with

the date by which (you say) you will reveal your statement, and
a hash or another system that will provide strong evidence that your statement is what you will say it is.

And, of course, save the statement. (If you use hashing and don't save your exact statement, you of course won't be able to prove it, so not only will your registration be useless, but also it will seem as if you post-facto decided to hide your statement.) If there are some circumstances in which you would not reveal the statement--e.g., if it could be infohazardous--strongly consider

explaining that in your comment, or
instead of sharing the hash (or whatever) of your statement, sharing the hash of a message that (1) explains why you might not share your statement and (2) includes or links to the hash of your real statement.

Otherwise, it will look bad if the date arrives and you don't reveal your statement.

To retract or edit a statement, send me a private message within 24 hours of commenting; after that, it cannot be changed in my record (although you can of course comment to say that you no longer endorse it, or to add hashes with new statements).

Fancy methods may exist if you want to provably commit to sharing your statement.

It's your responsibility not to share infohazards. This includes ensuring that any hashes you share can't be reversed, even if computing power advances more than you expect in the time before you reveal your statement, and ensuring that you don't commit to sharing something that, it might turn out, you should not share.

Edit: it seems there are multiple ways to handle verification, but for now, here's the simplest, I think: a sheet with most of the relevant information from comments. Thus information cannot be edited or deleted (without my complicity). And I guess now that there's a sheet, you can PM me rather than comment here to be added if you prefer.
Minimum Viable Alignment
by HunterJay — 2022-05-07T13:18:49.850Z

What is the largest possible target we could have for aligned AGI?

That is, instead of creating a great and prosperous future, is it possible that we can find an easier path to align an AGI by aiming for the entire set of 'this-is-fine' kind of futures?

For example, a future where all new computers are rendered inoperable by malicious software. Or a future where a mostly-inactive AI does nothing except prevent any superintelligence from forming, or that continuously tries to use up all over the available compute in the world.
I don't believe there is a solution here yet either, but could relaxing the problem from 'what we actually want' to 'anything we could live with' help? Has there been much work in this direction? Please let me know what to search for if so. Thank you.
[Linkpost] diffusion magnetizes manifolds (DALL-E 2 intuition building)
by Paul Bricman — 2022-05-07T11:01:55.967Z
Link: https://paulbricman.com/reflections/diffusion-magnetizes-manifolds

In trying to make sense of why diffusion works so well, I explored three complementary models related to magnetism, Hansel & Gretel, and landing airplanes.
scipy.optimize.curve_fit Is Awesome
by niplav — 2022-05-07T10:57:28.278Z

cross-posted from niplav.github.io

I recently learned about the python function scipy.optimize.curve_fit,
and I'm really happy I did.

It fulfills a need I didn't know I'd always had, but never
fulfilled: I often have a dataset and a function with some
parameters, and I just want the damn parameters to be fitted
to that dataset, even if imperfectly. Please don't ask any
more annoying questions like “Is the dataset generated by a
Gaussian?”
or “Is the underlying process
ergodic?”, just fit the
goddamn curve!

And scipy.optimize.curve_fit does exactly that!

You give it a function f with some parameters a, b, c, … and a
dataset consisting of input values x and output values y, and it
then optimizes a, b, c, … so that f(x, a, b, c, …) is as close as
possible to y (where, of course, x and y can both be numpy arrays).

This is awesome! I have some datapoints x, y and I believe it's
generated by some obscure function, let's say of the form
f(x,a,b,c)=a⋅x⋅sin(b⋅x+c), but I don't know
the exact values for a, b and c?

No problem! I just throw the whole thing into curve_fit
(scipy.optimize.curve_fit(f, x, y)) and out comes an array of optimal
values for a, b, c!

What if I then want c to be necessarily positive?

Trivial! curve_fit comes with an optional argument called bounds,
since b is the second argument, I call
scipy.optimize.curve_fit(f, x, y, bounds=([-numpy.inf, -numpy.inf, 0], numpy.inf)),
which says that curve_fit should not make the second argument smaller
than zero, but otherwise can do whatever it wants.

So far, I've already used this function
two
times,
and I've only known about it for a week! A must for every wannabe
data-scientist.

For more information about this amazing function, consult its
documentation.
Quote request: "if even the Sun requires proof"
by Smaug123 — 2022-05-07T10:30:13.068Z

I remember reading a quote, I think from the Buddhist or some other Eastern tradition, about how if you've argued yourself into believing the Sun doesn't exist, you've gone horribly wrong. It went something like "Nothing can be known, if even the Sun requires proof". I think I read it either on LessWrong or on gwern.net several years ago, but for the life of me I can't find it. I remember it being phrased more poetically than prosaically.

Does anyone know what quote I'm referring to, and where to find it?
But What's Your *New Alignment Insight,* out of a Future-Textbook Paragraph?
by David Udell — 2022-05-07T03:10:12.648Z

This is something I've been thinking about a good amount while considering my model of Eliezer's model of alignment. After tweaking it a bunch, it sure looks like a messy retread of much of what Richard says here; I don't claim to assemble any new, previously unassembled insights here.

Tl;dr: For impossibly difficult problems like AGI alignment, the worlds in which we solve the problem will be worlds that came up with some new, intuitively compelling insights. On our priors about impossibly difficult problems, worlds without new intuitive insights don't survive AGI.

Object-Level Arguments for Perpetual Motion

I once knew a fellow who was convinced that his system of wheels and gears would produce reactionless thrust, and he had an Excel spreadsheet that would prove this - which of course he couldn't show us because he was still developing the system. In classical mechanics, violating Conservation of Momentum is provably impossible. So any Excel spreadsheet calculated according to the rules of classical mechanics must necessarily show that no reactionless thrust exists - unless your machine is complicated enough that you have made a mistake in the calculations.

And similarly, when half-trained or tenth-trained rationalists abandon their art and try to believe without evidence just this once, they often build vast edifices of justification, confusing themselves just enough to conceal the magical steps.

It can be quite a pain to nail down where the magic occurs - their structure of argument tends to morph and squirm away as you interrogate them. But there's always some step where a tiny probability turns into a large one - where they try to believe without evidence - where they step into the unknown, thinking, "No one can prove me wrong".

…

Hey, maybe if you add enough wheels and gears to your argument, it'll turn warm water into electricity and ice cubes! Or, rather, you will no longer see why this couldn't be the case.

"Right! I can't see why couldn't be the case! So maybe it is!"

Another gear? That just makes your machine even less efficient. It wasn't a perpetual motion machine before, and each extra gear you add makes it even less efficient than that.

Each extra detail in your argument necessarily decreases the joint probability. The probability that you've violated the Second Law of Thermodynamics without knowing exac

...
D&D.Sci Divination: Nine Black Doves
by aphyer — 2022-05-06T23:02:01.266Z

This is an entry in the 'Dungeons & Data Science' series, a set of puzzles where players are given a dataset to analyze and an objective to pursue using information from that dataset.

STORY (skippable)

The omens are troubling for we Romans this year! Black doves have been spotted circling above major cities in all three of the Empire's main provinces. Not only that, news from abroad confirms that they have been seen across the world, from the wastes of Britannia in the West to the lush shores of Persia in the East.

There is confident general agreement that the black doves are a Very Bad Omen. No two people quite agree on why, though. Some say that they are emissaries of Pluto, sent out to escort the souls of the dead, and that their coming means Pluto must expect many to die. Others say that the birds have foreseen Jupiter's displeasure, and are taking flight to escape it. Captive philosophers from the Grecian territories say something about how they are the shadows on the walls of a cave cast from the true forms outside - but since the philosophers also say that we are all trapped in a cave at all times, no-one really takes them seriously.

You are a diviner in Rome. Your unconventional data-driven approach to divination has not endeared you to the broader divination community, so business has been quite scarce. However, two weeks ago, the main College of Divination in Rome was struck by lightning and burned down. Few of the staff were actually harmed, but the broad view is that anyone who built their college somewhere it would get struck by lightning clearly isn't a very good diviner. As such, when the recent ill omens arose, the Emperor approached you (as the best-known non-College diviner) to ask for advice on how best to weather this situation. This could be your big break, and your chance to demonstrate the superiority of your school of Data-Driven Divination!

The Emperor has given you the dataset gathered by years of Imperial diviners, which tracks the Omens that were recorded and the Disasters that happened in each year.

The good news is that this dataset is extremely extensive (reaching back to the founding of the Imperial system of Divination, which they mark as Year 0 in their system) and comprehensive (Imperial surveyors gather an annual update on every interesting omen a...
Is there a convenient way to make "sealed" predictions?
by Daniel Kokotajlo — 2022-05-06T23:00:36.789Z

Many people in our community claim to have ideas for how to build AGI, or other things, that they deem infohazardous and so don't want to publish. It would be great if they could publicly register these ideas in an encrypted way, so that later when their predictions come true they can reveal the key and everyone can see that they called it and give them epistemic credit accordingly.
I know this is possible in principle, e.g. by using PGP and posting encrypted messages on your LW shortform and then later revealing the key.
But it would be nice if this was a convenient, hassle-free feature embedded in LW, for example.
Also: Is this a bad idea for some reason? Is the privacy not as secure as I think, such that people would be hesistant to make even these encrypted predictions? (I guess there is the matter of how to securely store the key...) Is there a way to make a prediction that will automatically be decrypted after N years?