trentmkelly/LessWrong-43k huggingface dataset:
On expected utility, part 1: Skyscrapers and madmen
by Joe Carlsmith — 2022-03-16T21:58:39.257Z

(Cross-posted from Hands and Cities)

Summary:

Suppose that you’re trying to do something. Maybe: get a job, or pick a restaurant, or raise money to pay medical bills.

Some people think that unless you’re messing up in silly ways, you should be acting “as if” you’re maximizing expected utility – i.e., assigning consistent, real-numbered probabilities and utilities to the possible outcomes of your actions, and picking the action with the highest expected utility (the sum, across the action’s possible outcomes, of each outcome’s utility multiplied by its probability).

But in combination with plausible ethical views, expected utility maximization (EUM) can lead to a focus on lower-probability, higher-stakes events — a focus that can be emotionally difficult. For example, faced with a chance to save someone’s live for certain, it directs you to choose a 1% chance of saving 1000 lives instead – even though this choice will probably benefit no one. And EUM says to do this even for one shot, or few shot, choices – for example, choices about your career.

Why do this? The “quick argument” – namely, that EUM maximizes actual utility, given repeated choices and independent trials – isn’t enough for few-shot cases. And neither are appeals to “collective action” across EUM-ish people with your values.

Rather, I think the strongest arguments for EUM come from a cluster of related theorems, which say something like: your choices conform to certain attractive ideals of rationality if and only if you act like an EUM-er. But the theorems themselves often go unexplained. Informal discussions typically list some set of axioms, and then state the theorem, but they leave the proof, and the basic dynamics underlying it, as a black box.

Early on in my exposure to such theorems, I found this frustrating. If I was going to make big decisions (especially one-shot decisions) using EUM as a guiding ideal, I wanted to understand its rationale more deeply. But the full proofs are often lengthy and difficult.

This series of essays is an attempt to write the type of thing I would’ve loved to read, at that point in my life.

Part 1 (Skyscrapers and madmen) presents a way of visualizing EUM that I call the “skyscraper model.” It also discusses (1) the failure of the “quick argument,” (2) reasons not to let your epistemic relationship to EUM stop at “apparently the math says blah,” and (3)

...
What do paradigm shifts look like?
by leogao — 2022-03-16T19:17:37.586Z

One major question that heavily influences the choice of alignment research directions is the degree to which incremental improvements are necessary for major paradigm shifts. Because alignment is largely preparadigmatic, we may require a paradigm shift before we can make substantial progress towards aligning superhuman AI systems, rather than merely incremental improvements. The answer to this question determines whether the best approach to alignment is to choose metrics and try to make incremental progress on alignment research questions, or to mostly fund things that are long shots, or something else entirely. It also informs what policy we should take with respect to capabilities acceleration as an externality to alignment work: the degree to which incremental improvements in capabilities lead to paradigm shifts in capabilities informs how much we should worry about incremental capabilities improvements as a byproduct.

Some possible ways-the-world-could-be include:

Incremental improvements have negligible impact on when paradigm shifts happen and could be eliminated entirely without any negative impact on when paradigm shifts occur. All or the vast majority of incremental work is visible from the start as low risk low reward, and potentially paradigm shift causing work is visible from the start as high risk high reward.
Incremental improvements serve to increase attention in the field and thus increase the amount of funding for the field as a whole, thereby proportionally increasing the absolute number of people working on paradigmatic directions, but funding those working on potential paradigm shifts directly would yield the same paradigm shifts at the same time
Incremental improvements are necessary to convince risk averse funding sources to continue funding something, since putting money into something for years with no visible output is not popular with many funders, and thus forces researchers to divert a certain % of their time to working on funder-legible incremental improvements.
Most paradigm shifts arise from attempts to make incremental improvements that accidentally uncover something deeper in the process. It is difficult to tell before embarking on a project whether it will only yield an incremental improvement, no improvement at all, or a paradigm shift.
Most paradigm shifts cannot occur until incremental improvements lay the founda

...
Thoughts on Crowdfunding with Refund Bonuses
by Yoav Ravid — 2022-03-16T16:29:08.237Z

Alex Tabarok suggests improving crowdfunding by letting fundraisers offer a bonus as compensation for anyone that gets a refund if the project fails. By doing that the fundraiser solves the coordination problem making non-funding the equilibrium:

if I think that you won’t contribute then I may decide not to contribute and if I don’t contribute then you may decide not to contribute. Neither of us can do better by contributing, given the other person is not-contributing, and so non-contributing is a Nash equilibrium

But with refund bonuses:

if I think that you won’t contribute then I want to contribute, to earn the refund bonus, and the same is true for you. Indeed, the only equilibria in the crowdfunding game with refund bonuses have the project being funded.

He also highlights the importance of early backing to successfully fund projects, and refund bonuses, and especially refund bonuses only for early backers, incentivize early backing.

This is supported by a lab experiment (sci-hub) that tested various refund bonus schemes against the no-refunds baseline using a "lab-based fundraising platform with many main features of real-life crowdfunding such as asynchronous multiple contribution pledges over continuous time, constant updating of individual and aggregate pledge amounts until a fixed deadline, and simultaneously launched multiple fundraising campaigns. Each campaign lasts for two minutes, during which ten participating subjects can pledge their (multiple) contributions without any timing restrictions" where for each project, each participant is assigned a random value between 20 and 100 experimental dollars[1], which they get if the project is funded. The funding threshold for each project was set at 300 (experimental) dollars. For each project, participants could make as many contributions of any amount they want, which would instantly display for all other 9 participants. "In every period two alternative projects were available for potential contributions, with differing refund bonus rules for each". Sessions included 20-30 periods. Overall, data was reported from 280 college students, non of which participated in more than one session.

The results were:

Baseline: no refund bonus (on Kickstarter the funding frequency is 39.5%)
F3: bonus of $3 for total individual contribution ≥ $30.
F6: bonus of $6 for total individual contribution ≥ $30.
FE30: bonus of $6 for

...
Can you only realize objects in your mind after they appear?
by Cui — 2022-03-16T16:12:39.122Z

When meditating, you try to “be present”. But, say a background noise happens. You can only be aware of the noise after the fact. Does this take you away from the present?

Or is it that you are actually observing the mind, what the mind is doing in the present. But then does observing objects of the mind (even non-judgmentally) take you away from the present? “Oh, I heard a background noise.”

Or is it more about being in a kind of flow with your awareness? Easy and relaxed. Just noting when things come up to the mind.
[Intro to brain-like-AGI safety] 8. Takeaways from neuro 1/2: On AGI development
by Steven Byrnes — 2022-03-16T13:59:19.729Z

(Last revised: July 2024. See changelog at the bottom.)

8.1 Post summary / Table of contents

Part of the

“Intro to brain-like-AGI safety” post series

.

Thus far in the series,

Post #1

set up my big picture motivation: what is “brain-like AGI safety” and why do we care? The subsequent six posts (

#2

–#7) delved into neuroscience. Of those, Posts

#2

–

#3

presented a way of dividing the brain into a “Learning Subsystem” and a “Steering Subsystem”, differentiated by whether they have a property I call

“learning from scratch”

. Then Posts

#4

–#7 presented a big picture of how I think motivation and goals work in the brain, which winds up looking kinda like a weird variant on actor-critic model-based reinforcement learning.

Having established that neuroscience background, now we can finally switch in earnest to thinking more explicitly about brain-like AGI. As a starting point to keep in mind, here’s a diagram from

Post #6

, edited to describe brain-like AGI instead of actual brains:

Diagram is from

Post #6

, with four changes to make it about brain-like-AGI rather than actual brains: (1) “lifetime” is replaced by “training run” in the top right (Section 8.2 below); (2) “genetically-hardcoded” is replaced by “[probably] human-written” in the bottom-right (Section 8.3–8.4 below); (3) references to specific brain regions like “amygdala” have been crossed out, to be replaced with bits of source code and/or sets of trained model parameters; (4) other biology-specific words like “sugar” are crossed out, to be replaced with anything we want, as I’ll discuss in later posts.

This and the next post will extract some lessons about brain-like AGI from the discussion thus far. This post will focus on how such an AGI might be developed, and the next post will discuss AGI motivations and goals. After that, Post #10 will discuss the famous “alignment problem” (finally!), and then there will be some posts on possible paths towards a solution. Finally, in Post #15 I’ll wrap up the series with open questions, avenues for future research, and how to get involved in the field.

Back to this post. The topic is: given the discussion of neuroscience in the previous posts, how should we think about the software development process for brain-like AGI? In particular, what will be the roles of human-written source code, versus adjustable parameters (“weights”) discovered b...
Replacing Natural Interpretations
by adamShimi — 2022-03-16T13:05:19.610Z

Epistemic status:

exploratory

If the Earth moves, why can’t we feel it? And why do object dropped from high places fall vertically instead of to the side (since the Earth is supposed to have moved)? These are not stupid questions. If you were first introduced to the idea of the Earth that moves around a background of thinking that it was immobile, they would surely be a point of contention. And were for many thinkers the reaction to Copernicus's physics, and to the Greek thinkers he was inspired by.

So why shouldn't that refute heliocentrism? Galileo argued, in a move that has been vindicated by the last 400 years of success in physics, that there’s a distinction between absolute movement and relative movement, and that we can only detect the latter. So the stone falls horizontally because it is the only relative movement with regard to us and the Earth; it's also following Earth on its trajectory, but we do too. And we don't feel the movement of the Earth because it's not a movement relative to us but a movement with us.

Paul Feyerabend, the anarchist philosopher of science, describes this move as Galileo replacing what Feyerabend calls a natural interpretation with another: the intuitive, naive correspondence between a movement and its observation is replaced with only the distinguishability of relative movements.

After reading about it, I now see this idea of switching natural interpretations in many fascinating places in the history of science. This is a short post expanding on it and analyzing the concept through a few of these examples.

Revealing Natural Interpretations

Here’s how Feyerabend introduces natural interpretations in

Against Method

:

(All quotes are from Against Method)

Making the additional simplifying assumption, we can now distinguish between sensations and those 'mental operations which follow so closely upon the senses', and which are so firmly connected with their reactions that a separation is difficult to achieve. Considering the origin and the effect of such operations, I shall call them natural interpretations.

So they are not the observation themselves, but the almost instantaneous and unconscious implications that are drawn from the observations. Which can come from internalizing a model of the world — a theory of physics for example.

Feyerabend uses this notion to discuss what Galileo does in his dialogue, and how he deals with ...
Some (potentially) fundable AI Safety Ideas
by Logan Riggs — 2022-03-16T12:48:04.397Z

There are many AI Safety ideas I'd like to pursue, but I currently don't have the time for. Feel free to take these ideas, apply for funding, and, if you get funded, you can thank me by reducing existential risk.
[Note: I am not a funding manager or disperse funds in any way]

Literature reviews for each AI Safety Research Agenda

It can be difficult to know what specifically to work on in AI Safety, even after finishing [AI Safety Camp/ AGI Fundamentals Course/ internships/ etc]. It would useful to have a pinned post on alignment forum that listed ~10 research agendas, including:

1. Clear problem statement (eg "Mesa-optimizers are ..., here's a link to Mile's youtube video on the subject")
2. Current work w/ links/citations
3. Current Open Problems
4. People currently working on the problem w/ contact information

At least updated every 6 months if not sooner.
It could be structured with

1. One high-quality manager who's making lit reviews for 1-2 fields, manages other people doing the same, consolidates their works, and thinks about the structure/ format of the documents
2. Three-five people making lit reviews for 1-2 different fields who are intrinsically interested in their field of choice.

If I were pursuing this idea (which again, I'm not), I would pick an alignment researcher's agenda I'm interested in, go through their works, writing small summaries of posts for my own sake, and write a rough draft of the lit review. I'd then contact them for a call (through lesswrong), sending a link to the document, and update/publish based off feedback.
If I were the manager, I'd make an open call for interviews, asking people their alignment research field of interest, a clear problem statement, and how that problem statement connects with reducing existential risk.
A failure mode could be a 90-page report that no-one reads or misses the core point driving the researcher. It'd be great to have a Maxwell equations reformulation, or at least a "that's a much better way to phrase it".

Fundability

If I wanted to signal my ability to complete the above project, I would either need previous academic experience, or connections to people who know grantmakers, or, barring that, to actually make a draft of one such literature review, saying "Pay me money, and I'll make more".
Though, again note, I'm not a funding manager.

AI Safety Conference

We don't have a conference. Let's make one. One coul...
[Quote] Why does i show up in Quantum Mechanics and other Beautiful Math Mysteries
by Gunnar_Zarncke — 2022-03-16T11:58:30.526Z

David Schneider-Joseph writes on Facebook:

In principle, this isn’t really any more mysterious than, say, -1/2 showing up somewhere. It’s unfortunate that imaginary numbers got called imaginary because they’re not really any less real than “real” numbers.

Numbers, even “simple” ones like the positive integers, are abstractions that we use when they map well onto reality. We never find the number 3 out there in the physical world. We may find three apples or a representation of the number 3, yet 3 itself is not to be found. But this is no problem: 3 is a concept that helps us make sense of situations in which we have three of something.

For someone used to using numbers to count things, the concept of a negative or fractional number seems sort of … imaginary. How can you have a negative number of apples? How can you have half a dog? The answer is that those sorts of numbers are designed for different situations: e.g., debts or backward motion in the case of negative numbers, divisible things like gallons or pizza pies in the case of fractions.

Similarly, complex numbers are just designed for a new sort of thing. Their algebra maps naturally to, among other situations, the behavior of periodic systems such as a wave (including very classical waves such as sound waves and vibrating strings), the magnitude and argument of the number representing the amplitude and phase of the wave. It just means an amplitude of 1 and a 90° angle from the starting point.

(I did some minor spelling corrections)
War Mode vs. Peace Mode
by konstell — 2022-03-16T03:36:30.826Z

Cross-posted from konstell.com

[Epistemic status: rough heuristics, probably needs refinement, comments are always welcome.]

I first got this dichotomy by reading someone's comment on how Mao was a wartime general but not a peacetime general. Of course, it's possible that a more accurate model is not binary but a spectrum, or maybe it has more dimensions. Nonetheless, over time, I've found this heuristic useful.

In war mode, the objective is survival. There's little room for ambiguity because one's utmost goal is not to get killed, which means anything new that pops up is treated as a potential threat until proven otherwise. It's prudent to assume the worst intentions from other people. Survival pressure is high; goodwill is low. Scarcity mindset and zero-sum games are common. Pragmatism is usually more useful than romanticism.

In peace mode, the objective is flourishing. There's lots of room for ambiguity because safety and stability are given. Anything new that pops up can be treated with curiosity. It's wise to assume the best intentions from other people, for it opens up possibilities. Survival pressure is low, though there might be pressure in status competitions. Abundance mindset and positive-sum games are common. Pragmatism might be seen as "uncool" compared to romanticism.

These two modes require distinct sets of skills, thus it's crucial to have an accurate model of which mode one is operating. For example, A negotiation is inherently war mode with some room for positive-sum games; open-source software is a product of peace mode; psychological trauma is living chronically in a war mode mindset, unable to adjust to peace mode.
Danger(s) of theorem-proving AI?
by Yitz — 2022-03-16T02:47:47.275Z

Posting this on behalf of user PrimalShadow on the AstralCodexTen Discord:

What are ways in which an AI that is trained to prove specific math theorems is dangerous? In particular, imagine you have an AI which is trained to take a prompt asking to prove a formal-language-predicate, and then it produces a machine-readable proof of said thing.

I would posit some answers to my own question:

The AI ends up finding bugs in your prover instead of actually solving problems. (This is a failure mode, but probably not dangerous in the sense of being unfriendly)
The AI is used by humans to do dangerous things, e.g. break cryptography. (This is a type of danger, but not what I would call unfriendliness either.)
The AI somehow learns about the outside world, and modifies its outputs in an attempt to manipulate that world - E.g. to acquire more computing power - E.g. to get the operator to ask it easier questions
The AI as described is not directly dangerous, but some tweak to it is dangerous I wonder if I'm missing any broad categories.

I responded with the suggestion of what you might call "paperclipping" (it deciding it requires some astronomical amount of compute to solve the problem, so takes over the world, or maybe even the galaxy/beyond, to maximize compute), which PrimalShadow noted already falls into "The AI somehow learns about the outside world". He also pointed out that the AI wouldn't need to be trained on non-mathematical data, which would presumably reduce any risk of that happening.

Neither of us could easily think of a scenario where this trivially goes wrong, and we're both curious what the potential risks are from the perspective of more experienced people in the field.
Ukraine #4: Prediction Market Movement Modeling
by Zvi — 2022-03-15T22:20:01.111Z

I previously looked at various prediction markets related to Russia’s invasion of Ukraine. Scott Alexander provides an update on some prices, notices he is confused in various ways by the motions of various prices.

It seems easy enough to mostly explain any given single market and its movements. The real challenge is to combine all the markets and understand what story they are telling you together. This includes the history of their price movements as part of that story.

As usual, the Metaculus predictions cover more ground and go where others fear to tread, while any real-money markets are better where you can get them.

I am going to focus on the markets related to who is likely to win, as they tell the most interesting story. That story includes the lack of a quick Russian victory.

Will Kyiv fall to Russian forces by April 2022?

Note that the real money version of this is trading lower at 9%.

I have consistently thought this market was too high, and I still think that. The only way Kyiv falls in the next two weeks is a general surrender, that seems vanishingly unlikely. There is no way to take the city by force within two weeks given they have it very well defended – they claim to have 80,000 men, which makes me wonder why they even need that many. Naively seems to me like the second half of that only makes supplies a bigger issue and you’d rather use them elsewhere.

This is a steady climb downwards, and Scott wonders about whether it is a predictable and thus false pattern.

To some extent it is the automatic updating that Kyiv did not fall and does not appear about to fall. Each day of this makes it more likely Kyiv can hold out longer, reducing chances of a fall each day going forward, so things should reduce faster than the proportion of time expired. There’s also what looks like a clear shift away from trying that hard to take Kyiv quickly, and to focus on encircling Ukraine’s forces in the Donbass, which seems more realistic as a goal, and there isn’t time here to refocus.

What about longer-term versions of the same question?

Will Russia control Kyiv on June 1, 2022?

Notice that this started out not much higher than the first prediction. On March 3, Metaculus thought there was a 60% chance Kyiv would fall in March, and this question first settles in around 72%.

One could argue that the Russians might take the city and then leave without occupying it before June ...