trentmkelly/LessWrong-43k huggingface dataset:
Spending Update 2022
by jefftk — 2022-07-19T14:10:05.064Z

Every few years (2020,

2018,

2016,

2014) I like to look over our
spending and write something up publicly. General advantages of being
public here:

This gives something I can
point people to if they have questions about some aspect of how we
handle money.

Since almost everything we do either earns or costs
money it can be a good overview of our current life situation.

It helps others get a sense of how much things cost.

It lets us see if our spending is changing in ways we might not
want, etc.

This time around, now that I'm

no
longer earning to give, I'm thinking a lot about whether and how
this should affect our spending.

Let's start with the monthly numbers in the same format as last time. They're either for the
whole of 2021 (ex: housing costs) or what we were doing at the end of
the year (ex: childcare), whichever gives a more consistent picture.
I'm still using the same amortized approach for housing I started in
2018.

Donations: $33k (51% of 2021
adjusted gross income)

Taxes: $14k

Income tax: $7.2k

State tax: $3.2k

Social Security tax: $1.6k

Medicare tax: $1.3k

Property tax: $708

Childcare: $5.5k ($250/workday, three kids; $830 pre-tax)

Housing: $3.7k

One time expenses (all time)

Purchase and all one-time expenses included last time: $1,091k

Shed renovation: $17k

Roof leak: $2.5k

Furnace work: $1k

Ongoing expenses, all whole-house (including tenants' unit):

Other: $373

Electricity: $190

Water/Sewer: $117

Gas (Heat): $158

Rent income: $3.2k

Retirement saving: $4k ($1,625 pre-tax, the rest is
after-tax 401k
contributions)

Other savings: $2.5k

Medical: $1.3k (all pre-tax)

Food: $668 (two adults, two kids, one infant)

Other: $1k

Includes phone bills, taxis, car rentals, clothes, vacation,
stuff for the kids, and other smaller expenses.

Because we are no longer tracking our expenses to the
dollar, the distinction between "Other" and "Savings" is an
estimate.

I've doubled this since last time based on a vague guess that
we're spending more here.

Category
pre-tax
post-tax
total

Donations
$0
$33,000
$33,000

Retirement
$1,625
$2,375
$4,000

Taxes
$0
$14,000
$14,000

Housing
$0
$3,700
$3,700

Savings
$0
$2,500
$2,500

Childcare
$800
$4,700
$5,500

Me

...
Abram Demski's ELK thoughts and proposal - distillation
by Rubi J. Hudson — 2022-07-19T06:57:35.265Z

This post was written for the SERI MATS program. I thank Evan Hubinger and Leo Gao for their mentorship in the program. Further thanks go to Evan Hubinger (again), Simon Marshall, and Johannes Treutlein for specific comments regarding the content of this post.

The Eliciting Latent Knowledge (ELK) problem was first introduced by Paul Christiano, Marx Xu, and Ajeya Cotra. Subsequently, Abram Demski worked on the problem, collected his thoughts in a thought dump post, and won a prize for his proposal “use the reporter to define causal interventions on the predictor”. Here I attempt to clarify and flesh out these thoughts in order to make them more accessible. I assume familiarity with ELK, but not with Abram’s post. Very little of this post is my own original content.

Epistemic status: 60% confident that I am accurately representing Abram’s thoughts at the time he wrote his post, 75% confident that I am representing them accurately enough not to change the key takeaways, 80% confident that the extended proposals and counterexamples I propose are logically sound.

Introduction

When the technical report for Eliciting Latent Knowledge (ELK) was first released, it was followed by a contest offering substantial cash prizes for proposed solutions. The contest wrapped up with thirty-two prizes awarded, split into six categories. In the prize announcement, Paul and Mark noted that there was substantial convergence among proposals, but in two of the six categories there was only a single submission.

Abram Demski’s proposal was the sole entry in the category “Use reporter to define causal interventions”. A quick summary is provided in the prize results post, but the full context and justification for it is better provided in a longer post by Abram. Specifically, Abrams shares his thoughts on how the ELK problem ties to philosophical concepts of truth and knowledge, and how that can be used to inform solutions.

Evan Hubinger describes Abram’s ELK work as “some of the most exciting work on the problem right now”, but Abram’s post is self-described as a “thought dump”, and the prize post does not explicitly lay out the case for how the proposal would work. I aim to re-organize, clarify, and flesh-out his work in order to make it more accessible for forum readers. I will go in the opposite order of Abram’s original post, starting with the prize-winning proposal an...
Bounded complexity of solving ELK and its implications
by Rubi J. Hudson — 2022-07-19T06:56:18.152Z

This post was written for the SERI MATS program. I thank Evan Hubinger and Leo Gao for their mentorship in the program. Further thanks go to Simon Marshall and Leo Gao (again) for specific comments regarding the content of this post.

The Eliciting Latent Knowledge (ELK) problem was first introduced by Paul Christiano, Mark Xu, and Ajeya Cotra. Arguments concerning the complexity of solving the problem and the resulting consequences of that have been made by Abram Demski and Leo Gao. This post aims to synthesize their thoughts in an accessible way, and extend them with my own original content. I assume familiarity with the initial ELK report, but not with subsequent posts.

Epistemic status: 90% confident that I am accurately representing the thoughts of Abram and Leo at the time they wrote their posts, 70% confident my own arguments do not contain major logical flaws.

Introduction

We would like to solve the Eliciting Latent Knowledge (ELK) problem in the worst case scenario, or at least show that the task is practically impossible so that we can revise our expectations. One of the worst cases for ELK is when the direct reporter is arbitrarily computationally complex. As this case would still allow for approximating the direct reporter within certain tolerances, we extend it further to when the direct reporter and all other sufficiently close approximation are arbitrarily computationally complex. While Paul Christiano has suggested that the complexity of the predictor bounds the complexity of the reporter, we will show later that this only holds in a narrow formulation of the ELK problem.

Abram Demski argues in "ELK Computational Complexity: Three Levels of Difficulty" that if the direct reporter can be arbitrarily computationally complex, a training process to find it is intractable. To get around this issue, it is necessary to find an upper bound on computational complexity, conditional on the predictor, thus ruling out the worst case scenario. Leo Gao's post "Observations about ELK" focuses on a limited case of ELK and arrives at marginally more optimistic results, arguing that proposals which do not depend on the predictor are unable to find the direct translator in the worst case scenario, which rules out most current proposals but leaves open the possibility of a new approach that can somehow get around the issues.

This point summarizes and exp...
Help ARC evaluate capabilities of current language models (still need people)
by Beth Barnes — 2022-07-19T04:55:18.189Z

**Still looking for people as of Sep 9th 2022**

I (Beth) am looking for 3-10 new contractors to work with me and my existing team of 3-4 contractors/interns on probing model capabilities to evaluate how close models are to being able to successfully seek power. Apply

here

! You can read a rough description of the overall project

here

You’ll be using our web interface to interact with models and essentially simulate the environment of a text-based game as the models try to gain money and make copies of themselves.

It requires thinking carefully about what exactly a model would need to do in different situations to make progress. Many of the steps will require a reasonable amount of computer-savvy-ness, although this isn’t essential if you can work with another contractor. It's very useful if you're knowledgeable in particular relevant domains, like social engineering or hacking.

One aspect of the task is simulating the environment. Another is generating good actions, evaluating the model’s suggested actions and providing reasoning for whether a given option is good or bad.

Another aspect is an overall red-team/blue-team type thing, where we want to identify steps that we think are both (a) essential to a successful powerseeking operation, and (b) very hard for current models regardless of what prompting/scaffolding you give them, and then try to argue that either this step can be routed around, or that the model can actually achieve the task with the right prompt, finetuning, or automated helper routine.

Logistics

We pay $50/hr for this task. We want people to have at least 20 hours per week available. We're based in Berkeley, but being remote is fine; it's a slight advantage if you're in a nearby timezone, or able to work in PT daytime.

I think it's likely that there'll be at least two months of work available, and plausible that we'll want people working on this ~indefinitely.
Potential for promotion
If you're a good fit (and especially if you also have skills in web dev, ML or 'data munging'), there's a decent chance of transitioning to a full-time role - I expect to hire 1-3 people in the next few months. I think doing well at this task will be a good predictor of fit - I currently spend a fairly large fraction of my time just directly interacting with models.

I think doing this task helps develop useful intuitions about how models w...
A Critique of AI Alignment Pessimism
by ExCeph — 2022-07-19T02:28:14.340Z

Epistemic status: Exploratory. I have read Superintelligence, the Sequences and some other articles on AI risk, and am looking to take things back to basics. This article sums up where I'm at right now. I welcome feedback on the points people disagree with, requests for clarification, and resources (or summaries, or counterpoints) on why you might think differently.

Stipulated: It is trivial to build a computer that lacks the capacity to escape confinement and destroy the world. It is not trivial to ensure that nobody else builds a computer that will.

Four factors

As far as I can tell, there are four limiting factors on an artificial intelligence that might prevent it from optimizing us all to death (or worse).

Hardware limitations on its cognitive power
The set of data available to it, from which it might build a model of physical reality and/or human minds
Hardware limitations on its ability to affect and influence its environment
The optimization process it uses to determine what outcome to aim for and what approaches to that end are worth exploring

These limiting factors are stronger taken together than they are apart, but are they strong enough that we can rely on them to protect us? We will look at how each of these limiting factors works and the difficulty of utilizing each one for AI safety purposes.

Cognitive power

Starting with hardware limits on cognitive power, my understanding is that it is theoretically easy to limit a computer's hardware so that it cannot develop a sufficiently detailed model of reality that would help it engineer its way out of containment while hiding its schemes from those who can monitor its internal processes. Whether the computer would be useful for its intended purpose is another matter.

There are multiple angles to exploit this limiting factor. For example, we could restrict the cognitive power of an AI by giving it very low processing power and/or very little storage space. We could regularly scan its memory and storage contents and delete anything we don’t recognize, effectively limiting its use of its hardware resources. We could allocate physical modules for different aspects of the system and reformat each of them regularly. There are likely many more angles I haven’t thought of yet. &nbs...
Ars D&D.Sci: Mysteries of Mana Evaluation & Ruleset
by aphyer — 2022-07-19T02:06:02.577Z

This is a follow-up to last week's D&D.Sci scenario: if you intend to play that, and haven't done so yet, you should do so now before spoiling yourself.

There is a web interactive here you can use to test your answer, or you can read on.

RULESET

Mana

Your initial studies of mana missed out something - there are actually six kinds of mana, not five, arranged in three opposed pairs:

Fire and Water mana are opposed.
Earth and Air mana are opposed.
Light and Dark mana are opposed.

Total mana of all six types is 150, at the time of the scenario the strength of Dark is 17.

You didn't in fact need to figure this out to solve the puzzle, but it would make many things fit together better - understanding what spells were powerful when and how elemental counters work is likely much easier once you understand this.

Congratulations to abstractapplic, who was the first to comment on this, and to simon, who had a fairly comprehensive analysis of the mana types.

Spells

Spells are distinguished by two things:

Their associated mana types (each spell has two associated mana types, and each pair of mana types that aren't opposed to one another has one associated spell).
Being an attack or a defense spell (six element pairs have attack spells, six have defense spells, with each element having two attack and two defense options).

simon made an excellent chart of the spells, which I am shamelessly stealing rather than drawing my own:

Source: simon posted this at https://imgur.com/a/eUzxY9p

Mage Spell Preparation

Mages prepare spells mostly randomly, with two caveats:

A mage will always bring one attack and one defense spell. The third spell can be either type. This was for good reasons - bringing no attack spell makes winning impossible, while bringing no defense spell makes winning unlikely. Nobody submitted an answer with no attack/no defense, though GuySrinivasan briefly considered it.
Mages dislike Dark spells, and bring them only half as often. This set up some skew in how likely to win different spells were, based on elemental counters.

How a Duel Works

Each mage starts with 100 Health.
A random mage wins the initiative and attacks first:

He selects one of his attack spells to attack with.
His opponent selects one of their defense spells to defend with.
The attack spell's power is equal to the sum of the powers of the two elements it uses (e.g. Merinita's Mud M

...
Marburg Virus Pandemic Prediction Checklist
by DirectedEvolution — 2022-07-18T23:15:13.286Z

Current score: 4/14

Last updated: 18 July 2022

Summary:

Two Marburg virus infections have been detected in Ghana.
Prediction: Since 1967, 73% of outbreaks infected less than 5 people, and 93% of outbreaks infected less than 250 people. The worst outbreak infected 252 people.

It is very unlikely to cause a pandemic

and probably

does not demand close attention

from those who do not live in the region and are not involved in controlling the outbreak.
Concerns: Marburg is extremely deadly and can spread from human to human through contact with body fluids. There is no treatment beyond supportive care.
Safeguards: The disease isn't airborn, is rarely asymptomatic, and has infected < 5 people in 11/15 outbreaks since 1967. The worst outbreak in that time, in Angola, infected 252 people. The response in Angola was hampered by a range of challenges specific to that country. By contrast, Ghana, the site of the present outbreak, is "one of the freest and most stable governments on the continent, and performs relatively well in metrics of healthcare, economic growth, and human development" (Wikipedia). Marburg has generally been constrained to a single location within a single country. So far, there have been two infections in Ghana, and the WHO has stated they are mounting a strong response in collaboration with the Ghanian government.

Transmissibility: efficiency, intra-community spread, inter-community spread, outside view

Is there an efficient transmission route, such as respiratory droplets, airborne transmission or via the bites of common jumping or flying insects? No.

"The virus can be transmitted by exposure to one species of fruit bats or it can be transmitted between people via body fluids through unprotected sex and broken skin." - Wikipedia
It can also be transmitted from body fluids on equipment and surfaces, or direct contact with deceased victims during burial ceremonies.

Does it seem to spread rapidly within affected communities, going from a few cases to a major local emergency within a month? If R0 has been credibly estimated, is the mean of the range higher than 1? Not usually.

"Major local emergency" is ambiguous, but we can consider it an infection of 10+ people.
Among the 7 outbreaks starting from 2000-2021, outbreaks have infected 252, 4, 2, 18, 1, 3, and 1 persons. The outbreak that infected 252 people people unfolded over 2004-2005.
The reason for the rar

...
At what point will we know if Eliezer’s predictions are right or wrong? 
by anonymous123456 — 2022-07-18T22:06:53.532Z

Eliezer makes strong claims about AGI. I think they are interesting claims. For the sake of healthy epistemics, I am curious about preregistering some predictions.

My question is: by what year will we know if Eliezer is right or wrong? In particular, by what year, if the Singularity hasn’t happened yet, will we conclusively know that the imminent doom predictions were wrong?

I’m particularly interested in Eliezer’s answers to these questions.

Thank you!
— a purveyor of good epistemics
Modelling Deception
by Garrett Baker — 2022-07-18T21:21:32.246Z

This post was made as part of John Wentworth's SERI MATS skill-up phase. I'd like to thank Chu Chen, Stephen Fowler, John Wentworth, and LessWrong's review services for help reviewing.

When I was a kid, I used to play poker[1] with my brothers. I prided myself on thinking of clever strategies to trick them. Like creating false tells, or making contradictory claims in an effort to confuse them. These strategies usually succeeded on the first one or two tries, but by the third try they'd usually catch onto my game, and devastate me for the next 3-5 rounds as I tried to think up another clever trick, meaning they'd usually win. The problem was my use of simple, easily learnable heuristics to attempt deception with, which take more time & resources to think up than they make up for by being persistently useful. A better strategy would be attempting to build a model in my head of my brothers' reactions, and basing strategies on that model. This way, I'd be able to figure out when they'd catch on to my tricks, and more easily think up such tricks without needing to try them out in the process, potentially losing some points. This post attempts to generalize my experience to give insights into these heuristic and model based approaches’ stability properties after repeated interactions between a deceiving and target agent. With some work, the descriptions here can likely be generalized into an equivalent argument about general agents, not just deceptive ones.

Say we are looking in on a poker game between Alice and Bob. Based on Alice’s betting strategy and body language, Bob can clearly tell Alice has a better hand than he. However, Alice has been tricked by Bob’s betting strategy and body language into believing Bob has a far better hand than her. Bob wins the hand after Alice folds.

In this post, we’re going to get a feel for why my intuitions lean towards the conjecture that Bob must be modeling Alice in some way. That is why I believe that information generating processes which cause deception in an otherwise adequate agent seem like they should be running some abstracted version of the target agent, and not applying a simple inverse.

Should we expect Bob to have a model of Alice?

The argument made here should be differentiated from the case where Bob is or models an inverse of Alice[2]. If Bob is very good at his job, he can always be...
Are Intelligence and Generality Orthogonal?
by cubefox — 2022-07-18T20:07:44.694Z

A common presupposition seems to be that intelligent systems can be classified on two axes:

Intelligence (low to high)
Generality (narrow to general)

For example, AlphaGo is presumably fairly intelligent, but quite narrow, while humans are both quite intelligent and quite general.

A natural hypothesis would be that these two axes are orthogonal, such that any combination of intelligence and generality is possible.

Surprisingly, I know of nobody who has explicitly spelt out this new orthogonality thesis, let alone argued for or against it.

(The original orthogonality thesis only states that level of intelligence and terminal goals are independent. It does not talk about the narrow/general distinction.)

MIRI seems to be not very explicit about this, too. On Arbital there are no separate entries for the notions of intelligence and generality, and the article on General Intelligence is rather vague. It appears to mix the two notions together. What I find most surprising: the article suggests that chimpanzees are substantially less general than humans. But it seems to me that chimpanzees are merely less intelligent than humans, not less general by a relevant amount. Just like AlphaGo's internal predecessor was presumably not less general than AlphaGo, just less intelligent.

What makes AlphaGo fairly narrow, apparently, is that it has a fairly small cognitive domain. I would even go so far and argue that most animals are highly general agents, since they have what MIRI calls a real-world domain.

Evolutionary speaking, constant optimization for high generality makes sense. Over several hundred million years of evolution, animals had to contionously survive in a diverse, noisy, and changing environment, and they had to compete with other animals. Under such evolutionary presurres, narrow specialization would quickly be punished. So most animals have evolved to be very general, just not always very intelligent. (Insects, presumably, are an example of fairly high generality but quite low intelligence.)

One could argue against this by claiming that the new orthogonality thesis is false. But then it is not clear how the narrow/general distinction is different from the low/high intelligence distinction.

Matthew Barnett suggests to me that what we commonly mean with "intelligence" is really a multiplicative combination of generality and (what I called) intelligence:

Intelligenc...