trentmkelly/LessWrong-43k huggingface dataset:
Various Alignment Strategies (and how likely they are to work)
by Logan Zoellner — 2022-05-03T16:54:17.173Z

Note: the following essay is very much my opinion. Should you trust my opinion? Probably not too much. Instead, just record it as a data point of the form "this is what one person with a background in formal mathematics and cryptography who has been doing machine learning on real-world problems for over a decade thinks." Depending on your opinion on the relevance of math, cryptography and the importance of using machine learning "in anger" (to solve real world problems), that might be a useful data point or not.

So, without further ado: A list of possible alignment strategies (and how likely they are to work)

Edit (05/05/2022): Added "Tool AIs" section, and polls.

Formal Mathematical Proof

This refers to a whole class of alignment strategies where you define (in a formal mathematical sense) a set of properties you would like an aligned AI to have, and then you mathematically prove that an AI architectured a certain way possesses these properties.

For example, you may want an AI with a stop button, so that humans can always turn them off if the AI goes rogue. Or you may want an AI that will never convert more than 1% of the Earth's surface into computronium. So long as a property can be defined in a formal mathematical sense, you can imagine writing a formal proof that a certain type of system will never violate that property.

How likely is this to work?

Not at all. It won't work.

There is a aphorism in the field of Cryptography: Any cryptographic system formally proven to be secure... isn't.

The problem is, when attempting to formally define a system, you will make assumptions and sooner or later one of those assumptions will turn out to be wrong. One-time-pad turns out to be two-time-pad. Black-boxes turn out to have side-channels. That kind of thing. Formal proofs never ever work out in the real world. The exception that proves the rule is, of course, P=NP. All cryptographic systems (other than one-time-pad) rely on the assumption that P!=NP, but this is famously unproven.

There is an additional problem. Namely, competition. All of the fancy formal-proof stuff tends to make computers much slower. For example, fully holomorphic encryption is millions of times slower than just computing on raw data. So if two people are trying to build an AI and one of th...
Does the “ugh field” phenomenon sometimes occur strongly enough to affect immediate sensory processing?
by Rana Dexsin — 2022-05-03T13:42:46.225Z

Related to my comment on the parent question: is there documentation of specific attention minimization and/or blotting-out effects in immediate sensory processing related to past emotional aversion? I suspect the two of those, despite being listed as separate bullet points in the parent question, should be treated separately…

A more generalized form of this seems like it'd be the kind of dissociation that can occur in e.g. PTSD. Do some PTSD sufferers have sharper sensory issues surrounding the brain refusing to recognize certain stimuli?
Why humans don’t learn to not recognize danger?
by Joachim Bartosik — 2022-05-03T12:53:49.352Z

Very short version in the title. A bit longer version at the end. Most of the question is context.

Long version / context:

This is something I vaguely remember reading (I think on ACX). I want to check if I remember correctly/ where I could learn it in more technical detail.

Say you go camping in a desert. You wake up and notice something that might be a scary spider you take a look and confirm it's a scary spider indeed. This is bad, you feel bad.

Since this is bad, you will be less likely to do some things that led to you will be less likely to do things led to you feeling bad, for example you'll be less likely to go camping in a desert.

But you probably won't learn to:

avoid looking at something that might be a scary spider or
stop recognizing spiders

even though those were much closer to you feeling bad (about being close to a scary spider).

This is a bit weird if you think that humans learn to just get a reward usually you'd expect stuff that happened closer to the punishment to get punished more, not less.

What I recall is that there is a different reward for "epistemic" tasks. Based on accuracy or saliency of things it recognizes, not on whether it's positive / negative.

A bit longer version of the question:

Why don't humans learn to not recognize unpleasant things (too much)? Is there a different reward for some "epistemic" processes? Where could I learn more about this?
Monthly Shorts 4/2022
by Celer — 2022-05-03T04:10:19.975Z

Conflict

Elon Musk was once asked about the regulatory situation of providing satellite internet without the local country’s permission. His response was uniquely Muskian:

Elon Musk @elonmusk

@thesheetztweetz They can shake their fist at the sky

September 1st 2021

1,290 Retweets11,916 Likes

Now, it turns out, there are also other options. Dictators can, for example, launch electronic warfare measures against SpaceX’s operations. Fortunately…it turns out that SpaceX is better than the Russians and so Ukranian internet access continues.

Fun piece on military inter-service conflict (in favor), if that’s your jam.

One of the things I’ve had to grapple with, at my age, is understanding just how meaningful 9/11 is to people older than me. Two months of car crash deaths get shown on TV, and everybody goes completely mad. I go to a panel on national security work, and every single panelist and the moderator says that their inspiration to enter government service was 9/11. The Census Bureau handed over information on Arab neighborhoods to DHS (the story is more complicated than that: DHS seems to be both lying and incompetent and the Census Bureau did something both understandable and legally required, but this is the short version). We passed the Patriot Act, setting up massive denial of civil liberties by means both legal (new authorizations) and structural (empowering a type of agency that cares very little for such things at the expense of Justice and State, which do).

DHS has seized over $500 million in currency from people who didn’t follow said signage.

State and local taxation is usually regressive in America.

Code and Consequences

This request was intended to inform the implementation work. Instead, all hell broke loose.

…

After the 2019 CNSTAT meeting made clear that evaluators were not accounting for the biases of the published data, the Census Bureau attempted to inform stakeholders that they were not comparing their analyses to ground truth. Having nothing else to compare the data to, this information was not well received by either data users or other census advocates.

…

If the Census Bureau were to publish datasets that that included fractional or negative population counts, this was bound to confuse and upset external stakeholders, including lawyers and judges.

This is a surprisingly fascinating article on the introduction of differential privacy at the Census Bure...
Notes on Empathy
by David Gross — 2022-05-03T04:06:36.616Z

This post examines empathy, as part of a sequence of posts about virtues. It is mostly an exploration of what other people have learned about empathy, rather than my own research or opinions about it, though I’ve been selective about what I found interesting or credible. I wrote this not as an expert on the topic, but as someone who wants to learn more about it. I hope it will help people who want to know more about empathy and how to practice it skillfully.

This was a particularly interesting (and difficult) post to research and write. There is a wealth of literature about empathy, and I found a lot of it surprising. I feel kind of like the kid who opened up the back of dad’s watch to see how it works and now is sitting amidst a cubic yard of springs and gears wondering how to put it all back together.

What is empathy?

The word “empathy” is pretty new. It apparently was coined in 1908,[1] and only really caught on in the last half of the 20th century.[2] (The concept itself is not so new; earlier authors sometimes deployed the word “sympathy” to cover similar experiences.)

a Google Books Ngram view

The definition of empathy is contested.[3] What sort of thing empathy is also defies agreement: is it a sense or emotion (an immediate and visceral mirroring of another person’s state), or is it more like an intellectual feat (accurately discerning another person’s viewpoint), or maybe a social skill (the ability to respond appropriately to another’s condition)?

Some authors narrowly define empathy as a sense or feeling, like “the coexperience of another’s situation”[4] or “feeling what you think others are feeling.”[5] Others apply the term to a constellation of feelings, cognitive interpretations, and responses, for example “a social and emotional skill that helps us feel and understand the emotions, circumstances, intentions, thoughts, and needs of others, such that we can offer sensitive, perceptive, and appropriate communication and support”[6] or “the art of stepping imaginatively into the shoes of another person, understanding their feelings and perspectives, and using that understanding to guide your actions.”[7]

In some descriptions of empathy it is restricted to feeling distress upon noticing someone else exhibiting distress in circumstances such as pain, fear, or loss. But in others, you might just as easily empathize happily wit...
Looking for someone to run an online seminar on human learning
by Isaac King — 2022-05-03T01:54:54.129Z

I'm looking for someone with a background in education and/or cognitive science to run an online seminar for non-rationalists on how humans learn things and how to efficiently teach a subject to others. A few examples of the sort of content I'm thinking of are: Ebbinghaus's research on memory and spaced repetition, the difference between shallow and deep learning of a subject, Bloom's 2 sigma problem. The exact content would be up to you.

It would be a ~1 hour seminar on May 29th, run via Zoom or a similar platform. It pays $400 USD. If you're interested, please email me to discuss the details.
Home Antigen Tests Aren’t Useful For Covid Screening
by Elizabeth — 2022-05-03T01:30:05.921Z

Epistemic status: I strongly believe this is the right conclusion given the available data. The best available data is not that good, and if better data comes out I reserve the right to change my opinion.
EDIT (4/27): In a development I consider deeply frustrating but probably ultimately good, the same office is now getting much more useful information from antigen tests. They aren’t tracking with same rigor so I can’t comapre results, but they are now beating the bar of “literally ever noticing covid”.

In an attempt to avoid covid without being miserable, many of my friends are hosting group events but requiring attendees to take a home covid test beforehand. Based on data from a medium-sized office, I believe testing for covid with the tests people are using, to be security theater and provide no decrease in risk. Antigen tests don’t work for covid screening. There is a more expensive home test available that provides some value, and rapid PCR may still be viable.

It’s important to distinguish between test types here: antigen tests look for viral proteins, and genetic amplification tests amplify viral RNA until it reaches detectable levels. The latter are much more sensitive. Most home tests are antigen tests, with the exception of Cue, which uses NAAT (a type of genetic amplification). An office in the bay area used aggressive testing with both Cue and antigen tests to control covid in the office and kept meticulous notes, which they were kind enough to share with me. Here are the aggregated numbers:

The office requested daily Cue tests from workers. I don’t know how many people this ultimately included, probably low hundreds? I expect compliance was >95% but not perfect.

The results are from January when the dominant strain was Omicron classic, but no one got strain tested.

39 people had at least one positive Cue test, all of which were either asymptomatic or ambiguously symptomatic (e.g. symptoms could be explained by allergies) at the time, and 27 of which had recent negative cue tests (often but not always the day before, sometimes the same day)
Of these, 10 definitely went on to develop symptoms, 7 definitely did not, and 18 were ambiguous (and a few were missing data).
33 people with positives were retested with cue tests, of which 9 were positive.
Of those 24 who tested positive and then negative, 4 tested positive on tests 3 or 4.
Of the

...
Is evolutionary influence the mesa objective that we're interested in?
by David Johnston — 2022-05-03T01:18:06.927Z

Epistemic status: this is the result of me trying to better understand the idea of mesa optimizers. It's speculative and full of gaps, but maybe it's interesting and I'm not realistically going to have time to improve it much in the near future.

Humans are often presented as an example of "mesa optimisers" - organisms created to "maximise evolutionary fitness" that end up doing all sorts of other things including not maximising evolutionary fitness and transforming the world in the process. This analogy is usually accompanied by a disclaimer like this:

We do not expect our analogy to live up to intense scrutiny. We present it as nothing more than that: an evocative analogy (and, to some extent, an existence proof) that explains the key concepts.

I am proposing that if we focus on evolutionary "influence" instead of "fitness", we can flip both claims on their head:

Humans are extremely evolutionarily influential
We should take the evolution analogy seriously

Evolution is about how things change

I think evolutionary fitness is to some extent not the interesting thing about evolution. If the world was always full of indistinguishable rabbits and always will be full of indistinguishable rabbits, then rabbits are in some sense "fit", but evolution is also a boring theory because all it says is "rabbits". The interesting content of evolution is in what it says about how things change: if the world is full of regular rabbits plus one extra-fit rabbit, then evolution says that in a few years, the world will have few regular rabbits and lots of extra-fit rabbits.

I want to propose a rough definition of evolutionary influence that generalises this idea. There are a few gaps in the definition which I hope can be successfully resolved, but I haven't had the time to do this yet.

First, we need an environment. I currently think of an environment as "a universe at a particular point in time". The universe is:

A set T of "points in time"
A set Ω of configurations that the universe can have at a particular time
An update rule f:Ω→Δ(Ω) that probabilistically maps the configuration at one point in time to the configuration at the next (Δ(Ω) means "the set of probability distributions on Ω").

Given a time t∈T, the environment μt is a probability distribution on Ω.

An organism is a particular configuration of a "small piece" of...
Open & Welcome Thread - May 2022 
by Ruby — 2022-05-02T23:47:21.181Z

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you want to explore the community more, I recommend reading the Library, checking recent Curated posts, seeing if there are any meetups in your area, and checking out the Getting Started section of the LessWrong FAQ. If you want to orient to the content on the site, you can also check out the new Concepts section.

The Open Thread tag is here. The Open Thread sequence is here.
Predicting for charity
by Austin Chen — 2022-05-02T22:59:49.741Z

Excerpted from Above the Fold.

Prediction markets succeed when they require people to bet something they value, like money. But past attempts at real-money prediction markets like Intrade have been shut down by the CFTC. At Manifold Markets, we currently allow users to trade on Manifold Dollars (aka M$ or “mana”), an in-game currency specific to our platform. But one of the most common criticisms we hear is: “Why should I care about trading fake currency?”

It would be nice to find a middle ground where users can bet something of real value which doesn’t run afoul of financial regulations. Thinking about this, we were inspired by

donor lotteries

: if you can gamble with charitable donations, shouldn’t you be able to make bets with them? Thus,

Manifold for Good

was born.

You can now donate your M$ winnings to charity! Through the month of May, every M$ 100 you contribute turns into USD $1 sent to your chosen charity - we’ll cover all processing fees!

Why?

Manifold for Good solves two problems with today’s prediction markets. First, it allows you to bet using something valuable to you (i.e. donations to your favorite charity), which increases the incentive to bet correctly, relative to just virtual points. Second, it respects existing financial regulations, which has proven difficult for prediction markets in the past.

By providing an entertaining and impactful way to allocate money to charity, Manifold for Good can also increase the total amount of money donated. Just as donors participating in a charity bingo night are willing to pay extra for the value of entertainment, so too can Manifold’s markets provide a fun, motivating reason to participate in charitable activities.

What’s Next?

Think of Manifold for Good as an experiment! We’re seeing what the level of demand is for this kind of redemption for Manifold Dollars; let us know if you have any thoughts or suggestions.

In the future, we’d like to grow the program to increase the number of available charities. We currently support 30+ charities; if you have a charity recommendation, let us know and

we’ll pay a M$ 500+ bounty once we add it

!

We’d also love to offer donation matching to cause areas or charities — we think this would get users even more excited about forecasting and donating! If you would like to partner with us to fund an experiment like this, or be featured as a charity, please get in touch at give@mani...
Information security considerations for AI and the long term future
by Jeffrey Ladish — 2022-05-02T20:54:12.453Z

Summary

This post is authored by Jeffrey Ladish, who works on the security team at

Anthropic

, and Lennart Heim, who works on AI Governance with

GovAI

(more about us

at the end

). The views in the post are our own and do not speak for Anthropic or GovAI. This post follows up on Claire Zabel and Luke Muehlhauser’s 2019 post,

Information security careers for GCR reduction

.

We’d like to provide a brief overview on:

How information security might impact the long term future
Why we’d like the community to prioritize information security

In a following post, we will explore:

How you could orient your career toward working on security

Tl;dr:

New technologies under development, most notably artificial general intelligence (AGI), could pose an existential threat to humanity. We expect significant competitive pressure around the development of AGI, including a significant amount of interest from state actors. As such, there is a large risk that advanced threat actors will hack organizations — that either develop AGI, provide critical supplies to AGI companies, or possess strategically relevant information— to gain a competitive edge in AGI development. Limiting the ability of advanced threat actors to compromise organizations working on AGI development and their suppliers could reduce existential risk by decreasing competitive pressures for AGI orgs and making it harder for incautious or uncooperative actors to develop AGI systems.

What is the relevance of information security to the long term future?

The bulk of

existential risk

likely stems from technologies humans can develop. Among candidate technologies, we think that AGI, and to a lesser extent biotechnology, are

most likely

to cause human extinction. Among technologies that pose an existential threat, AGI is unique in that it has the potential to

permanently shift the risk landscape

and enable a stable future without significant risks of extinction or other permanent disasters. While experts in the field have

significant disagreements

about how to navigate the path to powerful aligned AGI responsibly, they tend to agree that actors that seek to develop AGI should be

extremely cautious

in the development, testing, and deployment process, given the failures could result in catastrophic risks, including human extinction.

NIST defines

information security

as “The protection of information and information systems from unau...
My thoughts on nanotechnology strategy research as an EA cause area
by Ben_Snodin — 2022-05-02T17:57:53.064Z

This is a cross-post from the Effective Altruism Forum.

Two-sentence summary: Advanced nanotechnology might arrive in the next couple of decades (my wild guess: there’s a 1-2% chance in the absence of transformative AI) and could have very positive or very negative implications for existential risk. There has been relatively little high-quality thinking on how to make the arrival of advanced nanotechnology go well, and I think there should be more work in this area (very tentatively, I suggest we want 2-3 people spending at least 50% of their time on this by 3 years from now).

Context: This post reflects my current views as someone with a relevant PhD who has thought about this topic on and off for roughly the past 20 months (something like 9 months FTE). Note that some of the framings and definitions provided in this post are quite tentative, in the sense that I’m not at all sure that they will continue to seem like the most useful framings and definitions in the future. Some other parts of this post are also very tentative, and are hopefully appropriately flagged as such.

Key points

I define advanced nanotechnology as any highly advanced future technology, including atomically precise manufacturing (APM), that uses nanoscale machinery to finely image and control processes at the nanoscale, and is capable of mechanically assembling small molecules into a wide range of cheap, high-performance products at a very high rate (note that my definition of advanced nanotechnology is only loosely related to what people tend to mean by the term “nanotechnology”). (more)

If developed, advanced nanotechnology could increase existential risk, for example by making destructive capabilities widely accessible, by allowing the development of weapons that pose a higher existential risk, or by accelerating AI development; or it could decrease existential risk, for example by causing the world’s most destructive weapons to be replaced by weapons that pose a lower existential risk. (more)

Timelines for advanced nanotechnology are extremely uncertain and poorly characterised, but the chance it arrives by 2040 seems non-negligible (I’d guess 1-2%), even in the absence of transformative AI. (more)

It seems likely that there’d be a long period of development with clear warning signs before advanced nanotechnology is developed, pushing against prioritising work in this area and pushin

...