trentmkelly/LessWrong-43k huggingface dataset:
On utility functions
by jodaru — 2023-02-10T01:22:47.603Z

Hello all,

I am new to alignment theory and was hoping to get a better understanding of utility functions. In particular I'm wondering why we talk about programs as always optimizing some utility function. Is this a known property of computer programs? Is there a theorem or something that says every computer program optimizes some function?

I'm also wondering, does this apply equally well to programs that can change their own code (or programs running on a quantum computer or other things like that, where it is not as deterministic as a typical python script if that makes sense)

Thanks!
Security Mindset - Fire Alarms and Trigger Signatures
by elspood — 2023-02-09T21:15:59.129Z

Series Overview and Goals

This is the second in a series of articles about applying traditional security mindset to the problems of alignment and AI research in general. As much as possible, we should try to mine the lessons from the history of security and apply them to the alignment problem. To the extent that security is orthogonal to alignment research, good security practices and leveraging existing security capabilities should still help extend timelines, hopefully long enough to allow time for alignment research to make sufficient progress. At the very least, it would be undignified for existential risk to be realized via a basic, preventable security failure.

Fire Alarms

There may be no fire alarm for imminent AGI, but what kind of useful fire alarms could we build with what we know today? What lessons can we draw from fire alarm design and use that could apply to an AGI early warning system? If we have a goal of being able to take some useful action, like leaving the building before being immolated by an unaligned superoptimizer of our own creation, then what kind of alarm features can help us achieve that?

Fire Alarm Detection Signatures and Features

Most smoke alarms work by detecting either disruption in the flow of ionized particles (good for fast-burning fires), or a beam of light (slow, smoldering fires). Some fire suppression systems are also triggered by heat. Some detectors combine a photoelectric and ionization signal for even more reliable signature detection.

What properties can we generalize from this to our AI safety system? Multiple diverse signals are good, if you don't know exactly what type of fire to expect. Each signal type is good at its own particular domain of detection, but has its own drawbacks.

Detection is automated. It does not require a human to overcome inattention blindness. The alarm cannot be bribed or socially engineered. It does not have cognitive bias. The alarm signal is clear and unambiguous; binary.

Fire Alarm Response Procedure

Once we’ve decided what signatures to alert on, what should people actually do when they hear the alarm? KatjaGrace has already written extensively about the psychology of alarm response. We’ll assume for this exercise that if those who are responsible for responding to the alarm have participated in its creation, then most of the social proof aspects won’t apply to the response.

Most people have...
Impostor syndrome: how to cure it with spreadsheets and meditation 
by KatWoods — 2023-02-09T21:04:52.843Z

I got rid of impostor syndrome after having it for nearly ten years, and I predict the results are replicable.

In this post I’ll explain how to do it yourself and describe the upcoming class I’m running where I’ll see if I can replicate it with a small group.

The very summarized version of the method is

Do loving-kindness meditation practice directed towards yourself for an hour a day for a week straight.
Do confidence practice (like loving-kindness, but for confidence. Explained below) for an hour a day for two weeks straight.
(Optional) Do occasional maintenance practice to “top up” your confidence or bounce back from setbacks.

Apply to attend the class here

.

What I tried before

I was a confident kid, then I did an internship at 80,000 Hours back in 2013. Living in Oxford gave me impostor syndrome almost instantly. Turns out that basing your confidence on being a big fish in a small pond only works if you stay in a small pond.

Between 2013 and 2022, I had more or less chronic low confidence and high anxiety around work. I tried everything to get rid of it: mindfulness, cognitive behavioral therapy, internal family systems, memory reconsolidation, exposure therapy, acceptance and commitment therapy, talk therapy, and just plain old reason. Nothing made a dent.

What it feels like to go from impostor syndrome to self-love

When I had impostor syndrome, every time I made a mistake or performed less than perfectly, I’d beat myself up mercilessly. Even when I succeeded by virtually anybody’s standards, I would always have a reason for why it didn’t really count, and couldn’t everybody see all the ways I’d messed up?

I was terrified of receiving feedback because then I would find out that secretly everybody hated me and thought I was useless, and they were just putting up with me because they had to. I felt I was always one step away from being kicked out of the community and being eaten by bears.

I've since experienced a 90% reduction in my work anxiety. After failures, I no longer feel debilitating self-recrimination; instead, I am inspired to think about what went wrong and improve.

In the past, when I looked at pictures of myself, the first thing that came to mind was how ugly I was. My initial reaction is now, "Aw. I love that girl."

I can now look clearly at the things that I’ve done and see the bad and...
Conditioning Predictive Models: Deployment strategy
by evhub — 2023-02-09T20:59:01.473Z

This is the sixth of seven posts in the Conditioning Predictive Models Sequence based on the paper “Conditioning Predictive Models: Risks and Strategies” by Evan Hubinger, Adam Jermyn, Johannes Treutlein, Rubi Hudson, and Kate Woolverton. Each post in the sequence corresponds to a different section of the paper.

6. Deployment strategy

Previously, we have been focusing on how to make conditioning predictive models as safe and competitive as possible. Now, we want to take a step back and discuss considerations for using conditioning predictive models to address AI existential risk and what sorts of difficulties we might run into doing so in practice.

In particular, just as AI safety researchers naturally think of using predictive models for advancing AI safety research, AI capabilities researchers might naturally jump to using predictive models for advancing capabilities. It may not even be necessary to generate additional research to build AGI with a powerful predictive model. Simply ignoring the previously-discussed ELK-related difficulties and training a model to take actions that lead to predicted futures that a predicted human approves of may be sufficient. Either way, the existence of powerful predictive models seems likely to rapidly contract AI timelines.

As a result, by the time predictive models can be used to predict a full solution to AI safety, the time available to do so is minimal—and as such, it is important to have fleshed out plans on how to use them safely well ahead of time.

Dealing with other, less careful actors

As we mentioned previously, using a predictive model to generate alignment research is only one possible use case—one that we restricted our attention to on the basis that we thought it contained the difficult aspects of using a predictive model safely. Restricting our attention to these sorts of particular conditionals—and figuring out how to do them safely—is fine if we have control over the ways in which our model will be used. If we don’t have that control, however—e.g. we are in a world where people are using predictive models in all sorts of different ways—then we have to consider what might happen when our predictive model is used in a much less careful way than described here and figure out how to either deal with or prevent that from happening.

We think that getting other actors to use predictive models at all should be quit...
Make Conflict of Interest Policies Public
by jefftk — 2023-02-09T19:30:01.282Z

If I'm interviewing someone for a position my job is to assess their
suitability as a potential employee, but if they're my cousin I might
be tempted to give them an overly favorable review. Most
organizations have Conflict of Interest (CoI) policies that describe
how to handle this sort of situation: it's common that someone might
have external relationships which lead to duties, interests, or
desires that conflict with what's best for their organization.

It's reasonably common for non-profits to publish their CoI policies (Hewlett,
Carnegie, Gates). Within effective altruism I do see some of this:

Animal Charity Evaluators has a great
page on their policies, which includes their full CoI policies.

Open Philanthropy shares their Relationship
Disclosure Policy, though I (and others)
don't see their full policy.

GiveWell has a public Conflict
of Interest Policy for Board Members, Officers, and Key Persons,
though probably also has an internal one for regular employees that
seems not to be public? They maintain a Relationship
Disclosures page, which I think is great.

Historically people
and organizations
within the EA movement have prioritized transparency, and while
there's been some
shift away from the most enthusiastic versions of this as we've better
understood the costs, there are still a lot of benefits. If
you're already going to the effort of drafting a policy like this,
making it public seems pretty useful:

EAs who are concerned about CoIs within the community and are
thinking about what norms they might try to influence can see what's
already formally in place.

Other organizations can reference it in trying to figure out
what sort of policy they want.

People who are worried a situation can see what policy was
(supposed to have been) followed.

On the other hand, many EA organizations don't seem to have public
policies. This includes ones that work in community building or
grant-making where they seem pretty important. Here are a few I
checked:

The Centre for Effective Altruism mentions
that they have a "written conflict of interest policy", but it
doesn't seem to be public.

The Effective Ventures Foundation doesn't list anything on
their site, but says in their 2021
IRS filings that "when a conflict of interest arises a plan for
regularly checking in with the involved parties is done monthly

...
Curated blind auction prediction markets and a reputation system as an alternative to editorial review in news publication. 
by ciaran  — 2023-02-09T18:48:18.493Z

If one were to build an open platform to compete with news media rather than social or blogging media, they would need some sort of accountability/quality control mechanism to mirror the role of editorial review at prestigious news platforms.

The best tool for aggregating information that we have devised is markets. This is especially the case in open systems. The biggest risk to a well-functioning market is collusion. The second major problem with prediction markets for information is that bettors can become more concerned with predicting what people think the truth is rather than predicting the actual truth. Any mechanism that was designed to employ markets in the place of editorial review would need to guard against these failings. With this in mind I propose this system:

A reporter with information on a climate event in Costa Rica writes an article and publishes it to the platform. They must stake a certain TBD amount of money on this article.
The article is published without editorial review (unlike how news media currently works).
Prospective fact checkers (post editorial reviewers) also stake money and state their specialist topics. Some of those that listed Costa Rican current affairs or climate as a topic are randomly chosen from the available pool of people. They are each given guidelines on how to judge an article and use these guidelines to give the article a trust score without colluding among themselves as they don’t who else's been asked to fact check. The article and the guidelines together likely represent the only Schelling point the fact checkers can converge on as the position of the market is unknown prior to settlement. Information about any consensus that might exist outside the truth is unknown. A fact checking assignment is like jury duty. Some of your stake is slashed if you renege on giving a score for a given article in your specialty. This requirement along with random selection should minimise any collusion.
The writer’s payout is determined by the score the fact checkers give him/her. The fact checkers’ payouts are determined by how close they are to the average score from the group. Everyone’s respective rep scores are also updated. Those that drop below a certain trust score will not have their articles listed on the platform or be chosen for post editorial review. Articles from the best performers will be amplified on the platfor

...
Tools for finding information on the internet
by RomanHauksson — 2023-02-09T17:05:28.770Z
Link: https://roman.computer/finding_information/

Edit 2023-05-09: I recorded a presentation for EA Software Engineers about this post. In it, I demonstrate each of the tools and discuss some extra ones at the end, namely content blockers, userscripts, and alternative front-end websites.

Isn't the internet such a magically useful tool? Thirty years ago, if you wanted to know how many plays Shakespeare wrote, you would have to physically walk to your local library and find a relevant book. Now, you can find the answer in less than ten seconds, at any time, wherever you are.

However, the internet is not a truthful, superintelligent oracle. Rather, it's a dangerous jungle of knowledge you must learn to navigate if you wish to find the truth. Good information is censored, hidden behind paywalls or within piles of spam, and difficult to differentiate from untrustworthy information. This post won't be a complete guide on how to navigate the world wide web of knowledge, but it will give you some tools I've discovered over the years that you can throw in your digital rucksack to aid your journey.

Search engines

The great internet sage Gwern Branwen wrote an advanced guide on finding ref­er­ences, pa­pers, and books on­line.
The search engines Brave Search and Kagi have the features "Goggles" and "Lenses" respectively, which are presets that filter or re-rank entire categories of websites in your results.
SearXNG is a highly customizable internet metasearch engine.
Perplexity uses natural language processing to answer your query with a paragraph (with sources) and allows you to ask followup questions.
Metaphor allows you to find websites by writing creative and long-form prompts, also using NLP.
Elicit is a research assistant that helps you find relevant research papers, also using NLP.

Bypassing restrictions

Sometimes you know exactly where to find a piece of information, but it's locked behind a paywall or deleted from the internet.

Unddit displays deleted comments and posts on Reddit.
Internet Archive is a non-profit library of free books, movies, websites, et cetera. It's famous for the Wayback Machine, which displays past archived snapshots of a given URL.
Bypass Paywalls is a browser extension to help bypass paywalls on selected sites.
The subreddit r/piracy has a wiki with loads of resources on obtaining copyrighted material for free.
Anna's Archive is a shadow library metasearch engine that aggregates results from websites

...
Covid 2/9/23: Interferon λ
by Zvi — 2023-02-09T16:50:01.022Z

There are two big pieces of pandemic news this week.

One is that there is growing alarm about the dangers of avian flu, or H5N1. Likely mammal-to-mammal transmission has been identified, a lot of wild animals are infected, and there is a real danger that this will become a pandemic. I will cover this in its own post.

The other is that there are trial results for a new Covid treatment, Interferon λ. It looks super effective. Cuts hospitalizations and deaths in half, with no major side effects.

Despite this, there is zero expectation by anyone that it will be available any time soon, and little surprise about this reaction. You see, the trial was done by academics and was in Brazil and Canada. So it doesn’t count. Sorry.

Executive Summary

We have a new drug that cuts Covid risk of hospitalizations and deaths by about half with no major side effects.

You can’t have it. Maybe ever.

Because FDA. FDA Delenda Est.

Let’s run the numbers.

The Numbers

Predictions

Predictions from Last Week: 210k cases (-8%) and 3,150 deaths (-8%)

Results: 238k cases (-6%) and 3,052 deaths (-11%)

Predictions for Next Week: 220k cases (-7%) and 2,750 deaths (-10%).

Last week I forgot to manually adjust Florida for cases, which I’ve fixed. That’s why the percentages above don’t match. Also means my prediction last week for cases was worse than I thought – cases were actually slightly up and I predicted a drop.

I expect deaths to continue to drop on schedule. Valentine’s Day is not a reporting-relevant holiday, although it is still a date it is important not to forget.

Deaths

Cases

Interferon λ

As noted up top: It looks super effective. Cuts hospitalizations and deaths in half, with no major side effects. Despite this, there is zero expectation by anyone that it will be available any time soon, and little surprise about this reaction.

Here is their results summary.

A total of 933 patients were assigned to receive pegylated interferon lambda (2 were subsequently excluded owing to protocol deviations) and 1018 were assigned to receive placebo. Overall, 83% of the patients had been vaccinated, and during the trial, multiple SARS-CoV-2 variants had emerged. A total of 25 of 931 patients (2.7%) in the interferon group had a primary-outcome event, as compared with 57 of 1018 (5.6%) in the placebo group, a difference of 51% (relative risk, 0.

...
EIS II: What is “Interpretability”? 
by scasper — 2023-02-09T16:48:35.327Z

Part 2 of 12 in the

Engineer’s Interpretability Sequence

.

A parable based on a true story

In 2015, a Google image classification app classified many photos depicting black people as gorillas.

Image from WSJ.

Remember

Google’s infamous blunder

from 2015 in which users found that one of its vision APIs often misclassified black people as gorillas? Consider a parable of two researchers who want to understand and tackle this issue.

Alice is an extremely skilled mechanistic interpretability researcher who spends a heroic amount of effort analyzing Google’s model. She identifies a set of neurons and weights that seem to be involved in the detection and processing of human and gorilla faces and bodies. She develops a detailed mechanistic hypothesis and writes a paper about it with 5 different types of evidence for her interpretation. Later on, another researcher who wants to test Alice’s hypothesis edits the model in a way that the hypothesis suggests would fix the problem. As it turns out, the hypothesis was imperfect, and the model now classifies many images of gorillas as humans!
Bob knows nothing about neural networks. Instead of analyzing the network, he looks at the dataset that the model was trained on and notices a striking lack of black people (as was indeed the case in real life

(Krishnan, 2020)

). He suggests making the data more representative and training the model again. When this is done, it mostly fixes the problem without side effects.

The goal of this parable is to illustrate that when it comes to doing useful engineering work with models, a mechanistic understanding may not always be the best way to go. We shouldn’t think of something called “interpretability” as being fundamentally separate from other tools that can help us accomplish our goals with models. And we especially shouldn’t automatically privilege some methods over others. In some cases, highly involved and complex approaches may be necessary. But in other cases like Alice’s, the interesting, smart, and paper-able solution to the problem might not only be harder but could also be more failure-prone. This isn’t to say that Alice’s work could never lead to more useful insights down the road. But in this particular case, Alice’s smart approach was not as good as Bob’s simple one.

Interpretability is a means to an end.

Since I work on and think about interpret...
The Engineer’s Interpretability Sequence (EIS) I: Intro
by scasper — 2023-02-09T16:28:06.330Z

Part 1 of 12 in the Engineer’s Interpretability Sequence.

If we want to reduce near and long term risks from AI, we should care a lot about interpretability tools. This is a very uncontroversial claim to make inside the AI safety community. Almost every agenda for safe advanced AI incorporates interpretability in some way. The key value of interpretability tools is that they aid in human oversight by enabling open-ended evaluation.

Short of actually deploying a system, any method of evaluating it can only be a proxy for its actual performance. The most common way to evaluate a model is by its performance in some test set or environment. But test sets alone can fail to reveal – and often incentivize – undesirable solutions involving overfitting, biases, deception, etc. This highlights the need for other ways to evaluate models, and an interpretability toolbox full of effective tools may go a long way.

Some of the seeds of the AI safety community’s interest in interpretability were planted by

Distill

in 2017. But 2022 was an inflection point with a massive new surge in interest and work on interpretability tools.

Anthropic

was founded a little over a year ago.

ARC

started less than a year ago.

Redwood

has begun to push for much more interpretability work, including with the

REMIX

program. We are seeing a

number

of

pushes

to get many more people involved in interpretability work. And as someone on the ground, I have subjectively observed a surge in interest over 2022. And the popularity of interpretability hasn’t been limited to the AI safety community. There is now so much work in interpretability that we now have a dataset of 5199 interpretability papers

(Jacovi, 2023)

. See also a survey of 300+ of them from some coauthors and me

(Räuker et al., 2022)

.

Growth in the interpretability literature by year from

Jacovi (2023)

.

But despite all this work, interpretability research has limitations. One of the goals of this sequence is to argue that:

Interpretability research both within the AI safety space and at large is not very productive and may be on course to stay this way.

This is intentionally baitey, and I mean to make this point with a large amount of detail and nuance over the course of this sequence. But one striking thing about interpretability research is that:

For all the interpretab...