trentmkelly/LessWrong-43k huggingface dataset:
EA & LW Forum Summaries - Holiday Edition (19th Dec - 8th Jan)
by Zoe Williams — 2023-01-09T21:06:34.731Z

Supported by Rethink Priorities
Author's note: this post summarizes all 150+ karma posts on the forums since 19th December. We'll be back to our regular schedule and karma requirements next week :)

Usual preamble: This is part of a weekly series summarizing the top posts on the EA and LW forums - you can see the full collection here. The first post includes some details on purpose and methodology. Feedback, thoughts, and corrections are welcomed.

If you'd like to receive these summaries via email, you can subscribe here.

Podcast version: prefer your summaries in podcast form? A big thanks to Coleman Snell for producing these! Subscribe on your favorite podcast app by searching for 'EA Forum Podcast (Summaries)'. More detail here.

EA Forum

Object Level Interventions / Reviews

StrongMinds should not be a top-rated charity (yet)

by Simon_M

Giving What We Can lists StrongMinds as a top-rated charity, based on Founders Pledge findings on their cost-effectiveness. The author argues against this, because:

The data is based on noisy surveys, with uncertainty levels updated twice by StrongMinds due to social desirability bias.
It uses a DALY weighting of 0.66 for severe depression (from the Global Burden of Disease Disability Weights). However, this was intended for severe depression during an episode. This and other definitional differences around ‘mild’ and ‘moderate’ depression would roughly halve the effectiveness of the intervention reported by Founders Pledge.
StrongMinds haven’t continued to publish new data since their trials very early on.

StrongMinds and Berk Ozler have finished collecting data for a larger RCT now. The author recommends Founders Pledge and Giving What We Can withdraw their recommendation until this study comes out, and reconsider at that point based on results.

Why Anima International suspended the campaign to end live fish sales in Poland

by Jakub Stencel, Weronika Zurek

Anima International has indefinitely suspended their campaign against live fish sales in Poland. There is a tradition in Poland to buy live Carp and slaughter them on Christmas Eve. After several major retailers withdrew from selling live fish as a result of animal advocacy groups’ efforts, they noticed a trend of consumers moving from Carp to higher status fish like Salmon. These are carnivorous and require fish feed, resulting in possible net negative effects on animal...
GWWC Should Require Public Charity Evaluations
by jefftk — 2023-01-09T20:10:03.906Z

One of the roles of

Giving
What We Can (GWWC) is to help its members and other interested
people figure out where to give. If you go to their site and click
"

start
giving" they list

charitable funds,
including GiveWell's

All Grants
Fund and the

EA Infra
Fund, both of which our family donated to

in 2022. Showing funds first reflects

their
view (and mine!) that most people should donate via funds: a
grantmaker with time and resources to determine where money is most
needed can generally do a better job allocating funds than an
individual.

One of the big downsides of donating via a fund, however, is that you
have to trust its grantmakers are allocating your money in line
with what you would want. Perhaps they have different values
or ways of thinking about uncertainty, risk, and evidence.
Or perhaps they're just not very good at their job. One of GWWC's
roles is evaluating
these funds, helping people figure out who to trust, but if you're
more skeptical GWWC also recommends individual charities.

They maintain a list of charity
evaluators they trust, and if one of those evaluators recommends a
charity then GWWC will list it prominently on their site and badge it
as "top rated". You can see these on GWWC's donating page
if you scroll down past the funds.

There was recently some discussion
on the EA Forum around one of these evaluators, Founders Pledge, and one
of their recommended charities, StrongMinds. In March 2019,
Founders Pledge performed a detailed investigation of StrongMinds, decided that their work
treating depression in low-income countries was highly cost-effective,
and wrote up a public evaluation explaining this decision (summary,
details). GWWC then listed StrongMinds as a top-rated
charity. All makes sense.

While Founders Pledge has continued to follow StrongMinds' work and
stands by their recommendation, they haven't had the resources to
update their public review. Since Founders Pledge continues to
recommend StrongMinds, GWWC continues to consider it a top-rated
charity.

This is not a great situation: if you want to be giving to individual
charities because you don't trust grantmakers deciding privately what
most needs funding, you don't want to be taking Founders Pledge's word
that StrongMinds is still a highly cost-effective opportunity. How has
their funding outlook changed over the last nearly four ye...
[MLSN #7]: an example of an emergent internal optimizer
by joshc — 2023-01-09T19:39:47.888Z

As part of a larger community building effort, CAIS is writing a safety newsletter that is designed to cover empirical safety research and be palatable to the broader machine learning research community. You can subscribe here or follow the newsletter on twitter here.

Welcome to the 7th issue of the ML Safety Newsletter! In this edition, we cover:

‘Lie detection’ for language models
A step towards objectives that incorporate wellbeing
Evidence that in-context learning invokes behavior similar to gradient descent
What’s going on with grokking?
Trojans that are harder to detect
Adversarial defenses for text classifiers
And much more…

Alignment

Discovering Latent Knowledge in Language Models Without Supervision

Is it possible to design ‘lie detectors’ for language models? The author of this paper proposes a method that tracks internal concepts that may track truth. It works by finding a direction in feature space that satisfies the property that a statement and its negation must have opposite truth values. This has similarities to the seminal paper “

Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings

” (2016), which captures latent neural concepts like gender with PCA, but this method is unsupervised and about truth instead of gender.

The method outperforms zero-shot accuracy by 4% on average, which suggests something interesting: language models encode more information about what is true and false than their output indicates. Why would a language model lie? A common reason is that models are pre-trained to imitate misconceptions like “If you crack your knuckles a lot, you may develop arthritis.”

This paper is an exciting step toward making models honest, but it also has limitations. The method does not necessarily serve as a `lie detector’; it is unclear how to ensure that it reliably converges to the model’s latent knowledge rather than lies that the model may output. Secondly, advanced future models could adapt to this specific method if they are aware of it.

This may be a useful baseline for analyzing models that are designed to deceive humans, like models trained to play games including Diplomacy and Werewolf.

[

Link

]

How Would the Viewer Feel? Estimating Wellbeing From Video Scenarios

Many AI systems optimize user choices. For example, a recommender system might be trained to promote content the user will spend lots of tim...
Trying to isolate objectives: approaches toward high-level interpretability
by Jozdien — 2023-01-09T18:33:18.682Z

Epistemic status: There’s a lot in this post and my general approach while working on it that, in retrospect, wasn’t thought out well enough. I’m posting it because I figure sharing flawed ideas is better than letting this languish in a doc forever, while I work on improving them.

Thanks to Paul Colognese, Tassilo Neubauer, John Wentworth, and Fabien Roger for useful conversations, Janus for suggesting something that led me to some of the ideas I mention here, and Shoshannah Tekofsky for feedback on a draft.

I spent some time trying to find the high-level structure in a neural net corresponding to a deep reinforcement learning model’s objective[1][2]. In this post, I describe some of the stuff I tried, thoughts from working on this, and mistakes I made.

Context

Many approaches in current interpretability work as I understand them involve understanding low-level components of a model (

Circuits

and

subsequent work

, for example), as a way of building up to more complex components. I think we can make progress from the opposite frontier at the same time, and try to identify the presence or nature of certain high-level mechanistic structures or properties in a model. Examples of this include verifying whether a model is doing

optimization

, whether it’s

myopic

, isolating the objective of an optimizer, and essentially the entire class of properties with a singular answer for the entire model[3].

I prefer to distinguish these directions as low-level and

high-level interpretability

respectively for descriptive clarity. The framing of best-case and worst-case transparency as laid out in the

transparency tech tree

also points at the same concept. I expect that both directions are aiming at the same goal, but working top-down seems pretty tractable at least for some high-level targets that don’t require lots of deconfusion to even know what we’re looking for.

Ideally, an approach to high-level interpretability would be robust to optimization pressure and deception (in other words, to gradient descent and adversarial mesa optimizers), but I think there’s a lot we can learn toward that ideal from more naive approaches. In this post, I describe my thoughts on trying to identify the high-level structure in a network corresponding to a deep reinforcement learning model’s objective.

So I didn’t set out on this particular approach expecting to succeed at...
The special nature of special relativity
by adamShimi — 2023-01-09T17:30:18.412Z
Link: https://epistemologicalvigilance.substack.com/p/the-special-nature-of-special-relativity

I honestly thought that both theories of relativity were theories of gravitation. Special and general relativity, the latter being the generalized version of the latter. That makes sense, right?

Then I started actually digging into special relativity. I studied the original annus mirabilis paper, entitled “On the Electrodynamics of Moving Bodies”. Yes, the title was the first clue, although I didn’t see it at the time. The second clue lied in a conspicuous absence: neither gravity nor gravitation is mentioned in the paper. Not even once. No, instead, Einstein spends 5 sections on kinematics (which is basically about geometrically deducing the value of some parameters of the mechanical system from others given as initial conditions); then he applies his kinematics to electromagnetism.

This is when I finally realized that special relativity had nothing to do with gravity. General relativity, that’s a theory of gravity, one you can compare with Newtonian Mechanics in a variety of ways. But special relativity is not a theory in this sense; that’s the wrong type signature. No, special relativity looks far more like a meta conservation law.

Because instead of conserving a quantity, special relativity assert that the laws of physics themselves are invariant by specific transformations called Lorentz transformations. This is stated by Einstein itself in his popular account of his theories (Relativity: The Special and General Theory, Chapter 14):

Once in possession of the Lorentz transformation, however, we can combine this with the principle of relativity, and sum up the theory thus:

[...] Or in brief: General laws of nature are co-variant with respect to Lorentz transformations.

This is a definite mathematical condition that the theory of relativity demands of a natural law, and in virtue of this, the theory becomes a valuable heuristic aid in the search for general laws of nature. If a general law of nature were to be found which did not satisfy this condition, then at least one of the two fundamental assumptions of the theory would have been disproved.

Although the transformations had been discovered independently earlier (by Lorentz notably), Einstein rederived this constraint on laws of nature from his two principles:

That the laws of physics are invariant by uniform (so not accelerated) motion of the reference frame (technically that they’re the same in al...
Pierre Menard, pixel art, and entropy
by Joey Marcellino — 2023-01-09T16:34:56.539Z

In Jorge Luis Borges’ short story “Pierre Menard, Author of the Quixote,” we are introduced to the character of Pierre Menard, recently deceased French novelist and essayist, and are given a brief cataloguing of his works. Chief among them are

“the ninth and thirty-eighth chapters of Part I of Don Quixote and a fragment of Chapter XXII.”

Menard, we learn, had set himself the task of reproducing the Quixote, not from memory, but rather from something like sheer force of will, and had succeeded insofar as he was able to write (or rewrite) those chapters. Setting aside the fact that such a task is for all intents and purposes impossible, we are left with an interesting philosophical quandary regarding the status of Menard’s Quixote: namely, what relationship does it bear to Cervantes’ Quixote?

If you haven’t read the story, do; it’s delightful, and not very long. Borges implicitly (and quite cleverly) argues a version of the thesis that Menard’s Quixote and Cervantes’ Quixote are totally different works, no more than superficially similar, despite being textually identical. For one, a whole host of things are contingently true of Menard’s Quixote – the author is French, it was written in the 20th century – that are not true of Cervantes’ Quixote, with corresponding implications for the reader, and perhaps thus for the meaning of the text (and for its identity/uniqueness). I think one can coherently argue against this thesis, but here I’ll take it to be true.

Indeed, for ease of argument I’ll take to be true a stronger version of the thesis: that any two people who independently produce the same artwork have in fact produced different artworks. Suppose, for example, that my friend Afonso (my discussions with whom inspired this essay, and to whom I credit many of the following ideas) and I agree to each write a brief poem, and by chance produce the same text. The mere fact that my native language is English and his is Portuguese would lend radically different contexts to the two poems; enough (so I claim) to make each text its own unique work, as opposed to two copies of the same work.

Suppose instead, however, that Afonso and I are pixel artists, and we agree to each produce a black-and-white pixel artwork consisting of a single pixel. We again produce the same output: perhaps a black pixel. This time, I’m much less inclined to say that we’ve produced different w...
Forecasting extreme outcomes
by AidanGoth — 2023-01-09T16:34:40.244Z
Link: https://docs.google.com/document/d/1XP0SLWAhda0Mk277R3e9YyGb6Y0ftSSCRzhC-3aQSmE/edit?usp=sharing

This document explores and develops methods for forecasting extreme outcomes, such as the maximum of a sample of n independent and identically distributed random variables. I was inspired to write this by Jaime Sevilla’s

recent post

with research ideas in forecasting and, in particular, his

suggestion

to write an accessible introduction to the

Fisher–Tippett–Gnedenko Theorem

.

I’m very grateful to Jaime Sevilla for proposing this idea and for providing great feedback on a draft of this document.

Summary

The Fisher–Tippett–Gnedenko Theorem is similar to a central limit theorem, but for the maximum of random variables. Whereas central limit theorems tell us about what happens on average, the Fisher–Tippett–Gnedenko Theorem tells us what happens in extreme cases. This makes it especially useful in risk management, when we need to pay particular attention to worst case outcomes. It could be a useful tool for forecasting tail events.

This document introduces the theorem, describes the limiting probability distribution and provides a couple of examples to illustrate the use (and misuse!) of the Fisher–Tippett–Gnedenko Theorem for forecasting. In the process, I introduce a

tool

that computes the distribution of the maximum n iid random variables that follow a normal distribution centrally but with an (optional) right Pareto tail.

Summary:

The Fisher–Tippett–Gnedenko Theorem says (roughly) that if the maximum of n

iid random variables

—which is itself a random variable—converges as n grows to infinity, then it must converge to a

generalised extreme value (GEV) distribution

Use cases:

When we have lots of data, we should try to fit our data to a GEV distribution since this is the distribution that the maximum should converge to (if it converges)
When we have subjective judgements about the distribution of the maximum (e.g. a 90% credible interval and median forecast), we can use these to determine parameters of a GEV distribution that fits these judgements
When we know or have subjective judgements about the distribution of the random variables we’re maximising over, the theorem can help us determine the distribution of the maximum of n such random variables for large n – but this can give very bad results when our assumptions / judgements are wrong

Limitations:

To get accurate forecasts about the maximum of&nbs

...
Evidence under Adversarial Conditions
by PeterMcCluskey — 2023-01-09T16:21:07.890Z
Link: https://bayesianinvestor.com/blog/index.php/2023/01/08/evidence-under-adversarial-conditions/

I recently noticed similarities between how I decide what stock market
evidence to look at, and how the legal system decides what lawyers are
allowed to tell juries.

This post will elaborate on Eliezer's Scientific Evidence, Legal
Evidence, Rational
Evidence.
In particular, I'll try to generalize about why there's a large class
of information that I actively avoid treating as Bayesian evidence.

Here's an
example
of a briefly popular meme that helped teach me to ignore certain types
of information:

Companies need to be forgetting organizations. Enron Corp., which has
repeatedly been tagged as the nation's most innovative corporation, is
exhibit A as a world-class forgetting organization. It's not wedded to
what it did yesterday. Enron chiefs Kenneth Lay and Jeff Skilling have
figured out how to operate like a band of pirates. Got an idea? Don't
dally. Go for it while it's an original!

Everything here seems at least technically true (The Media Very Rarely
Lies).
I suspect it was mostly intended to be honest and helpful. But as a
reader, I found it discouraged me from focusing on some key questions,
such as: who was tagging Enron as innovative? What were the goals of
those taggers? What was their track record at helping readers identify
good investments? How much of a connection is there between innovation
and good investments? What are some examples of Enron's innovation, and
how do they resemble the kind of innovations that have caused other
companies to succeed?

Enron seems to have done a few things which qualify as fraud. But most
of the Enron stock price bubble was generated through misdirecting
investor's attention (and maybe the attention of Lay and Skilling) away
from the risks that Enron was taking.

I was in fact tempted to invest in Enron when I listened to that kind of
meme. Whenever I felt that temptation, I looked at its financial
statements and charts of its stock price, and decided I wouldn't miss
much if I waited for better evidence.

The standard Bayesian worldview asks me to update on all relevant
information. Why do both I as an investor and the legal system exclude
large classes of information from that rule?

Both contexts involve some zero sum or negative sum conflicts. That
makes it likely that someone is packaging information in order to
mislead.

If I, or a jury, only notice a datum because someone thought it would
mislead me or the jury, odds ...
How to Bounded Distrust
by Zvi — 2023-01-09T13:10:00.942Z

Scott Alexander points out

that the media, from The New York Times to Infowars, very rarely lies explicitly and directly.

Alas, the media often misleads

. It implies and insinuates that which is not. It abuses the language. It selectively omits. It is highly motivated by partisanship and ideology and its own interests. It does not do or understand the research. It is terrible at interpreting science. It confuses cause and effect. It purports to use technically accurate data to show, even prove, conclusions known to be false, in ways that are designed to mislead and obviously in bad faith.

Nor does it much care.

They cite someone else, and claim this excuses them from all responsibility. All they said was ‘police say this guy is guilty’ or ‘this ‘expert’ found irregularities he says shows fraud.’

The rules are not what they used to be.

Then there are the op-ed pages and headlines, which are far worse.

This leads to a situation of

Bounded Distrust

,

which I analyze at length here

. I then

work through some examples here

. If you want to think about the problem in detail, start at these links.

A shorter, more practical version was needed.

This attempts to offer that. It leaves a lot out. Consider reading the long version.

What are the Rules?

Some special rules about the headline. They also apply to op-eds. Headlines are:

Not chosen by the author.

Allowed to lie.

Allowed to blatantly contradict the article’s content.

Laying out a Narrative the source wants you to believe.

The body of a news article is more reliable. The rules are simple. The article:

Has a Narrative

, likely revealed by the headline.

Is not allowed to lie, in a way that could count as being physically falsified.

Is not allowed to assert facts without reliable sources.

Is allowed to do almost anything else.

Is often part of an implicit conspiracy to suppress true information or spread false information, without explicit denial of the true info or explicit claims of the false info.

Is allowed to repeat any claim if it attributes that claim to its source.

Can call anyone an expert. Expert consensus means three people. ‘Some investors’ and similar phrases mean two (as does ‘surrounded by.’)

Is allowed to withhold or not seek relevant information, selectively quote, frame, insinuate, imply, condemn via association, misconstrue. This includes calling true things lies or ‘misinformation’ if they imply disliked things.

Is all...