trentmkelly/LessWrong-43k huggingface dataset:
A problem with group epistemics
by Mckay Jensen — 2023-01-18T17:06:07.114Z
Link: https://quevivasbien.github.io/posts/problem-with-group-epistemics.html

(Cross-posted from my site.)

My goal here is to quickly describe a potential problem that groups of people may have when trying to collectively figure out the truth of some subject.

The basic idea is that people may use each others' beliefs to inform their own beliefs, without properly accounting for the fact that others' beliefs are based on some of the same evidence as their own beliefs.

In what follows I present a simplified example of how this might happen, a bit of math to guide our thinking, and some illustrative plots.

The basic takeaway is that a failure to "de-correlate" others' beliefs from the supporting evidence leads to sub-optimal epistemics. In particular:

Groups that make this error may be overconfident in their beliefs.

This can make it harder to recover accurate beliefs if the first people to explore a topic reach conclusions far from the truth.

An illustrative example

Suppose that Alice is trying to figure out whether the following claim is true:

AGI will be developed sometime in the next 20 years.

Alice starts with some prior, and after investigating this claim updates her belief to be more confident that it is true.

Later, Bob comes along and wants to know whether this claim is true. He investigates the claim, using, for the most part, the same sources of information that Alice used, and arrives at a belief similar to Alice's posterior. He then notes that Alice also updated her belief to be more confident of the claim's truth, and uses that as additional evidence; he thus ends up even more confident than Alice that AGI will be developed in the next 20 years.

Bob's last step is of course an error: if Alice and Bob used the same information to form their beliefs, Bob shouldn't use Alice's belief to inform his own belief -- the information expressed through Alice's posterior belief is the same as the information in the evidence that Bob already reviewed. Bob needs to "de-correlate" the information he sees from the information in Alice's belief -- in this case, he will find that the two sources of information are perfectly correlated, meaning that they are redundant.

We probably aren't so naive as to completely fail to decorrelate the information in other people's beliefs from what we see; but it seems likely that we make this mistake to some extent. We may, therefore, want to be a bit skeptical of common beliefs formed in a group where raw evi...
Why you should learn sign language
by Noah Topper — 2023-01-18T17:03:24.090Z
Link: https://naivebayes.substack.com/p/why-you-should-learn-sign-language

Sorry if this is somewhat off topic, but I have a small dream of sign language becoming a rationalist shibboleth and serving as a jumping off point for sign to spread more in society at large. I know this is pretty unlikely, but it seems worth a shot. I gave a presentation on it at a LessWrong Community Weekend, and people seemed interested. Sign language is also super intellectually interesting in its own right and is a very fun learning experience! Anyway, on with the show.

(EDIT: A few extra notes based on comments. First, I actually think ASL is pretty easy to learn from the video lectures I link below. I did okay in a one-on-one conversation with a deaf person after that. Second, you'll obviously have to weigh up if learning to sign is worth it for you. I have just tried to argue that it's more useful and enriching than you might otherwise have thought. Finally, this mostly is just a Neat Thing. The only "rationalist" aspects would come from an interest in learning neat things, and my argument that society would be better optimized if everyone knew how to sign. Sorry again for being off topic.)

Introduction

I’ve been learning sign language for four years now, and I swear it’s fun like nothing else in the world. At the risk of projecting my interests onto others, I’d like to make the case for learning to sign. I’ll focus primarily on American Sign Language (ASL) since it’s what I know, but most of what I say should generalize to other sign languages too.

If you want to learn ASL, I highly recommend

Dr. Bill Vicars’ materials here

. Vicars has a PhD in deaf education, is deaf himself, and has published his core ASL 1 - 4 lectures online, plus much more, free of charge. He is seriously excellent.

What is sign language anyway?

First, I should explain what sign language is. The basics are probably pretty clear: it’s a language where you communicate using the shape and movement of your hands, rather than spoken or written words (although really other body parts are involved too, like facial expressions). So that’s what sign language is. But more importantly, what is it not?

ASL is not a

conlang

. Conlangs (or constructed languages) are purposefully invented and planned for some purpose, such as

Esperanto

for international communication, or

Lojban

for unambiguous communication.

While we’re at it, ASL is not a code for English. For example, Braille is a...
Flying With Covid
by jefftk — 2023-01-18T17:00:03.542Z

I flew out to CA for a

gig
this weekend, and two of my kids

came along. We got into SFO
late Thursday night,

drove to
San Jose to sleep, and got to the gig on Friday. We took rapid tests,
all negative. A late night, and the kids fell asleep backstage in the
keyboard case:

Saturday morning one was mostly fine, but the other was sleepy ("jet
lag"). She stayed sleepy all day, napping more backstage, and she
wasn't eating much ("different food than she usually eats"). After
choosing what food we should get for dinner she wouldn't eat it and
was cranky ("seems more 'sick' than 'jet lag'"). Another rapid
test [1] immediate positive:

So I'm 3,000 miles from home with two kids, one of which has the
contagious disease everyone's spent the last three years
avoiding. We're scheduled to check out by noon tomorrow, drive back to
SFO, and take a red-eye home. What do I do?

Two options:

Isolate in place.

Fly home anyway.

Earlier in the pandemic this would have been a clear choice: isolate
in place. It was required by public health order, and especially
before vaccination traveling would have been too much risk to others
and contribution to pandemic spread.

On the other hand, pre-covid I think almost everyone would have gone
home. If you read
advice
it was mostly about how to decide whether to cancel a vacation or how
to keep a sick kid comfortable on a plane. While there's some "if you're
sick, don't fly", the examples were whether to begin a trip, not
whether to return home. Discussion of "well enough to fly" was about
whether you were up to it, not about risk to others. And of course
there's no discussion of when you should wear a mask to protect others
in the terminal or on the plane.

At this point, however, we are partway through a transition to
treating covid as a normal disease. While the CDC recommendation is
still not
to fly, this was also what they recommended for people with
similar illness pre-covid,
and they recommend other things that most people aren't doing like
masking when local covid levels are high. Airlines are back to
charging change
fees even with a positive test.

Isolating in place is at least four more days in CA, more if someone
else gets sick (which did happen). Maybe six days in expectation?
With flight change fees ($300), rental car extension ($400), and more
days in the hotel ($600) t...
Prototype of Using GPT-3 to Generate Textbook-length Content
by Rafael Cosman — 2023-01-18T14:25:02.444Z
Link: https://github.com/RafaelCosman/gpt-generated-textbooks

The code is all open source (see the Github). As an example for how this performs, here is The Art of Better Thinking: Overcoming Cognitive Biases and Unlocking Rationality. To read the full book (and have slightly improved formatting), you can view it directly on Github.

[Disclaimer: this book was written by AI and may contain significant errors. It should not be relied upon for any reference or educational use]

The Art of Better Thinking: Overcoming Cognitive Biases and Unlocking Rationality

1. Introduction to the Art of Rationality & Overcoming Cognitive Biases

A. Definition of Rationality

What is Rationality?

Rationality is the ability to think logically and objectively, and to make decisions that are based on evidence and reason, rather than emotion. It involves the ability to analyze facts and evidence, draw conclusions, and make decisions that are based on the evidence and logical reasoning. Rationality is an important skill for making decisions in life, as it helps us to make sound decisions and avoid falling prey to cognitive biases and logical fallacies.

Examples of Rational Thinking

Rational thinking can be seen in many aspects of life. For example, a rational decision-maker will consider both the pros and cons of a situation before making a decision. They will also consider the potential consequences of their decision, and weigh the risks and rewards of each option. They will also consider other relevant factors, such as the cost, time, and effort involved in each option.

In addition, rational thinking involves the ability to recognize and eliminate logical fallacies in arguments. This includes recognizing when someone is making an argument based on false assumptions or faulty logic, and being able to identify and refute such arguments.

Benefits of Rational Thinking

Rational thinking has many benefits. It helps us to make sound decisions, and to avoid falling prey to cognitive biases and logical fallacies. It also helps us to think more objectively and to view situations from multiple perspectives. It also helps us to think more critically and to consider the implications of our decisions. Finally, it helps us to develop better problem-solving skills, as we are able to analyze complex problems and identify potential solutions.

B. Overview of Cognitive Biases

What is a Cognitive

...
How many people are working (directly) on reducing existential risk from AI?
by Benjamin Hilton — 2023-01-18T08:46:29.884Z
Link: https://80000hours.org/problem-profiles/artificial-intelligence/#neglectedness

Summary

I've updated my estimate of the number of FTE (full-time equivalent) working (directly) on reducing existential risks from AI from 300 FTE to 400 FTE.

Below I've pasted some slightly edited excepts of the relevant sections of the 80,000 Hours profile on preventing an AI-related catastrophe.

New 80,000 Hours estimate of the number of people working on reducing AI risk

Neglectedness estimate

We estimate there are around 400 people around the world working directly on reducing the chances of an AI-related existential catastrophe (with a 90% confidence interval ranging between 200 and 1,000). Of these, about three quarters are working on technical AI safety research, with the rest split between strategy (and other governance) research and advocacy. [1]We think there are around 800 people working in complementary roles, but we’re highly uncertain about this estimate.

Footnote on methodology

It’s difficult to estimate this number.

Ideally we want to estimate the number of FTE (“full-time equivalent“) working on the problem of reducing existential risks from AI.

But there are lots of ambiguities around what counts as working on the issue. So I tried to use the following guidelines in my estimates:

I didn’t include people who might think of themselves on a career path that is building towards a role preventing an AI-related catastrophe, but who are currently skilling up rather than working directly on the problem.
I included researchers, engineers, and other staff that seem to work directly on technical AI safety research or AI strategy and governance. But there’s an uncertain boundary between these people and others who I chose not to include. For example, I didn’t include machine learning engineers whose role is building AI systems that might be used for safety research but aren’t primarily designed for that purpose.
I only included time spent on work that seems related to reducing the potentially existential risks from AI, like those discussed in this article. Lots of wider AI safety and AI ethics work focuses on reducing other risks from AI seems relevant to reducing existential risks – this ‘indirect’ work makes this estimate difficult. I decided not to include indirect work on reducing the risks of an AI-related catastrophe (see our problem framework for more).
Relatedly, I didn’t include people working on other problems that might indirectly affect the chance

...
EA & LW Forum Summaries (9th Jan to 15th Jan 23')
by Zoe Williams — 2023-01-18T07:29:06.656Z

Supported by Rethink Priorities
This is part of a weekly series summarizing the top posts on the EA and LW forums - you can see the full collection here. The first post includes some details on purpose and methodology. Feedback, thoughts, and corrections are welcomed.

If you'd like to receive these summaries via email, you can subscribe here.

Podcast version: prefer your summaries in podcast form? A big thanks to Coleman Snell for producing these! Subscribe on your favorite podcast app by searching for 'EA Forum Podcast (Summaries)'. More detail here.
Author's note: you'll notice a few differences in today's post:

The EA and LW forum sections have been combined, given an increasing amount of cross-posting. You can now find everything by topic, regardless of which forum it was posted on.
There's a new 'special mentions' section. The karma requirements for the main section have gone up based on karma inflation (and will continue to do so over time), but each week I'll call out a few posts in this section I think were undervalued.

Philosophy and Methodologies

Moral Weights according to EA Orgs

by Simon_M

* = the author expects the organization would not endorse the figure shown (their best guess in GiveWell’s case, and pending further research for HLI).

The Capability Approach

by ryancbriggs

The Capability Approach to human welfare is about increasing the options, or ‘functionings’, available to people (eg. eating, being a doctor, being a parent). This is in contrast to commonly used measures in EA like subjective wellbeing (SWB) or preference satisfaction.

The author argues this is a valuable approach that is neglected in EA, but influential in international development more broadly. It’s agnostic about what people choose to do with the options - meaning it suits everyone, and avoids optimizers like EAs pushing our favorite functionings (eg. SWB) on people with different preferences.

Because there are so many potential functionings and ways to rate them, it’s messy to measure.

Some

indexes

have tried, on both a country and individual level. You can also note that being alive and having resources (eg. money) lead to many options, and work on those. This matches up with GiveWell and Open Philanthropy prioritizing income gains and lives saved as privileged metrics, likely influenced by the capabilities approach.

GWWC Should Require Public Charit

...
OpenAI’s Alignment Plan is not S.M.A.R.T.
by Søren Elverlin — 2023-01-18T06:39:18.350Z

In response to Eliezer Yudkowsky's challenge, I will show how the alignment research approach outlined by OpenAI lacks common desiderata for effective plans. Most of the deficiencies appear to be difficult or impossible to fix, and we should thus expect the plan to fail.

Meta-level description of what makes a plan good places great emphasis on the goals/objectives of the plan. George T. Doran suggests goals be S.M.A.R.T.: Specific, Measurable, Achievable, Relevant, and Time-Bound.

Specific: OpenAI's description of the goals as AI that is "Value aligned" and "Follow human intent" could be elaborated in much greater detail than these 5 words. Yet making them specific is no easy task. No definition exists of these words in sufficient detail to be put into computer code, nor does an informal consensus exist.

Measurable: There currently exists no good way to quantify value alignment and intent-following. It is an open question if such quantification is even possible to do in an adequate way, and OpenAI does not seem to focus on resolving philosophical issues such as those required to make value alignment measurable.

Achievable: The plan suggests a relatively narrow AI would be sufficient to contribute to alignment research, while being too narrow to be dangerous. This seems implausible: Much easier problems than alignment research have been called AGI-complete, and general reasoning ability is widely thought to be a requirement for doing research.

Relevant: The plan acknowledges existential risk from advanced AI, but the proposed goal is insufficient to end the period of acute risk. This gap in the plan must be closed. I do not think this is trivial, as OpenAI rejects MIRI-style pivotal acts. My impression of OpenAI is that they hope alignment can be solved to the extent that the "Alignment tax" becomes negative to such an overwhelming degree that deceptively aligned AI is not built in practice by anyone.

Time-bound: The plan is not time-bound. OpenAI's plan is conceptualized as consisting of 3 pillars (Reinforcement Learning from Human Feedback, AI-assisted evaluation, AI doing alignment research), but the dependencies mean this will need to be 3 partially overlapping phases. The plan lacks information about timing or criteria for progressing from each phase to the next: When does OpenAI intend to pivot towards the last phase, AI-based alignment research?

Other proble

...
Formal definition of Ontology Mismatch?
by  — 2023-01-18T05:52:31.615Z

Does anyone have a clean, mathematical definition of ontology mismatch which situates the problem in the statistical learning theory?

There's this post on the alignment forum where from what I can understand ontology identification is defined as a classifier which has a conservative decision boundary given some questions and an error-free dataset, but it'd be nice to have something cleaner?
Transformer Mech Interp: Any visualizations?
by Joyee Chen — 2023-01-18T04:32:33.085Z

After getting to the the part of a demo (one of Neel Nanda's interp demos) where they talk about the idea of a Logit Lens, and Layer Attribution, I have a bit of trouble visualizing it as I could have for simpler concepts (e.g. residual streams, which were indeed drawn as my primary method of comprehension). Anybody have good resources for illustrations? (I know Nanda had a great colorful runthrough but it was only at a high level and for encoder-decoder machines, and thus not generalizable)
Neural networks generalize because of this one weird trick
by Jesse Hoogland — 2023-01-18T00:10:36.998Z
Link: https://www.jessehoogland.com/article/neural-networks-generalize-because-of-this-one-weird-trick

Produced under the mentorship of Evan Hubinger as part of the

SERI ML Alignment Theory Scholars Program

- Winter 2022 Cohort

A big thank you to all of the people who gave me feedback on this post: Edmund Lau, Dan Murfet, Alexander Gietelink Oldenziel, Lucius Bushnaq, Rob Krzyzanowski, Alexandre Variengen, Jiri Hoogland, and Russell Goyder.

Statistical learning theory is lying to you: "overparametrized" models actually aren't overparametrized, and generalization is not just a question of

broad basins

.

The standard explanation thrown around here for why neural networks generalize well is that gradient descent settles in flat basins of the loss function. On the left, in a sharp minimum, the updates bounce the model around. Performance varies considerably with new examples. On the right, in a flat minimum, the updates settle to zero. Performance is stabler under perturbations.

To first order, that's because loss basins actually aren't basins but valleys, and at the base of these valleys lie "rivers" of constant, minimum loss. The higher the dimension of these minimum sets, the lower the effective dimensionality of your model.[1] Generalization is a balance between expressivity (more effective parameters) and simplicity (fewer effective parameters).

Symmetries lower the effective dimensionality of your model. In this example, a line of degenerate points effectively restricts the two-dimensional loss surface to one dimension.

In particular, it is the singularities of these minimum-loss sets — points at which the tangent vanishes — that determine generalization performance. The remarkable claim of singular learning theory (the subject of this post), is that "knowledge … to be discovered corresponds to singularities in general" [1]. Complex singularities make for simpler functions that generalize further.

The central claim of singular learning theory is that the singularities of the set of minima of the loss function determine learning behavior and generalization. Models close to more complex singularities generalize further.

Mechanistically, these minimum-loss sets result from the internal symmetries of NNs[2]: continuous variations of a given network's weights that implement the same calculation. Many of these symmetries are "generic" in that they are predetermined by the architecture and are always present. The more interesting symmetries are non-generic s...
Progress links and tweets, 2023-01-17
by jasoncrawford — 2023-01-17T21:31:50.314Z
Link: https://rootsofprogress.org/links-and-tweets-2023-01-17

The Progress Forum

Tyler Cowen AMA
“In Praise of Fast Food” by Rachel Laudan (excerpts and link)
Construction of the World Trade Center (Brian Potter)
The Pull of Cities (Anton Howes)
Gift subscriptions to Jim Pethokoukis’s Substack

Opportunities

The O’Shaughnessy Fellowship: 1 year, $100k for ambitious people

Links

The Economist on progress studies and other “new tech worldviews”
New Substack on nuclear power by Jack Devanney
Quantum computers may break RSA encryption sooner than expected (via @tegmark) [UPDATE: this is probably bogus]
How much does a gas stove shorten your life? Maybe ~53 days (by @dynomight)
Time-to-violent-death of Roman emperors displayed a bathtub curve
Caro is still working on the 5th and final volume of his LBJ bio
Nathan Myhrvold wrote a five-volume anthology on bread (!)

Queries

What should Dwarkesh ask Marc Andreessen? (@dwarkesh_sp)
Recommendations for things to read on well-run scientific labs?
Why didn’t the predicted demise of radiologists happen? (@BenGoldhaber)
Good sources showing how labor-intensive industries tend to move to where low-cost labor is? (@_brianpotter)
What would it take for 2022-2090 to be as transformative for medicine/biology as 1870-1950? (@Willyintheworld)
What are more examples of individual grant programs such as Thiel Fellowship or Emergent Ventures? (@William_Blake)
What does the “progressive” vision of the future look like today? (@lo_commotion)
Can anyone find a source for this quote?

Tweets

The devastating human consequences of the Ehrlichs’ campaign against “overpopulation.” (@daniel_eth asks, why has this man not been canceled yet?)
“Traditional foods” are not very old
In the long run, we need a heat-management system for the Earth
Amazing progress on tap water connections in rural India

Quotes

Things we take for granted: that glass is transparent
Sanger’s Rule for technical advances in scientific experimentation
Building bridges and keeping the water running are underrated
Penicillin was stalled for a decade after the initial attempt to extract it failed
Bitumen was the Super Glue™ of the third millennium BC
Technologies often start out with “trivial“ uses and become necessities
No society has held technological leadership for very long
“I have yet to hear anyone even mention the theoretical possibility that we could respond to a new variant … by trying to vaccinate people before they could get infected”
A Department of Drugs and a

...