trentmkelly/LessWrong-43k huggingface dataset:
Long Covid Risks: 2023 Update
by Elizabeth — 2023-05-06T18:20:01.259Z

Back in 2021 I wrote a post estimating the risk of long covid. Recently a client hired me to do an update, focusing on changes induced by Paxlovid and vaccination. This was a <5h project and the literature wasn’t very rich so nothing I say here is conclusive, but nothing I said last time was conclusive either so let’s enjoy this together.

Some caveats:

I spent 5 hours on this, and that includes client-specific work I’m not including here.

Research that met my standard was really scarce; ultimately each conclusion is based on a single study. My goal was data that includes a large population not selected for having long covid, where reporting was automated so you don’t need to worry about response bias. In practice, this means I used data from large medical systems with integrated reporting, like the American Department of Veteran Affairs, national medical systems, and HMOs. Surveys from long covid support groups were ignored with prejudice.

Summary

Vaccination helps, a bit: Given a medically diagnosed infection (which means it was serious enough to actually get you to the doctor), up-to-date vaccination lowers the risk of long covid by about 20% (this does not include the reduction in risk of having diagnosable covid in the first place, which is substantial).

Paxlovid helps, more: Nirmatrelvir, which is one of two drugs that make up Paxlovid, reduces long covid risk by about 30% for medically diagnosed infections (which means it was serious enough to actually get you to the doctor). An optimist might hope the other drug (which is in the same class, although most commonly used as an adjuvant) is also useful and round this to 50%.

Most symptoms are temporary: Long covid does tend to get better over time, but how quickly depends on the symptom. At one year post-infection, the rate of heart issues is nearly indistinguishable from controls, but cognitive issues have a 50% chance of persisting.

Calculate your absolute risk: Your absolute risk depends on your age and comorbidities. The measured risk for 70-year-old men (not controlling for comorbidities) of developing at least one serious sequelae of medically diagnosed covid n (which means it was serious enough to actually get the patient the doctor) is ~12%. If you want to norm this for your own demographic, you can get a very crude estimate by entering your demogra...
Is "red" for GPT-4 the same as "red" for you?
by Yusuke Hayashi — 2023-05-06T17:55:20.691Z

Penned by Yusuke Hayashi, an independent researcher hailing from Japan, this article bears no affiliation to the authors whose scholarly works are referenced herein. Demonstrating intellectual autonomy, the analysis presented is unequivocally distinct from the cited publications.

Have you ever wondered if the "red" you perceive is the same "red" someone else experiences? A recent study explores this question using a distance measure called the Gromov-Wasserstein distance (GWD).

Is my “red” your “red”?: Unsupervised alignment of qualia structures via optimal transport[1]
Paper (Preprint by Kawakita et al. published in PsyArXiv): https://psyarxiv.com/h3pqm
Slide (Presentation by Masafumi Oizumi at Optimal Transport Workshop 2023): https://drive.google.com/file/d/1Z5QTxayqJkmYyOgirrbJRRflY8HOy8ox/view?usp=sharing

In the paper "Is my "red" your "red"? : Unsupervised Alignment of Qualia Structures by Optimal Transport" (Kawakita et al., 2022) proposes a new approach to assessing the similarity of sensory experiences between individuals. This approach, based on GWD and optimal transport, allows for the alignment of qualia structures without assuming a correspondence between individual experiences. As a result, it provides a quantitative means of comparing the similarity of qualia structures between individuals.

This image is an excerpt from a slide created by Dr. Masafumi Oizumi, the author of the paper "Is my 'red' your 'red'?: Unsupervised alignment of qualia structures via optimal transport."

This image is an excerpt from a slide created by Dr. Masafumi Oizumi, the author of the paper "Is my 'red' your 'red'?: Unsupervised alignment of qualia structures via optimal transport."

This image is an excerpt from a slide created by Dr. Masafumi Oizumi, the author of the paper "Is my 'red' your 'red'?: Unsupervised alignment of qualia structures via optimal transport."

By applying this method, it is possible to compare the similarity of subjective experiences, or qualia structures, between individuals using GWD, a distance measure employed in machine learning research. While this study compares sensory experiences between humans, the same approach could be used to compare the sensory experiences of humans and the advanced AI model, GPT-4. We are now ready to revisit the question posed in the title:

Is "red" for GPT-4 the same as for you?

If asked whether large language models (LL...
The Broader Fossil Fuel Community
by Jeffrey Heninger — 2023-05-06T14:49:38.633Z

In his blog post, Why Not Slow AI Progress?,[1] Scott Alexander notes that AI Safety researchers and AI Capabilities researchers seem to be part of the same community. This is kind of weird: fossil fuel companies and environmentalists seem to be much more socially distinct. Scott gave several reasons why this is the case and why we might want this to be the case.

There is another reason that these are the same community: AI Safety researchers and AI Capabilities researchers are philosophically much closer to each other than fossil fuel companies and environmentalists. In particular, both parts of the Broader AI Community share the following belief:

The long-term goal of humanity should be a technological utopia, and the most useful tool for building Our Glorious Future[2] is AGI.

The main disagreement between the two parts of the community is about how hard it will be to align the AGI with human values.

This statement is not at all obvious to people outside of the Broader AI Community. Reasonable people can and do disagree with all parts of it:[3]

Should humanity have long-term goals?
Should our goals be utopian?
Are the problems that we will have to solve to get to Our Glorious Future mostly technological or mostly social?
Is the best way to achieve Our Glorious Future through AGI?

AI Safety researchers and AI Capabilities researchers tend to answer these questions the same way, while other intellectual communities and the general public have more varied views. If anything, it seems as though the futurism of AI Safety researchers is more extreme.

The comparison to fossil fuels is telling. A similar sentence for these communities would be:

A short- to medium-term goal of humanity should be a society with abundant energy, and the most useful tools achieving this are fossil fuels.

This belief is not shared by both fossil fuel companies and environmentalists.

I will place environmentalists into three categories, based on how they related to this belief:

De-growth environmentalists reject that society should have abundant energy. Instead, they want a Return to Nature with less economic activity overall. I am skeptical of this position because I doubt that most of these environmentalists recognize how much more human suffering is implied by their position, but this does seem to be a major force within the environmental movement.
Alternative energy environmentalists&nb

...
Estimating Norovirus Prevalence
by jefftk — 2023-05-06T11:40:04.588Z

How many people currently have Norovirus? What about in June 2021, to
pick a historical month? How could we get an estimate?

The CDC says there are 19-21M
cases annually but Norovirus is strongly seasonal; you're much
more likely to get it in the winter than the summer. How much more
likely?

The CDC tracks Norovirus and other outbreaks through the National Outbreak Reporting
System (NORS) and you can download the full data through 2021.
Since 2012 I see 36,341 outbreaks, of which 20,876 (57%) were
classified as Norovirus. Categorizing those by month, and plotting
with winter in the middle, we can see January has 10x the outbreaks of
August:

Infections are probably proportional to outbreaks, so we could get an
estimate for each month by weighting by the number of outbreaks. But
we should be careful: the CDC overall estimate is pre-covid, and by
going through 2021 we've included a period when Norovirus, like most
infectious diseases, was heavily suppressed:

Let's say we want to know the prevalence during, say, June 2021. Not
as in what fraction caught it at some point during that month (that
would be the incidence) but on any day what fraction of people were
currently infected. We'll start by converting the 20M number of annual
infections into a per-capita number by dividing by the corresponding
population, 299M people [1] to get an annual incidence of 6.7k per
100k (20M / 399M). A Norovirus illness lasts about
two days, so this is equivalent to an average prevalence of 37 per
100k (6.7k * 2 / 365). Pre-covid there were an average of 174
outbreaks per month, [2] and in June 2021 there were 76 outbreaks
reported, so for June 2021 we can estimate 16 per 100k people (37 per
100k * 76 / 174).

Why do I care how many people have Norovirus? I want to know how a
disease's prevalence (how many people currently have it) translates into
relative abundance in wastewater sequencing [3] (what fraction of sequencing reads match the pathogen) to be able to
answer questions like "how much would it cost to flag a pathogen that
sheds into sewage like pathogen X when Y% of people are currently
infected". If we have a decent estimate for the number of infected
people, we can connect that to the relative abundance we see in
sequencing data, and model the connection. Getting an estimate for
Norovirus is one step in this process, and since it's a
...
Alignment as Function Fitting
by A.H. — 2023-05-06T11:38:04.245Z

In this post, I spell out some fairly basic ideas, that seem important to me. Maybe they are so basic that everyone finds them obvious, but in the spirit of this piece, I thought I would write them up anyway.

tl;dr I start off by looking at using linear regression to model data. I examine how interpolation and extrapolation using a linear model can go wrong and why you need some extra assumptions in order to justify interpolation/extrapolation using a linear model. I write about the analogies between the failure modes of linear regression and AI failure modes. I conclude that problems in AI alignment often boil down to the problem of extrapolating outside of training data, which applies equally to linear regression and complex AI models.

Suppose we have a set of datapoints of the form (x,y) and we wish to study this set of data and use it to make predictions about unseen datapoints. We will be given a value of x that is not in our dataset and attempt to predict the corresponding value of y. This is a classic problem for which one would use regression analysis, the simplest form of which is linear regression, fitted using the ordinary least squares method. I will consider a couple of examples where this prediction method goes wrong. These are not unique to linear regression and are indicative of broader failure modes.

Justifying Interpolation

Suppose our data is given on the plot below, in blue. We perform a linear regression and use it to find a line of best fit which minimizes the residual sum of squares (RSS). The regression line is shown in red.

We can get a rough idea of how well our regression line fits the data by seeing how big the RSS is. Since the RSS is not zero, we know that the fit is not perfect, but it is small enough that we might feel happy using our line to make predictions.

Suppose we are required to predict the value of y when x=5. This is an example of interpolation. Interpolation has several strict mathematical definitions, but here I will use it to mean "making predictions of values within the range of the dataset". Since we have datapoints for x>5 and x<5, the point x=5 lies within this range. We can use the linear regression line to predict the value of y when x=5. This is indicated using the black dashed line and gives us a prediction that,&...
My preferred framings for reward misspecification and goal misgeneralisation
by Yi-Yang — 2023-05-06T04:48:49.257Z

TL;DR

I was feeling quite dissatisfied with a bunch of categories like “inner and outer misalignment” and “reward misspecification and goal misgeneralisation”
I think most of this dissatisfaction stems from them not being able to correctly describe situations where there are multiple points of failure or gaps.
Hence, I propose using a root cause type of framing (inspired by Richard Ngo and Rohin Shah) instead which I think makes descriptions of failures more specific and nuanced.

Assumptions and caveats

I'll be referencing RL agents throughout and not other types of agents.
I'm assuming my target audience has at least some basic knowledge from tghe AGI Safety Fundamentals (AGISF) course.
I have hardly any ML experience and have time boxed myself when making this forum post so I expect to make some mistakes along the way. Also, this was originally made in a slide-deck so I may have lost some fidelity during the transfer.
This forum post is my AGISF course capstone, so it would be super valuable for me to figure out how I should update towards thinking about AI alignment and my own career plan. Feedback is very much welcomed.

Much thanks to Chan Jun Shern, Chin Ze Shen, and Mo Putera for their feedback and inspiration. All mistakes are mine.

If you'd like to listen to the presentation instead, you can watch it here, which is 18 minutes long with the option of speeding up. Here's the slide-deck.

My journey through different kinds of framings

When I first learned about about AI safety, the first analogy that really clicked with me was the myth of King Midas. He wished that everything he touches becomes gold, and well, he got his wish. Soon, he was starving to the death.

Then, I learned about inner and outer misalignment. Here’s a quick recap:

Outer misalignment happens when there’s a discrepancy between the designer’s intended goal (what’s in our heads) and the specified goal (what we say).

An example of this is the King Midas myth, where what King Midas probably wanted is to be wealthy, but gave a pretty bad wish.

Inner misalignment happens when there’s a discrepancy between the specified goal (what we say) and the agent’s revealed goal (the agent’s actual goal or proxy goal).

An example of this is a pet making a mess of the house when you’re not around. The pet owner might have specified to not destroy any furniture. However, the pet might just re

...
You don't need to be a genius to be in AI safety research
by Claire Short — 2023-05-06T02:32:02.164Z

The aim of this article is to share my experiences within the AI safety community to those in research and operations in the field, with the goal of creating a more inclusive, supportive, and positive environment. The intention of this article is to allow those in a more privileged position to think about these considerations and potentially implement new practices in their organizations or work in order to create a more welcoming space for those not traditionally represented here. I’ve also had some incredibly supportive and fulfilling experiences within this field which are worth celebrating. However in order to create the most inclusive space that we can, I think the following points are important to address.

Using ‘genius’ is not a reliable metric for researchers

When people claim to be searching for ‘genius’ researchers or that you need to be a genius to contribute in a meaningful way to AI safety, it’s unclear which specific metrics they are using for evaluation. A number of factors contribute to the ambiguity and subjectivity surrounding the definition of "genius," including people's individualized perspectives shaped by their upbringing, the complexity of intelligence, our growing understanding of cognitive processes, the context-dependence of genius, and the lack of clear criteria to determine genius. These factors make it challenging to establish a universally accepted definition of genius.

Regardless of the criteria used to determine genius, it is critical to understand that this quality isn’t the only one necessary to contribute in a meaningful way.

Recruiting researchers is a task that requires careful evaluation of multiple factors such as skill, experience, ability to work with others, and ingenuity. However certain organizations, program managers, and coaches in the AI safety field pay too much attention to the benchmark of ‘genius’ to recruit or encourage researchers. This type of thinking is harmful and unnecessary because it leads to a lack of diversity, narrow focus, elitism, imposter syndrome, and waste of resources in the pursuit of this elusive trait.

By allowing this idea to continue, we run the risk of recruiting for a narrow set of skills or attributes, thus limiting potential innovation and diversity of thought, while also potentially slowing research progress. Instead, it's important to actively seek talented individu...
Naturalist Collection
by LoganStrohl — 2023-05-06T00:37:42.774Z

There’s a kind of rhythm to the four phases of naturalism. To me, they feel a bit like oscillation between zooming out and zooming in, or like inhaling and exhaling. The first phase (Locating Fulcrum Experiences) has a broad exploratory feel, like zooming out; it’s sort of meandering, touching on this and that until I’ve found the right thing to hone in on. The second phase (Getting Your Eyes On) emphasizes focused attention, and involves a lot of precision and detail. It’s more like zooming in.

This third phase, “Collection”, zooms out again. After mining a small number of experiences for all the detail you can perceive, the next step is to train yourself to notice every single instance of your fulcrum experience, no matter when, where, or how it happens.

You may find that the experiences you previously studied in detail represented a narrow subset of all the instances that happen; the modal instance might be far more subtle, might happen in a context you didn’t anticipate, or might be quite different in some other way. By the end of this phase, you’ll likely perceive some large patterns in the way this experience shows up in your life, and you’ll be very familiar with the circumstances that give rise to it.

How To Collect Experiences: Gamified Noticing

In the most basic version of collection, you’ll simply tap your leg (or make some other planned gesture) when you notice the experience you’re watching for. As you may recall from a previous essay, I call this “marking”. You don’t necessarily have to do anything special with your mind when you mark an experience; you can just endeavor to explicitly notice the experience every time (that is, to notice that you’ve noticed it).

But the standard version of collection, the one I do when I’m serious about a study, is a little bit gamified. Rather than just making a marking gesture, you’ll also keep a tally of how many times you noticed over the course of a day. Your goal each day is to beat your high score.

There are a few good ways to keep a noticing tally. One way to do it is just to count in your head, which apparently works fine for some people. Or you could get a little tally device such as a knitting counter[1], a golf counter, or a counter ring[2], and keep that with you. There are also phone apps[3] that mimic golf and knitting counters. Personally, I typically use a tiny notebook[4] t...
Explaining “Hell is Game Theory Folk Theorems”
by electroswing — 2023-05-05T23:33:20.977Z

I, along with many commenters, found the explanation in Hell is Game Theory Folk Theorems somewhat unclear. I am re-explaining some of the ideas from that post here. Thanks to jessicata for writing a post on such an interesting topic.

1-shot prisoner’s dilemma.

In a 1-shot prisoner’s dilemma, defecting is a dominant strategy. Because of this, (defect, defect) is the unique Nash equilibrium of this game. Which kind of sucks, since (cooperate, cooperate) would be better for both players.

Nash equilibrium.

Nash equilibrium is just a mathematical formalism. Consider a strategy profile, which is a list of which strategy each player chooses. A strategy profile is a Nash equilibrium if no player is strictly better off switching their strategy, assuming everyone else continues to play their strategy listed in the strategy profile.
Notably, Nash equilibrium says nothing about:

What if two or more people team up and deviate from the Nash equilibrium strategy profile?
What if people aren’t behaving fully rationally? (see

bounded rationality

)

Nash equilibria may or may not have predictive power. It depends on the game. Much work in game theory involves refining equilibrium concepts to have more predictive power in different situations (e.g.

subgame perfect equilibrium

to handle credible threats,

trembling hand equilibrium

to handle human error).

n-shot prisoner’s dilemma.

OK, now what if people agree to repeat a prisoner’s dilemma n=10 times? Maybe the repeated rounds can build trust among players, causing cooperation to happen?

Unfortunately, the theory says that (defect, defect) is still the unique Nash equilibrium. Why? Because in the 10th game, players don’t care about their reputation anymore. They just want to maximize payoff, so they may as well defect. So, it is common knowledge that each player will defect in the 10th game. Now moving to the 9th game, players know their reputation doesn’t matter in this game, because everyone is going to defect in the 10th game anyway. So, it is common knowledge that each player will defect in the 9th game. And so on… This thought process is called

backwards induction

.

This shows that the unique Nash equilibrium is still (defect, defect), even if the number of repetitions is large. Why might this lack predictive power?

In the real world there mi

...