trentmkelly/LessWrong-43k huggingface dataset:
Sequencing Swabs
by jefftk — 2024-02-01T01:50:17.938Z

Update 2024-02-26: due to a

serious
typo all the results in this post were off by a factor of 100.
Sequencing from individuals still looks promising, but by less than it
did before. I've updated the numbers in the post, and added notes
above the charts to explain how they're wrong. Thanks to Simon Grimm
for catching my mistake.

While this is about an area I work in, I'm speaking for myself and
not my organization.

At the Nucleic Acid
Observatory we've been mostly looking at metagenomic sequencing of
wastewater as a way to identify a potential future 'stealth'
pandemic, but this isn't because wastewater is the ideal sample type:
sewage is actually really—the joke writes itself. Among other
things it's inconsistent, microbially diverse, and the nucleic acids
are have degraded a lot. Instead the key advantage of wastewater is
how practical it is to get wide coverage. Swabbing the noses and/or
throats of everyone in Greater Boston a few times a week would be
nuts, but sampling the
wastewater gets you 3M people in a
single sample. [1]

Imagine, though, that people were enthusiastic about taking samples
and giving them to you to sequence. How much better would that be?
What does "better" even mean? In Grimm
et al. (2023) we operationalized "better" as RAi(1%):
what fraction of shotgun metagenomic sequencing reads might come from
the pathogen of interest when 1% of people have been infected in the
last week. For example, in our re-analysis of Rothman et
al. (2021) we found a RAi(1%) of ~1e-7 for SARS-CoV-2,
which means we'd estimate that in a week where 1% of people contracted
Covid-19, one in 10M sequencing reads would come from the virus.
Let's say you were able to get throat swabs instead; what might
RAi(1%) look like?

The ideal way to determine this would be to swab a lot of people, run
untargeted sequencing, analyze the data, and link it to trustworthy
data on how many people were sick. But while public health
researchers do broad surveillance, collecting and testing swabs for
specific things, as far as I can tell no one has combined this with
metagenomic sequencing. [2] Instead, we can get a rough estimate from
looking at studies that just sequenced sick people.

In Lu et
al. (2021) they ran shotgun metagenomic RNA sequencing on throat
swabs from sixteen Covid patients in Wuhan. Patients with higher
viral load, and so lower Ct ...
Leading The Parade
by johnswentworth — 2024-01-31T22:39:56.499Z

Background

Terminology: Counterfactual Impact vs “Leading The Parade”

Y’know how a parade or marching band has a person who walks in front waving a fancy-looking stick up and down? Like this guy:

The classic 80’s comedy Animal House features a

great scene

in which a prankster steals the stick, and then leads the marching band off the main road and down a dead-end alley.

That is not the guy who’s supposed to have that stick.

In the context of the movie, it’s hilarious. It’s also, presumably, not at all how parades actually work these days. If you happen to be “leading” a parade, and you go wandering off down a side alley, then (I claim) those following behind will be briefly confused, then ignore you and continue along the parade route. The parade leader may appear to be “leading”, but they do not have any counterfactual impact on the route taken by everyone else; the “leader” is just walking slightly ahead.

(Note that I have not personally tested this claim, and I am eager for empirical evidence from anyone who has, preferably with video.)

A lot of questions about how to influence the world, or how to allocate credit/blame to produce useful incentives, hinge on whether people in various positions have counterfactual impact or are “just leading the parade”.

Examples

Research

I’m a researcher. Even assuming my research is “successful” (i.e. I solve the problems I’m trying to solve and/or discover and solve even better problems), even assuming my work ends up adopted and deployed in practice, to what extent is my impact counterfactual? Am I just doing things which other people would have done anyway, but maybe slightly ahead of them? For historical researchers, how can I tell, in order to build my priors?

Looking at historical examples, there are at least some cases where very famous work done by researchers was clearly not counterfactual. Newton’s development of calculus is one such example: there was simultaneous discovery by Leibniz, therefore calculus clearly would have been figured out around the same time even without Newton.

On the other end of the spectrum, Shannon’s development of information theory is my go-to example of research which was probably not just leading the parade. There was no simultaneous discovery, as far as I know. The main prior research was by Nyquist and Hartley about 20 years earlier - so for at least two decades the ...
Proposal for an AI Safety Prize
by sweenesm — 2024-01-31T18:35:48.130Z

After reading Yudkowsky’s list of lethalities, I searched for some kind of prize out there that might be similar to what he suggests in his point #40, to motivate truly impactful AI safety work:

“promising to pay big money retrospectively for good work to anyone who produces it.”

I found a few prizes that had been awarded already or to which applications were closed. I also found people talking about potential prizes that, as far as I know, never came to be.

Here I outline what I think a decent AI Safety Prize could look like. I’m happy to hear suggestions for improvements on this proposal, or potential issues you see with it, so please do write them in the comments if you have them. I may send this off to XPRIZE as a prize suggestion.

Outline of an AI Safety Prize

This AI Safety Prize would be focused on establishing the ability to produce a safe, "aligned" AGI system (as indicated, but not proven, by a system that showed apparent alignment for at least 3 months, see below). It would not, of course, be a guarantee that no misaligned AGI ever came online.

The total prize should be at least $10M, to catch enough attention, with at least $5M of that being possible for one individual or team to win. Who would pay for it? Companies that ended up using the developed AI safety technique(s). Participating companies would agree to pay out a certain prize amount if they used a safety advancement in their AGI/near-AGI product for at least 3 months with no apparent misalignment/safety issues showing up. If companies didn’t end up using a particular technique, they wouldn’t pay out for it. For closed models, this would require honesty on their part, but I suspect the small relative prize quantities (compared to the huge economic upsides of AGI) and possible downsides of bad PR would motivate companies to stay honest. Companies could also decide to solicit funding for their portion of the prize from donors.

Safety concepts would have to be actionable by someone reasonably skilled in the art. For example, saying “we should use reinforcement learning with feedback from elephants” wouldn’t qualify for a prize, while providing a specific mechanism/algorithm that could do this (detailed plans for brain chips plus peanut reward functions?) would. Advancements that make it into a product to enhance its performance, not its safety, wouldn’t count. Dual use concepts (those that en...
Literally Everything is Infinite
by Spiral — 2024-01-31T18:31:48.664Z

Hi all,

I was on a walk, digesting a lot of "Western" and "Eastern" philosophy I have been reading over the last couple months, and a mental paradigm emerged for me. Not in blaze of light or spiritual experience kind of way, but more in a small thought that grew larger the more I questioned it. I am not claiming to be the first to suggest this, though I haven't personally heard it explored further before.

The thought is "Everything is Infinite. Literally everything."
This will probably sound stupidly simple, pointlessly vague - but give it a shot - I believe the consequences of this have practical ramifications.
The underlying logic is that while beginnings, endings, and limits are very helpful conceptual tools for humans, they don't actually exist in reality.
Take this assumption seriously, draw out the consequences, and see where it takes you. Please critique anything here obviously, because this is only useful a paradigm if it holds up.

Here are some examples. Each one builds on the previous one, so I would recommend deciding if you agree with one before moving on to the next:

"Zoom" is Infinite
By Zoom I mean zooming into the micro level, and zooming out the macro. We haven't been able to find the smallest building blocks of the universe, nor the largest. Even if we did find the smallest - what is genuinely stopping us from zooming in further?

Objects are Boundless
The paradigm in a number of Eastern philosophies, along with what appears more evident as we are able to see further into the micro is that the "hard edges" we assumed defined objects, are actually quite blurred. Following on from the previous point, if we can never reach the smallest building block, then how can we define the edges of an object, lined with those smallest building blocks? Again, while being able to isolate "objects" as objects through language and concepts has incredible utility for us, it is not actually the true reality. Don't worry, I know you've heard these ideas a thousand times, but keep taking it further.

Time is Infinite
What we perceive as time never had a beginning and will never have an end. You may argue that there must be beginnings and ends, because our experience begins at birth and ends at death. Though the key here is that, while the states of things transform (which conceptually seem like a beginning and end), the things themselves never do. Before you were born, you wer...
What fuels your ambition?
by Cissy — 2024-01-31T18:30:53.274Z
Link: https://www.moremyself.xyz/p/ambition

Growing up, I had a fraught relationship with ambition. My deepest fear was not fulfilling my potential. To paraphrase Paul Graham in his

Cities & Ambitions

, I was one of those ambitious kids who possessed ambition that seemed to precede anything specific to be ambitious about. I just knew I wanted to do something great. I was singularly focused on working hard, being the best I could be, and maximizing my impact in any way I could.

For an immigrant in 21st century America, there are traditional standards of impact to strive towards: become a doctor, lawyer, engineer or go into business. I began my undergraduate career studying economics on the pre-law track. When it became clear to me that economics was largely theoretical and that many of my most ambitious peers were pursuing finance, I swiftly transferred to the business school.

When companies began recruiting on campus, I gravitated towards the highest “impact” jobs (read: highest status and highest paying). I pursued roles at investment banks and investment funds. When I landed my first full-time job at an investment fund, I poured my energy into proving my value as a new graduate. I developed a strong work ethic, prioritizing work over everything else to lay the groundwork for making partner at the firm one day.

When

my interests and ambitions no longer aligned

with my career trajectory in finance, I tried my hand at working in venture capital and then in early stage tech. With each pivot, I convinced myself that this new job and company was it — where I would unleash my full ambition and where it would finally flourish.

I would become infatuated with my new role and responsibilities, dreaming of how my future would unfold at the company. Naturally, I’d benchmark my self-worth against the progress I was making towards that reality, measured in the form of expanded responsibilities, promotions, and salary raises. This is what I believed ambition to be: being achievement-driven and outcomes-orientated with a commitment to relentless hustle.

In this paradigm, my ego drove my ambition which was fueled by a lack of worthiness. I thought to be worthy, I must prove that I was extraordinary by societal standards. If I changed the world then I’d be baseline worthy.

When we’re motivated by deep insecurity, we strive towards collecting accomplishments that are deemed impressive by others, but rarely does that lead ...
"Genlangs" and Zipf's Law: Do languages generated by ChatGPT statistically look human?
by Justin-Diamond — 2024-01-31T18:30:32.171Z
Link: https://arxiv.org/abs/2304.12191

OpenAI's GPT-4 is a Large Language Model (LLM) that can generate coherent constructed languages, or "conlangs," which we propose be called "genlangs" when generated by Artificial Intelligence (AI). The genlangs created by ChatGPT for this research (Voxphera, Vivenzia, and Lumivoxa) each have unique features, appear facially coherent, and plausibly “translate” into English.

This study investigates whether genlangs created by ChatGPT follow Zipf's law. Zipf’s law approximately holds across all natural and artificially constructed human languages. According to Zipf’s law, the word frequencies in a text corpus are inversely proportional to their rank in the frequency table. This means that the most frequent word appears about twice as often as the second most frequent word, three times as often as the third most frequent word, and so on. We hypothesize that Zipf’s law will hold for genlangs because (1) genlangs created by ChatGPT fundamentally operate in the same way as human language with respect to the semantic usefulness of certain tokens, and (2) ChatGPT has been trained on a corpora of text that includes many different languages, all of which exhibit Zipf's law to varying degrees. Through statistical linguistics, we aim to understand if LLM-based languages statistically look human. Our findings indicate that genlangs adhere closely to Zipf’s law, supporting the hypothesis that genlangs created by ChatGPT exhibit similar statistical properties to natural and artificial human languages.
AI, Intellectual Property, and the Techno-Optimist Revolution
by Justin-Diamond — 2024-01-31T18:30:32.153Z
Link: https://www.researchgate.net/publication/377030301_AI_Intellectual_Property_and_the_Techno-Optimist_Revolution

The emergence of generative Artificial Intelligence (“AI”) is putting computers at odds with creators and raising novel questions about the future of Intellectual Property (“IP”) law. Though scholars are scrambling to anticipate the legal consequences of AI, few have attempted to describe how prevailing tech-philosophies might shape AI regulation via persuasion and/or regulatory capture.

In this paper, I describe one such “techno-optimist” perspective and consider its idealized application to the thorniest issues of AI in IP. In short, how would a legal system influenced by techno-optimism treat issues of copyright and patent law in a world of rapidly accelerating AI? After examining (1) text & data mining (“TDM”) regulations, (2) copyright infringement by AI models, and (3) the patent system & patentability of AI generated innovations, I conclude that a techno-optimist approach to IP law would necessitate diminished data & copyright protections, but have indeterminate if not paradoxical implications for the patent system.
My Alignment "Plan": Avoid Strong Optimisation and Align Economy
by VojtaKovarik — 2024-01-31T17:03:34.778Z

Summary: Many people seem to put their hopes on something like the following "plan":

(Optional) Step 0: Hope, or assume, that sharp left turn will not happen. Proceed with capabilities until we get ~human-level AI. Then jump to Step 3.
Step 1: Convince everybody to avoid building the kind of AI that could undergo sharp left turn.
Step 2: Use AI for automating processes that we trust to be safe.
Step 3: Solve the problems caused by automating the economy.

If this is true, I think there should be more acknowledgment of this fact, and more discussion of the failure modes of this plan.

Epistemic status: Descriptive, rather than normative.

Descriptive, rather than Normative

I label the epistemic status of this post as "descriptive, rather than normative". What do I mean by that? And what do I mean by alignment "plan"?

While I thought a lot about AI alignment, I still have many uncertainties about the topic. And I don't have any plan, for helping us build beneficial AGI, that I would be optimistic about. But I keep working on this, and I have opinions and preferences over which projects to undertake. So, the question I ask in this post is: To the extent that my actions and beliefs seem to be in line with any plan at all, what "plan" do they seem to be following?

I should disclaim that my actual beliefs are a bit more nuanced than the description given here. But for the sake of brevity, I will stick with the simpler formulations below.

The main reason I write this post is that I suspect that many other people might be putting their hopes into a "plan" similar to what I describe. (In the case of alignment researchers, this might be explicit and due to the absence of better ideas. In the case of capabilities researchers, this might happen implicitly, as a result of background assumptions and not having thought about the topic.) To the extent that this is the case, I think it would be useful to acknowledge that this is what is happening, such that we can discuss the plan explicity. To the extent that other people have a significantly different plan, I would be curious to know what the plan is.

Finally, note that I make no claim that the plan described here, or even my more nuanced version of it, is good. In fact, I do not think it is good --- I just don't have a better one. And I think that explicitly describing the plan is the first step towards improving it.

My Alignment "

...
Where freedom comes from
by Logan Kieller — 2024-01-31T16:53:18.274Z
Link: https://logankieller.substack.com/p/where-freedom-comes-from

We are agents engaging with an environment. Based on the environment we, as agents, act.

Action will be determined by two aspects: magnitude and direction.

Magnitude is the execution, the effort, or the immensity of the action.

Direction is the reasoning, the strategy, or the path you ultimately choose to take.

Choosing to write is my action, choosing this topic is my direction, and choosing to spend hours researching, writing, discussing, and editing is my magnitude.

A mix of internal (factors of the agent) and external (factors of the environment) forces will catalyze and instantiate actions, each with its own unique magnitude and direction.

A good agent will choose the appropriate actions to respond to the environment, but a great agent will choose actions to intentionally change the environment.

As an agent, the environments you encounter throughout life present opportunities for action, yet not all of these environments initiate action evenly. The key to living a free and fulfilling life is understanding how action is initiated.

The first catalyst for an action is a shove.

A shove is a rude and obtuse external force that pushes you to do something.

It provides you with a prescribed magnitude and direction. In my essay “What fills a vacuum?” I would label this as something that claims a stake in your life. If my boss asks me to complete a menial task, I know exactly what I have to do and at what intensity; this is a shove.

This shove gives you a direction of action, and it requires some magnitude (which is usually evident).

The second catalyst for an action is a tunnel.

A tunnel is an external restriction that limits which direction you can go but has no claim on the magnitude.

It is a specific action you have control of the effort on. When your roommate asks you to tidy up something, they are providing you with a direction, yet you can choose how much effort to apply to the action. It becomes a trade-off, rather than something you have little control over.

The third catalyst for an action is a displacement.

Similarly, a displacement is an external force that pushes you to do something, yet unlike a shove, it is agnostic to the action itself, but it does demand an action is taken. It creates an exclusionary zone, a metaphorical “no loitering” sign. You cannot stay here but it doesn’t care where exactly you go. This instantiates a requisite magnitude, b...
Per protocol analysis as medical malpractice
by braces — 2024-01-31T16:22:21.367Z

“Per protocol analysis” is when medical trial researchers drop people who didn’t follow the treatment protocol. It is outrageous and must be stopped.

For example: let’s say you want to know the impact of daily jogs on happiness. You randomly instruct 80 people to either jog daily or to simply continue their regular routine. As a per protocol analyst, you drop the many treated people who did not go jogging. You keep the whole control group because it wasn’t as hard for them to follow instructions.

At this point, your experiment is ruined. You ended up with lopsided groups: people able to jog and the unfiltered control group. It would not be surprising if the eager joggers had higher happiness. This is confounded: it could be due to preexisting factors that made them more able to jog in the first place, like being healthy. You’ve thrown away the random variation that makes experiments useful in the first place.

This sounds ridiculous enough that in many fields per protocol analysis has been abandoned. But…not all fields. Enter the Harvard hot yoga study, studying the effect of hot yoga on depression.

If the jogging example sounded contrived, this study actually did the same thing but with hot yoga. The treatment group was randomly assigned to do hot yoga. Only 64% (21 of 33) of the treatment group remained in the study until the endpoint at the 8th week compared to 94% (30 of 32) of the control. They end up with striking graphs like this which could be entirely due to the selective dropping of treatment group subjects.

What’s depressing is that there is a known fix for this: intent-to-treat analysis. It looks at effects based on the original assignment, regardless of whether someone complied or not. The core principle is that every comparison should be split on the original random assignment, otherwise you risk confounding. It should be standard practice to report the intent-to-treat and many medical papers do so---at least in the appendix somewhere. The hot yoga study does not.

It might be hard to estimate this if you’re following people over time and there’s a risk of differential attrition---you're missing data for a selected chunk of people.

Also, hot yoga could still really work! But we just don’t know from this study. And with all the buzz, there’s a good chance this paper ends up being worse than useless: leading to scaled-up trials with null findings th...