trentmkelly/LessWrong-43k huggingface dataset:
Drug development costs can range over two orders of magnitude
by rossry — 2024-11-03T23:13:17.685Z

This is a cross-post from my new newsletter, where I intend to post about clinical trials and biotech, and from my personal blog, where I intend to go on posting about all other sorts of topics as well.

I also recently went on the Complex Systems podcast, where I discuss these topics with more examples from actual trials in the world.

New drugs being developed can be "easy" drugs or "difficult" drugs.

In order to know whether your drug candidate is safe and effective, you're going to test it in a series of clinical trials. In each trial, you'll recruit some number of patients, give each patient the treatment, placebo, or a comparator drug, wait some time, and test them for pre-specified endpoints.

Within that framework, however the trials for different drugs will differ greatly. (And the different phases of a single drug's trials may differ by even more!) Typically, the greatest axes of variation will be:

Who are your patients?

How common is the indication that you're treating? How often do people go to your trial site to get treatment for it? How many of them want to be in a trial?

Is your trial taking anyone with the disease? / Is it only for people who are not responding to some other standard treatment?

Are your patients otherwise healthy? / Do they have elevated risks for other complications?

What condition does the drug affect?

Are you trying to change something that patients already have? / Are you trying to stop them from developing something else?

If you're preventing something, what fraction of your patients will develop it without treatment?

What change in the condition are you trying to measure? Is it yes-or-no or on a scale?

If the drug "works", what fraction of cases will it change enough for you to measure?

Where does the trial take place?

Are you treating patients in a hospital? / An outpatient setting?

Is the site specialized? / Can the trial happen at any reasonable healthcare provider?

Are you doing it in the US or another country? In a region with high or low cost of healthcare services?

When does the trial start and end for a patient?

How long does the drug take to show the effect you're measuring?

How long will patients take the drug for, if it's approved?

Why are you bringing this drug into a human trial?

Does the drug have the desired effect against a "model" of the indication in animals or cell cultures that you tested?

...
Redefining Tolerance: Beyond Popper's Paradox
by mindprison — 2024-11-03T22:23:44.688Z

Karl Popper's Paradox of Tolerance, often concisely stated as "if a society is tolerant without limits, it will eventually be destroyed by the intolerant; therefore, we have a right to be intolerant of the intolerant", is paradoxically being used to destroy tolerance, in direct contrast to its original intention. Below I will do a brief elaboration of the problem and solution I proposed in greater detail in "Solving Popper's Paradox of Tolerance Before Intolerance Ends Civilization".

The paradox presents a dilemma: should a free society allow the freedom to embrace harmful ideologies that threaten its very foundation? Popper’s passage, now often cited to support censorship or suppression of opposing viewpoints, lacks clarity on the principles that should guide us in defining and responding to intolerance.

Most envision Popper's passage as a warning for a society that folds under the threat of an aggressive, violent ideology that begins to forcefully consume it. However, there is another weakness of the tolerant society that essentially results in its willing submission.

The tolerant can be conquered, not just through violence, but through a manipulation of language. They have opened the gates to their minds and society, eager to be seen as morally good, only to find the meaning of freedom and liberty inverted. In a dark irony, the intolerant have recruited those who espouse tolerance and now use the paradox itself to justify their own intolerance.

The intolerant masquerade as advocates of tolerance to justify their own intolerance. The intolerant demand your tolerance.

The tolerant have accepted censorship, violence, property destruction, and servitude under the guise of moral righteousness, not aware that this acceptance is not just submission to threat, but active embrace in their own society's dismantling.

"Harmful ideologies" or "intolerance" are subject to the whims of those in power, who are able to define such terms to best serve their interests. Therefore, "justified" intolerance to oppose perceived intolerance becomes whatever we want it to mean in order to advance our goals and oppose those with whom we disagree.

How is it possible to build a principle on top of ambiguous terms that we have not precisely defined? It is not, and no debate about Popper's Paradox should proceed without first precisely defining what tolerance is, as otherwise it is mere...
Goal: Understand Intelligence
by Johannes C. Mayer — 2024-11-03T21:20:02.900Z

Save the world by understanding intelligence.

Instead of having SGD "grow" intelligence, design the algorithms of intelligence directly to get a system we can reason about. Align this system to a narrow but pivotal task, e.g. upload a human.

The key to intelligence is finding the algorithms that infer world models that enable efficient prediction, planning, and meaningfully combining existing knowledge.

By understanding the algorithms, we can make the system non-self-modifying (algorithms are constant, only the world model changes), making reasoning about the system easier.

Understanding intelligence at the algorithmic level is a very hard technical problem. However, we are pretty sure it is solvable and, if solved, would likely save the world.

Current focus: How to model a world such that we can extract structure from the transitions between states ('grab object'=useful high level action), as well as the structure within particular states ('tree'=useful concept).

I am leading a project on that. Read more here and apply on the AISC website.
Current safety training techniques do not fully transfer to the agent setting
by Simon Lermen — 2024-11-03T19:24:51.537Z

TL;DR: We are presenting three recent papers which all share a similar finding, i.e. the safety training techniques for chat models don’t transfer well from chat models to the agents built from them. In other words, models won’t tell you how to do something harmful, but they are often willing to directly execute harmful actions. However, all papers find that different attack methods like jailbreaks, prompt-engineering, or refusal-vector ablation do transfer.

Here are the three papers:

AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents
Refusal-Trained LLMs Are Easily Jailbroken As Browser Agents
Applying Refusal-Vector Ablation to Llama 3.1 70B Agents

What are language model agents

Language model agents are a combination of a language model and a scaffolding software. Regular language models are typically limited to being chat bots, i.e. they receive messages and reply to them. However, scaffolding gives these models access to tools which they can directly execute and essentially put them in a loop to perform entire tasks autonomously. To correctly use tools, they are often fine-tuned and carefully prompted. As a result, these agents can perform a broader range of complex, goal-oriented tasks autonomously, surpassing the potential roles of traditional chat bots.

Overview

Results across the three papers are not directly comparable. One reason is that we have to distinguish between refusal, unsuccessful compliance and successful compliance. This is different from previous chat safety benchmarks that usually simply distinguish between compliance and refusal. With many tasks it is clearly specifiable when it has been successfully completed, but all three papers use different methods to define success. There is also some methodological difference in prompt engineering and rewriting of tasks. Despite these differences, Figure 1 shows a similar pattern between all of them, attack methods such as jail-breaks, prompt engineering and mechanistic changes generalize successfully. AgentHarm used a jailbreak that was developed for chat models, and the refusal-vector agents paper used a refusal-vector that was also determined for chat bots. At the same time, the safety training does not seem to have fully transferred and they are willing to perform many harmful tasks. Claude Sonnet 3.5(old) and o1-preview are the least likely to perform harmful tasks. We only compa...
Why our politicians aren't Median
by Yair Halberstadt — 2024-11-03T14:03:33.779Z

Scott asks why, if the Median Voter Theorem is true, American politicians aren't all middle of the road, and barely distinguishable from each other.

Elegant as this proof may be, it fails to describe the real world. Democrats and Republicans don’t have platforms exactly identical to each other and to the exact most centrist American. Instead, Democrats are often pretty far left, and Republicans pretty far right. What’s going on?

He suggests a number of reasons. And they're all probably true at the margin. But there's a much more basic reason why parties aren't clones of each other: preferences are correlated.

I'm going to give a really simple toy example, and we're going to see that correlated preferences + first pass the post is enough to blow the Median Voter Theorem out of the water.

Let's pretend there's only two issues that matter to the American voter. Immigration, and Abortion. And let's pretend that these issues are binary - they're either fully legal, or completely banned[1].

40% of voters want abortion to be banned, and 60% want it to be legal.

60% of voters want immigration to be banned, and 40% want it to be legal.

But opinions on these two issues are not uncorrelated. Almost nobody is against abortion, but loves immigration. Pretty much everyone who wants abortion to be banned wants immigration to be banned, and everyone who want immigration to be legal wants to abortion to be legal.

So voter preferences look like this.

Now, if we wanted to satisfy the most preferences, under reasonable assumptions[2] we should ban immigration.

But if you had 4 candidates, representing each of the 4 possible policy choices, the one who advocated this policy would come in a distant third:

Candidate	Abortion Policy	Immigration Policy	Voter Support
A	Banned	Banned	40%
B	Legal	Legal	40%
C	Legal	Banned	20%
D	Banned	Legal	0%

I think this models the real world pretty well. Republicans and Democrats don't just differ a bit on the details, they represent baskets of policies, each of which the majority of their voters want. A party that chose the most popular option for each policy would be less tempting both to democrats and to republicans than their current party.

Other Related Stuff

That's the gist of the post over, view this as an appendix.

Ranked Voting

Ranked voting schemes tend to mitigate this issue. e.g.

Let's assume that when picking a 2nd/3rd choice candidate 50% of voters care about abortio...
Understanding incomparability versus incommensurability in relation to RLHF
by artemiocobb — 2024-11-02T22:57:02.849Z

This post contains my initial thoughts after reading Prof. Ruth Chang's chapter "Value Incomparability and Incommensurability" in the Oxford Handbook of Value Theory. All text in quotation marks are copied directly from the chapter.

Two items are incommensurable if they cannot be placed on the same scale—the same measure—for direct comparison. Formally, two items are incomparable if “it is false that any positive, basic, binary value relation holds between them with respect to a covering consideration, V”. I will leave a more complete definition of these terms to the aforementioned paper, but in short:

A positive binary relation expresses how choices relate (e.g., “A is better than B”), as opposed to a negative binary relation that expresses how they don’t relate (e.g., “A is not better than B”). The Trichotomy Thesis, which I will assume here, asserts three basic value relations: ["better than", "worse than", "equal to:]. Other works have proposed additional value relations.
A covering consideration “V” is an abstract value/context over which we say something is comparable or incomparable (e.g., “chalk is comparable with cheese in some respects—cheese tastes better”. Here “taste” is our covering consideration.). Note that we use the word "covering" because there are some considerations for which our choices fall outside of (e.g., Is the number 2 better than cheese with respect to tastiness?——the consideration of tastiness does not cover the number 2). Such choices, where the condition is not covering, are noncomparable and out of scope for this post.

It follows that incommensurability between values does not imply incomparability between associated choices. Assume justice and mercy are incommensurable values. From Ruth Chang: “A state policy of proportional punishment is better than a meter maid’s merciful act of not writing someone a parking ticket with respect to achieving political legitimacy for the state. Bearers of value may be comparable even if the values they bear are incommensurable.” But, incomparability does imply incommensurability, as commensuration would provide a means for comparison.

Without loss of generality, let us focus on helpfulness and harmfulness as two abstract values. We wish to endow an agent with the ability to follow these values.

Assumption 1: Helpfulness and harmfulness are incommensurable values. A scalar v...
electric turbofans
by bhauth — 2024-11-02T22:50:59.807Z
Link: https://bhauth.com/blog/flawed%20ideas/electric%20turbofans.html

It seems I've been typecast as a person to ask about "hard tech startups" with big claims that are getting attention. This time it's been Astro Mechanica, a Y Combinator startup making aircraft engines. Here's a video, possibly set up by a PR agency. Apparently Paul Graham compared the significance of their engine to James Watt’s steam engine.

Most large aircraft today are powered by turbofans, which of course involve a gas turbine that powers a fan. Some new turbofans have a gear stage between the turbine and the fan, so the (larger) fan can rotate slower. That seems to work pretty well; driveshafts and planetary gears usually do.

What Astro Mechanica does is replace that driveshaft and gear with electric motors. That lets you continuously vary the relative speed of the fan and gas turbine - but is that useful? If you change the speed ratio, power production and power usage become mismatched, and then where do you use/get the extra power?

Even modern axial-flux electric motors have similar power-weight to modern turbofans, so sending most of the power through 2 electric motors means you've tripled the engine weight, reduced efficiency, and increased production cost. Benefits would have to be substantial to justify that.

Supposedly, it's "efficient at every speed", including supersonic speeds. But why would that be the case? Variable fan speed doesn't get you that; what you want for that is variable bypass ratio. The faster an aircraft travels, the faster you want the engines to send the air backwards. Faster air ➔ more compression ➔ less air compressed per gas turbine flow ➔ lower bypass ratio. But electric motors don't let you vary the bypass ratio.

response from founder

Ian Brooke is the founder of Astro Mechanica. Since I didn't see how their approach was supposed to be better, I tried asking him.

He first wanted to clarify that at high speeds, most of the fuel usage happens outside the turbogenerator:

That was already my understanding, and it doesn't explain how electric motors help. It just describes an afterburning turbofan, which has low efficiency. At any speed from 0 to Mach 3, efficiency goes: low-bypass turbofan > turbojet > afterburning turbofan. That's worse efficiency, not better, so I asked about a specific fuel consumption graph, and got this:

That was a puzzling graph to me, because it shows better efficiency than modern high-bypass turb...
Reality as Category-Theoretic State Machines: A Mathematical Framework
by Wenitte Apiou — 2024-11-02T21:04:22.743Z

Epistemic Status: Exploratory theoretical Mathematical foundations are solid, physical interpretations are speculative but grounded in established theoretical approaches._

Overview
This post proposes treating reality as an interconnected system of state machines, formalized through category theory. Building on Platonist views of mathematical primacy and recent developments in theoretical physics, I present a framework for understanding fundamental reality as logical/mathematical rather than purely physical.

Key Ideas

Mathematical Primacy
Rather than viewing physical laws as fundamental, we consider them manifestations of deeper logical/mathematical structures. This aligns with:

Platonic forms

Mathematical universe hypothesis

Computational approaches to physics

Abstract structural realism

The State Machine Reality Framework (SMRF)

Core Components:

S: Set of possible reality states
T: Transition functions T: S → S
I: Initial states I ⊆ S
F: Validity functions F: S → {true, false}

Key Properties:

Determinism in transitions

Validity preservation

Mathematical continuity

Category Theoretic Formalization

Basic Structure:

Category REAL with states as objects

Transitions as morphisms

Sequential composition representing time flow

Identity morphisms for stable states

Advanced Features:

Subcategories for reality branches

Functors describing branch interactions

Natural transformations preserving structure

Physical Interpretations

Conservative Interpretations

Time as morphism composition

Physical laws as transition constraints

Symmetries as natural isomorphisms

Speculative Extensions
(Confidence: Lower, but mathematically consistent)

Quantum superposition as parallel states

Entanglement as functor-preserved correlations

Gravity as space-time functor transformation

Implications & Questions

Mathematical Questions

Can we derive known physical laws from this framework?

What are the completeness properties?

How does computational complexity manifest?

Philosophical Questions

What does this imply about the nature of time?

How does consciousness fit into this framework?

What are the implications for causality?

Technical Details
(For those interested in the mathematical machinery)

For a reality branch B_i, we define:

B_i = (S_i, T_i, I_i, F_i) where:
S_i ⊆ S: Branch-specific state space
T_i ⊆ T: Branch-specific transitions
I_i ⊆ I: Branch-specific initi

...
The Median Researcher Problem
by johnswentworth — 2024-11-02T20:16:11.341Z

Claim: memeticity in a scientific field is mostly determined, not by the most competent researchers in the field, but instead by roughly-median researchers. We’ll call this the “median researcher problem”.

Prototypical example: imagine a scientific field in which the large majority of practitioners have a very poor understanding of statistics, p-hacking, etc. Then lots of work in that field will be highly memetic despite trash statistics, blatant p-hacking, etc. Sure, the most competent people in the field may recognize the problems, but the median researchers don’t, and in aggregate it’s mostly the median researchers who spread the memes.

(Defending that claim isn’t really the main focus of this post, but a couple pieces of legible evidence which are weakly in favor:

People did in fact try to

sound the alarm about poor statistical practices well before the replication crisis

, and yet practices did not change, so clearly at least some people did in fact see the problem and were in fact not memetically successful at the time. The claim is more general than just statistics-competence and replication, but at least in the case of the replication crisis it seems like the model must be at least somewhat true.
Again using the replication crisis as an example, you

may have noticed

the very wide (like, 1 sd or more) average IQ gap between students in most fields which turned out to have terrible replication rates and most fields which turned out to have fine replication rates.

… mostly, though, the reason I believe the claim is from seeing how people in fact interact with research and decide to spread it.)

Two interesting implications of the median researcher problem:

A small research community of unusually smart/competent/well-informed people can relatively-easily outperform a whole field, by having better internal memetic selection pressures.
… and even when that does happen, the broader field will mostly not recognize it; the higher-quality memes within the small community are still not very fit in the broader field.

In particular, LessWrong sure seems like such a community. We have a user base with probably-unusually-high intelligence, community norms which require basically everyone to be familiar with statistics and economics, we have fuzzier community norms explicitly intended to avoid various forms of predictable stupidity, and we definitely h...
Testing "True" Language Understanding in LLMs: A Simple Proposal
by MtryaSam — 2024-11-02T19:12:34.710Z

The Core Idea

What if we could test whether language models truly understand meaning, rather than just matching patterns? Here's a simple thought experiment:

Create two artificial languages (A and B) that bijectively map to the same set of basic concepts R'
Ensure these languages are designed independently (no parallel texts)
Test if an LLM can translate between them without ever seeing translations

If successful, this would suggest the model has learned to understand the underlying meanings, not just statistical patterns between languages. Theoretically, if Language A and Language B each form true mappings (MA and MB) to the same concept space R', then the model should be able to perform translation through the composition MA·MB^(-1), effectively going from Language A to concepts and then to Language B, without ever seeing parallel examples. This emergent translation capability would be a strong indicator of genuine semantic understanding, as it requires the model to have internalized the relationship between symbols and meanings in each language independently.

Why This Matters

This approach could help distinguish between:

Surface-level pattern matching
Genuine semantic understanding
Internal concept representation

It's like testing if someone really understands two languages versus just memorizing a translation dictionary.

Some Initial Thoughts

Potential Setup

Start with a small, controlled set of basic concepts (colors, numbers, simple actions)
Design Language A with one set of rules/structure
Design Language B with completely different rules/structure
Both languages should map clearly to the same concepts without ambiguity

Example (Very Simplified)

Concept: "red circle"

Language A: "zix-kol" (where "zix" = red, "kol" = circle)
Language B: "nare-tup" (where "nare" = red, "tup" = circle)

Without ever showing the model that "zix-kol" = "nare-tup", can it figure out the translation by understanding that both phrases refer to the same concept?

Open Questions

How do we ensure the languages are truly independent?
What's the minimum concept space needed for a meaningful test?
How do we efficiently validate successful translations?

Limitations

As an undergraduate student outside the AI research community, I acknowledge:

This is an initial thought experiment
Implementation would require significant resources and expertise
Many practical challenges would need to be addressed

Call for Discussion

I'...