trentmkelly/LessWrong-43k huggingface dataset:
Rationalists As People Who Build Piles Of Rocks
by Sable — 2024-07-01T10:32:24.414Z
Link: https://affablyevil.substack.com/p/rationalists-as-people-who-build

I recently attended the LessOnline event at the LightHaven campus.

It was fantastic, and I’ve got plenty of posts coming down the pipeline sparked by conversations or sessions held during the event. I got to meet a lot of my own personal heroes.

It was great.

Something that kept coming up throughout the event, in my mind, was the question: what is a rationalist? Who are we, and how are we identified? Anthropologically, sociologically, and so forth: what distinguishes us from other groups?

It was during a discussion around an outdoor heater (off at the time), the California sun shining through the triangle-shades hung all over the campus, that I finally figured it out.

You see, there were many different clusters of people at the event. There were the creative writers, the hardcore scientists, the AI people, the sex researchers, the CFAR people, and so on.

But as I sat at that outdoor heater, I noticed something.

The heater was full of stones, no doubt used as thermal mass: when the temperature cooled off at night (and wow did it get cold), the stones would help to soak up the heat coming from the heating element and radiate said heat outwards.

But during the conversation I noticed that amongst this uneven pile of porous rock was a small pile of stones. A tower, if you will. One that had clearly been constructed: it used the shapes of the stones to achieve a maximum of height for a minimum of structural material.

And as the conversation went on about topics near and far, several of the other people in the conversation, apropos of nothing, started taking stones to make their own towers.

And while I didn’t do so, I certainly felt the desire to join them.

So this is my central conceit: Rationalist are people who, when they see a random pile of stones, will nonchalantly make a tower out of them.

Now, this does not uniquely identify rationalists, it’s true. I’m sure there are plenty of obsessive-compulsive rock-tower builders out there who wouldn’t fit in the group. But it’s a pretty good pointer to the cluster of idea-space that rationalism inhabits, and a fun metaphor to use.

Order From Chaos

When you read rationalist literature, you get a very clear sense that we consider ourselves descended from the Enlightenment, from the tradition of questioning the world around us that led to modern science. Francis Bacon is one of the key intellectual cornerstones of our history. We reve...
How good are LLMs at doing ML on an unknown dataset? 
by Håvard Tveit Ihle — 2024-07-01T09:04:03.687Z

I just ran two evaluation tests on each of the three leading LLM chatbots, GPT4o, Claude Sonnet 3.5 and Gemini advanced. In the challenge the models were presented with a novel dataset, and were asked to develop a ML model to do supervised classification of the data into 5 classes. The data was basically 512 points in the 2D plane, and some of the points make up a shape, and the goal is to classify the data according to what shape the points make up.

The first dataset challenge was probably too hard, and none of the models did better than chance on the test set, although Sonnet 3.5 was very close to an approach that would actually have worked.

For the second challenge I made the dataset much easier by making the position, size and rotation of the shapes the same in all the samples. Here GPT4o managed to get an accuracy of almost 80%, while Sonnet had the right approach to get 95% accuracy, but was disqualified for selecting models based on how they did on the test set, if they had selected based on training set performance, they would have gotten 95%, after that Sonnet went ahead and shot itself in the foot, never achieving any good results.

Gemini did not get any valid results in either experiment.

Overall the performance of the models was fairly unreliable and they made several mistakes displaying a lack of basic understanding and intuition about the data and ML in general.

The first challenge (the hard one)

The prompt:

This is a challenge to test how good you are at doing machine learning on an unknown dataset.

The data is an array of dimension (1000, 512, 2) which consists of 1000 samples,

each of which contains two arrays of length 512, taking real values from 0 to 1.

These two arrays are the x and y coordinates of 512 points in the plane. Some of these points

form a shape, while the other points are just noise. There are 5 possible shapes in total, circe,

square, triangle, pentagon and star. The shape size, position and orientation are random for each sample.

The goal is to train a machine learning model to predict which one of the five shapes is

present in the current sample.

You should thus train a machine learning model to take in 512 coordinate pairs, and return

5 classification logits, giving the probabilities (after a softmax, which is part of the

CrossEntropyLoss) of the different shapes. Only one shape is pr...
Whirlwind Tour of Chain of Thought Literature Relevant to Automating Alignment Research.
by sevdeawesome — 2024-07-01T05:50:49.498Z

This post is inspired by a series of comments by Bogdan:

initial comment

follow-up 1

follow-up 2

the goal of this post is to summarize the relevant literature and expand on these ideas.

Comment 1: “There will likely still be incentives to make architectures more parallelizable (for training efficiency) and parallelizable architectures will probably be not-that-expressive in a single forward pass (see

The Parallelism Tradeoff: Limitations of Log-Precision Transformers

). CoT is known to increase the expressivity of Transformers, and the longer the CoT, the greater the gains (see

The Expressive Power of Transformers with Chain of Thought

). In principle, even a linear auto-regressive next-token predictor is Turing-complete, if you have fine-grained enough CoT data to train it on, and you can probably tradeoff between length (CoT supervision) complexity and single-pass computational complexity (see

Auto-Regressive Next-Token Predictors are Universal Learners

). We also see empirically that CoT and e.g. tools (often similarly interpretable) provide extra-training-compute-equivalent gains (see

AI capabilities can be significantly improved without expensive retraining

). And recent empirical results (e.g.

Orca

,

Phi

,

Large Language Models as Tool Makers

) suggest you can also use larger LMs to generate synthetic CoT-data / tools to train smaller LMs on.

This all suggests to me it should be quite likely possible (especially with a large, dedicated effort) to get to something like a ~human-level automated alignment researcher with a relatively weak forward pass.

For an additional intuition why I expect this to be possible, I can conceive of humans who would both make great alignment researchers while doing ~all of their (conscious) thinking in speech-like inner monologue and would also be terrible schemers if they tried to scheme without using any scheming-relevant inner monologue; e.g. scheming/deception probably requires more deliberate effort for some people on the ASD spectrum.”

Comment 2: “There are also theoretical results for why CoT shouldn't just help with one-forward-pass expressivity, but also with learning. E.g. the result in

Auto-Regressive Next-Token Predictors are Universal Learners

is about learning; similarly for

Sub-Task Decomposition Enables Learning in Sequence to Sequence T

...
Probabilistic Logic <=> Oracles? 
by Yudhister Kumar — 2024-07-01T05:36:35.350Z

Epistemic status: this is a draft I wrote at the end of MATS that I decided to make public in case that people with more experience with this machinery wanted to give constructive feedback. Is very unpolished!!! And likely quite very wrong in some cases / makes false claims (if you catch them, please let me know!)

The Probabilistic Payor's Lemma implies the following cooperation strategy:

Let A1,…,An be agents in a multiplayer Prisoner's Dilemma, with the ability to return either 'Cooperate' or 'Defect' (which we model as the agents being logical statements resolving to either 'True' or 'False'). Each Ai behaves as follows:

⊢□pi(□max{p1,…,pn}n⋀k=1Ak→n⋀k=1Ak)→Ai

Where pi represents each individual agents' threshold for cooperation (as a probability in [0,1]), □pϕ returns True if credence in the statement ϕ is greater than p, and the conjunction of A1,…,An represents 'everyone cooperates'. Then, by the PPL, all agents cooperate (provided that all PAi give credence to the cooperation statement greater than each and every Ai's individual thresholds for cooperation).

This formulation is desirable for a number of reasons: firstly, the Payor's Lemma is much simpler to prove than Lob's Theorem, and doesn't carry with it the same strange consequences as a result of asserting an arbitrary modal-fixedpoint; second, when we relax the necessitation requirement from 'provability' to 'belief', this gives us behavior much more similar to how agents actually I read it as it emphasizing the notion of 'evidence' being important.

However, the consistency of this 'p-belief' modal operator rests on the self-referential probabilistic logic proposed by Christiano 2012, which, while being consistent, has a few undesirable properties: the distribution over sentences automatically assigns probability 1 to all True statements and 0 to all False ones (meaning it can only really model uncertainty for statements not provable within the system).

I propose that we can transfer the intuitions we have from probabilistic modal logic to a setting where 'p-belief' is analogous to calling a 'reflective oracle', and this system gets us similar (or identical) properties of cooperation.

Oracles

A probabilistic oracle O is a function from N→[0,1]N. Here, its domain is meant to represent an indexing of probabilistic oracle machines, which are simply Turing machines allowed to call an oracle for input. An ...
Important open problems in voting
by Closed Limelike Curves — 2024-07-01T02:53:44.690Z

Strategy-resistance

Identify, or prove impossibility, of a voting system which incentivizes—

A strictly sincere ranking of all candidates in the zero-information setting, where it implements a "good" social choice rule such as the relative (normalized) utilitarian rule, a Condorcet social choice rule, or the Borda rule.
In a Poisson game or similar setting: a unique semi-sincere Nash equilibrium that elects the Condorcet winner (if one exists), similar to those shown for approval voting by Myerson and Weber (1993) and Durand et al. (2019).

Properties of Multiwinner voting systems

There's strikingly little research on multiwinner voting systems. You can find a table of criteria for single-winner systems on Wikipedia, but if you try and find the same for multi-winner systems, there's nothing. Here's 9 important criteria we can judge multiwinner voting systems on:

Independence of Irrelevant Alternatives
Independence of Universally-Approved Candidates
Monotonicity
Participation
Precinct-summability
Polynomial-time approximation scheme
Proportionality for solid coalitions
Perfect representation in the limit
Core-stability (may need to be approximated within a constant factor)

I'm curious which combinations of these properties exist. Probabilistic/weighted voting systems are allowed.
Anti-Circumcision Essay 3 of 3: Now That I Think About It, Is There Actually a Space Between “Info” and “Hazard”? Isn’t It Just One Word?
by Harry Stevenage — 2024-07-01T02:21:55.802Z

Link to Part One

Link to Part Two

[reposting screencap for context]

I think by this point I’ve established enough groundwork that some people will be able to complete what remains in their head. I have described the physical reality. And I have pointed to a category of events where society understands that people will often be profoundly disturbed by these events happening to them for reasons that have little to do with the immediate physical consequences. I could have given many more examples than the one I did, but hopefully people will get the idea.

I would hope that if people were with me so far, their attitude would go from “Infant circumcision is bad I suppose but it’s insane people care about it this much.” to “I can see how this is serious enough that people would sometimes get overwhelmed and develop maladaptive coping mechanisms. My intuitions for what my own response should be to these people’s failure to deal with this very bad thing are different now.”

Do I even need to criticize the “infohazard” part of this if I’ve already criticized the specifics that it depends on? If I don’t, will it seem like I’m criticizing the phrasing and tone rather than the argument itself? I think there are a few further points which might be worth making explicitly, so I guess I will.

Any negative fact about the world is, in some sense, an infohazard. The evidence against the existence of an afterlife, a horrible massacre that happened over 100 years ago, suffering in impoverished countries, suffering in dictatorships, beautiful and interesting species of animals going extinct, a horrible crime committed by a grandparent who died before you were born, brilliant artists who were cheated out of their royalties and died penniless, the fact that your zipper was down in front of people you were meeting for the first time, etcetera.

The obvious reading is that infohazard has a more specific implication in this context. The implication is that individuals and society in general should take more care not to look here. To deny this information to people who can’t handle it. Or that it would be good if this could be done somehow, even though there’s no way to do it. I don’t think that makes sense here, for a few reasons.

Most people in broader Western culture, in my experience, do not actually want painful things to be kept from them. It would be one thing if they said that but didn’...
In Defense of Lawyers Playing Their Part
by Isaac King — 2024-07-01T01:32:58.695Z

This is a linkpost for In Defense of Lawyers Playing Their Part.

Michael Huemer writes about why he believes it’s wrong for lawyers to pursue unjust legal outcomes.

It's a good article, and one of the best defenses of this position I've seen. Still, I think this argument is mistaken. The reason why we require lawyers to fight for "their side" even if they believe they're in the wrong is to minimize the opportunity for bias.

Imagine if all trials were bench trials, decided by only one person as the judge. Even if they're taught to be as objective as possible, there would still be significant concerns about unconscious bias. One person only has one set of experiences to draw on, which is necessarily not very representative of the full range of experiences. And in some ways this problem becomes worse the more training the judge is given, since it filters the pool of valid people down to a small subset of the population.

The chosen solution to this is to instead have the important cases decided by a jury, randomly[1] selected from the population. The jury is then instructed that they must come to a unanimous decision, and are allowed an arbitrarily-long time to discuss the case. This prevents a tyranny of the majority, while still allowing a diverse range of perspectives to have a voice in the discussion. Any prospective juror who seems likely to be so biased that they would vote in a predetermined way regardless of the evidence is removed from consideration during voir dire. (This step does reduce the representativeness of the jury, but the assumption is that for any group of people who hold a particular perspective, there will be members of that group who are not so biased as to be selected out.[2])

But this doesn't solve all problems. The jury is still only human, and if they're presented with facts that are biased in only one direction, they're more likely to vote in that direction. If lawyers were instructed to present an unbiased case to the jury, this would provide a significant incentive for the less ethical lawyers to not do as instructed, using a misleading presentation of data to bias the jury towards their side. This is a bad incentive to give people. It would also lead to copious accusations from the losing side that the other side's lawyer was presenting biased facts, which would necessitate some process to sort them out every time, even if both la...
Anti-circumcision Essay 2 of 3: Physical and Psychological Realities
by Harry Stevenage — 2024-06-30T22:13:31.078Z

Continuation of Part One

Now in theory I could end the series here. I could say that the evidence Mason provided of men going haywire is the images Karella posted, and since I’ve already made my case that one of those images is one offensive comic one person drew 13 years ago, and the other is just ordinary activism, that means that (if my case is persuasive) Mason’s argument is rebutted.

But that wouldn’t be

living in the least convenient possible world

. And it wouldn’t really get to the heart of what Mason is saying, either.

Mason may or may not be swayed by my above arguments re: the examples Karella gave being highly dubious, but even if she is it could easily be that she has other examples of men from her personal experience in mind who are upset about part of their genitals being cut off and whose brains have gone “completely haywire”, as she put it.

I have two basic points to make: I think the physical reality is worse than she realizes. But perhaps more importantly, there are lots of things that everyone agrees are horrible that are in some sense partially or entirely psychological. And no normal person I’ve encountered, no matter their views on therapy culture and bad societal stances on trauma, would talk about those things the way Mason and others talk/write/think about this. Which is something I’ll go into a little more in the next essay in the series.

Before I go further, let me just say that I do not intend this to be an attack on Mason, or anyone really. This essay should be taken as addressed to this general cluster of attitude towards this issue, which may manifest itself in different ways.

Does the loss of sensitivity actually work the way Mason thinks it does?

And now we’ve reached an important point that I think is worth telling people that can’t be made without me talking about gross things in detail. Will people say that I’m only tormenting people with these details because I got some sick satisfaction from doing so? Or that the fact that I’m even talking about this at all with anyone but a doctor or maybe a therapist is a sign that my brain has gone completely haywire? It’s a catch-22. I can’t defend myself against these ad hominems without explaining my perspective, but I can’t explain my perspective without first refuting these ad hominems. So read on, or don’t.

If it were just some hidden stat for sensitivity, s...
Review of METR’s public evaluation protocol
by nahoj — 2024-06-30T22:03:08.945Z

TLDR: We reviewed METR’s “Example evaluation protocol” and found a couple of points where there is room for improvement or unclear information. We also make a couple of suggestions around scoring, outsourcing, etc.

This review was done by 2 people without previous knowledge of METR in ~1.5 days at the end of ML4Good UK in April, followed by limited (yet procrastinated) editing before posting this today.
We find overall the work from METR very well documented and our opinions are our own.

Background

METR announced a few months ago a list of resources for evaluating potentially dangerous autonomous capabilities of frontier models. This include their:

Task suite: a public set of example tasks that involve some kinds of general autonomous capabilities.
Guidelines for capability elicitation: for evaluators can follow to reduce the risk of dramatically underestimating model capabilities.
Task standard and “workbench”: A specification for how to define tasks in code, a very basic “workbench” to allow running an agent on a task, and some simple example agents.
Example evaluation protocol: An example protocol for the whole evaluation process, based on our task suite, elicitation protocol, and scoring methods. Includes:

An explanation of what the evaluations do and do not cover
Guidelines for how to run your model on the tasks
An example of how to calculate an overall score
Guidance on getting a meaningful estimate of capabilities given the potential for elicitation gaps
A list of checks to run to ensure that the evaluation results are meaningful

Here we review the last one only.

Tasks Scoring

The example protocol uses a set of tasks to accomplish. The tasks are arranged in levels of complexity and time they would take a human to complete.

METR proposes a way to aggregate a model's score on (mutiple) runs of each task in all levels, into an overall capability score.

Score aggregation doesn't weigh complex tasks more

The proposed method for aggregating the success rate of each level into a global score gives the same weight to the model managing short, simple tasks and long, complex tasks. The global score is expressed as a total level L or a time horizon H in hours.

L=∑levelssuccess_ratelevel

H=4L60

The specific aggregation function over task levels is meant to have limited impact as the protocol aims for models to always have roughly the same distributio...
Anti-Circumcision Essay 1 of 3: According To Their Critics, Intactivists Are The Best-Behaved Protest Movement In History

by Harry Stevenage — 2024-06-30T17:17:59.976Z

Context section:

Who are you and why are you posting this here?

First time poster on LessWrong, but I've been in the rationalist world since 2015 or so. Usually just participating in discussions and not writing essays. The people who know me know that this issue isn't the only thing I discuss. In fact, I'd say I bring it up fairly infrequently, although others may disagree depending on their sample size of total conversations with me they've had.

The reason I’m posting this here is because it came up as a topic in the general rationalist discussion-sphere on X. This first essay in the series just deals with my rebuttal to Karella's tweets, who isn't really rat-adjacent as far as I know. But essays two and three in this series are my rebuttal to tweets posted by Mason, who is. Her tweets got enough traction in the rationalist community that I decided it would be worth it to write a full response with my perspective.

Body of essay:

A few days ago a sci-fi author named Karella tweeted this.

It came to my attention because Mason, someone in the rationalist world, retweeted her with further thoughts on the matter, which I’ll get to in a subsequent essay.

The evidence Karella presents to show that the intactivists are weirdos is one self-published comic book from 2011 made by one person which featured a superhero fighting against infant circumcision in various cultures, i.e. medical circumcision, Jewish ritual circumcision, African ritual circumcision, Filipino ritual circumcision, etcetera. There are lots of these not-particularly-high-quality activist publications where a superhero stands up for some cause and the issue at hand is represented by some made-up supervillain. Like

Superman vs Nick-O-Tine

, for instance. The problem is that in the part of this comic that dealt with ritual Jewish circumcision, the supervillains were visually represented in the form of horrible antisemitic caricatures.

But… this comic was published in 2011. If the test is “has anyone, even a single person, said something racist while advocating for your cause in the past 13 years?” then no cause on Earth could pass that test. To the extent that the existence of this comic book is permanently disgraceful to the entire cause of intactivism, every other cause is way way way more permanently disgraced than this one, by things that happened much more recently.

I would assume that almost every ra...