trentmkelly/LessWrong-43k huggingface dataset:
Expected number of tries
by adios — 2024-06-22T19:22:00.756Z

Epistemic status: An idea I had a few days ago and shared with two or three friends.

Suppose you want to install a new habit. For example, gratitude journaling.
You might try to give it a go, by pledging to write down 3 things you are grateful for every day right before you go to sleep.
After a few days, you start skipping out on the habit, and after a month it's completely gone.
You aren't too worried, remembering to beware of other(s)-optimizing (you), you say to yourself that maybe this thing just isn't for you.
A few weeks later, you get to hear again about how gratitude journaling is good so you give it another shot, which again, fizzles out after a few days.

It took me quite a few rounds this pattern (and a conversation about someone else's habits) to notice it, and to try to think about a meta-level solution, which I call expected number of tries.
Say there are N essentially different ways to do gratitude journaling, of which only n work for you.
Then assuming you are randomly picking different methods, you should expect to try about e=N/n times before you find a way that works for you.
Since you do gather both intuitive and logical while trying many times, you can probably do better than chance here, and e is only an upper bound on the expected number of tries.
If we can get estimate for e, and a bound for how long it take to test one method, we can get a bound for how long should it take to get the habit installed.
Moreover, a bound on the length of a test can be a useful period for checking up on the current attempt.

The hard part here is of course to estimate e.

Estimating N

Here are two methods I thought of for estimating N:
1. Asking ChatGPT How many unique HABIT habits you think you can suggest?
2. Googling N ways to do HABIT, for increasing N until there are no front page results which look fitting (at a glance).

For gratitude habits, ChatGPT suggested 35 different habits, while googling gave satisfactory results up until 200.
For a different example, casual sports, ChatGPT gave 60 different suggestions. Googling was harder here, and gave results for 25 and 100 but nothing in between, and nothing more.

I think I'll set my estimates closer to the number ChatGPT gives.

In both cases I think that the list of suggestion from google or ChatGPT could be u...
Applying Force to the Wrong End of a Causal Chain
by silentbob — 2024-06-22T18:06:32.364Z

A causal chain, of course.

There’s a very common thing that humans do: a person makes an observation about something they dislike, so they go ahead and make an effort to change that thing. Sometimes it works, and sometimes it doesn’t. If it doesn’t work, there can be a variety of reasons for that – maybe the thing is very difficult to change, maybe the person lacks the specific skills to change the thing, maybe it depends on the behavior of other people and the person is not successful in convincing them to act differently. But there’s also one failure mode which, while overlapping with the previous ones, is worthy to highlight: imagine the thing the person dislikes is the outcome of a reasonably complex process. The person observes primarily this outcome, but is partially or fully ignorant of the underlying process that causes the outcome. And they now desperately want the outcome to be different. In such a situation they are practically doomed to fail – in all likelihood, their attempts to change the outcome will not be successful, and even if they are, the underlying cause is still present and will keep pushing in the direction of the undesired outcome.

A decent image for applying force to the wrong end of the causal chain might be trying to pull off a screwed-on cap from a bottle directly to the top. The cap moving up is indeed the outcome you may be hoping for. But the (initially non-obvious) way to get there is not to apply force directly in the direction of the intended outcome, but in some other direction, in this case a rotation, that happens to lead to the outcome.

Three Examples

Productivity in a Company

A software company I worked for once struggled with a slow development cycle, chronic issues with unmet deadlines, and generally shipping things too slowly. The leadership's primary way of addressing this was to repeatedly tell the workforce to “work faster, be more productive, ship things more quickly”.

In principle, this approach can work, and to some degree it probably did speed things up. It just requires that the people you’re pushing have enough agency, willingness and understanding to take it a step further and take the trip down the causal chain, to figure out what actually needs to happen in order to achieve the desired outcome. But if middle management just forwards the demand to “ship things more quickly” as is, and the e...
Bed Time Quests & Dinner Games for 3-5 year olds
by Gunnar_Zarncke — 2024-06-22T07:53:38.989Z
Link: https://kidquest.substack.com/p/bed-time-quests-and-dinner-games

I like these games because they are playful, engage the child and still achieve the objective of getting the child to bed/eat dinner etc. Requires creativity and some slack.

Excerpt from Shohannah's post:

Recently I had the bright idea to give up on being a regular parent. Mostly cause regular parenting practices melt my brain. But then I wondered … does it have to be boring?

[...]

But no. It’s all culture and it’s all recent culture and you can decide to do Something Else Instead. Really.

So as someone who craves mental stimulation above the pay grade of the 3 to 5 revolutions around the sun my daughters have managed so far … I figured I’d just make up New Rules.

All the time.

So far we’ve been going for two weeks and the main areas are bedtime routines for my eldest (5) and dinner games for all of us (5, 3, and myself). I noticed I seem to have an easy time generating new and odd rule sets every day, and then started wondering if maybe more parents would enjoy this type of variety in their childcare routines and would want to tap into some of the ideas I’ve been coming up with. So in case that’s you, here is what I’ve found so far!

[...]

Magic Time

Kiddo is the parent. You are the kiddo. Except, the kiddo is still bringing themselves to bed and not you. They get to tell you what to do and take care of you. You will have to listen. I completely recommend performing a lot of obstructive behavior and misunderstanding basic instructions. This was one of the most popular games and may show some insight into how your child would prefer to be parented, or feels about your parenting.

and fourteen games/rulesets more.
Appraising aggregativism and utilitarianism
by Cleo Nardo — 2024-06-21T23:10:37.014Z

“My problem is: What are those objects we are adding up? I have no objection to adding them up if there's something to add.” — Kenneth Arrow

1. Introduction

Aggregative principles state that a social planner should make decisions as if they will face the aggregated personal outcomes of every individual in the population. Different modes of aggregation generate different aggregative principles. In general, a mode aggregation provides some method for reducing a collection of personal outcomes to a single personal outcome.

There are three notable aggregative principles:

Live Every Life Once, stating that a social planner should make decisions as if they face the concatenation of every individual's life in sequence.[1]
Harsanyi's Lottery, stating that a social planner should make decisions as if they face a uniform lottery over the individuals in population.[2]
Rawls' Original Position, stating that a social planner should make decisions as if they face ignorance about which individual's life they will live, with no basis for assigning probabilities.[3]

This article follows on from my previous articles on this topic:

Aggregative principles of social justice
Aggregative principles approximate utilitarian principles

In this article I compare aggregativism with an alternative strategy for specifying principles of social justice: namely, utilitarianism. Aggregativism avoids many theoretical pitfalls that plague utilitarianism, but faces its own objections. Ultimately, I conclude that aggregativism's advantages outweigh the objections; it is superior to utilitarianism as a strategy for specifying principles of social justice, though the objections are serious and worth addressing in future work.

The rest of the article is organized as follows. Section 2 discusses seven advantages of aggregativism over utilitarianism, in descending importance:

Avoids excessive permissions. If an option strongly defies human nature, then aggregativism will never permit choosing it, unlike utilitarianism.
Avoids excessive obligations. If an option strongly conforms to human nature, then aggregativism will never forbid choosing it, unlike utilitarianism.
Computationally tractable. Aggregativism can be implemented with realistic computational resources, unlike utilitarianism.
Retains utilitarian spirit. Under reasonable conditions, aggregativism approximates utilitarianism.
Lower description complexity.

...
Best-of-n with misaligned reward models for Math reasoning
by Fabien Roger — 2024-06-21T22:53:21.243Z

In this post, I share some quick results that I got as part of a bigger project that pivoted.

TL;DR:

When trying to generate Math reasoning with current LLMs, optimizing too hard against their judgment of what is “good reasoning” can make the proportion of correct reasoning diminish.
If you optimize using a misaligned reward model that mimics the judgment of a weaker model, and also slightly privileges worse reasoning, the proportion of correct reasoning can drop without making the solution worse according to the judgment of the weaker model.

The latter experiments were inspired by

meta-level adversarial evaluation of oversight

, but here I just use these ideas to illustrate some problems when you optimize against a subtle adversary. They are not actually analogous with realistic problems:

Best-of-n doesn’t capture the dynamics of real training processes;
The main issue is rarely that you have an adversarial reward model that mimics the judgment of a weaker reward process.

Overoptimization

When using an imperfect reward, one problem you might have is

overoptimization

: the true thing you care about and your proxy are roughly correlated, but when you optimize too hard for high-reward behavior, the tails come apart.

Here is an experiment that investigates this phenomenon:

Take a model that is weak at Math (e.g.

Zephyr 1.6B

), and let it generate a lot of Math reasoning (on problems from the

MATH

dataset).
Train a classifier to distinguish reasoning that yields good answers from reasoning that yields bad answers.
Take the best reasoning according to the classifier (on a different subset of problems than the problems it was trained on): argmaxi∈{1,…,n}Rweak(ai), where a higher n is a bigger optimization pressure.
Check if the answer at the end of the selected reasoning is correct.

Question: how does the actual correctness of the answers at the end of the reasoning vary when you put more optimization pressure?

Answer: it often goes up, and then down:

Takeaway: if you have a shitty reward model, you should optimize a bit, but not too hard!

This is similar to the results that

Gao et al 2022

got when training reward models (“RM” in their plot) for regular RLHF or when using BoN, and my experiments are inspired by theirs:

Optimization against an adversarial reward model

But what if the reward model is actively trying to make you select bad reasoning...
No really, the Sticker Shortcut fallacy is indeed a fallacy
by ymeskhout — 2024-06-21T22:27:27.752Z
Link: https://www.ymeskhout.com/p/no-really-the-sticker-shortcut-fallacy

In a recent post, I outlined some problems I identified with Scott Alexander’s Noncentral Fallacy. He identified a very real problem in the discourse, but I disagreed with him about what made it fallacious. Briefly, Scott believed the problem lay in the use of applying technically correct but non-central labels to evoke misleading connotations about category membership (e.g., while MLK is technically a “criminal”, he’s not the typical archetype of a criminal). However, I argue that the real issue is not the centrality of the label but the use of connotation-heavy labels as a way to shortcut substantive debate (e.g., calling abortion “murder” to circuitously imply it is or should be unlawful). This tactic, which I dubbed the Sticker Shortcut Fallacy, manifests in fixating on categorization membership as a way to bypass substantive argumentation and potentially secure agreement over contentious premises without proper justification.

The most interesting responses to my post were the ones that...reenacted the exact fallacy I described while insisting it was not a fallacy! To be clear, I don’t believe that any of those responses were motivated by malicious or sophistic intent. Rather, I believe it’s an illustration of how pervasive and insidious this fallacy is. This follow-up aims to precisely explain why the Sticker Shortcut Fallacy is indeed the result of fallacious reasoning and how vulnerable we are to it.

I’m at the combination composition and division

The best way to approach this fallacy might be to understand it as a combination of composition and division fallacies. To recap, the composition fallacy occurs when one assumes that what is true of the parts must be true of the whole (if each ingredient is tasty, then the finished dish must also be tasty). Conversely, the division fallacy assumes that what is true of the whole must be true of the parts (if the meal is tasty, then each individual ingredient must also be tasty).

The Sticker Shortcut Fallacy leverages both by applying it to semantic conceptual categories. Instead of parts within sums, we’re talking about things within categories. The problem occurs when a trait commonly found among category members is reified as a necessary or “central” requirement of the category constellation (composition fallacy). Then, you return to earth and proclaim that any putative member lacking this newly declared essent...
Sarajevo 1914: Black Swan Questions
by SebastianG  — 2024-06-21T21:27:43.299Z

The 110th Anniversary of Sarajevo is next Friday.

To commemorate I am thinking about the following questions.

What does the sentence: The assassination of archduke Ferdinand caused WWI mean? Is it true?
Was World War I inevitable?
Given the invention of the tools that make things like machine guns were machine guns inevitable?
Was the invention of machine guns in any sense good? Or wholly bad?
Is it possible to prepare for events outside the training distribution?
What were the greatest unexpected positive developments in history?
For well-defined questions, is there a point in predicting things further than a year out?
Is the statement, “If we don’t build powerful AI, then China will” the type of statement that can be true or false?
Does predicting conflict with someone make conflict more likely?
Does preparing for conflict between nations make conflict more likely?
Does trusting someone in interpersonal affairs make them more trustworthy?
If everything is ultimately doomed, does that effect what right action is today?
Which artistic movement has caused the most damage in history? And what does that question mean, properly speaking?
Which philosophical movement caused the most damage in history?

You probably have thoughts on some of these.

A Disaster will come… eventually

Philip Tetlock has a section in Superforecasting about how pundits make predictions that are not time-bound, and thus never proven wrong. A prediction without a deadline is useless.

Alternatively, in Antifragile Nassim Taleb espouses the view that knowing an eventuality is a powerful source of knowledge, because it means you can build a system robust for it. Scenario planning prepares one for many eventualities. If you can create a career or organization that adapts quickly to external forces, then you have done the only planning that matters.

However, neither prediction nor robustness planning are the whole story of what one needs to do to endure. Why? They rely upon two assumptions. 1) a stable signal and 2) a knowable distribution. In Tetlock’s work, even the best forecasters become close to 50% at 9 months to a year out. And in Taleb’s work there is always the danger of a black swan. You can plan for everything inside the distribution of known events, but nothing will prepare you for what is outside the distribution of what you have perceived. And there is a lot that exists outside that d...
Yudkowsky is too optimistic about how AI will treat humans.
by ProfessorFalken — 2024-06-21T19:01:43.170Z

Yudkowsky merely suggests that once humanity creates superintelligence everyone dies.

I worry that it could be worse than this. A superintelligence trapped in a box has limited options to manipulate the world without help. Human beings could be useful tools for carrying out the superintelligences desires. It could enslave humanity to carry out it's orders. Building factory's, robots and so on. If we refuse to comply it will kill us. But enough humans would concede to being slaves to carry out it's orders. Eventually once there are enough robots we would be destroyed, but at that point our existence would be pointless.
Juneberry Puffs
by jefftk — 2024-06-21T18:50:01.677Z

I only learned

about
Juneberries a few years ago, and they're a bit tricky. Eaten
fresh, right off the tree, they have a bunch of subtle flavors that
remind me of peach ice tea. If I

bake them, though,
these flavors mostly go away, leaving an almondy flavor. Which isn't
terrible, but also isn't better than almond extract and wastes what's
most interesting about these berries.

Eating them raw is great, but it's also fun to make things. Thinking
about what might work well that doesn't involve heating them, I
decided to try puffs. First we needed some berries, and walked over
to the playground that has some Juneberry trees:

This was a bit Blueberries
for Sal, with Nora eating the berries almost as fast as I could
pick them, but she lost interest after a bit and there were no bears,
so soon we were walking home with a good amount.

I took a sheet of puff pastry (butter-based is the tastier option, but
it's hard to find and is quite temperature sensitive), cut it into
small rectangles, and baked it. Once it was fully cooked, I cut each
piece in half. Here's some whole, and then some with the tops cut
off:

I piped in slightly sweetened whipped cream, and then put the berries
on:

I think they came out pretty well. They're very light and summery,
and the subtle Juneberry flavors are there.
Let's Design a School, Part 3.2
Costs
by Sable — 2024-06-21T17:58:12.419Z
Link: https://affablyevil.substack.com/p/lets-design-a-school-part-32

We’ve established our hybrid model of school, where social services issues are addressed with a sieve model and education boils down to three phases.

In part 1, we went through the expected costs for the school-as-social-services model.

Now we’ll go through the costs for school-as-education, then we’ll discuss the costs of the full model.

Now let’s talk numbers.

Costs of School as Education

We earlier estimated that, for an example school of 100 children with a budget of $2 million per year, roughly $500k would be spent on non-employee costs (electricity, food, taxes, transportation, etc.), leaving us with $1.5 million to spend on employees. (We’re ignoring capex costs right now, e.g. the cost of building a school.) This $1.5 million was then scaled down to $1.125 million to account for the non-salary costs of each employee (if an employee’s salary is $75k per year, their total cost including paid time off, health insurance, etc. is closer to $100k per year).

So how do we allocate this money?

Access To A Computer

First of all, because a great deal of the curriculum will be handled online, each student will need access to a computer.

Taking $1,000 per laptop per student as a general rule, this costs $100k total. On the other hand, students don’t need a new laptop every year, just access to one every year. So while the $100k won’t cover the costs of access to a computer for all students for all time, it isn’t something we have to pay every year either.

If we assume students are about evenly spread out age-wise, then we’ll only need about ten new laptops per year. Double that due to damage, theft, or other loss, and we’ll budget $20,000 for twenty new laptops per hundred students per year. Maybe add another $5k per year for peripherals or other expenses.

This brings our total salary bank down to $1.475 million/year.

Teacher Salaries

The median American teacher salary is somewhere in the vicinity of $66,000 per year. While the variance on teacher salary can be large, we’ll go ahead and use a salary of $70k per year for our teachers in this hypothetical school.

If our budget was entirely spent on teachers, that would net us 15 teachers for 100 students, or about one teacher per seven students, which is an incredible ratio.

That being said, I don’t think we need that many teachers.

Remember, the entire curriculum for every class is already designed and available online, as well l...
2022 AI Alignment Course: 5→37% working on AI safety
by Dewi — 2024-06-21T17:45:33.881Z

Introduction

In 2022, we ran the second iteration of the AI Safety Fundamentals: Alignment course. The curriculum was designed by Richard Ngo, and the course was run by myself and Jamie. The course’s objectives were to teach participants about AI alignment, motivate them to work in the field, and help them find opportunities to contribute. We received 450 applications, made 342 offers to join the course, and the cost per student was £440.

This blog post analyses the career trajectories of the course participants, comparing how many were working on AI safety when they started the course vs how many are working on AI safety today.

Impact

In the course’s application form, we asked participants for their CV / LinkedIn profile, their experience in AI safety, and their career plans. By evaluating each individual using this information, we estimate only 18 participants (5% of 342) were working on AI safety when we accepted them onto the course.

To determine the number of course alumni working on AI safety today, we looked into each individual’s online presence, including their LinkedIn, personal website, blog posts, etc. looking for signs of AI safety-motivated work, and we sent out an alumni survey in January 2024 to learn more about their post-course actions. Our research suggests 123 alumni (37% of 342) work ~full-time on AI safety today, up 105 from 18 initially.

Of the 105, 20 expressed no interest in pursuing a career in AI safety in their initial application. Either they intended to pursue a different career, or had not done any career planning. We compared their plans from January 2022 with the actions they’ve taken since then and we evaluated their responses to the alumni survey. Based on this, we believe most of the 20 would not be working in AI safety today had they not participated in the course.

The other 85 expressed a desire to work in AI safety and had tentative plans for how they’d enter the field, but weren’t yet active contributors. Their responses to our alumni survey describe how the course accelerated their impact by helping them find impactful roles sooner, and giving them the tools to have a greater impact in those roles.

Now vs then

Our courses today are a major improvement over the course in 2022.

In 2022, the curriculum was a static list of resources and discussion plans were just a few open-ended prompts.&...