trentmkelly/LessWrong-43k huggingface dataset:
Focus on the Hardest Part First
by Johannes C. Mayer — 2023-09-11T07:53:33.188Z

Consider reading this instead.

Here is some obvious advice.

I think a common failure mode when working on AI alignment[1] is to not focus on the hard parts of the problem first. This is a problem when generating a research agenda, as well as when working on any specific research agenda. Given a research agenda, there are normally many problems that you know how to make progress on. But blindly working on what seems tractable is not a good idea.

Let's say we are working on a research agenda about solving problems A, B, and C. We know that if we find solutions to A, B, and C we will solve alignment. However, if we can't solve even one subproblem, the agenda would be doomed. If C seems like a very hard problem, that you are not sure you can solve, it would be a bad idea to flinch away from C and work on problem A instead, when A seems so much more manageable.

If solving A takes a lot of time and effort, all of that time and effort would be wasted, if you can't solve C in the end. It's especially worrisome when A has tight fightback loops, such that you constantly feel like you are making progress. Or when it is just generally fun to work on A.

Of course, it can make sense to work on A first if you expect this to help you solve C, or at least give you more information on its tractability. The general version of this is illustrated by considering that you have a large list of problems that you need to solve. In this case, focusing on problems that will provide you with information that will be helpful for solving many of the other problems can be very useful. But even then you should not lose sight of the hard problems that might block you down the road.

The takeaway is that these two things are very different:

Solving A as an instrumental subgoal in order to make progress on C, when C is a potential blocker.

Avoiding C, because it seems hard, and instead working on A because it seems tractable.

Though I expect this to be a general problem that comes up all over the place. ↩︎
The Promises and Pitfalls of Long-Term Forecasting
by GeoVane — 2023-09-11T05:04:50.006Z

Disclaimer: We will be speaking at the Manifest Conference in Berkeley, CA (Sept. 22-24) about long-term forecasting, its promises, and its pitfalls. Below is an excerpt from last week's edition of our newsletter, Predictions, detailing some of what we plan to speak about at the event. This week's edition can be found here.

If you are reading this newsletter, you probably have a certain amount of intellectual buy-in on the concept of quantified forecasting. However, much of the social sciences remain skeptical, to say the least, about the practice.

Take the domain of international relations, where Philip Tetlock got his start and serves as the basis for much of the quantified forecasting research. Many scholars have raised objections about whether or not we can prediction international relations which can be best captured by Robert Jervis’ System Effects published in 1997.

System Effects

The essence of the book (which should probably get its own dedicated post at some point) is that the realm of international politics deals with a system where its elements are interconnected, such that a change in one part of the system makes changes in other parts of it, and that the system contains properties and behaviors which are different than those of its part (in other words, that the total is greater than the sum of its parts).

As a result of systems effects, in the international system (and any complex, interconnected system for that matter) we have:

Delayed and indirect outcomes

Emergent characteristics — Relationship between elements are based on relationships to other elements

Non-integrable function — you cannot understand the whole thru its parts

Unintended outcomes

Nonlinearities — unexpected breaks from the past given history

Feedback loops

Regulation being difficult

Does Jervis believe these system effects doom prediction? Not entirely, especially in his 1997 book. Although by his 2012 revisit, Jervis takes a slightly more negative tone, writing:

...My approach has an ambiguous stance towards prediction” and that “being realistic about the limits of our ability to know how we can reach desired ends can make us freer to act on our ideals. When it is not possible to see around the bend, to use Jones-Rooy and Page’s phrase, perhaps it is better not to try.

What is the response by the forecasting space to these claims? In the same journal edition as Jervis’ 2012 update, Phili...
Logical Share Splitting
by DaemonicSigil — 2023-09-11T04:08:32.350Z
Link: https://pbement.com/posts/laws_of_trading.html

Are mathematicians just not trying hard enough?

The Riemann hypothesis is one of the most important open problems in math. There's a $1 million prize from the Clay mathematics institute for a proof or disproof of the Riemann hypothesis. At the time of writing, it remains unsolved. From this, we may conclude that one cannot simply buy a solution to difficult mathematical problems.

Or could we? How do we know that buying a difficult maths proof is impossible? Perhaps the Clay mathematics institute is somehow not asking the question in the right way. And it's true that the value of a million dollars has been eroded over time by inflation. One might guess that a Riemann proof would be worth at least 100 million. Would that be enough to conjure it from the collective intelligence of humanity?

Simply directly declaring a $100 million reward for a solution would probably not work. For one thing, there's the issue of corollary-sniping where the prize wouldn't give anyone an incentive to publish solutions to hard intermediate steps of the problem, since the prize as a whole only goes to the one who solves the entire problem as a whole. For another, even the million dollar prize on its own would be plenty of reason for a money-motivated person to solve the problem if a solution were within their grasp. The issue is not merely one of funding, we humans are somehow failing to combine our efforts properly.

Prediction markets are pretty cool

One of the standard ways to buy knowledge is prediction markets. Can we try that here? John Wentworth describes here a scheme for using markets to get mathematical proofs. Here's a scheme that's very similar in the overall idea, though the exact rules are slightly different:

Shares on mathematical propositions are traded on the market. Propositions should be the kind of things that might be theorems, i.e. they are syntactically meaningful, and contain no free variables, though it's fine if they are false or unprovable.

Shares on provable propositions are worth $1, at least if anyone knows or can find the proof. How this works out in practice is that a share in ⊤ can be redeemed in exchange for $1, together with the next rule.

Shares in logically equivalent propositions can be exchanged for each other. So if it's the case that A⟺B then a share in A can be exchanged for a share in B. This is a trading rule that will obviously have to be

...
What are some good language models to experiment with?
by tailcalled — 2023-09-10T18:31:50.272Z

Like if I want to experiment with a steering technique, it would be useful to have a language model that is small, capable, but not so finetuned that it becomes inflexible. (Or maybe ideally, a model which has both a finetuned and a non-finetuned variant.)

I've seen some people use GPT-2. Is that recommended? Are there any alternatives?
Playing the game vs. finding a cheat code
by Metacelsus — 2023-09-10T18:11:56.077Z
Link: https://open.substack.com/pub/denovo/p/playing-the-game-vs-finding-a-cheat

This is a linkpost from my blog De Novo.

Imagine a new Pokémon game has just come out, and you really want to catch a Zapdos. It’s listed in the game’s Pokédex, so you know it must be possible to catch, but you’re not sure how.

You could either:

Play the game normally. There are some hints that Zapdos like to hang out in thunderclouds so you could try looking there.
Randomly mash buttons and look for any weird glitches which you can exploit to give yourself a Zapdos.

Zapdos is a legendary lightning-bird Pokémon that looks pretty cool.

Advantages of playing normally

You know it’s possible to get a Zapdos this way, it will just take a while and you’ll need to explore thunderclouds a lot.
By exploring thunderclouds, you might learn more about them.
If you get a Zapdos using a glitch, it’s possible that the save file might be corrupted in some way and your Zapdos won’t work quite right.

Advantages of button mashing

Other players have found glitches to get other Pokémon, and although nobody has got a Zapdos yet, it seems plausible that it could work.
By finding glitches, you might learn more about the internal logic of the game.
If you have the code of the game to examine, you can more easily find glitches.
Once you’ve found the proper sequence of button mashes, it’s a lot easier to get a Zapdos than by playing the game. This is good if you want to get more of them, or if you want to share the cheat code you’ve found with other people.

Gotta grow ‘em all

Now, I don’t care so much about catching rare Pokémon, but I do really want to grow an oocyte in cell culture. Again, I have two choices for my general approach:

Try to mimic the developmental signaling processes of the ovary, going from pluripotent stem cells, to primordial germ cells, to oogonia, to oocytes.
Use transcription factors to reprogram the cells into oocytes (including activating meiosis).

Both approaches are valid!

The first approach has been very successful in mice, shown in several groundbreaking papers[1] by the labs of Mitinori Saitou and Katsuhiko Hayashi. Although it will certainly be more challenging in humans due to the longer time required for development, I think it’s promising. For this to work, a strong knowledge of human ovarian development is key, and until recently, this was not available due to lack of data on human fetal ovaries.

The second approach has also been somewhat successful. The Haya...
Cruxes on US lead for some domestic AI regulation
by Zach Stein-Perlman — 2023-09-10T18:00:06.959Z

Written quickly. Suggestions welcome.

A possible risk of some US AI regulation is that US regulation would differentially slow US AI progress and that would be bad. This post explores the factors that determine how much US regulation would differentially slow US AI progress and how bad that would be.

Note that the differentially slowing US problem only applies to regulation that slows US AI progress (toward powerful/dangerous systems), such as strong regulation of large training runs. The US can do things like facilitate incident reporting and clarify AI labs' liability for harms without slowing domestic AI progress, and some regulation (especially restricting the publication of AI research and sharing of model weights) would differentially slow foreign AI progress!

Note that international coordination on AI safety mostly avoids this problem.

Cruxes

If I was making a model of the differentially slowing US problem, these would be its factors.

(Here "China" often can mean any foreign state. Actual-China seems most relevant because it's well-positioned to lead on AI in worlds where strong US regulation slows US AI progress.)

How much would US regulation differentially slow US AI progress? (This and its subcruxes depend on the details of the regulation.)

To what extent does the regulation (or legible-safety in general) slow progress?[1]

In the abstract?
To what extent is leading labs' behavior already congruent with the regulation?[2]

Would China voluntarily follow US regulation?[3] Would the US be able to

extraterritorialize

its regulation effectively?
To what extent does US AI progress boost Chinese AI progress (via e.g. publishing research or leaking insights)?

How bad would differentially slowing US AI progress be?

How bad would it be directly?

How far behind is China; how long would it take to catch up (after pricing in the possibility of relevant future events like stronger US export controls)?

How far behind is China now?
How effectively will US/allies deny compute to China?[4]

How strong will US/allied export controls be?
How effectively will they be enforced?
Will US/allies restrict access to cloud compute for AI training? (How effectively?)
Will China attempt to obtain compute illegally? (How effectively?)

At a given level of compute access, how committed is China to frontier AI?

How much safer are leading US labs than leading Chinese labs?[5]
How muc

...
Using Negative Hallucinations to Manage Sexual Desire
by Johannes C. Mayer — 2023-09-10T11:56:24.906Z

[Edit 2023-10-05]: Thanks to jimmy's comments I realized that I was abusing this technique. I agree with jimmy that integrating your sexuality instead of suppressing it is better, based on my partial attempts and successes so far. This seems painfully obvious in hindsight. However, this technique works as advertised for me, and I expect I will be using it in the future. If you want to use this technique I advise you to tread lightly. It is very powerful. I was able to completely suppress sexual thoughts for at least 2 weeks in a row iirc.

[Edit]: I think I have mistitled this post. It should not contain the negative hallucination part, because it is not required at all for the technique to change your perception of your body. It's enough to imagine a fictional scene that contains the properties named below. This really seems to trip people off.

If you often experience strong sexual desire that is distracting you from work, consider trying this:

Imagine you have no primary sexual organs. It's just a flat piece of skin down there. Furthermore, imagine that you don't experience any sexual arousal no matter how somebody touches you.

Also imagine the same for the object of your sexual desire (usually another person).

This does not need to be applied to positive emotions that are not sexual desire. E.g. it is fine to feel a positive emotion when you imagine hugging somebody.

In my experience doing this is super effective. It shuts down my sexual desire within seconds. I have used it many times now so I am pretty sure that it works, and I am not imagining it.

I expect this technique works well, because it only requires that you adhere to the constraints specified in the first two paragraphs. You don't need to control what you are thinking any further. You can imagine rubbing yourself against somebody else, flat skin on flat skin. But it just feels kind of pointless and so you stop. It's like imagining rubbing your elbow against the elbow of another person. It's not something you are in danger of imagining too much.

I find this technique really useful for preventing myself from sexualizing people, and to not get distracted by sexual thought. I was pretty good at it before, but using this technique feels at least an order of magnitude better.

All of this, of course, breaks down once somebody touches you in an erotic way or unmistakably informs you that your primary sexua...
Feature proposal: Export ACX meetups
by Viliam — 2023-09-10T10:50:15.501Z

Currently there are two places that display upcoming meetups of Astral Codex Ten:

Community Section of LessWrong
articles at ACX, such as "Meetups Everywhere 2023: Times & Places"

Note that there are almost 200 meetups this autumn. Sometimes a mistake happens, and meetups that are displayed at one place are missing from the other place. As a consequence, some people miss the information about an upcoming meetup.

I don't know how exactly this works today. My best guess is that someone prepares a spreadsheet, and then imports it to both places. That part seems to work nicely.

Problems happen when the information changes later. People announce new ACX meetups after the deadline. Also, meetup organizers can edit the information on LW, but the changes do not automatically propagate to ACX.

ACX runs on Substack, which is a generic blogging platform. There is no specialized support for meetups, only a WYSIWYG editor for articles.

If these assumptions are generally correct, I propose to modify the process like this.

Step one: After collecting information to the spreadsheet, import the data to LessWrong. Then throw the spreadsheet away, because now LessWrong becomes the source of truth. (Meetup organizers can edit their announcements.)

Any meetups that are announced after the deadline, must be created on LessWrong. It is possible to reuse the same process: collect them to a (new!) spreadsheet, and import it. But they can also be created manually, whatever is more convenient.

Step two: Make an export from LessWrong for Astral Codex Ten, in format that is most convenient for Scott to import. Probably rich text or HTML code, which he could simply copy and paste to the article, without having to edit it further. That means, that the exported meetups would be organized in Scott's preferred way (alphabetically, groupped by continent and city) and properly formatted.

That means, at first, Scott would create a new article, copy the data from LessWrong, and write an introduction. After changes, Scott would delete the imported part from the existing article, and copy the new export from LessWrong.

Afterwards he can compare the old export and the new export (e.g. using this tool) and announce the major changes in the next Open Thread. This part cannot be fully automated, because there are various kinds of possible changes, some of them important (new meetup, cancelled meetup, change of...
AI presidents discuss AI alignment agendas
by TurnTrout — 2023-09-09T18:55:37.931Z
Link: https://www.youtube.com/watch?v=02kbWY5mahQ

None of the presidents fully represent my (TurnTrout's) views.

TurnTrout wrote the script. Garrett Baker helped produce the video after the audio was complete. Thanks to David Udell, Ulisse Mini, Noemi Chulo, and especially Rio Popper for feedback and assistance in writing the script.
Probabilistic argument relationships and an invitation to the argument mapping community
by lunatic_at_large — 2023-09-09T18:45:47.001Z

I've been thinking about the probabilistic properties of how ideas are connected to each other, and I think that the Lesswrong community might be interested in the same kind of questions that interest me. I'd like to introduce[1] a mathematical framework for talking about random arguments and the ideologies that are roughly-self-consistent within them. Using this framework to answer practical questions requires a lot of data that I don't have. I think that the best source of this data is from argument mapping programs, and that one of the most direct applications of this theory is to make computational argument mappers that can provide their users with non-immediately-obvious insights into their arguments. I've noticed some interest in the rationalist space for argument mapping programs and it seems that a good number of people are creating them. For this reason, I'd like to invite anyone who is in the business of creating argument mappers to get together and chat about how these kinds of computational tools could be integrated -- I made a Discord server at https://discord.gg/2DuBbmEQ5X but am open to other platform suggestions. Maybe we could collect enough data to start addressing some interesting philosophical questions!

Motivation

Here are some questions that I don't feel that I currently have the tools to reason about:
(1) Should we expect debates to converge as we spend more time/effort thinking about them? If we pick a question with a yes/no answer "at random" and look at the probability that we think the answer is "yes" over time (assuming we live in a static world where no real-life developments impact our reasoning), then does this probability usually have a limit as time goes to infinity? Is it likely to exhibit some kind of "false convergence" where, for example, it hovers near 1 for debate lengths between 10 minutes and 10,000 hours but at 10,001 hours a new consideration could suddenly emerge and change the arguer's probability of the answer being "yes" to near 0?

(2) How should we expect ideologies to evolve over time? Let's think of ideologies as "roughly-self-consistent" sets of ideas. Suppose you take a community that has been thinking about some cluster of questions for time T. Suppose you observe roughly three competing ideologies. Let's say you return to this community after they've been thinking for time 2T. How would you expect the se...
How teams went about their research at AI Safety Camp edition 8
by Remmelt — 2023-09-09T16:34:05.801Z

AI Safety Camp connects new collaborators worldwide to discuss and decide on a concrete research proposal, gear up online as a team, and try their hand at AI safety research during intensive coworking sprints.

Fourteen teams formed at this year's virtual camp. Here are their original proposals. Below are summaries at the end of the camp. Each team presented their findings at our final online weekend together. Some published a paper or post since. Most are continuing work, so expect a few more detailed and refined write-ups down the line.
Do you have AI Safety research ideas that you would like others to work on? Is there a project you want to do and you want help finding a team to work with you? Apply here.

Want to fund organisers who had to freeze their salary? Email remmelt@aisafety.camp.

Uncertainty -> Soft Optimization

Research lead: Jeremy Gillen

Team Participants’ names: Benjamin Kolb, Simon Fischer, Juan Azcarreta Ortiz, Danilo Naiff, Cecilia Wood

Short write-up: For AISC 2023, our team looked into the theoretical foundations for soft optimization. Our goal at the beginning was to investigate variations of the original quantilizer algorithm, in particular by following intuitions that uncertainty about goals can motivate soft optimization. We ended up spending most of the time discussing the foundations and philosophy of agents, and exploring toy examples of Goodhart’s curse.

Our discussions centered on the form of knowledge about the utility function that an agent must have, such that expected utility maximization isn’t the correct procedure (from the designers perspective). With well calibrated beliefs about the true utility function, it’s always optimal to do EU maximization. However, there are situations where an agent is very sensitive to prior specification, and getting this wrong can have a large impact on the true utility produced by the agent. Several other unrelated threads were pursued, such as an algorithm with high probability of above-threshold utility, and the relationship between SGD and soft optimization.

Post: Soft-optimization, Bayes and Goodhart

Discussing and Crystallizing a Research Agenda Based on Positive Attractors & Inherently Interpretable Architectures

Research lead: Robert Kralisch

Team Participants’ names: Anton Zheltoukhov (TC), Sohaib Imran, Johnnie Pascalidis and David Liu

Short write-up: For AISC 2023, our team w...