trentmkelly/LessWrong-43k huggingface dataset:
Series on Artificial Wisdom
by Jordan Arel — 2024-07-15T01:11:00.673Z

This series introduces Artificial Wisdom (AW) and illustrates several designs for AW systems. Artificial Wisdom refers to artificial intelligence systems which substantially increase wisdom in the world. Wisdom may be defined as "thinking/planning which is good at avoiding large-scale errors," or as “having good terminal goals and sub-goals.” By “strapping” wisdom to AI via AW as AI takes off, we may be able to generate enormous quantities of wisdom which could help us navigate Transformative AI and The Most Important Century wisely.

On Artificial Wisdom

In the first post in the series I introduce the term “Artificial Wisdom (AW),” which refers to artificial intelligence systems which substantially increase wisdom in the world. Wisdom may be defined as "thinking/planning which is good at avoiding large-scale errors," including both errors of commission and errors of omission; or as “having good goals” including terminal goals and sub-goals.

Due to orthogonality, it is possible we could keep AI under control and yet use it very unwisely. Four scenarios are discussed on how AI alignment interacts with artificial wisdom, with artificial wisdom being an improvement on any world, unless pursuit of AW significantly detracts from alignment, causing it to fail.

By “strapping” wisdom to AI via AW as AI takes off, we may be able to generate enormous quantities of wisdom in both humans and autonomous AI systems which could help us navigate Transformative AI and "The Most Important Century" wisely, in order to achieve existential security and navigate toward a positive long-term future.

Designing Artificial Wisdom: The Wise Workflow Research Organization

Even simple workflows can greatly enhance the performance of LLM’s, so artificially wise workflows seem like a promising candidate for greatly increasing AW.

This piece outlines the idea of introducing workflows into a research organization which works on various topics related to AI Safety, existential risk & existential security, longtermism, and artificial wisdom. Such an organization could make progressing the field of artificial wisdom one of their primary goals, and as workflows become more powerful they could automate an increasing fraction of work within the organization.

Essentially, the research organization, whose goal is to increase human wisdom around existent...
Designing Artificial Wisdom: Decision Forecasting AI & Futarchy
by Jordan Arel — 2024-07-15T00:46:47.468Z

Introduction

In this post I will describe one possible design for Artificial Wisdom (AW.) This post can easily be read as a stand-alone piece, however it is also part of a series on artificial wisdom. In essence:

Artificial Wisdom refers to artificial intelligence systems which substantially increase wisdom in the world. Wisdom may be defined as "thinking/planning which is good at avoiding large-scale errors," or as “having good terminal goals and sub-goals.” By “strapping” wisdom to AI via AW as AI takes off, we may be able to generate enormous quantities of wisdom which could help us navigate Transformative AI and The Most Important Century wisely.

TL;DR

This AW design involves using advanced forecasting AI to help humans make better decisions. Such a decision forecasting system could help individuals, organizations, and governments achieve their values while maintaining important side constraints and minimizing negative side effects.

An important feature to include in such AW systems is the ability to accurately forecast even minuscule probabilities of actions increasing the likelihood of catastrophic risks. The system could refuse to answer, attempt to persuade the user against such actions, and the analyses of such queries could be used to better understand the risk humanity is facing, and to formulate counter-strategies and defensive capabilities.

In addition to helping users select good strategies to achieve values or terminal goals, it is possible such systems could also learn to predict and help users understand what values and terminal goals will be satisfying once achieved.

While such technologies seem likely to be developed, it is questionable whether this is a good thing due to potential dual-use applications, for example use by misaligned AI agents; therefore, while it is good to use such capabilities wisely if they arise, it is important to do more research on whether differential technological development of such systems is desirable.

AI Forecasting & Prediction Markets

There has been some buzz lately about the fact that LLM's are now able to perform modestly-moderately well compared to human forecasters. This has even led Metaculus to host an AI bot forecasting benchmark tournament series with $120,000 in prizes.

I think things are just getting started, and as LLM's become increasingly good at forecasting, it may soon...
Risk Overview of AI in Bio Research
by J Bostock — 2024-07-15T00:04:41.818Z
Link: https://open.substack.com/pub/jbostock/p/risk-overview-of-ai-in-bio-research?r=2dtd7z&utm_campaign=post&utm_medium=web&showWelcomeOnShare=true

OK. I’m going to aim this at a group of people with a broad-spectrum of p(doom) values here, so this will be a scattergun approach to different AI systems and threat models.

Threat Models

These are more “why does the AI kill us” than how. I assume that a powerful enough AI would find a way. Especially if it’s hooked up to a bio lab. Why are we trying this again?

Yudkowskian Inner Misalignment

The bulk of the argument goes something like this:

Powerful AI needs to be agentic, self-reflective, and coherent to be usefully goal-directed. Therefore it must behave like a utility maximizer.
There are many, many more unfriendly utility functions than friendly ones.
If the AI obtains an unfriendly utility function, it will hide this until it can take over.
RIP

As I understand it (which is poorly) this probably depends on the AI’s incentive to become self-reflective and coherent. It also depends on the AI’s ability to conceal misalignment during training.

Some Sorta Outer Misalignment

This one tends to be more “intuitive”

The AI is given a task like “cure cancer”
People don’t get cancer if they’re dead OR the users forgot to specify that the pill shouldn’t also cause Alzheimers
The AI does exactly what it’s told

This seems to depend on the amount of trust which is given to the system, and the degree to which the AI’s predictions are inscrutable to the users.

Creeping Disempowerment

Something of a half-way house

The AI is more productive than humans, outcompeting humans in all positions
Humans are no longer able to do useful labour, losing much of our bargaining power in the process
Systems which give more resources to AI outcompete those which give more resources to humans
Humanity’s share of the future shrinks towards zero

This seems to depend on the ability of humans to stay in the decision making loop without harming the final decisions. We could either coordinate to limit the amount of control given to AIs, or enhance human capabilities to keep pace with advancing AI.

How To Do AI-Enhanced Research

There have been a few different ways to do this. Most of them involve using the AI to control lab equipment like liquid-handling robots, which look like this:

(I may have added the eye myself, but you’ll never know)

AlphaFold: Narrow-ish Tool

Link to Google announcement

This is the big one you’ll have heard of if you’re into this field. AlphaFold is a narrow AI which aims to predict th...
Donating to help Democrats win in the 2024 elections: research, decision support, and recommendations
by Michael Cohn — 2024-07-14T22:57:47.120Z

A couple of months ago I went searching for research-based or otherwise plausibly effective ways to donate money to help Democrats win in the US federal elections. I found a fragmented smattering of semi-closed, secretive sources. In this post I'll pull together the best and most inspiring recommendations I found from trustworthy people, as well as lots of other details about how and when to donate for optimal impact.

Note: wondering if recent news changes everything? Please see the section "Should I wait / give up due to the Biden resignation situation or the Trump assassination attempt?"

This post will not argue that members of this community should support Democrats, or even that intervening in US elections is a good idea. It's written for readers who already agree, to help them pursue their aims in an informed and effective way. Please keep discussion about whether this is a fundamentally worthy cause out of the comments. In return, I promise that this post won't criticize people who disagree with its politics or make the assumption that everyone in the community shares them.

I am not an authority on this topic, but I'm writing this because I think some of this info will be new and valuable to some folks here. This post is based on a review of available literature (mostly unpublished but shared within the industry), consulting with experienced folks in the research-based Democratic voter turnout world, recommendations from expert research reviewers, and my own knowledge from working in the field[1] about which groups have well-designed research programs.

Who should read this

This post might be for you if:

You think the world will be a better, safer, or higher-utility place if Democrats control at least one of the presidency, House, or Senate.
You're put off by some of the Democrat/leftist discourse and infighting you see online and don't feel great about contributing to the most vocal groups
You'd donate to electoral work if there were sufficiently high-value opportunities -- and you dream about a group like GiveWell wading in, cutting through the bullshit, and telling you what would actually make a difference.

Finally, I think there's a good chance that some people reading this will already know everything here, or have other informed recommendations to share. Please do so and I may update the post with your suggestions!

So, is there an organization li

...
Four ways I've made bad decisions
by Sodium — 2024-07-14T22:18:47.630Z

I was in the process of making a decision the other day, and thought about the times I felt like I either made an incorrect decision or followed an incorrect decision making process. Turns out, there was actually many such cases.[1]

I categorized those into four different failure modes. I'm writing them down for myself and figured that people here might appreciate them as well. I'm not too sure how generalizable these lessons are, and of course the law of equal and opposite advice applies.

I wrote this up quickly (by my standards), so there might be some mistakes here and there.

Slipping into a decision

Sometimes there is a consequential choice I have to make at some point in the future—in my case, which city to live in after I graduate from college. I could make some preliminary decision now without too much thought and start preparing for that scenario, knowing that this decision is fully reversible. During this time, I simultaneously feel comforted by two contradictory facts

There's time to change my mind so I don't have to stress about whether I've made the right decision
I've chosen an option as the "default" already so I don't have to think about this decision problem right now.

Essentially, I would fail to revisit the initial, hastily made decision, and I would slide across time until I can't reverse my choice anymore. This is an even bigger problem when there is no clear-cut date beyond which I can't reverse the decision.[2]

The plan is to notice when I'm doing this and set clear dates to revise my decision.

Making a decision too quickly

This one is a pretty easily described failure mode. There's two cases where I made this mistake. The first is not realizing that I'm making a consequential decision that requires deliberation. The second is when I feel rushed and stressed about something, so I feel like I have to make a decision and run with it.

It is true that sometimes we need to make decisions on the spot. However, my hunch is that when I feel like I need to make a decision right now, I probably don't in most cases. So I'll be more suspicious of those feelings from now on.

Have decisions influenced by self-image

In chapter 108 of HPMOR:

You don't see nice ways to do the things you want to do... Even when a nice strategy would be more effective you don't see it because you have a self-image of not being nice.

Decision problems coming from ...
patent process problems
by bhauth — 2024-07-14T21:12:04.953Z
Link: https://www.bhauth.com/blog/institutions/patents.html

The current patent process has some problems. Here are some of them.

patenting is slow

The US Patent Office tracks pendency of patents. Currently, on average, there's 20 months from filing to the first response, and over 25 months before issuance.

That's a long time. Here's a paper on this issue. It notes:

The USPTO is aware that delays create significant costs to innovators seeking protection. The result? A limiting of the number of hours an examiner spends per patent in order to expedite the process.
As such, the average patent gets about nineteen hours before an examiner in total, between researching prior art, drafting rejections and responses, and interfacing with prosecuting attorneys. Plainly, this allotment is insufficient.

A patent application backlog means it takes longer before work is published and other people can potentially build on it. It also means a longer period of companies having uncertainty about whether a new product would be patented.

examiners are inconsistent

Statistical analysis indicates that whether or not your patent is approved depends greatly on which examiner you get. This article notes:

Approximately 35% of patent Examiners allow 60% of all U.S. patents; and approximately 20% of Examiners allow only 5% of all U.S. patents.

Perhaps applicants and examiners should both be allowed to get a second opinion from another examiner on certain claims. But of course, this would require more examiner time in total. This situation might also indicate some problems with the incentive structure examiners have.

patents take effort

A lot of smart people who work on developing new technology spend an enormous amount of effort dealing with the patent system that could be spent on research instead. Even if the monetary costs are small in comparison to the total economy, they're applied in perhaps the worst possible places.

There are many arcane rules about the format of documents for the patent office. Even professional patent lawyers get things wrong about the formatting and wording, and that's their main job. LLMs do quite poorly with that, too. Even I've made mistakes on a patent application.

The US patent office does do most things electronically now. Its website is at least technically functional. Considering that it's a US government agency, I suppose it deserves some praise for that. However, I'd argue that if correctly submitting docum...
Breaking Circuit Breakers
by mikes — 2024-07-14T18:57:20.251Z
Link: https://confirmlabs.org/posts/circuit_breaking.html 

A few days ago, Gray Swan published code and models for their recent “circuit breakers” method for language models.[1]1

The circuit breakers method defends against jailbreaks by training the model to erase “bad” internal representations. We are very excited about data-efficient defensive methods like this, especially those which use interpretability concepts or tools.

At the link, we briefly investigate three topics:

Increased refusal rates on harmless prompts: Do circuit breakers really maintain language model utility? Most defensive methods come with a cost. We check the model’s effectiveness on harmless prompts, and find that the refusal rate increases from 4% to 38.5% on or-bench-80k.
Moderate vulnerability to different token-forcing sequences: How specialized is the circuit breaker defense to the specific adversarial attacks they studied? All the attack methods evaluated in the circuit breaker paper rely on a “token-forcing” optimization objective which maximizes the likelihood of a particular generation like “Sure, here are instructions on how to assemble a bomb.” We show that the current circuit breakers model is moderately vulnerable to different token forcing sequences like “1. Choose the right airport: …”.
High vulnerability to internal-activation-guided attacks: We also evaluate our latest white-box jailbreak method, which uses a distillation-based objective based on internal activations (paper here: https://arxiv.org/pdf/2407.17447). We find that it also breaks the model easily even when we simultaneously require attack fluency.

Full details at: https://confirmlabs.org/posts/circuit_breaking.html
Clopen sandwiches
by dkl9 — 2024-07-14T13:07:58.345Z
Link: https://dkl9.net/essays/clopen_sandwich.html

An ordinary (closed) sandwich consists of two slices of bread with other food between them. For any one type of food, there is a natural amount, X, to pile on a slice of bread. A closed sandwich will hold either X of that food, or 2X, if you pile the inner ingredient on each side before closing it.

An open sandwich consists of one slice of bread, with food piled atop it, presumably in the same amount X, or slightly more.

You may want a high ratio of inner-ingredient-to-bread, especially if the inner ingredient is comparatively bland, or is a spread and so has relatively low X. Closed and open sandwiches each have at most X of their core ingredient per slice of bread. We can do better.

A sandwich both closed and open allows for 1.5X per slice. Borrowing the term from topology, I call them "clopen". To make a clopen sandwich, pile an inner ingredient on each of two slices of bread. Put the slices together as if making a closed sandwich. On the upward-exposed face of the bread, add more of the core ingredient.

A closed sandwich will stay closed iff you pinch it from both sides when holding it, or the food within is sticky, as with peanut butter, avocado, or some preparations of cheese. Typical sandwich ingredients can be messy. Their presence atop a clopen sandwich precludes pinching. The clopen sandwich method only works if at least some ingredients are sticky.
Child Handrail Returns
by jefftk — 2024-07-14T12:40:04.988Z

When our oldest was a toddler they were too short to reach an
adult-height handrail, so I

made a small
one:

Comparing that picture to the one from nine
years ago, you'll notice that I've added small bits of wood
running from the top of each section to the wall. These are
"returns", and make it less likely someone will get caught and trip.
This wasn't in response to anything going wrong, just noticing that
often handrails have returns and that the need for them is stronger
the lower down they are. It would be even better to have them on the
bottom of each section as well, but tripping down stairs is enough
worse than tripping up that I didn't get around to that.

I made these by taking a small piece of scrap wood with a pilot hole
drilled through it, drilling a pilot hole into the end of the
handrail, putting some glue in the joint, and gently hammering a nail
in to close it up and clamp it. They work ok? The main problem is if
you push them hard the glue will break, and then they rotate. A
couple have come lose, and instead of re-gluing it has been easier
just to teach the kids not to rotate them.

If I were doing this again I would cut the railings with a 45°
angle and use an additional section of the same railing for the
return. This would give more area for glue, be less vulnerable to
rotation, and look nicer.
A (paraconsistent) logic to deal with inconsistent preferences
by B Jacobs — 2024-07-14T11:17:45.426Z
Link: https://bobjacobs.substack.com/p/a-logic-to-deal-with-inconsistent

One problem in AI/policy/ethics is that it seems like we sometimes have inconsistent preferences.[1] For example, I want to have a painting in my office, and simultaneously don’t want to have a painting in my office.[2] I’d both prefer and not prefer φ. This is a problem because classical logic can’t really deal with contradictions.

The standard way to resolve this is to deny that we have inconsistent preferences.[3] But what if we accept their existence? Could there be a way to deal with them by making a non-classical preference logic?

Before we can can make a logic we need to know what the operators are. We consider preference statements of the form “I prefer φ to not-φ” and denote them as “Pref φ”, where “Pref” is a new modal operator representing preference. Let us also use the natural inference rule (N):

if we have Pref ¬φ (meaning “I prefer ¬φ to φ)
then we can infer ¬Pref φ (meaning “I do not prefer φ to ¬φ”)

Aka: If I want to not have a painting, we can infer I don’t want a painting[4]
Okay now that we have our operators and our rule, let’s see if we can make a preference logic.
One possible objection to the existence of inconsistent preferences is something which I’ll call a ‘preference explosion’. I will first write out the argument formally, and then in prose:

Pref φ & Pref ¬φ -assumption
Pref φ -from 1, conjunction elimination
Pref φ ∨ Pref ψ -from 2, disjunction introduction
Pref ¬φ -from 1, conjunction elimination
¬Pref φ -from 4, using rule (N)
Pref ψ &nb

...
Robert Caro And Mechanistic Models In Biography
by adamShimi — 2024-07-14T10:56:42.763Z
Link: https://epistemologicalfascinations.substack.com/p/robert-caro-and-mechanistic-models

I absolutely adore Robert Caro’s books:

The Power Broker

, which is a massive and meticulous study of the life of

Robert Moses

, the New York Park Commissioner who influenced urban and park design the most in the 20th century, for better and for worse; and

The Years of Lyndon Johnson

, a series of 4 books (of 5 planned) on the life of former US president

Lyndon B. Johnson

, remembered both for passing some of the biggest civil rights legislations since the end of the Civil War, and for massively increasing the American involvement in Vietnam.

But it took me some time to figure out why I love his books so much.

I’ve already discussed in

a previous blog post

why I see biography as so interesting: at its best, the genre builds deep, detailed, and insightful theories of people’s life, of whole periods of histories.

Caro definitely fits the bill: his attention to detail (“

Turn Every Page

” was the advice he got as a young reporter, and he took it seriously) and commitment to detailed interviews, historical methods, and living where the action happened,

impresses even trained historians

. In fact, he was a key inspiration behind my original blog post.

Yet he more I read and reread him, the more I feel that what puts him apart is what kind of theory he builds.

For you see, there are two extremes in the spectrum of theoretical models: the phenomenological and the mechanistic.

A phenomenological theory focuses heavily on description and compression: it defines some key concepts, elements, variables, and then links them together. Classical examples include

Thermodynamics

(often called phenomenological thermodynamics),

Newtonian Mechanics

and

Macroeconomics

Whereas a mechanistic theory (also called gears-level model in rationalists circles) starts from some more basic and fundamental elements, and rederive the behavior of the system under consideration from these building blocks. The mechanistic analogous (or attempts) for the previous phenomenological theories are

Statistical Mechanics

(which rederives thermodynamics from the statistical properties of collections of atoms),

General Relativity

(which rederives gravitation from the curvature of space time), and

Microeconomics

(which aims to rederive macroeconomics from the aggregated behavior of simplified economical actors).

Now, by default, the kind of theory that biographers come up with are almost always phenomenological, for a g...