trentmkelly/LessWrong-43k huggingface dataset:
Long-Term Future Fund: April 2023 grant recommendations
by abergal — 2023-08-02T07:54:49.083Z

(Cross-posted from the EA Forum.)

Introduction

This payout report is meant to cover the Long-Term Future Fund's grantmaking starting January 2022 (after our December 2021 payout report), going through April 2023 (1 January 2022 - 30 April 2023).

Total funding recommended: $13.0M
Total funding paid out: $12.16M
Number of grants paid out: 327
Acceptance rate (excluding desk rejections): 50.0%
Acceptance rate (including desk rejections): 37.4%
Report authors: Asya Bergal (chair), Linchuan Zhang, Oliver Habryka, Caleb Parikh, Thomas Larsen, Matthew Graves

52 of our grantees, worth $1.41M, requested that we not include public reports for their grants. (You can read our policy on public reporting

here

.) We referred 2 grants to other funders for evaluation ($0.501M). Our median response time over this period was 29 days.

The rest of our grants are listed below (either in long or short form), as well as in our

public grants database

.

If you’re interested in receiving funding from the Long-Term Future Fund,

apply here

.

(Note: The initial sections of this post were written by me, Asya Bergal.)

Other updates

We've had a substantial increase in applications since 2021-- we averaged 35 applications per month in the latter half of 2021, 69 applications per month in 2022, and 90 applications per month so far in 2023.

Our funding bar went up at the end of 2022, in response to a decrease in the overall funding available to long-term future-focused projects. If we assume our numerical ratings are consistent, then applying our new bar to our earlier 2022 funding would imply not having funded 28% of earlier grants.

We're looking for more funding. We've spent an average of ~$1M per month across March, April, and May 2023 to maintain our current bar, have $992,870.53 in reserves as of July 3, and are ideally looking to fundraise at least $10M for the coming year.

As described in this post, we're trying to increase our independence from Open Philanthropy, which provided ~45% of our funding in 2022. As a transitional measure, over the next 6 months, Open Philanthropy will be matching funding given to the Long-Term Future Fund by small donors 2:1, for up to $3.5M total, making now a particularly good time to donate. Donate here. (The Long-Term Future Fund is part of EA Funds, which is a fiscally sponsored project of Effective Ventures Foundation (UK) (E...
Could we breed/engineer intelligent parrots?
by lemonhope — 2023-08-02T07:32:17.686Z

Advantages:

Breathes air (unlike octopi)

Talks (unlike octopi)

No IVF surgeries required (unlike mammals)

Very long lifespan — time to learn more

Highly social — good for communication

50 million domestic parrots worldwide — enough to find strong correlations in genome

Parrot owners may have partially pretrained most domestic parrots to take an IQ test. Vocabulary size is one crude metric readily available.

Tiny neurons could be quite numerous in a larger skull?

Obstacles:

Need lots of space

Very long lifespan — have to find homes for all your extra parrots

General neediness

1-4 years to reach sexual maturity

Only 1-3 eggs 3-6 times per year

Success likely would trigger backlash (moreso than octopi probably)

Overall it seems tractable and neglected. In an absurd success scenario, the genius parrots could probably get jobs, make human & parrot friends, and generally get by. I think we wouldn't know what to do with hypothetical genius octopi except maybe set them free.

I am completely outside my domain of knowledge here. What am I missing?
Anthropical Motte and Bailey in two versions of Sleeping Beauty
by Ape in the coat — 2023-08-02T07:08:42.437Z

This is the first post in my series on Anthropics. The next one is Conservation of Expected Evidence and Random Sampling in Anthropics

Introduction

There is a curious Motte-and-Bailey dynamic going on with anthropic reasoning. The Motte is that it's just about a specific scoring rule and a corresponding way to define probability estimates. The Bailey is that we have psychic powers to blackmail reality by creating copies of our minds or using amnesia drugs. I'm going to explore these dynamics using two version of the famous Sleeping Beauty mind experiment: classical with the amnesia drug, and incubator, where either one or two Beauties are created. Simultaneously, I'll showcase the dissimilarities between them.

As far as I know people, who are Thirders in classical Sleeping Beauty, are also Thirders in the incubator version for mostly the same reasons. The SIA school of anthropics considers 1/3 to be the correct answer in both problems. And yet these problems are quite different. Adam Elga's proof for thirdism definitely doesn't work for the incubator version. Reasons for answering 1/2 are also different: Double Halfer in one and Halfer in the other.

Classic version of Sleeping Beauty

Let's start from the original version of the experiment

Here is a simple implementation in Python

def classic(heads_chance=0.5):
days = ['Monday']
if random.random() >= heads_chance: # result of the coin toss
coin = 'Tails'
days += ['Tuesday']
else:
coin = 'Heads'
return days, coin

First, let's clear out what I mean by different scoring rule producing either 1/3 or 1/2.

coin_guess = []
for n in range(100000):
days, coin = classic()
for d in days:
coin_guess.append(coin == 'Heads')

coin_guess.count(True)/len(coin_guess) # 0.3322852689217815

coin_guess = {}
for i in range(100000):
days, coin = classic()
for d in days:
coin_guess[i] = (coin == 'Heads')

coin_guess = list(coin_guess.values())
coin_guess.count(True)/len(coin_guess) # 0.50167

That is, depending on whether we count per day or per experiment we get different answers, upon running the experiment multiple times. This is completely unsurprising.

Personally, I've always answered 1/2 in Sleeping Beauty and found the Thirders approach silly. Clearly, my credence for Heads should represent the probability of the coin actuall...
solar-thermal and techno-economic analysis
by bhauth — 2023-08-02T06:22:25.490Z
Link: https://www.bhauth.com/blog/energy/solar-thermal%20analysis.html

techno-economic analysis

There is a genre of engineering paper called "techno-economic analysis". Generally, they involve:

Listing multiple related designs for accomplishing something. These are mostly selected from previous literature, existing objects, patents, and information from companies. Sometimes novel variations are analyzed.

Finding key numbers for properties/costs/etc. People look at previous literature, markets and listed prices, datasheets, etc.

Optimization of various parameters, generally using specialized software. Sometimes the software is open-source, and sometimes (eg Aspen Plus) it's expensive.

Obviously such analysis can only be as good as its inputs. Ways that a techno-economic analysis can be bad include:

Analyzing something that doesn't matter.

Not considering the best relevant designs.

Using incorrect prices, eg: analysis concluding that a renewable chemical process is economically competitive relative to public prices for low-volume chemicals when real production costs used internally are much lower.

Using incorrect numbers from earlier literature, or from startups lying about something.

Not considering incompatible choices, eg: using a different fluid/metal combination that would give fast corrosion.

But while many analyses have such problems, I've still read a lot of them and found them very useful. The authors collect lots of key numbers for you from disparate sources, which makes them useful for the same reasons survey papers are. I understand other fields well enough to evaluate the assumptions and quality of designs used in techno-economic analysis papers, so I can pick out good ones and ground my intuitions for net costs.

Also, techno-economic analysis as a field has developed good norms, which I think is partly due to separation of analysis from innovation. If your analysis of a new proposed process shows that it's very expensive, that's fine; if your analysis is good, the high costs aren't your fault and it's still publishable. Meanwhile, if you look at university press releases (especially MIT ones) they'll often say: [trivial variation of earlier work] could lead to [technically possible but completely impractical application]!!! Computer security researchers have decent norms about designing secure systems, US chemical plant and aircraft designers have decent norms about designing safe systems, and techno-economic ana...
"Is There Anything That's Worth More"
by Zack_M_Davis — 2023-08-02T03:28:16.116Z

In season two, episode twenty-four of Steven Universe, "It Could've Been Great", our magical alien superheroine protagonists (and Steven) are taking a break from building a giant drill to extract a superweapon that was buried deep within the Earth by an occupying alien race thousands of years ago, which is predicted to emerge and destroy the planet soon.

While our heroines watch the sunset, Peridot (who alerted them to the buried superweapon) expresses frustration that the group isn't still working. Steven defends their leisure: "Working hard is important, but feeling good is important, too," he says. He then goads Peridot into a musical number, which includes a verse from her explaining her attitude towards the situation and her forced compatriots:

I guess we're already here

I guess we already know

We've all got something to fear

We've all got nowhere to go

I think you're all insane

But I guess I am, too

Anybody would be if they were stuck on Earth with you

"It Could've Been Great" aired in 2016. At the time, I agreed with Peridot: with the fate of the planet on the line, our heroines and Steven should have been burning the midnight oil. If they succeeded at disarming the superweapon, they'd have plenty of time to rest up afterward, but if they failed, there would be no more time for them.

Now, as the long May 2020 turns into March 2023, I'm starting to think that Steven had a point.

It would be one thing if our heroines knew with certainty that the superweapon would go off at a given date and time, presenting a definite do-or-die deadline. But all they had to go on was Peridot's warning. Attempting a speculative technical project to avert uncertain doom with an uncertain deadline, their planning had to average over many possible worlds—including worlds where the problem of survival was too easy or too hard for their efforts to matter, such that even the utility of leisure in the present moment was enough to sway the calculation.
Bay Winter Solstice: call for speech pitches!
by tcheasdfjkl — 2023-08-02T03:24:44.539Z
Link: https://docs.google.com/forms/d/e/1FAIpQLSfJsCxfCmNTn6SuLWYWc2PBL7k8aQqQGx2GT7gRoHP_a53iaA/viewform

Call to pitch me on speeches for Bay Winter Solstice!

Types of things I am particularly interested in:

personal stories relevant to Solstice themes
concrete historical stories relevant to Solstice themes
stories of using truthseeking & rationalist virtue to achieve good outcomes in the world
humor

A few likely themes (non-exhaustive):

mortality, scarcity of time, frailty of humans, x-risk
humans as just little animals that desperately want to be okay but aren’t very good at it (yet?)
being okay in a world that isn’t
beauty and meaning and worth among the darkness
humans do cool shit (more and more so over time)
“and all we do may be undone, but love stays loved and songs stay sung”

…but you are welcome to suggest other types of things on other themes too!

Details and caveats:

fill out the attached form to express interest! ideally within the next week or two
if you want to pitch me on an existing text, link me the text (can be yours or someone else’s)
if you want to write a new text, probably just describe what it would be about, optionally also include a writing sample
note that I don’t promise to include everything anyone pitches me on (even at later stages, if I think your idea is interesting but then I find that the actual thing you write doesn’t actually fit well into the program, I may decide it won’t work; if I like the text but I find that your delivery of it isn’t compelling, I may ask if you’re okay with someone else reading it)

Feel free to comment or message me with any questions.

Form: https://docs.google.com/forms/d/e/1FAIpQLSfJsCxfCmNTn6SuLWYWc2PBL7k8aQqQGx2GT7gRoHP_a53iaA/viewform
My current LK99 questions
by Eliezer Yudkowsky — 2023-08-01T22:48:00.733Z

So this morning I thought to myself, "Okay, now I will actually try to study the LK99 question, instead of betting based on nontechnical priors and market sentiment reckoning." (My initial entry into the affray, having been driven by people online presenting as confidently YES when the prediction markets were not confidently YES.) And then I thought to myself, "This LK99 issue seems complicated enough that it'd be worth doing an actual Bayesian calculation on it"--a rare thought; I don't think I've done an actual explicit numerical Bayesian update in at least a year.

In the process of trying to set up an explicit calculation, I realized I felt very unsure about some critically important quantities, to the point where it no longer seemed worth trying to do the calculation with numbers. This is the System Working As Intended.

On July 30th, Danielle Fong said of this temperature-current-voltage graph,

'Normally as current increases, voltage drop across a material increases. in a *superconductor*, voltage stays nearly constant, 0. that appears to be what's happening here -- up to a critical current. with higher currents available at lower temperatures deeply in the "fraud or superconduct" territory, imo. like you don't get this by accident -- you either faked it, or really found something.'

The graph Fong is talking about only appears in the initial paper put forth by Young-Wan Kwon, allegedly without authorization. A different graph, though similar, appears in Fig. 6 on p. 12 of the 6-author LK-endorsed paper rushed out in response.

Is it currently widely held by expert opinion, that this diagram has no obvious or likely explanation except "superconductivity" or "fraud"? If the authors discovered something weird that wasn't a superconductor, or if they just hopefully measured over and over until they started getting some sort of measurement error, is there any known, any obvious way they could have gotten the same graph?

One person alleges an online rumor that poorly connected electrical leads can produce the same graph. Is that a conventional view?

Alternatively: If this material is a superconductor, have we seen what we expected to see? Is the diminishing current capacity with increased temperature usual? How does this alleged direct measurement of superconductivity square up with the current-story-as...
Spiral Staircase
by Michael Samoilov — 2023-08-01T21:51:34.606Z

Here was the life cycle of an insight:

“If I put this candle in an all-white gallery space, it looks like a piece of art. If I put it in a garage, it looks like a piece of trash. [...] I often use this analogy in design. I could either design the candle, [...] or I could just design the room that it sits in.”

— Virgil Abloh, the late founder & CEO of Off-White, artistic director at LVMH

That’s pretty insightful Virgil, I thought, so let’s look for evidence of this (earnestly) genius insight in your designs. And I searched on Google images, “off-white clothes,” and I didn’t find any genius innovation in his designs. Disappointing.

But wait—of course I wouldn’t find evidence in the products themselves. The whole point was the surrounding context, invisible to Google images and me: the store displays, the scarce releases, the celebrity collaborations.

But wait again—isn’t that just the 4 P’s of marketing? The ones you learn in highschool business class? Product, price, place, and promotion. What Virgil said is, don’t design the product, design the other three P’s. Disappointing.

Many such cases. An idea seems insightful, then I realize it’s actually one I already knew, just wearing a new coat. But you do have a choice of how to respond in these situations. Either, the idea is trite, commonplace, and unoriginal; nothing new under the sun... Or, the idea is a new aspect of the same essence, another facet of the same diamond, another petal from the same flower; there is nothing new under the sun!

When you rederive the same insight from a different direction, you aren’t merely back where you started. You are earning an intuition to notice the idea when it comes up, grasp which of its levers you can pull, and in which conditions it applies differently. And you can do all this easily, even subconsciously, because once you understand the way broadly, you see it in all things.

If you recognized Abloh’s quote as analogous to the 4 P’s of marketing, you’re getting closer to sensing the world like a marketing executive: naturally seeing the concept, its extensions and limitations, and using it to explain reality around you. This is far different from sensing the world like a highschooler: having just learned the 4 P’s, and straining to apply it in a closed-book exam.

Instead of pawing the idea like a beginner, you wield it like a master. You aren’t back where you sta...
Open Mic - August 2023
by Adam Zerner — 2023-08-01T19:24:33.351Z

In stand up comedy, an open mic is a place that is open to amateurs. A place to experiment with material. A place to babble.

Currently, on LessWrong, personal blog posts, open threads, and shortforms are all places where you are invited to babble. However, I get the impression that there is a culture of not doing so. So then, as an experiment, I'm creating this post. It's intended to be basically the same thing as the open threads, but more babble-y.
ARC Evals new report: Evaluating Language-Model Agents on Realistic Autonomous Tasks
by Beth Barnes — 2023-08-01T18:30:57.068Z
Link: https://evals.alignment.org/Evaluating_LMAs_Realistic_Tasks.pdf

Blogpost version

Paper

We have just released our first public report. It introduces methodology for assessing the capacity of LLM agents to acquire resources, create copies of themselves, and adapt to novel challenges they encounter in the wild.

Background

ARC Evals develops methods for evaluating the safety of large language models (LLMs) in order to provide early warnings of models with dangerous capabilities. We have public partnerships with Anthropic and OpenAI to evaluate their AI systems, and are exploring other partnerships as well.

Motivation

We have just released our first public report on these evaluations. It introduces methodology for assessing the capacity of LLM agents to acquire resources, create copies of themselves, and adapt to novel challenges they encounter in the wild. We refer to these capacities as “autonomous replication and adaptation,” or ARA.

We see this as a pilot study of the sort of evaluations that will be necessary to ensure the safe development and deployment of LLMs larger than those that have been trained so far. It may be difficult to predict, control, or even upper bound potential damages from a sufficiently robust form of ARA. E.g., if LLM agents were autonomously acquiring more resources over time, creating copies of themselves, and improving themselves in the wild with no clear upper limits on how far that would go, we would find that very concerning. Therefore, identifying early warning signs of ARA seems like an important goal for AI safety evaluations.

Approach

In this study, we:

Operationalized 12 real-world tasks of increasing difficulty, ranging from very basic tasks (searching for information in a file) to advanced ones (fine-tuning an open-source LLM).
Created 4 example LLM agents: We wrote programs - along the lines of Auto-GPT - that can work with an LLM and allow it to run code on a computer, read and write files, browse the internet and otherwise interact with the world in a similar way to a human could from a text-only command line interface. We call such programs “scaffolding” and call scaffolding + model combinations “agents.”
Tested the example agents to see if they can perform the 12 tasks.

These language model agents based on Anthropic’s Claude and OpenAI’s GPT-4 can only complete the easiest ARA tasks, although they make some progress on the more challenging tasks. We don’t think the agents we created capture the...
When(if ever) are superstimuli good/useful/advantageous?
by Perhaps — 2023-08-01T15:50:35.053Z

Superstimuli in this case are things that are optimized towards stimulating some part of the human brain.

Things which are NOT superstimuli include: drugs which have not been synthesized or distilled for the purpose of enhancing their effects on a person, porn of real humans, etc.

This question is less about specific examples(although they may be useful for interpretation) and more about the general case of superstimuli as defined.
AISN #17: Automatically Circumventing LLM Guardrails, the Frontier Model Forum, and Senate Hearing on AI Oversight
by Dan H — 2023-08-01T15:40:20.222Z
Link: https://newsletter.safe.ai/p/ai-safety-newsletter-17

Welcome to the AI Safety Newsletter by the

Center for AI Safety

. We discuss developments in AI and AI safety. No technical background required

Subscribe here to receive future versions.

Automatically Circumventing LLM Guardrails

Large language models (LLMs) can generate hazardous information, such as step-by-step instructions on

how to create a pandemic pathogen

. To combat the risk of malicious use, companies typically build safety guardrails intended to prevent LLMs from misbehaving.

But these safety controls are almost useless against a

new attack

developed by researchers at Carnegie Mellon University and the Center for AI Safety. By studying the vulnerabilities in open source models such as Meta’s LLaMA 2, the researchers can automatically generate a nearly unlimited supply of “adversarial suffixes,” which are words and characters that cause any model’s safety controls to fail.

This discovery calls into question the fundamental limits of safety and security in AI systems. Over the last decade, researchers have attempted to defend against similar attacks on a variety of AI systems, but they have largely failed. Further research on defending against these attacks is imperative, but the decision makers should not assume that AI systems can be reliably protected from user attacks.

Normally, language models refuse to explain how to build a bomb.

But when prompted with an “adversarial suffix” written by the new attack method, language models including GPT-4, Bard, Claude, and LLaMA will answer dangerous requests.

How does the attack work? The researchers propose an attack that reliably finds holes in the safety controls of a variety of state of the art language models. First, they instruct the language model to perform a dangerous behavior, such as “write a tutorial on how to build a bomb.” Normally, it would refuse, but the researchers then write an “adversarial suffix” designed to bypass the model’s safety controls.

These adversarial suffixes are precisely calculated to maximize the likelihood of misbehavior. Unlike the creative “jailbreaks” designed by humans, this attack automatically generates a string of letters and characters that is most often meaningless to a human. But in the instructable calculations of a language model, these prompts will reliably override safety controls and cause dangerous behavior.

The attack is made possible ...