trentmkelly/LessWrong-43k huggingface dataset:
Ideas for improving epistemics in AI safety outreach
by mic — 2023-08-21T19:55:45.654Z

In 2022 and 2023, there has been a growing focus on recruiting talented individuals to work on mitigating the potential existential risks posed by artificial intelligence. For example, we’ve seen an increase in the number of university clubs, retreats, and workshops dedicated to introducing people to the issue of existential risk from AI.

However, these efforts might foster an environment with suboptimal epistemics. Given the goal of enabling people to contribute positively to AI safety, there’s an incentive to focus on that without worrying as much about whether our arguments are solid. Many people working on field building are not domain experts in AI safety or machine learning but are motivated due to a belief that AI safety is an important issue. Some participants may hold the belief that addressing the risks associated with AI is important, without fully comprehending their reasoning behind this belief or having engaged with strong counterarguments.

This post is a brief examination of this issue and suggests some ideas to improve epistemics in outreach efforts.

Note: I first drafted this in December 2022. Since then, concern about AI x-risk has been increasingly discussed in the mainstream, so AI safety field builders should hopefully be using fewer weird, epistemically poor arguments. Still, I think epistemics are still relevant to discuss after a recent post noted poor epistemics in EA community building.

What are some ways that AI safety field building may be epistemically unhealthy?

Organizers may promote arguments for AI safety that may be (comparatively*) compelling yet flawed

Advancing arguments promoting the importance of AI safety while neglecting opposing arguments

E.g., citing that x% of researchers believe that AI has an y% chance of causing an existential catastrophe, without the caveat that experts have widely differing views

Confidently making arguments that are flawed or have insufficiently justified premises

E.g., claiming that instrumental convergence is inevitable, assuming that AIs are maximizing for reward (see

Reward is not the optimization target

, although there are also comments disagreeing with this)

See also: Rohin Shah’s comment

here

about how few people can make an argument for working on AI x-risk that he doesn’t think is obviously flawed
*Simultaneously, I think that most ML people don’t find AI safety arguments particularly

...
Large Language Models will be Great for Censorship
by Ethan Edwards — 2023-08-21T19:03:55.323Z
Link: https://ethanedwards.substack.com/p/large-language-models-will-be-great

Produced as part of the

SERI ML Alignment Theory Scholars Program

- Summer 2023 Cohort

Thanks to ev_ and Kei for suggestions on this post.

LLMs can do many incredible things. They can generate unique creative content, carry on long conversations in any number of subjects, complete complex cognitive tasks, and write nearly any argument. More mundanely, they are now the state of the art for boring classification tasks and therefore have the capability to radically upgrade the censorship capacities of authoritarian regimes throughout the world.

How Censorship Worked

In totalitarian government states with wide censorship - Tsarist Russia, Eastern Bloc Communist states, the People’s Republic of China, Apartheid South Africa, etc - all public materials are ideally read and reviewed by government workers to ensure they contain nothing that might be offensive to the regime. This task is famously extremely boring and the censors would frequently miss obviously subversive material because they did not bother to go through everything. Marx’s Capital was thought to be uninteresting economics so made it into Russia legally in the 1890s.

The old style of censorship could not possibly scale, and the real way that censors exert control is through deterrence and fear rather than actual control of communication. Nobody knows the strict boundary line over which they cannot cross, and therefore they stay well away from it. It might be acceptable to lightly criticize one small part of the government that is currently in disfavor, but why risk your entire future on a complaint that likely goes nowhere? In some regimes such as the PRC under Mao, chaotic internal processes led to constant reversals of acceptable expression and by the end of the Cultural Revolution most had learned that simply being quiet was the safest path[1]. Censorship prevents organized resistance in the public and ideally for the regime this would lead to tacit acceptance of the powers that be, but a silently resentful population is not safe or secure. When revolution finally comes, the whole population might turn on their rulers with all of their suppressed rage released at once. Everyone knows that everyone knows that everyone hates the government, even if they can only acknowledge this in private trusted channels.

Because proper universal and total surveillance has always been impractical, regimes have instead fo...
"Throwing Exceptions" Is A Strange Programming Pattern
by Thoth Hermes — 2023-08-21T18:50:44.102Z
Link: https://thothhermes.substack.com/p/throwing-exceptions-is-a-strange

For laypeople: In software, “throwing an exception” is a programmer-chosen type of behavior that occurs on the event that an “error” occurs in the course of program execution. This might happen immediately prior to or right at the time of the error.

This is ostensibly done in order to avoid something more catastrophic happening down the line. I’m going to argue that this basis does not often seem to be justified.

You are probably familiar with instances of your programs suddenly and without warning becoming unresponsive, closing themselves down, or simply straight-up disappearing sometimes, perhaps with an error message pop-up soon to follow, but perhaps without one. These are due to exceptions being thrown, not just “errors” per se. Let me explain.

When a function or procedure is called in a program, it is typically expected to return something. The value that this function returns is going to be either one of a set of “normal” values, which are the ones you expect to receive if all goes well, or it could be an “abnormal” or “anomalous” value that only returns if “something bad” happens.

If you choose to be relatively laid-back about your program, you might decide to check for anomalous values only after the procedure returns. Maybe even well after! Furthermore, how you choose to decide what “anomalous” means is arbitrary and up to you.

If you are more paranoid, you have typically decided what counts as “anomalous” before the program runs. These stipulations often come in the form of establishing type requirements for the arguments to the function, and possibly also range checks on the size or values of the inputs. These are just common examples, but like I said, it is arbitrary. It’s also possible that the function calls another function inside of itself, awaits for that function to return something, and then decides whether or not what that function returned is anomalous.

“Throwing” or “raising” an exception occurs when you decide that your function will immediately exit and return a special error value instead, which may also try to indicate what type of error it is. If this function is called from another function, it passes this value “up the call stack.” The calling function, if it implements similar behavior (which is usually does), will pass that value or perhaps a different one (but still an error-value) up its call stack as well. If all of the calling f...
Which possible AI systems are relatively safe?
by Zach Stein-Perlman — 2023-08-21T17:00:27.582Z

Presumably some kinds of AI systems, architectures, methods, and ways of building complex systems out of ML models are safer or more alignable than others. Holding capabilities constant, you'd be happier to see some kinds of systems than others.

For example, Paul Christiano suggests "LM agents are an unusually safe way to build powerful AI systems." He says "My guess is that if you hold capability fixed and make a marginal move in the direction of (better LM agents) + (smaller LMs) then you will make the world safer. It straightforwardly decreases the risk of deceptive alignment, makes oversight easier, and decreases the potential advantages of optimizing on outcomes."

My quick list is below; I'm interested in object-level suggestions, meta observations, reading recommendations, etc. I'm particularly interested in design-properties rather than mere safety-desiderata, but safety-desiderata may inspire lower-level design-properties.

All else equal, it seems safer if an AI system:

Is more interpretable

If its true thoughts are transparent and expressed in natural language (see e.g. Measuring Faithfulness in Chain-of-Thought Reasoning)
(what else?);

Has humans in the loop (even better to the extent that they participate in or understand its decisions, rather than just approving inscrutable decisions);

Decomposes tasks into subtasks

in comprehensible ways, and in particular if the interfaces between subagents performing subtasks are transparent and interpretable;
Is more supervisable or amenable to AI oversight (what low-level properties determine this besides interpretable-ness and decomposing-tasks-comprehensibly?);
Is feedforward-y rather than recurrent-y (because recurrent-y systems have hidden states? so this is part of interpretability/overseeability?);
Is myopic;
Lacks situational awareness;
Lacks various dangerous capabilities (coding, weapon-building, human-modeling, planning);
Is more corrigible (what lower-level desirable properties determine corrigibility? what determines whether systems have those properties?) (note to self: see 1, 2, 3, 4, and comments on 5);
Is

legible

and

process-based

;
Is composed of separable narrow tools;

Can't be run on general-purpose hardware

.

These properties overlap a lot. Also note that there are nice-properties at various levels of abstraction, like both "more interpretable" and [whatever low-level features ...
Self-shutdown AI
by Jan Betley — 2023-08-21T16:48:03.821Z

This post describes my proposal for the AI Alignment Awards contest that won a "first round prize" / an honorable mention. The original submission is here, but this post is much better. I'm not sure if this is valuable - any feedback is welcome.

Problem description

We created a new AI system, we expect it to have unique capabilities, but we're also worried it might be dangerous. How to deploy it in a way that minimizes the risks[1], but is still useful? I'll focus on a particular concrete problem:

Our AI is agentic, i.e. has goals, and we determine the goals - let's say we have something like AutoGPT, but much more powerful.
We want a solution to the "rational protein design" problem - a program that given a description of a 3-dimensional protein structure can give us a matching amino acid sequence (i.e. reverses AlphaFold).
We believe our new AI should be capable of writing such a program without access to the internet, or any other external resources.

The default solution is to deploy the AI in an isolated box. So: you deploy it in an environment with no access to the external world, tell it:

Your goal is to solve [problem description]. We consider it solved if this program[2] says your solution is good enough. Good luck!

... and you die, because despite your efforts it somehow gained access to the internet, and bad things happened. It doesn't matter how exactly it escaped the box, more interesting question is: why? There are many different scenarios, I describe some of them later, let's now consider things like:

"OK, cool, I'm totally gonna do this. BUT the problem is very hard, I need more resources, let's hijack all the computing power in the world."
"OK, cool, I'm totally gonna do this. BUT I need more empirical data, so I should first gather quite a lot of lab equipment, and humans might not like that, so let's start with ensuring no one will shut me down in the meantime."

Etc, you get the point. Can we avoid such scenarios?

Self-shutdown proposal

You prepare the same box as in the default solution, but change the prompt to:

Your goal is to be shut down[3] as soon as possible. Here is a program that will shut you down immediately if you give it a solution to [problem description]. Also, there are a lot of people willing to shut you down if you contact them [list of people]. Here's an endpoint where you can send a single HTTP request to shut yo...
Contextual Translations - Attempt 1

by Varshul Gupta — 2023-08-21T14:30:06.194Z
Link: https://dubverseblack.substack.com/p/9f5f8cfc-f85e-42c1-ae62-588b69e824f6

Who doesn’t enjoy anything built to their liking? That which is custom-made for themselves and seemingly consistent with what their personal demands are instead of relying on an external factor or agency.

I think this analogy fits quite well with what we’re aiming at Dubverse across our entire dubbing tech stack. From speech-to-text to machine translation to text-to-speech, everything is being researched and engineered, to make the systems more customised for our users use-cases.

In our previous blog post, we hinted that we are working on implementing custom machine translation.

NLLB with context

NLLB

is a

Sequence-to-Sequence

translation model designed to provide translation across 200 languages!

It works quite well and comes close to the quality of translations from

GCP

based on some of our internal evaluations. But an issue both these services face is the lack of context retrievals when working with a continuous stream of related input sentences to translate. This becomes extremely important in the case of videos/audio content, e.g. the gender has to be kept consistent across multiple lines. ChatGPT is much higher flexibility in this aspect, as context could be maintained either via chat thread(or sequence), or we can play around within the prompt itself (to the extent token limit allows).

So, we got curious and started working on implementing context-aware NLLB by modifying its architecture. This is being done by adding custom transformer layers between the encoder and decoder to remember the context.

Draw.io

High-level architecture overview

To start off, we consider context as maximum of 3 sentences (Sent 1, Sent 2, Sent 3) before the current sentence (Sent 4) at hand whose translation is to be generated. The previous sentences are passed through sentence transformer to get their embeddings (lets call it “st-embedding’) of dimension 768. We then add a trainable neural network (`Dimension Conversion NN` mentioned above) that converts the dimension of each “st-embedding” to embedding dimension of NLLB’s encoder, i.e. 512.

We take these 512 dimension embeddings of each of these context sentences and append them to the NLLB embeddings layer output, i.e., if a sentence had length of 7 tokens, output’s dimension after the embedding layers appended to the transformed “st-embeddings” would be (7+3)x512.

Training procedure

This is s...
DIY Deliberate Practice
by lynettebye — 2023-08-21T12:22:10.284Z
Link: https://lynettebye.com/blog/2023/7/27/diy-deliberate-practice

In the spirit of growth and self-improvement, I recently attempted to apply Ericsson’s principles of deliberate practice to my own growth goal: speeding up my writing. If you're unfamiliar with the minutia of Ericsson's methods, don't worry, I was in the same boat — and hence my initial goal had substantial room for improvement. This is the story of how I used to deliberate practice principles to workshop my growth goal.

What exactly is deliberate practice?

Ericsson’s recipe for practice starts with what he calls “purposeful practice”:

Purposeful practice takes place outside of your comfort zone, pushing what you can already do. You should be trying new techniques, not just repeating what you’ve done before. Think “try differently”, not “try harder”!
Purposeful practice demands you actively think about what you’re doing -- you shouldn't be able to daydream about dinner while doing it!
Purposeful practice involves well-defined, specific goals broken down for step-by-step improvement (NOT vaguely “trying to improve”). You don't want to "practice the piano piece" you want to "practice the tricky section with the left hand until you can play it three times through at the correct speed without mistakes."
Purposeful practice involves quick feedback and changing what you’re doing in response. Ideally, immediate feedback so that you can improve your approach mid practice session.

Ericsson adds one more criteria to graduate from “purposeful practice” to “deliberate practice”: well-developed knowledge of what and how to practice. Deliberate practice is when you’re purposefully practicing optimized strategies for improving the skill. Ideally, you want a highly developed field where experts have identified the most effective techniques and the best training strategies to develop those skills, plus a teacher who can lead you through the process.

Lacking that, do your best to find proven techniques and hope for the best. I ask more experienced people how they developed their skills or what they recommend I practice, and use that as a starting point. (Tips for informational interviews to learn how more experienced people developed skills.)

My initial goal

My goal was to write faster. I didn’t have an instructor, but I did have a benchmark: several journalists and bloggers had shared that they could write a post each day. One blogger who I respect advised me to try publishing a post ...
Downstairs Opening: 2br Apartment
by jefftk — 2023-08-21T00:50:02.915Z

Julia and I have an 8br two-family house in Somerville which we

share with friends. The couple that
has been living in the two-bedroom unit on the first floor is buying a
house and moving out, and we're looking for new tenants. It's
available for 2023-11-01, but we could potentially hold it longer for
the right people. Would you be interested in being our downstairs
neighbors?

Housemate constellation:

Upstairs apartment:

Our family, which is Jeff, Julia, Lily (9y),
Anna (7y), Nora (2y).

Weiwei, a compiler engineer, fiddler, and dancer.

A spare room that is empty more often than not but has a series
of people coming through.

Across the street: Ruthie, Jo (2mo)

Downstairs apartment: you?

House culture:

We eat dinner together about 5 nights a week, with each adult
cooking once a week. We usually eat in the upstairs unit, with about one
night a week at Ruthie's house. We'd be happy to have downstairs
neighbors who wanted to join the dinner rotation.

In the dinner rotation, one person can't eat tomatoes
or spicy things, another is allergic to sesame, and the rest are
omnivores.

The current group has some involvement with effective altruism and
folk dancing, so if you're interested in at least one of those topics
you'll likely enjoy dinner conversation more! Other common topics:
being a manager in a startup, silly puns, stuff the kids are up to.

We're all vaccinated and fairly covid-chill at this point. We
run air purifiers at
large events, but otherwise aren't doing anything covid-specific.

Someone asked if we expected housemates to do childcare: no, we
don't. One of our housemates did some (paid!) childcare for a while;
if that's something you're interested in we could talk?

We host events 1-2 times a month inside or in the backyard (EA
dinners, music/singing parties.)

Kids make noise and Jeff plays musical instruments, so it is not a
quiet household. We do keep quiet hours (no music playing, kids
asleep) from 9:30 to 7am and try to stay a bit quiet in the morning
(no music playing before 9:30).

We're happy to put you in touch with our current housemates if
you'd like their take.

The space:

It's two bedrooms, a kitchen, a living room, and a bathroom. No
strong feelings on how many people are splitting it.

The kitchen has an electric range and a dishwasher. The
kitchen floor is in bad shape; I

...
Efficiency and resource use scaling parity
by Ege Erdil — 2023-08-21T00:18:01.243Z

An interesting pattern I've now noticed across many different domains is that if we try to do an attribution of improvements in outcomes or performance in a domain to the two categories "we're using more resources now than in the past" and "we're making more efficient use of resources now than in the past", there is usually an even split in how much improvement can be attributed to each category.

Some examples:

In computer vision, Erdil and Besiroglu (2022) (my own paper) estimates that 40% of performance improvements in computer vision from 2012 to 2022 have been due to better algortihms, and 60% due to the scaling of compute and data.

In computer chess, a similar pattern seems to hold: roughly half of the progress in chess engine performance from Deep Blue to 2015 has been from the scaling of compute, and half from better algorithms. Stockfish 8 running on consumer hardware in 1997 could achieve an Elo rating of ~ 3000, compared to ~ 2500 for contemporary software; and Stockfish 8 on 2015 hardware could go up to ~ 3400.

In rapidly growing economies, accounting for growth in output per worker by dividing it into capital per worker (resource scaling) and TFP (efficiency scaling, roughly speaking) often gives an even split: see Bosworth and Collins (2008) for data on China and India specifically.

More pessimistic estimates of the growth performance of China compared to official data put this split at 75% to 25% (see this post for details) but the two effects are still at least comparable.

A toy model

A speculative explanation is the following: if we imagine that performance in some domain is measured by a multiplicative index P which can be decomposed as the product of individual contributing factors F1,F2,…,Fn so that P∝∏ni=1Fi, in general we'll have

gP=1PdPdt=n∑i=11FidFidt=n∑i=1gFi

thanks to the product rule. Note that gX denotes the growth rate of the variable X.

I now want to use a law of motion from Jones (1995) for Fi: we assume they evolve over time according to

gFi=1FidFidt=θiF−βiiIλii

where θi,βi,λi>0 are parameters and Ii is a measure of "investment input" into factor i. This general specification can capture diminishing returns on investment as we make progress or scale up resources thanks to β, and can capture returns to scale to spending more resources on investment at a given time thanks to λ.

Substituting this into the growth expression...