trentmkelly/LessWrong-43k huggingface dataset:
So, I Want to Be a "Thinkfluencer"
by DragonGod — 2022-08-20T18:39:05.825Z

Epistemic Status

Rough and unpolished. I might refine it later.

Introduction

I want to optimise something like: "expected positive impact on a brighter world". Probably, the best way I can do this is through direct work and not via earning to give. I think I have substantial intellectual endowment (more on that later), and like I'm quite privileged (more on that later as well). So, I should choose a career plan that maximises (expected utility) of my positive contribution to a brighter world conditional on the space of possible people I can be.

Near term (within the next 30 years), I want to pursue a career trajectory as an AI safety researcher and a "radical transhumanist thinkfluencer". For the AI safety research part, I'm currently learning abstract maths (currently category theory) [maybe I'll write another post motivating that, but I basically want to try my hand at Agent Foundations style research and I think I have the intellectual endowment for this to have considerable positive value in expectation]. I'll be starting a CS Masters at the end of September (if my student VISA is granted), I'll probably take a gap year (for some intensive learning project probably) then pursue a PhD in CS/AI/computational neuroscience (not mathematics. I'm under the impression I can autodidact any mathematics I find myself compelled to learn).

I'm 24 now, so I'm hoping to start my career trajectory at 32 (8 years forms a natural/compelling Schelling point [At 24 I'm a quite different person from the person I was at 16. And he is in turn a remarkably different person from the person I was at 8. I can thus expect to be a pretty different person at 32. "Who is the person I want to be at 32? What do I need to do to become that person?").

I quit my job as a web developer at the end of July. I don't plan to return to software development (I found it frustrating, and I think it's neither my absolute nor comparative advantage).

Interlude

To disambiguate what I mean by "radical transhumanist thinkfluencer" a bit, I want to help sell the following ideas:

The current state of the world is very suboptimal
Vastly better world states are possible
We can take actions that would make us significantly more likely to reach those vastly better states
We should do this

I'd like to paint concrete and coherent visions for a much brighter future (not concrete utopias, but general ways that we can ma

...
The 'Bitter Lesson' is Wrong
by deepthoughtlife — 2022-08-20T16:15:04.340Z

There are serious problems with the idea of the 'Bitter Lesson' in AI. In most cases, things other than scale prove to be extremely useful for a time, and then are promptly abandoned as soon as scaling reaches their level, when they could just as easily combine the two, and still get better performance. Hybrid algorithms for all sorts of things are good in the real world.

For instance, in computer science, quicksort is easily the most common sorting algorithm, who uses a pure quicksort? Instead they add on an algorithm that changes the base case, or handles lists with small numbers of entries, and so on. People could have learned the lesson that quicksort is just better than these small-list algorithms once you reach any significant size, but that would have prevented improving quicksort.

Another, unrelated personal example. When I started listening to Korean music, it bothered me that I couldn't understand what they were singing, so on some of my favorite songs, I searched fairly significantly for translations.

When I couldn't find translations for some songs, I decided to translate them myself. I didn't know more than a phrase or two of Korean at the time, so I gave them to an AI (in this case, Google Translate, which had already transitioned to deep learning methods at the time.)
Google Translate's translations were of unacceptably low quality of course, so I used it as a word or short phrase reference. Crucially, I didn't know Korean grammar at the time either (which is in a different order than English.) It took me a long time to translate those first songs, but not anywhere near enough to be much training on the matter.

So how did my translations of a language I didn't know compare to the DL used by Google? It was as different as night and day in favor of my translation. It's not because I'm special, but because I used those methods people discarded in favor of 'just scale'. What's a noun? What's a verb? How do concepts fit together? Crucially, what kind of thing would a person be trying to say in a song, and how would they say it? Plus, checking that backwards, if they were trying to say this thing, would they have said it in that way? I do admit to a bit of prompt-engineering even though I didn't know what that was at the time, but that means I knew that a search through similar inputs would give a better result instinctively.

I hav...
The Loire Is Not Dry
by jefftk — 2022-08-20T13:40:01.237Z

Several of my friends have shared this paste (or shares of it) on
Facebook:

This is the current state of the Loire, the longest river in
France. This has not happened before in at least the 2000 years since
literate people inhabited France. The Romans would have written about
this. The medieval Franks would have written about this. To the best
of our historical knowledge, nowhere in the past 2000 years has the
Loire run dry, and likely never long before that. The drought that now
grips southwestern Europe may well be unprecedented in recorded human
history.

This is in the heart of wine country where grapes grow in abundance
and wheat waves like golden seas- but not now. Now the wheat burns and
the grapes whither to raisins on the vine. This is the end of
days. And on Monday morning I'll return to work and pretend this isn't
happening. It's complete madness.

Along with this picture:

But it's all bunk. While the Loire going dry would be a big deal,
it's not dry, and this picture isn't unusual.

The picture is of parallel path of the Loire where its crossed by the
Pont de
Varades bridge. The main flow of the river is just East of this
picture, off to the left:

Google
Maps

Translating some tweets from Thibault Laconde, @EnergieDevlpmt
on Twitter:

"It spans a shallow dead arm which is often dry. Here are pictures
from May 2011 and August 2009: ..."

"The Montjean station was built in 1842, so we have lots of historical
data. It had lower flows (below 95m3/s) during the drought of 1976
(with a minimum of 73m3/s on the 22 August) but also in 1870, 1905,
1906, 1921, 1947, 1949, 1950, etc."

"The current flow is low but not unprecedented. It corresponds almost
exactly to the decadal low VCN3. That is to say, at a constant
climate, we expect to reach it over 3 days on average every 10 years."

There is definitely a drought, and the heatwave
has broken many records, but these water levels are not anywhere close
to a "haven't seen in 2000 years" event.
Refine's Second Blog Post Day
by adamShimi — 2022-08-20T13:01:47.190Z

Yesterday was the second blog post day at Refine. It came after the first week of research and iteration on incubatees ideas and questions.

It still went well. There are less posts than the first day, but even the incubatees that didn't write a post ended up working on interesting research in the process of trying to write a blog post, or in reading a lot of interesting stuff that excited them.

Here is the list of blog posts by Refine participants:

PreDCA: vanessa kosoy's alignment protocol
What if we approach AI safety like a technical engineering safety problem
Benchmarking Proposals on Risk Scenarios

Here's the list of blog posts by friends of Refine and members of my Epistemology Team at Conjecture:

No One-Size-Fit-All Epistemic Strategy
Epistemic Artefacts of (conceptual) AI alignment research
No One-Size-Fit-All Epistemic Strategy
by adamShimi — 2022-08-20T12:56:23.261Z

This post is part of the work done at Conjecture.

This post has been written for the second Refine blog post day, at the end of the first week of iterating on ideas and concretely aiming at the alignment problem.

What constantly frustrates me about the old school philosophers of science, the Poppers, the Kuhns, even the Feyerabend, is their obsession with finding a few simple "rules of science" to apply anywhere and everywhere. This obsession for the overly general (even in the case of the anarchist Feyerabend) leads to leaving much value and insight on the table because all these compressions misses relevant insights in the history of science.

Popper erects falsification and self-contradiction as the death of a theory, missing all the cases where throwing the theory away was not the correct move. (Examples: Kékulé on the structure of Benzene, Galileo and Copernic on the moving Earth)
Kuhn pretty much defines a mature science as having only a single paradigm, where many of the advances in science required and thrived off pluralism (Examples: the methodological omnivory of historical scientists, the distance problem and chemical formula determinations in Chemistry)
Feyerabend concludes from his insightful criticism of Popper's view that there is no rule in science and that "Anything Goes", thus refusing to see regularities that exists within the history of science (See levels of pluralism for an example of drawing on these regularities)

Once again, the issue is not that these thinkers contributed nothing: Popper pushed falsification to interesting places (like evolutionary epistemology); Kuhn articulated beautifully the role of normal science; Feyerabend made accurate and crucial criticisms of the existing methodologies.

The issue is instead that none of them just thought that the strategies that should be used in science, the criteria for what makes sense and is accepted, might depend on the context. Not in a relativist sense of having different notions of truth, but instead saying that epistemic strategy X, which works in field F, might not work in field G.

Nowhere is this lack of context sensitivity clearer than in the old-school belief that Physics is a representative science.

To see why I think this doesn't work, let's look at one concrete example: the dull function hypothesis. In the words of A. Zee, this is the strategy that assumes that when you have to find a...
What if we approach AI safety like a technical engineering safety problem
by zeshen — 2022-08-20T10:29:38.691Z

This post has been written for the second

Refine

blog post day, at the end of the first week of iterating on ideas and concretely aiming at the alignment problem. Thanks to Adam Shimi, Paul Bricman, and Daniel Clothiaux for helpful discussion and comments.

Introduction

This post aims to provide a general introduction to technical safety used in engineering related industries, describe some commonly used tools, and attempt to draw parallels between technical safety and AI safety. I am uncertain about the value of thinking in this framework towards building safe AI, but I expect this post to at least be useful as a

productive mistake

.

To prevent confusion:

I use the term ‘technical safety’ to refer loosely to dealing with safety issues in engineering domains (I am only slightly familiar with technical safety in the upstream oil and gas sector).
I use the term ‘AI safety’ to mean ensuring AI deployment does not lead to catastrophic consequences, similar to Steve Byrnes’

definition

.

For the purpose of this post, I will assume a slow takeoff scenario where AI capabilities scales relatively slowly while never being fully aligned. At some point, a relatively powerful AI is created - powerful enough to do a lot of things but not powerful enough to instantly wipe us all out. Perhaps it has goals that do not generalize well, and starts optimizing for goals that are misaligned to our true goals. This leads to bad outcomes, perhaps events that unfortunately cause some fatalities. There continues to be a series of tragic accidents that increase in severity as the AIs get more and more capable over time, but we still have time to learn from these incidents and make safety improvements towards future systems.

The concepts in this post may not be directly applicable to an AI FOOM scenario, where the first AGI recursively improves itself and suddenly becomes extremely powerful and wipes out humanity, leaving us no opportunity to course-correct.

Technical safety in a nutshell

The principle of technical safety is built upon

Heinrich’s Law

, which states “in a workplace, for every accident that causes a major injury, there are 29 accidents that cause minor injuries and 300 accidents that cause no injuries.” The numbers are unimportant, but the idea is that minimizing the near-misses (accidents that cause no injuries) leads to a reduction in minor accidents...
Vague concepts, family resemblance and cluster properties 
by Q Home — 2022-08-20T10:21:31.475Z

My definition of a "vague concept" (A):

It creates a group of things that share family resemblance or "family difference".
It doesn't have any meaning outside of context. But in a specific context it obtains specific meaning.
It's very hard to understand through the framework of causation: what causes things to be A/non-A?
When you consider more and more examples of A, the boundary between A and non-A becomes increasingly arbitrary and complicated. But in a specific context the boundary is simple.

Examples of vague concepts: games, health, human values, personality, words and some hypotheses.

I think it may be very important to study ways to think about vague concepts: it's related to hypotheses generation and human values. So it may be related to AGI and AI alignment.

I'll discuss some examples and share my thoughts about inner working of vague concepts.

Examples

Games

An example about "games" is famous in philosophy. But keep in mind that my interpretation may differ:

https://en.wikipedia.org/wiki/Family_resemblance

It argues that things which could be thought to be connected by one essential common feature may in fact be connected by a series of overlapping similarities, where no one feature is common to all of the things. Games, which Wittgenstein used as an example to explain the notion, have become the paradigmatic example of a group that is related by family resemblances.

When you consider a bunch of "games", it's easy to see the common features. But as you consider more and more "games" and things that are sometimes called "games", it turns out that everything can be a game. The boundary between "games" and "non-games" becomes more and more arbitrary and complicated.

And yet no matter how much you stretch the concept (e.g. say something like "love is just a game"), in a specific context the meaning is clear enough. In a specific context the boundary between "games" and "non games" is quite simple. Does this boundary have a fractal dimension? (I'm half-joking)

I also think it's important to consider "family difference": maybe you have just a single object A and a group of objects (B). When you compare A to an object from (B), they're always different enough. But you can't formulate a universal difference between A and all objects from (B). Maybe because you don't know all the objects in (B).

Sorites paradox

Sorites paradox is at least partially relevant here.

A typical

...
Benchmarking Proposals on Risk Scenarios
by Paul Bricman — 2022-08-20T10:01:53.248Z

This post has been written for the second Refine blog post day, at the end of the first week of iterating on ideas and concretely aiming at the alignment problem.

Thanks to Adam Shimi, Dan Clothiaux, Linda Linsefors, Nora Ammann, TJ, Ze Shen for discussions which inspired this post.

I currently believe prosaic risk scenarios are the most plausible depictions of us messing up (e.g.

Ajeya Cotra's HFDT takeover scenario

). That's why I find it exciting to focus on making progress on those ones in particular, rather than scenarios involving ideal agents in esoteric circumstances, exotic brain-like architectures, etc. That said, my risk model is very much in its infancy, largely based on

bitter lessons

from cognitive modeling, expert systems, and other non-ML approaches to learning and reasoning I briefly explored in undergrad.

In the context of this post, making progress on prosaic risk scenarios means coming up with proposals which decrease the estimated probability of extinction compared to baselines of minimal safety efforts. As a person with background in ML, I find it quite natural to operationalize this goal through benchmarks which you can then test proposals against. With this approach in mind, I'm then excited about pushing the state-of-the-art on prosaic risk scenarios.

However, one might argue that overfitting solutions to perform well on specific scenarios might lead to brittleness when encountering novel failure modes which we never thought of. The proposal wouldn't be robust, it would generalize poorly, something something getting closer to a moon landing by climbing a tree. One way of tackling this issue would be to compile a comprehensive evaluation harness which contains scenarios which individually test for broad classes of potential issues (e.g. deception in general as opposed to a specific deceptive behavior). Another way of approaching this would be to frame scenarios as hits on the attack surface exposed by a proposal. As scenarios move from concrete details to general failure modes, the associated hit moves from resembling a point to being area-of-effect style. Modeling the attack surface effectively would then help gauge the performance of the proposal.

Another concern of using risk scenarios as benchmarks for proposals would be the lack of objective metrics. You don't get a clean accuracy with high reproducibility, you...
Reducing Goodhart: Announcement, Executive Summary
by Charlie Steiner — 2022-08-20T09:49:23.881Z

I - Release announcement

I have just posted a very thorough edit-slash-rewrite of my Reducing Goodhart sequence. Many thanks to people who gave me feedback or chatted with me about related topics, and many apologies to the people who I told to read this sequence "but only after I finish editing it real quick."

If you're interested in the "why" and "what" questions of highly intelligent AIs learning human values, I can now recommend this sequence to you without embarrassment. And if you skimmed the old sequence but don't really remember what it was about, this is a great time to read the new and improved version.

II - Super short pitch

I worked on this because I was tired of talking nonsense about what it would mean for a value learning AI to "beat Goodhart's law." If you would also like to not talk nonsense about beating Goodhart's law and learning human values, this sequence will help.

III - Executive summary, superfluous for AF regulars

What does it mean for an AI to learn human preferences and then satisfy them? The intuitive way to approach this question is to treat "human preferences" as fixed facts that the AI is supposed to learn, but it turns out this is an unproductive way to think about the problem. Instead, it's better to treat humans as physical systems. "Human preferences" are parts of the models we build to understand ourselves.

Depending on how an AI models the world, it might infer different human preferences from the same data - you can say a reluctant addict either wants heroin or doesn't without actually disputing any raw data, just changing perspective. This makes it important that value learning AI models humans the way we want to be modeled. How we want to be modeled is itself a fact about our preferences that has to be learned by interacting with us.

A centerpiece of this sequence is Goodhart's law. Treating humans as physical systems and human preferences as emergent leads to a slightly unusual definition of Goodhart's law: "When you put pressure on the world to make it extremely good according to one interpretation of human values, this is often bad according to other interpretations."

This perspective helps us identify bad behavior that's relevant to Goodhart's law for value learning AI. We should build value learning AI that is sensitive to the broad spectrum of human values, that allows us to express our meta-preferences, and that is conse...
Less Threat-Dependent Bargaining Solutions?? (3/2)
by Diffractor — 2022-08-20T02:19:11.405Z

In the previous two posts, we went over various notions of bargaining. The Nash bargaining solution. The CoCo value. Shapley values. And eventually, we managed to show they were all special cases of each other. The rest of this post will assume you've read the previous two posts and have a good sense for what the CoCo value is doing.

Continuing from the last post, the games which determine the payoff everyone gets (not to be confused with the games that directly entail what actions are taken) are all of the form "everyone splits into two coalitions S and N/S, and both coalitions are trying to maximize "utility of my coalition - utility of the opposite coalition"".

Now, in toy cases involving two hot-dog sellers squabbling over whether to hawk their wares at a beach or an airport, this produces acceptable results. But, in richer environments, it's VERY important to note that adopting "let's go for a CoCo equilibria" as your rule for how to split gains amongst everyone incentivizes everyone to invent increasingly nasty ways to hurt their foes. Not to actually be used, mind you. To affect how negotiations go.

After all, if you invent the Cruciatus curse, then in all those hypothetical games where your coalition faces off against the foe coalition, and everyone's utility is "the utility of my coalition - the utility of the foe coalition"... well, you sure can reduce the utility of the foe coalition by a whole lot! And so, your team gets a much higher score in all those games.

Of course, these minimax games aren't actually played. They just determine everyone's payoffs. And so, you'd end up picking up a whole lot of extra money from everyone else, because you have the Cruciatus curse and everyone's scared of it so they give you money. In the special case of a two-player game, getting access to an option which you don't care about and the foe would pay $1000 to avoid, should let you demand a 500$ payment from the foe as a "please don't hurt me" payment.

But Which Desiderata Fails?

Step 1 in figuring out how to get an outcome which is Not That is to look at the list of nice properties which the CoCo solution uniquely fulfills, and figure out which one to break.

As it turns out from picking through the paper, the assumption that must be broken is the axiom of "gaining access to more actions shouldn't lead to you getting less value". As a quick intuitiv...