trentmkelly/LessWrong-43k huggingface dataset:
Why is increasing public awareness of AI safety not a priority?
by FinalFormal2 — 2022-08-10T01:28:44.068Z

Has anyone done anything in this domain? Is there a public face for AI safety that we can promote? I wanted to pester my favorite streamer about looking into AI safety, but I don't know who I would refer him to.

It seems so obvious to me that this should be a priority, especially for anyone who cannot engage in research themselves.

It seems like this might be a result of an aversion to bad press, but the truth is that bad press would be significantly better than what we have now. As far as I can see we have no press.

This is disappointing.

Sometimes I seriously consider the possibility that you guys are all larping.
Manifold x CSPI $25k Forecasting Tournament
by David Chee — 2022-08-09T21:13:15.889Z
Link: https://www.cspicenter.com/p/introducing-the-salemcspi-forecasting

Just wanted to share that

Manifold Markets

has partnered with CSPI and the Salem Center UT to create a forecasting tournament that anyone can participate in. The top prize is a research grant for $25,000 to one of the top 5 most profitable users after 1 year.

The questions being asked will mostly pertain to economics, political and social issues.

If you would like to learn more and participate you should check out Richard Hanania's breakdown of the full details

here

. And

this is the website

if you just want to have a peruse.
Proposal: Consider not using distance-direction-dimension words in abstract discussions
by moridinamael — 2022-08-09T20:44:13.256Z

Epistemic Status: Less Wrong Classic™, a bold and weird proposal to possibly improve your rationality a little bit, maybe.

Overview

It is common to use words and phrases implying distance, directionality, and dimension when discussing ideas and concepts that do not inherently possess these properties. This usually doesn't help and often actively harms our ability to communicate and reason correctly. I am running an experiment to see the effect it has on my thinking and speaking habits.

Rationale

We implicitly translate a thought from its natural form as a clean, clear, distinct concept into a spatial/continuum metaphor all the time, casually and without regard. A mathematician would not try to pull a mathematical object with a thousand dimensions, half of which are binary or non-Euclidean dimensions, into a two-dimensional projection without carefully thinking about the implications of doing so, but we do the equivalent of this all the time, casually and without regard for subtle consequences.

When you mentally impose dimension/distance/direction onto an idea, you risk bringing it into the part of your mind where you are allowed to freely leverage spatial metaphors when thinking about the idea. You will likely have no awareness of all the ways in which you are abusing these spatial/geometric metaphors, because humans are not really smart enough to keep track of all the implications of our casually asserted metaphors. I think that even genius-level humans can be extremely blind to the damage that's done when they implicitly translate a thought from its natural form as a clean, clear, distinct concept into a spatial/continuum metaphor.

I think there's a reason we like to compress things to one or two dimensions and use spatial metaphors. The Curse of Dimensionality applies to humans as well as machines. The way we deal with it is by squinting until all but a couple of the dimensions go away and then reasoning as if only a couple of dimensions exist. Perhaps this is not very generous to humans, but I don't think it's too far off the mark. We come with pre-built circuitry for reasoning in three or fewer dimensions, so it's not surprising that we want to compress things.

It makes sense to use "continuum" language iff what you're talking about has natural and clear continuum properties. "Is this paint color 'more red' than that paint color?" Otherwise you are, at best, m...
How would two superintelligent AIs interact, if they are unaligned with each other?
by Nathan1123 — 2022-08-09T18:58:16.198Z

Hello,

As I read more from this forum and other areas about ethical AI and Decision Theory, I start to imagine what future scenario there could be if multiple AGIs are created simultaneously, by actors who are not coordinating with each other. We've seen similar issues come up in the history of technology, particularly computer technology, where multiple competing standards are created around the same time which are mutually incompatible.

So we imagine we have two AGIs, which are intelligent enough to communicate with humans and each other but are built on very different utility functions, as well as different approaches to Decision Theory. From the perspective of their respective creators, their AGI is perfectly aligned with human morality, but due to different sets of assumptions, philosophies or religions the two actors defined their outer alignment in two different ways.

The reason this seems like a bad situation is because (from what I understand) FDT works on an assumption that other actors use a similar utility function as itself. Thus, the two agents would start with the assumption that the other agent uses the same utility function (which is a false). This bad assumption would fall into miscommunication and conflict, as each agent believes the other is acting immoral or defective.

This seems oddly similar to the way human conflicts arise in the real world (through miscommunication), so an AGI being capable of having that problem incidentally makes them more human-like.

What do you think would be the result? Has this thought experiment been entertained before?
Disagreements about Alignment: Why, and how, we should try to solve them
by ojorgensen — 2022-08-09T18:49:57.710Z

Tldr

AI Alignment does not follow the standard scientific method.
Therefore, we might not agree on which research agendas have the potential to align AGI systems.
We should work intentionally to build consensus around the foundations of different research agendas.
Ways to do this might include more clearly stating the assumptions behind a given framework, further focussing on distilling, red-teaming certain frameworks, or conferences focussing on the foundations of different frameworks.

Introduction

I often see the phrase "AI Alignment is pre-paradigmatic" used a lot within the community, suggesting that at some point AI Alignment will exist as some clear paradigm. To me, this implies that some of the current, core disagreements around the foundations of the field will eventually clear up and that there will be some degree of understanding around the different approaches towards Alignment, how they fit together, and the assumptions behind them.

I don’t think it is controversial to state we are not currently in this position as a field. This is exemplified by posts such as Soares’ “

On how various plans miss the hard bits of the alignment challenge

”, the

MIRI conversations

, and (to an extent) Christiano’s “

Where I agree and disagree with Eliezer

”.

The core idea of this post is that I don't think that we should necessarily expect the field to deal with these internal disagreements without intentional work to do so. We want to be sure that we have aligned AGI, not just aligned it within some framework and hoped for the best. I think this means that there should be more work trying to understand the assumptions made by different researchers, combined with precise methods to reason about these. This reminds me of the "rowing and steering" forum post - trying as hard as we can within a framework is different to trying to decide which frameworks are built on solid foundations, and thus which we should be trying to pursue.

The main thrust of this post is to explain why I think the Alignment community should dedicate at least some resources to "steering" as opposed to "rowing", and to give some ideas as to how we could go about doing this. I will also briefly consider why we might not want to do this.

Thanks to Holly Muir, Hanna Palya, Anson Ho, and Francis Priestland for feedback on previous drafts of this post.

Epistemic Status: I wrote this post whilst trying to form better i...
Progress links and tweets, 2022-08-09
by jasoncrawford — 2022-08-09T17:35:44.939Z
Link: https://rootsofprogress.org/links-and-tweets-2022-08-09

Links

The challenge of corrosion and how engineers fight it (YouTube)
Arnold Kling claims that we over-consume medical services in the US
The lessons of Xanadu (on my personal blog)

Queries

Has anybody done serious, detailed work on the mechanisms that drive technological s-curves? (@Ben_Reinhardt)
Who are the smartest people working on battery storage? Best writing on the topic? (@juliadewahl)
What are examples of projects getting done really fast by organizations that are normally very slow? (@_brianpotter)
Is there a Thiel Fellowship type program for PhD candidates to bail out of academia? (@tayroga)

Tweets & retweets

“Appeal to nature” fallacies are ethically problematic for progress (thread by @RafaRuizdeLira)
Josh Barro on Matt Yglesias on “induced demand” (@jbarro)
US immigration as an important-but-not-urgent policy disaster (@sama)
The US basically stopped building large-scale water infrastructure in the 1980s (@_brianpotter)
Is it possible to find venture capital for AI research org with strong safety focus?
by AnonResearch — 2022-08-09T16:12:29.234Z

I wrote here a while back with this throwaway asking for help: AI researchers from Russia looking for a new home.
We still have a few options, but I wonder if it's possible in the current climate to find funding for a startup doing fundamental AI research with focus on alignment, or any startup needs to be product-focused to get funding.
What do you think? Do you have any potential contacts I can reach out to?
Many Gods refutation and Instrumental Goals. (Proper one)
by aditya malik — 2022-08-09T11:59:23.905Z

Making another due to complaints about lack of clarity in the previous one.

Many Gods refutation (there are too many AIs to care about a particular one. Even if you acausally trade with one, another one might punish you for not following it)

Instrumental Goals for AI ( AIs have Instrumental goals, being if you donate to AI research, all of them would benefit)

My take is many gods refutation still works due to us having no real idea about mindspace of possible AI, thus the AI could just punish for not following it in particular. Also, a small preference today could result in a vastly different in the future, further providing objective to punish for the AI.

What do you think? Any replies would be appreciated.
Content generation. Where do we draw the line?
by Q Home — 2022-08-09T10:51:37.446Z

If you want to be affected by other people, if you want to live in a culture with other people, then I believe 4 statements below are true:

You can't generate all of your "content".

This would basically mean living in the Matrix without other people. Consumerist solipsism. Everything you see is generated, but not by other people.

You can't 100% control what content you (can) consume.

For example, if you read someone's story, you let the author control what you experience. If you don't let anybody control what you experience, you aren't affected by other people.

"Symbols" (people's output) in the culture shouldn't have an absolutely arbitrary value.

Again, this is about control. If you want to be affected by other people, by their culture, you can't have 100% control over the value of their output.

"Right now I feel like consuming human made content. But any minute I may decide that AI generated content is more valuable and switch to that" - attitudes like this may make the value of people's output completely arbitrary, decided by you on a whim.

Other people should have some control over their image and output.

If you want to be affected by other people, you can't have 100% control over their image and output. If you create countless variations of someone's personality and exploit/milk them to death, it isn't healthy culturewise.

You're violating and destroying boundaries that allow the other person's personality to exist. (inside of your mind or inside the culture)

...

But where do we draw the line?

I'm not sure you can mix the culture of content generation/"AI replacement" and the human culture. I feel that with every step weakening the principles above the damage to the human culture will grow exponentially.

The lost message

Imagine a person you don't know. You don't care about them. Even worse, you're put off by what they're saying, you don't want to listen. Or maybe you just isn't interested in the "genre" of their message.

But that person may still have a valuable message for you. And that person still has a chance to reach you:

They share their message with other people.
The message becomes popular in the culture.
You notice the popularity. You check out the message again. Or someone explains it to you.

But if any person can switch to AI generated content at any minute, transmitting the message may become infinitely harder or outright impossible.

"But AI can generate a bet...
What are some alternatives to Shapley values which drop additivity?
by eapi — 2022-08-09T09:16:08.470Z
Link: https://math.stackexchange.com/questions/4480545/what-are-some-alternatives-to-shapley-values-which-drop-additivity

I originally asked this on math.stackexchange; after reading Diffractor's Unifying Bargaining sequence (Part 1 here) I'm wondering if there are more insights floating about, so I'm repeating it here.

Shapley values seem to be the standard answer to "how should a coalition split the rewards of their cooperation", but I'm curious about alternatives.

The standard characterization of Shapley values says that Shapley values are the unique coalition payments which satisfy a bunch of properties. Three of them (efficiency, symmetry, and null player) seem pretty necessary for any "reasonable" or "practical" coalition payment rule, but the last one (linearity) does not.

If I didn't care for linearity (or its close synonyms, additivity and aggregation):

What sorts of payment rules become available?
What other properties of Shapley values are maintained?
What other properties would produce a uniquely characterized payment rule?

Alternatively, are any of the other properties also reasonable to drop (for instance, symmetry)? What do you end up with?
Radio Bostrom: Audio narrations of papers by Nick Bostrom
by PeterH — 2022-08-09T08:56:47.629Z
Link: https://forum.effectivealtruism.org/posts/zXdvCQHtyZqGu4XwD/radio-bostrom-audio-narrations-of-papers-by-nick-bostrom

We've started a project to record audio narrations of the most important papers by Nick Bostrom.

The recordings will be added to nickbostrom.com. They will also be distributed as a podcast with its own website, brand and YouTube channel.

Once we have 10-20 episodes, we'll start promoting the podcast more widely. We'll also get the podcast on the first page of Google for "Nick Bostrom" and related keywords.

✨ The first four episodes are available now! ✨

They are:

The Vulnerable World Hypothesis (2019)

Sharing the World with Digital Minds (2020)

The Fable of The Dragon Tyrant (2005)

Existential Risk Prevention as Global Priority (2012)

This project is supported by a grant from the FTX Future Fund Regranting Program.[1]

How to listen

You can subscribe to the podcast on Apple Podcasts, Spotify, or by RSS.

You can also search in your favourite podcast app. Please tell us if it doesn't show up for you.

Finally, you can listen online at radiobostrom.com.

About this project

Continue reading if you'd like to know more about this project...

The plan

Record high-quality human narrations of ~20 of the most important Bostrom papers (hopefully more).

Distribute recordings as as podcast, and via nickbostrom.com

Create an "Introduction to Nick Bostrom series" of 5-10 recordings, which becomes one of the best places for people to start engaging with Bostrom's work.

Market the podcast and intro series somewhat (perhaps very) ambitiously.[2]

(Perhaps, one day) Record high-quality critical discussions of various Bostrom papers or ideas, with invited guests.

Timeline

Early August: soft-launch to EA Forum and other stakeholders; seek feedback.

August: continue recording 15-20 of Bostrom’s most important papers.

Late August or early September: finish 15+ recordings, supporting materials, and release. After that: start the initial marketing push.

Team

Peter Hartree, Project Lead

Shaun Acker, Narrator

Conor Barnes, Assistant Web Developer

Chloé Kalonek, Visual identity

Katy Moore, Audio-proofing

Perrin Walker, Narrator

TBC (we're about to hire a third, female narrator)

Matthew van der Merwe and Fin Moorhouse are kindly serving as advisors.

Some ways you can help...

(1-5 mins) Subscribe to the podcast, and share it with at least 3 friends who might like it.

(1 min) Like and retweet this tweet, so that more people see it.

(1-10 mins) Write a 5-star review on

...
Team Shard Status Report
by David Udell — 2022-08-09T05:33:48.658Z

Team Shard is a nebulous alignment research collective, on paper siloed under John Wentworth's SERI MATS program, but in reality extending its many tendrils far across the Berkeley alignment community. "Shard theory" -- a name spoken of in hushed, mildly confused tones at many an EA hangout. This is their story (this month).

Epistemic status: A very quick summary of Team Shard's current research, written up today. Careful summaries and actual results are forthcoming, so skip this unless you're specifically interested in a quick overview of what we're currently working on.

Introduction

This past month, Team Shard began its research into the relationship between the reinforcement schedules and learned values of RL agents. Our core MATS team is composed of yours truly, Michael Einhorn, and Quintin Pope. The greater Team Shard, however, is legion -- its true extent only dimly suggested by the author names on its LessWrong writeups.

Our current path to impact is to (1) distill and expound shard theory and preregister its experimental predictions, (2) run RL experiments testing shard theory's predictions about learned values, and (3) climb the interpretability tech tree, starting with finetuned-on-values-text large language models, to unlock more informative experiments. In the 95th percentile, best-case possible world, we learn a bunch about how to reliably induce chosen values in extant RL agents by modulating the agent's reinforcement schedule and are able to probe the structure of those induced values within the models with interpretability tools.

Distillations

If you don't understand shard theory's basic claims and/or its relevance to alignment, stay tuned! A major distillation is forthcoming.

Natural Shard Theory Experiments in Minecraft

Uniquely, Team Shard already has a completed (natural) experiment under its belt! However, this experiment has a couple of nasty confounds, and even without those it would only have yielded a single bit of evidence for or against shard theory. But to summarize: OpenAI's MineRL agent is able to, in the best case, craft a diamond pickaxe in Minecraft in 4 minutes (!). Usually, the instrumental steps the MineRL agent must pursue to craft the diamond pickaxe … lie on the most efficient path to crafting the diamond pickaxe. So we can't disentangle from the model's ordinary gameplay data whether the model terminally values the journey or ...
Announcing: Mechanism Design for AI Safety - Reading Group
by Rubi J. Hudson — 2022-08-09T04:21:50.551Z

We’re starting a new reading group for people interested in applying mechanism design tools to technical AI alignment. If you’re interested in joining, you can apply here by August 22nd (applying takes less than five minutes). If you have recommendations for papers to discuss, please mention them in the comments.

What We’re Doing

Mechanism design is the study of how to reach desirable outcomes or equilibria in the face of differing incentives and incomplete information. Many AI safety researchers have expressed enthusiasm about the potential of using these tools for work in alignment, but relatively little work has been done in the intersection. We believe this is partially due to a lack of potential researchers with expertise in both technical AI safety and mechanism design, and partially due to a lack of shovel-ready problems. The goal of this reading group is to make progress on both fronts.

There are three main areas that this reading group will cover:

Work at the intersection of technical AI safety and mechanism design, and where it can be expanded
Current work in technical AI safety, and how mechanism design tools can help
Current work on mechanism design, and how it can be applied to technical AI safety

The plan is to start with papers in the intersection, then alternate between papers on technical AI safety and papers on mechanism design while keeping the broader perspective in mind. Note that although we believe AI governance work is important and contains many applications for mechanism design, that will not be the focus of this reading group.

Who We Are and Who We Want

I (Rubi) am entering the 2nd year of a PhD in Economics this fall, and am currently working on technical AI safety in Berkeley through the SERI MATS program. Other likely participants include three Economics PhD students at top schools and a Math undergraduate student currently taking part in the SERI Summer Research Fellowship. Our hope for this reading group is to connect with people who have similar interests and create the potential for future collaborations.

Based on current expressions of interest, we expect the modal participant in the reading group to be a PhD student in economics, focusing on economic theory, who has read through the AGI Safety Fundamentals curriculum (or an equivalent, such as Eleuther's). If that sounds like you, definitely apply! However, these should not be consi...
What are some Works that might be useful but are difficult, so forgotten?

by TekhneMakre — 2022-08-09T02:22:41.891Z

It seems like the internet in general, and social media broadly, e.g. LessWrong, has this problem that there's a lot of data. The data has to be filtered somehow. The main way data is filtered is by (1) recency, (2) backlinks, and (3) upvotes.

One problem is that these filters don't distinguish between stuff that's not upvoted or backlinked because it's not so good / interesting / important / novel / true / useful / etc., vs. stuff that's not upvoted or backlinked because it was hard to understand (or, easy to misunderstand as not good) and therefore didn't meet the threshold for being signal boosted. In other words, we aggregate signals of "I don't know about this yet" roughly the same way we aggregate signals of "This isn't good enough that it should be remembered much".

This seems like potentially a big problem, or rather, a big opportunity. Maybe there's good ideas that didn't spread much that we could revive. Maybe ideas would develop faster if people could expect that other people would remember "there's an open question here" rather than just not remembering. But IDK how big of an opportunity it is, so I wonder:

What are some examples of works (of any kind whatsoever) that were / are hard to understand, and so didn't get signal boosted when they were new enough to have attention directed to them to become understood, but instead got lost in the shuffle?

One of the famous examples around here is [Infrabayesianism](https://www.alignmentforum.org/posts/83DimRqppcaoyYAsy/job-offering-help-communicate-infrabayesianism). Maybe some of [Donald Hobson's work](https://www.lesswrong.com/users/donald-hobson) gives examples. What else?