trentmkelly/LessWrong-43k huggingface dataset:
Anki deck for learning the main AI safety orgs, projects, and programs
by Bryce Robertson — 2023-09-30T16:06:33.045Z

Having a high-level overview of the AI safety ecosystem seems like a good thing, so I’ve created an Anki deck to help people familiarise themselves with the 167 key organisations, projects, and programs currently active in the field.

Why Anki?

Anki is a flashcard app that uses spaced repetition to help you efficiently remember information over the long term. It’s useful for learning and memorising all kinds of things – it was the the main tool I used to learn German within a year – and during the time that I’ve been testing out this deck I feel like it’s already improved my grasp of the AI safety landscape.

What you’ll learn

The deck is based on data from the AI Existential Safety Map run by AED – if you’re not familiar with them, you’ll learn who they are in this deck.

Each card includes:

The org’s full name
Its nickname/acronym (where applicable)
Its logo
A brief description of what it does
A link to its website for further info (accessed through the ‘Edit’ button for that card)

How to access

You can download the deck here.

Accuracy and feedback

Given the difficulty of summarising an entire org/project into one or two sentences, the descriptions come with the caveat of being necessarily reductive. They aim to capture the essence of each entity but may not fully encompass the breadth or nuance of their work. I encourage you to visit the link included in each card if you’d like a more comprehensive understanding of that particular org.

That being said, if you think any content should be modified then please comment them below, along with any problems/suggestions for the deck in general.

If the general feedback is that this seems to be useful to people, then I may in the future create one covering the most prominent people in AI safety as well.

Thank you to @George Vii for testing out the deck in advance, and credit to all the volunteers who have contributed to the AI Existential Safety Map. This project was completed while a grantee of CEEALAR.
The Lighthaven Campus is open for bookings
by habryka — 2023-09-30T01:08:12.664Z
Link: https://www.lighthaven.space/

Lightcone Infrastructure (the organization that grew from and houses the LessWrong team) has just finished renovating a 7-building physical campus that we hope to use to make the future of humanity go better than it would otherwise.

We're hereby announcing that it is generally available for bookings. We offer preferential pricing for projects we think are good for the world, but to cover operating costs, we're renting out space to a wide variety of people/projects.

How do I express interest?

Chat with us (you can use LessWrong intercom in the bottom right)
Submit an inquiry via this form
Comment on this post

What kinds of things can I run here?

Team Retreats (15-80 people)

We offer cozy nooks with firepits, discussion rooms with endless whiteboards, plus lodging. The space is very modular, and we can put up walls and dividers that will separate out your own private section of the campus.

Parties & Events (10 - 500 people)

From private dinners to 500 person parties. Sound setup, music, use of private kitchens, snacks/catering.

Conferences (50-600 people)

20+ session spaces and lodging for up to 80 attendees.

Lodging (10 - 80 people usually)

We have 45 bedrooms sleeping up to 80 people that can be booked together with or independently from events.

Other

You could host all kind of things here, like choir rehearsals, dinners, LARPing, etc.

What does it cost?

We determine pricing on a case-by-case basis, but a good approximation is $100 - $250 per person per day for retreats conferences and $25 - $75 per person for parties.

We offer some groups large discounts (including free), if we think what you are doing makes the world better. Do really reach out to us if price is an issue. We will often be able to work something out.

You can use your own caterer, or we can provide catering. Our default caterer offers meals from $20/meal to $50/meal that most visitors have found decently satisfying.

We engage in some price discrimination. If we think your best alternative to our venue would be much more expensive, we may charge more than our listed price. We try to find a fair price that splits the difference between our own costs and the value you get out of the space.

We also provide more bespoke services, for an additional charge (see below). Overall we try to be reasonable about pricing, and don't expect to make much profit on the space.

What do other people say about the space?

Some ...
Headphones hook
by philh — 2023-09-29T22:50:04.962Z

Some months ago I got a 3D printer. (An Anycubic Kobra Go, which was pretty high up the list of "best value for money for a starter printer" that I found through /r/3Dprinting at the time.) I haven't used it much, but recently I used it to solve an itch. I wanted my headphones to be in easy reach, so I designed and printed a hook that I could slide over the edge of the table near my desk. This is the first thing I've designed and I'm pretty happy with how it came out!

The curve was pretty awkward. I was working with OpenSCAD (maybe something else would have been easier?) which doesn't have an easy way that I found to draw nice curves. (Lines in general seem annoying, you can extrude a 2d shape to 3d but not a 1d shape to 2d?)

I decided to go with an Archimedean spiral, and did a bunch of math to figure out how the various parameters had to relate to each other.1 I ended up with equations that I'd have had to solve numerically because they probably had no closed form. Then rather than writing a simple script to give me the answers I just eyeballed it, figuring I could redo it properly if I felt like it. Seems basically fine though.

I ended up printing three copies. The first had a nozzle jam about half way through so it came out shitty. You can see it in the background of the second pic. It actually works, but it doesn't have enough friction on the table so it sags down. I can use it for looping cable over, though a deeper hook would be better at that. The second was mostly fine, but I decided I wanted 100% infill on the curve for strength and for some reason I did that by making the walls thick. In hindsight that's silly, it's pretty flexible and when you bend it a bit it starts to split along the middle. So I did one with normal walls and normal 100% infill and it's absolutely fine. These two are the ones actually holding it up in the second picture, I might print a third to get less pressure on the padding.

I could have made something simpler if I wanted to fasten it to the table, either with screws or duct tape. But I like that it doesn't need that.

The exact design probably won't be any use to anyone who doesn't have the same model of cheap folding table as me. But in case it's interesting or helpful, here's the .scad:

.scad file

top_depth = 1.5;
bottom_depth = 1.5;
vert_depth = 1.5;
hook_depth = 3;
width = 10;

top_length = 100;
edge_height = 15.54;
b1_

...
Paul Christiano's views on "doom" (video explainer)
by Michaël Trazzi — 2023-09-29T21:56:01.069Z
Link: https://youtu.be/K8SUBNPAJnE

I have just published a video explainer of Paul Christiano's Lesswrong post: My views on "doom".

This is the second Lesswrong post I make a video explainer about, the first one being Richard Ngo's Clarifying and Predicting AGI.

Feel free to mention in the comments if you have any feedback on this kind of content or would like to suggest another (AI related) Lesswrong post I could make a video about.
The Retroactive Funding Landscape: Innovations for Donors and Grantmakers
by Dawn Drescher — 2023-09-29T17:39:23.874Z
Link: https://impactmarkets.substack.com/p/the-retroactive-funding-landscape

Summary: This is an overview of existing solutions for retroactive funding – also known as outcomes-based funding: What are the main problems that each solution targets and how does it try to solve them? You may learn of new funding models or better understand the differences between them. The article is particularly interesting for donors of all sizes, grantmakers, and investors, as well as project developers across global development, forecasting, and AI safety.

Please send me your name, email address, and approximate annual donation budget in a private message (or through this form) if you are interested in using AI Safety Impact Markets to inform your giving! We publish only the sum of all budgets. It’s crucial for us to motivate expert donors to share their donations.

Disclaimer: This article covers many projects in the retroactive funding space. I’m a cofounder of AI Safety Impact Markets, so that’s the project that I know really well. I I’m much less knowledgeable of the other projects.

This is an edited transcript of an eponymous talk. I’ve held this talk at EAGx Berlin 2023 among other venues.

The Problems

The basic idea behind retroactive funding is that there is some sort of buyer (also known as retrofunder) that provides funding as an incentive. This could be a government, some sort of protocol in the blockchain space, a foundation, a startup in a growth sector that’s doing R&D, a company that has a corporate social responsibility (CSR) budget, impact investors, or donors – all sorts of altruistic actors. The idea is always to commit to paying out some sort of reward to create an incentive for project developers and investors. (I’ll always say investor when someone is at least partially profit-oriented and funder or buyer otherwise.)

With this incentive in place, these project developers can go to investors and ask for funding in the form of equity or debt in order to realize their project. Then, if the project is successful, the buyers can reward both of these parties. These financial flows are indicated in the diagram – but of course, they only happen conditionally on this project actually producing something that the buyer values.

This solves a few different problems, and depending on the temperament of the buyers, some of these might be relevant in particular cases or they might not be relevant.

Knowledge & Overhead

One problem that is p...
Bids To Defer On Value Judgements
by johnswentworth — 2023-09-29T17:07:25.834Z

Consider two claims:

“broccoli is good for you”
“broccoli decreases cholesterol”

Even though the former might be considered a lossy summary of the latter, the two feel very different; they pull very different levers in my brain. “Broccoli decreases cholesterol” pulls levers like:

Is the claim even true? Does broccoli really decrease cholesterol? Would I expect to hear people claim this even in worlds where it is false?
How much does broccoli decrease cholesterol? Is it a tiny effect size? Also how much broccoli?
Where did this information come from? Was it perhaps among the endless stream of bullshit nutrition studies?
Relative to what baselines? Is broccoli substituted for something, or added? What’s the population?
Do I want lower cholesterol? Do I want it more than I want to eat food tastier than broccoli?

(Probably other people will not have these exact same levers, but I expect most people instinctively respond to “eating broccoli decreases cholesterol” with some kind of guess about where that information came from and how trustworthy it is.)

The other version, “Eating broccoli is good for you”, not only doesn’t pull those levers, it feels like… the sentence is making a bid to actively suppress those levers? Like, those levers are all part of my value-judgement machinery, and the sentence “broccoli is good for you” is making a bid to circumvent that machinery entirely and just write a result into my value-cache.

This is a “bid to defer on a value judgement”: the sentence is a bid to directly write a value-judgement into cache, without going through my own internal value-judgement machinery. If I accept that bid, then I’m effectively deferring to the speaker’s value-judgement.

The Memetic Parasite Model

If broccoli is good for you (and presumably for most other humans, in general), then sharing that information is a friendly, helpful, prosocial action.

More generally: if a value judgement is correct, then passing it along is typically a friendly, helpful, prosocial action. After all, it will help other people to make more “good” decisions if they have more correct information cached about what’s “good”/”bad”.

But this gives rise to a potential parasitic meme dynamic:

Alice, at some point, hears that broccoli is good for you. She caches that value judgement.
When talking to Bob, Alice notices that it would be helpful and prosocial for her to tell Bob that brocco

...
Announcing FAR Labs, an AI safety coworking space
by Ben Goldhaber — 2023-09-29T16:52:37.753Z

FAR Labs is a coworking hub in downtown Berkeley for organizations and individuals working on AI safety and related issues. Since opening the space in March 2023, we have grown to host approximately 30 members. Our members are primarily drawn from four anchor organizations, but we also host a number of independent researchers and research teams.

Now that our initial setup is complete, we are pleased to announce an open

call for applications

from individuals or organizations.

Our initial aims for FAR Labs:

First and foremost it should be a place to do great work. Our members are working on challenging problems, and we want to improve their effectiveness, reduce distractions, and provide a professional environment for them to work in. That includes providing a variety of workspaces (private offices, dedicated desks and hot-desks, general areas), catering, and other office amenities such as a gym.
A warm, intellectually generative culture. Having interesting and fun conversations is one of the best parts of working in a shared environment, and championing a culture that enables those interactions is incredibly important to us.
Supporting collaborations between members, other alignment organizations, and outside collaborators (e.g. academics, or industry researchers). While membership is tied to actively working on AI safety (technical or governance) or related areas (e.g. field building, advocacy, fundraising), we also want to make a space that’s welcoming to many viewpoints, which we expect to benefit both members and visitors.

FAR AI

’s broader mission is to support research and initiatives that promote trustworthy and safe AI systems. FAR Labs is an investment in operations and coordination. By creating research environments and good operational scaffolding, we can accelerate safety research and x-risk reduction across projects and orgs.

For the past six months that’s looked like setting up the space and getting the basics in place (office, food, equipment). Moving into 2024 the Labs team will begin offering programs for members – as well as others in the AI safety ecosystem – for developing relevant skills for research and operational excellence. We’re particularly excited about identifying best practices and providing training to help members in building and scaling high performing teams.

FAR Labs runs at cost/a slight loss[1]; we’re aiming for a fully member ...
A tool for searching rationalist & EA webs
by Daniel_Friedrich — 2023-09-29T15:23:33.255Z
Link: https://ratsearch.blogspot.com/

UPDATE (November 24) - If you have ChatGPT plus, you can try the GPT-bot version. This bot primarily searches through EA webs, if it fails to find anything relevant, it broadens the search to the whole web (more information here).
______________
I've made a simple website where you can limit your Google search to webs related to EA and rationalism. There are three modes:

Basic webs (EA Forum, LessWrong, Alignment Forum, 80.000 Hours, ACX, SSC)
Search by topics (webs, which Google thinks are related with topics like Effective Altruism)
Search webs + blogs (basic webs, blogs listed here and here, webs of EA-adjacent orgs listed here) (webs listed at ea.news)

Note that the first few results tend to be ads.

I've made the web with near-zero programming experience using Google's Programmable Search Engine app and ParseHub for webscraping links. You can access the full list of webs here.
Basic Mathematics of Predictive Coding
by Adam Shai — 2023-09-29T14:38:28.517Z

This is an overview of the classic 1999 paper by Rajesh Rao and David Ballard that introduced predictive coding. I'm going to focus on explaining the mathematical setup instead of just staying at a conceptual level. I'll do this in a series of steps of increasing detail and handholding, so that you can grasp the concepts by reading quickly, assuming you are familiar with the mathematical concepts. In addition I have implemented a convnet version of this framework in pytorch, which you can look at in this notebook:

Why I wrote this

The phrases "predictive coding" and "predictive processing" have squeezed their way outside of academia and into the public. They occur quite commonly on Lesswrong, and in related spaces. They are used to point towards a set of ideas that sound something like "the brain is trying to predict sensory input" or even more handwavy "everything about the mind and psychology is about prediction." This makes predictive coding[1] sound like a big theory of everything, but it was actually introduced into neuroscience as a very concrete mathematical model for the visual cortex.

If you are interested in these ideas, it is helpful to understand the mathematical framework that underlies the more grandiose statements. If nothing else, it grounds your thinking, and it potentially gives you a real mathematical playground to test any ideas you might have about predictive coding. Additionally, 1999 was quite a while ago, and in many ways I think the mathematics of predictive coding makes more intuitive sense in today's deep learning ways of thinking, than in the way originally presented. I've tried to port the original work into something that looks more at home in today's ways of thinking (e.g. the code on github i linked to implements a predictive coding network in pytorch and uses convnet structure etc.)

A note of clarification: there are other network implementations of predictive processing, but this is the original one that started everything. Karl Friston has said that this is the paper that led him to active inference.

A second note of clarification: My understanding of these terms is that predictive processing is an umbrella term that encompasses especially the more theory of everything type ideas, while predictive coding really refers to (a small set) of particular network architectures, like the one described here. But I do thi...
"Diamondoid bacteria" nanobots: deadly threat or dead-end? A nanotech investigation 
by titotal — 2023-09-29T14:01:15.453Z
Link: https://titotal.substack.com/p/diamondoid-bacteria-nanobots-deadly

Confidence level: I’m a computational physicist working on nanoscale simulations, so I have some understanding of most of the things discussed here, but I am not specifically an expert on the topics covered, so I can’t promise perfect accuracy.

I want to give a huge thanks to Professor Phillip Moriarty of the university of Nottingham for answering my questions about the experimental side of mechanosynthesis research.

Introduction:

A lot of people are highly concerned that a malevolent AI or insane human will, in the near future, set out to destroy humanity. If such an entity wanted to be absolutely sure they would succeed, what method would they use? Nuclear war? Pandemics?

According to some in the x-risk community, the answer is this: The AI will invent molecular nanotechnology, and then kill us all with diamondoid bacteria nanobots.

This is the “lower bound” scenario posited by Yudkowsky in his post AGI ruin:

The nanomachinery builds diamondoid bacteria, that replicate with solar power and atmospheric CHON, maybe aggregate into some miniature rockets or jets so they can ride the jetstream to spread across the Earth's atmosphere, get into human bloodstreams and hide, strike on a timer.

The phrase “diamondoid bacteria” really struck out at me, and I’m not the only one. In this post by Carlsmith (which I found very interesting), Carlsmith refers to diamondoid bacteria as an example of future tech that feels unreal, but may still happen:

Whirling knives? Diamondoid bacteria? Relentless references to paper-clips, or “tiny molecular squiggles”? I’ve written, elsewhere, about the “unreality” of futurism. AI risk had a lot of that for me.

Meanwhile, the controversial anti-EA crusader Emille Torres cites the term “diamondoid bacteria” as a reason to dismiss AI risk, calling it “patently ridiculous”.

I was interested to know more. What is diamondoid bacteria? How far along is molecular nanotech research? What are the challenges that we (or an AI) will need to overcome to create this technology?

If you want, you can stop here and try and guess the answers to these questions.

It is my hope that by trying to answer these questions, I can give you a taste of what nanoscale research actually looks like. It ended up being the tale of a group of scientists who had a dream of revolutionary nanotechnology, and tried to answer the difficult question: How do I actually bui...
Steering subsystems: capabilities, agency, and alignment
by Seth Herd — 2023-09-29T13:45:00.739Z

Human brains have steering subsystems. LLMs and most RL agents do not.

Steering systems

, as defined by Max H, are goal-directed AI systems, or optimizers. Here I focus on steering subsystems: the parts of human and AI cognitive systems most directly relevant to goal-direction. These work in three distinct ways (and probably more), each providing a different type and amount of agency, and associated capabilities.

Thinking about types of steering subsystems can clarify our conceptions of agency. Steering subsystems increase risks by adding capabilities. Notably, sophisticated steering subsystems create useful representations of goals. This allows them to break complex tasks into subgoals (e.g., [prevent human interference]:). Adding steering subsystems to otherwise non-agentic AI (like LLMs) may prove irresistible and dangerous, because it may allow rapid capability gains. But this scenario has an upside: aligning a steering subsystem is somewhat simpler than aligning the whole system it steers. Thus, alignment plans that focus on steering subsystems may have an advantage.

I spent a bunch of time trying to work out the brain mechanisms of complex cognition.[1] This work has some relevance for understanding some different types of steering subsystems and resulting types of agency.

Cognition is goal-directed in different ways when different steering mechanisms are used. There are several distinctions proposed by different cognitive sciences: model-based vs. model-free RL from machine learning; habitual vs. goal-directed behavior from animal neuroscience; automatic vs. controlled processing from cognitive psychology; and System 1 and System 2 thinking from behavioral economics. None of these distinctions seems to cleanly match the brain mechanisms creating different types of goal-directed cognition for human decision-making.[2] Therefore I'll describe the cognitive mechanisms directly.

Agency is not a binary; it is at least a spectrum. Humans use at least three types of steering:

Types of steering and agency

Systems trained with reward and reward predictions

No steering subsystems

Systems that predict outcomes of actions and their values

Limited steering subsystems

Systems that select possible high-reward outcomes as goals

Full steering subsystems
Hierarchical subgoal creation for planning
Implemented only recently, few barriers to

...