trentmkelly/LessWrong-43k huggingface dataset:
Would (myopic) general public good producers significantly accelerate the development of AGI?
by mako yass — 2022-03-02T23:47:09.322Z

This question is material to us, as we're building an impact certificate market (a major component in retroactive public goods funding), and if the answer is yes, we might actually want to abort, or — more likely — I'd want to put a lot of work into helping to sure up mechanisms for making it sensitive to long-term negative externalities.

Another phrasing: Are there any dependencies for AGI, that private/academic AI/AGI projects are failing to coordinate to produce, that near-future foundations for developing free software would produce?

I first arrived at this question with my economist hat on, and the answer was "of course, there would be", because knowledge and software infrastructure are non-excludable goods (useful to many but not profitable to release). But then my collaborators suggested that I take the economist hat off and try to remember what's actually happening in reality, in which, oh yeah, it genuinely seems like all of the open source code and software infrastructures and knowledge required for AI are being produced and freely released by private actors, in which case, us promoting public goods markets couldn't make things worse. (Sub-question: Why is that happening?)

But it's possible that that's not actually happening, it could be a streetlight effect: Maybe I've only come to think that all of the progress is being publicly released because I don't see all of the stuff that isn't! Maybe there are a whole lot of coordination problems going on in the background that are holding back progress, maybe OpenAI and Deepmind, the algorithmic traders, DJI, and defense researchers are all doing a lot of huge stuff but it's not being shared and fitted together, but a lot of it would be in the public cauldron if an impact cert market existed. I wouldn't know! Can we rule it out?

It would be really great to hear from anyone working on AI, AGI, and alignment on this. When you're working in an engineering field, you know what the missing pieces are, you know where people are failing to coordinate, you probably already know whether there's a lot of crucial work that no individual player has an incentive to do.
The Agent
by Ideopunk — 2022-03-02T22:43:58.676Z

Before the war I didn't really think about anything. I just did things and things just happened to me. Then yesterday my mother asked me if we should leave. "Before it's too late". That was the first time I ever really thought about something. This is what thinking was like for me:

- I pictured all the animals and trees and hay bales and garbage and driftwood and streams and bogs and winds and bridges and messages and tunnels between us and freedom.

- I pictured every horrible thing they would do to us.

- I pictured firing a gun.

- I pictured being shot with a gun.

- I pictured being buried under rubble.

- I pictured moving rubble off of my mother.

- I pictured reading a book from my childhood in a refugee camp.

- I pictured my mother so tired that I would carry her.

- I pictured us committing suicide together.

- I pictured learning a new language.

- I pictured the stars without smoke.

Then I balanced each of these on a giant scale, inside a calculator, which assigned numbers to each of these, and multiplied each by how likely they were, and calculated how good it would be to go, and how good it would be to stay.

It was much better to go. I told my mother so. She didn't ask how I decided. She packed us food, and off we went, into the heavy heat of the night, pausing only to listen to the whistling of the bombs.
March 2022 Welcome & Open Thread
by Ruby — 2022-03-02T19:00:43.263Z

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you want to explore the community more, I recommend reading the Library, checking recent Curated posts, seeing if there are any meetups in your area, and checking out the Getting Started section of the LessWrong FAQ. If you want to orient to the content on the site, you can also check out the new Concepts section.

The Open Thread tag is here. The Open Thread sequence is here.
The Wicked Problem Experience
by HoldenKarnofsky — 2022-03-02T17:50:18.621Z

I’ve spent a lot of my career working on wicked problems: problems that are vaguely defined, where there’s no clear goal for exactly what I’m trying to do or how I’ll know when or whether I’ve done it.

In particular, minimal-trust investigations - trying to understand some topic or argument myself (what charity to donate to, whether civilization is declining, whether AI could make this the most important century of all time for humanity), with little reliance on what “the experts” think - tend to have this “wicked” quality:

I could spend my whole life learning about any subtopic of a subtopic of a subtopic, so learning about a topic is often mostly about deciding how deep I want to go (and what to skip) on each branch.
There aren’t any stable rules for how to make that kind of decision, and I’m constantly changing my mind about what the goal and scope of the project even is.

This piece will narrate an example of what it’s like to work on this kind of problem, and why I say it is “hard, taxing, exhausting and a bit of a mental health gauntlet.”

My example is from the 2007 edition of GiveWell. It’s an adaptation from a private doc that some other people who work on wicked problems have found cathartic and validating.

It’s particularly focused on what I call the hypothesis rearticulation part of investigating a topic (steps 3 and 6 in my learning by writing process), which is when:

I have a hypothesis about the topic I’m investigating.
I realize it doesn’t seem right, and I need a new one.
Most of the things I can come up with are either “too strong” (it would take too much work to examine them satisfyingly) or “too weak” (they just aren’t that interesting/worth investigating).
I need to navigate that balance and find a new hypothesis that is (a) coherent; (b) important if true; (c) maybe something I can argue for.

After this piece tries to give a sense for what the challenge is like, a future piece will give accumulated tips for navigating it.

Flashback to 2007 GiveWell

Context for those unfamiliar with GiveWell:

In 2007, I co-founded (with Elie Hassenfeld) an organization that recommends evidence-backed, cost-effective charities to help people do as much good as possible with their donations.
When we started the project, we initially asked charities to apply for $25,000 grants, and to agree (as part of the process) that we could publish their application materials. This was

...
The Limits Of Medicine - Part 2 - Homogeneity Assumptions
by George3d6 — 2022-03-02T16:45:29.895Z
Link: https://www.epistem.ink/p/the-limits-of-medicine-part-2-homogeneity?s=w

This is part of a series of articles I'm writing to try and present a clear framework around where medicine is lacking and what easy wins might be found -- This one is "basic but necessary" in that I try to explain a concept that's rather familiar to most people here (controlled trials and their limitations). So you may want to skip it unless you're very unfamiliar with medical science.

Science started by observing highly regular events (patterns). Things like the movement of the celestial bodies, the tracks of arrows, the angles at which stones hold into arches.

These things vary with some parameters, rock composition changes the science of architecting cathedrals in Andalusia compared to the Loire Valley, location on Earth adds noise to the details of astronomy, but they vary very little.

Fundamental science is observing the pattern in “simplistic” events, ones where most parameters are essentially universal constants. When faced with most of the real world we aggregate many observations, call the similarities between them “pattern”, and the weird occurrences noise and errors, accounting for them with confidence and ranges and probabilities.

Biology is one such complex real-world system, and calling anything "causal" in biology is wrong, a naive simplification. It's complex correlations and interactions all the way down. It may literally be computationally impossible to boil biology down to the almost-deterministic realm of physics (simulations).

Aggregate Treatment

This leads us to study things like drugs and interventions by aggregating the effect across many people. Give acetaminophen to a man and it might magically cure his headaches, give it to another and it might slightly worsen it. Give it to 10,000 and ask them to rate their pain improvement and you can know something like:

Acetaminophen usually helps with headaches

Go into the detials and you can say something like:

Acetaminophen lessens headaches in 80% of individuals mildly to perfectly (1.4x compared to placebo), it slightly worsens them or has not effect in 20% (0.46 compared to placebo) of individuals.

Or even:

Acetaminophen lessens headaches in 80% of individual mildly to perfectly, it slightly worsens them or has not effect in 20% of individuals. But for bearers of rs12345678(A;G) it worsens headaches in 60% of cases and does nothing in 40% of cases.

The problem is that the more you dig down the more ...
[Intro to brain-like-AGI safety] 6. Big picture of motivation, decision-making, and RL
by Steven Byrnes — 2022-03-02T15:26:11.828Z

(Last revised: July 2024. See changelog at the bottom.)

6.1 Post summary / Table of contents

Part of the

“Intro to brain-like-AGI safety” post series

.

Thus far in the series,

Post #1

set out some definitions and motivations (what is “brain-like AGI safety” and why should we care?), and Posts

#2

&

#3

split the brain into a Learning Subsystem (cortex, striatum, cerebellum, amygdala, etc.) that “learns from scratch” using learning algorithms, and a Steering Subsystem (hypothalamus, brainstem, etc.) that is mostly genetically-hardwired and executes innate species-specific instincts and reactions.

Then in Post #4, I talked about the “short-term predictor”, a circuit which learns, via supervised learning, to predict a signal in advance of its arrival, but only by perhaps a fraction of a second. Post #5 then argued that if we form a closed loop involving both a set of short-term predictors in the extended striatum (within the Learning Subsystem) and a corresponding set of hardwired circuits in the Steering Subsystem, we can get a “long-term predictor”. I noted that the “long-term predictor” circuit is closely related to temporal difference (TD) learning.

Now in this post, we fill in the last ingredients—roughly the “actor” part of

actor-critic reinforcement learning

(RL)—to get a whole big picture of motivation and decision-making in the human brain. (I’m saying “human brain” to be specific, but it would be a similar story in any other mammal, and to a lesser extent in any vertebrate.)

The reason I care about motivation and decision-making is that if we eventually build brain-like AGIs (cf.

Post #1

), we’ll want to build them so that they have some motivations (e.g. being helpful) and not others (e.g. escaping human control and self-reproducing around the internet). Much more on that topic in later posts.

Teaser for upcoming posts: The next post (#7) will walk through a concrete example of the model in this post, where we can watch an innate drive lead to the formation of an explicit goal, and adoption and execution of a plan to accomplish it. Then starting in Post #8 we’ll switch gears, and from then on you can expect substantially less discussion of neuroscience and more discussion of AGI safety (with the exception of one more neuroscience post towards the end).

Unless otherwise mentioned, everything in this post is “things that...
The Round Table Manifesto
by Jonathan Moregård — 2022-03-02T11:42:24.847Z
Link: https://honestliving.substack.com/p/the-round-table-manifesto-pt-1?r=czmjo&s=w&utm_campaign=post&utm_medium=web

A 2 part sequence about optimal seating arrangements, pit-falls with long tables & adoption strategies for round tables.

Pt 1 - The Horrors of Long Tables

Pt 2 - Adoption of Round Tables
Ordinary and unordinary decision theory
by JonasMoss — 2022-03-02T11:39:30.522Z

There are a couple of paradoxes in decision theory, typically involving beings making perfect predictions, copies of yourself, and similar shenanigans. If you're unfamiliar with all of this, take a look at Newcomb's problem. Attempted resolutions of these paradoxes usually involve either the construction of complex decision theories (such as FDT ) or copious hand-waving and informal arguments (the philosophy literature on the subject).
This post has two messages.

Most decision problems, including paradoxes, are ordinary decision problems.
Unordinary decision problems can be found by testing with Jensen's inequality. These typically fall into two categories

Counterfactual game-theory and
Precommitment problems, when the size of the action space affects the utility.

I have three guiding principles.

Formulate your problems mathematically. Make sure all your quantities are defined. I'm not talking about complex math here. I'm just talking about doing the work. Sometimes it's not possible to formalize the problem in a way that keeps every aspect of the original, informal project description. And that's ok!
Ignore useless information. Find the minimal description of your problem and solve it.
Take action sets seriously. If a is an allowed action, you can do a, even if it is irrational.

Ordinary decision theory

We're dealing with a rational actor, presumably you, an action space A, and a set of random utilities Ua,a∈A. I take the action set seriously – it is possible to choose any of the actions in the set. We know the distribution of Ua for every a∈A. Let's say that an acta∗ is rational if it maximizes the expected utility EUa,i.e.,a∗∈argmaxaEUa.We'll assume that argmaxaEUa exists. (Defining a rational act is hard when a maximum does not exist. For instance when the act a gives deterministic utility Ua=∑ai=1i−2 when A=N, and even harder when the utility function is discontinuous at infinity. See "Satan's apple".)

You may want to pause for a second or two to figure out if you agree with this notion of a rational act. Is it the usual definition? Is it strange in any way?

Let's take a look at a simple example. Suppose you're given the option to flip either (A) a biased coin with probability 0.7 for heads, (B) a biased coin with probability 0.6 for heads. In either case, you ...
Summit-Seeking as Distinct From Mountaineering
by Sable — 2022-03-02T10:40:57.446Z

Epistemic Status: Proposing new terminology to clarify discourse surrounding potential new equilibria.

TL;DR: We should use the term summit-seeking to refer to the evaluation of equilibria different from the current one, and the term mountaineering to refer to the discussion of the practical considerations of moving from one equilibrium to another. Posts and conversations can avoid confusion by specifying which kind of discussion they are or wish to have.

I. An Object-Level Example of The Problem

I greatly enjoyed the series of guest posts on AstralCodexTen by Lars Doucet regarding the empirical claims of Georgism (1st, 2nd, 3rd).

The comment section of those posts, on the other hand, confused me. This post is about trying to identify that confusion, and providing a shorthand terminology for resolving it in future discussions so as to help us communicate better.

Briefly, so we may use concrete examples, Georgism posits that land, that which is already there without human labor or capital (think a literal plot of dirt), is importantly distinct from both, in that the value of a given piece of land - a location - is mostly due to what's around it, not what's on it. A house in a really good school district is worth more than that same house in a really bad school district; an empty plot of dirt in the middle of New York is worth far more than an empty plot of dirt in Siberia. Georgism then makes claims about how land ought to be taxed (as distinct from how property taxes currently work), for various moral and financial reasons.

For a complete (and much better) explanation, I encourage you to read Lars' original book review.

Lars' posts attracted a large and healthy debate, with the highlights I generally expect from Scott Alexander's commentariat. I also saw, however, a large number of comments with the nature of

I just bought a house, and if we switched to a Georgist Land Value Tax I'd be underwater on my mortgage.

Given that the nature of the post was about whether or not the claims of Georgism hold up empirically (or are at least plausible), this strikes me as a category error.

Replies to those comments also pointed this out, but a sizable percentage of the discourse remained concerned with the consequences of a switch from our current tax law to a more Georgist tax law. I've no wish to single anyone out, and I think the discou...
Can you help me find this book?
by mukashi — 2022-03-02T10:19:11.649Z

I think I read a post here in LW presenting two books by authors with different ideologies (one communist and another libertarian) reaching the same conclusion: we should stop financing the education system. One of them must be "The Case Against Education" by Caplan (which I finished recently and I loved). Does anyone know what's the other book, or even the post I am referring to?

I have tried google and the LW search for quite a while without success.
Ukraine Situation Report 2022/03/01
by lsusr — 2022-03-02T05:07:59.763Z

I love Russia, I love Russian and I love Russians. My favorite physics professor was a nuclear scientist from the Soviet Union. I studied Russian in college where a classmate gave me my first copy of Foreign Affairs Journal. Catherine the Great is #4 on my list of heroes.

I don't know much about Ukraine or Ukrainians, but it is hard not to love them too after the swagger and humor they have exhibited since the Russian invasion.

A week ago I wrote "The Russian Armed Forces is among the three most capable militaries in the world". Since then, I have been astonished by the incompetence of the Russian Armed Forces.

Russian Equipment

When I found out Ukraine issued 10,000 automatic rifles to civilians my first thoughts were "Those rifles could arm an insurgency but using them in conventional battle against Russian forces would be suicide". That's because there's more to fielding an effective soldier than giving a man or woman a gun. He or she needs to know what direction to point it.

Here is an interesting comment thread between people who (I assume) are mostly American infantry.

Having heard over the past 18-24 months about LSCO [Large Scale Combat Operations] and russia, I have to admit they have been exposed pretty badly. Also one of my friends noted during all of the combat footage we have not seen any night infantry combat, let alone any IR lasers. Now objectively I haven’t seen every single piece of combat footage and presumably there is night combat. ―regularguyofthenorth

I saw a report from Ukraine that said that Russian forces wait until dawn to attack because they don't have NVGs, nor the training on using them, not sure how accurate that is. ―IllustriusDot1866

Everything I've seen has said the same. No NODs, no optics, for anyone but leadership positions in infantry units. That's insane to me. I can't imagine life in a combat zone without NODs or optics. ―bang_the_drums

That was something that stuck out to me too - even for the UA. I see maybe 1:20 with anything but an iron sight and a dream. ―Roastage

I've seen an AS VAL [a Soviet assault rifle] and a VSS Vintorez [a silenced Soviet sniper rifle] as "combat trophies" so far, neither one of them had optics, which basically defeats the point of a suppressed gun shooting subsonic ammo that can penetrate armor.

I also saw the VDV [Russian Airborne Forces] guys at the Hostomel airport, one of them had an AK-12 wi

...
Musings on the Speed Prior
by evhub — 2022-03-02T04:04:42.134Z

Thanks to Paul Christiano, Mark Xu, Abram Demski, Kate Woolverton, and Beth Barnes for some discussions which informed this post.

In the ELK report, Paul, Mark, and Ajeya express optimism about penalizing computation time as a potentially viable way to select the direct translator over the human imitator:

Human imitation requires doing inference in the entire human Bayes net to answer even a single question. Intuitively, that seems like much more work than using the direct translator to simply “look up” the answer.

[...]

Compared to all our previous counterexamples, this one offers much more hope. We can’t rule out the possibility of a clever dataset where the direct translator has a large enough computational advantage to be preferred, and we leave it as an avenue for further research.

I am more skeptical—primarily because I am more skeptical of the speed prior's ability to do reasonable things in general. That being said, the speed prior definitely has a lot of nice things going for it, and I do think it's worth taking a careful look at both the good and the bad that the speed prior has to offer. Conceptually, what we want to pay attention to when evaluating a prior from an AI safety perspective is threefold: it needs to favor good models over bad models (e.g. the direct translator over the human imitator), it needs to be competitive to implement, and it needs to favor models with good generalization over models with bad generalization (e.g. the resulting models need to themselves be performance competitive).

Before I do that, however, an important preliminary: there are multiple different forms/types of speed priors, so when I say “the speed prior,” I really mean a class of priors including:

various combinations of circuit-size and circuit-depth complexity and

various combinations of Turing machine time and description complexity, where Turing machine time complexity can be measured either by taking the max number of steps taken across all inputs or the average number of steps taken on a particular distribution of inputs.

The basic structure of this post will be a dialog of sorts between a pro-speed-prior and an anti-speed-prior perspective. I'll start with some of the arguments in favor of the speed prior and how the anti-speed-prior perspective might respond, then give some arguments against the speed prior and how the pro-speed-prior perspective might ...
Are our community grouphouses typically rented, or owned?
by mako yass — 2022-03-02T03:36:58.251Z

Asking partially because the Auckland community really needs to start a grouphouse and could do with advice, and also I expect the answers to be interesting for a bunch of other reasons:

Intuitively, one might expect a mostly longtermist community to be interested in buying, rather than renting, and if they did, that could end up being quite interesting in economic and social terms, different residents may end up owning potentially different-sized 'shares' of the property, selling them when they leave (to the house, or to the new resident?) and buying them when they join. All of these things would be somewhat socially challenging, full of lessons, and worth talking about. It might also provide motivation to work on a less ambitious form of my Propinquity Cities, which can pretty much be paraphrased as a matchmaking system/placement solver for grouphouses.
Alternately (and this is the impression I get), maybe everyone just rents. It would be the path of least resistance and you can make some arguments about the benefits of dynamism. In which case I'd want to start a discussion of whether that really makes sense, and whether it has made sense so far.
Alternately, maybe one person owns the house, and everyone else pays rent to them?