trentmkelly/LessWrong-43k huggingface dataset:
Progress links and tweets, 2022-07-27
by jasoncrawford — 2022-07-27T17:20:25.268Z
Link: https://rootsofprogress.org/links-and-tweets-2022-07-27

Opportunities

Arcadia Science launches an Entrepreneur-in-Residence (EIR) program (thread from @ArcadiaScience)
The Longevity Prize goes to “outstanding progress in longevity research” (announcement from @allisondman)
Nicole Ruiz is making a list of opportunities in frontier tech (@nwilliams030)
And don’t forget that The Roots of Progress is seeking a CEO! Thanks to Max Roser and Virginia Postrel, among others, for signal-boosting

Announcements

UK’s new Advanced Research and Invention Agency (ARIA) adds Matt Clifford as chairman and Ilan Gur as CEO. Gur will step down as CEO of Activate. See also threads from @matthewclifford and @ilangur. Congrats to all!
Boom announces the new design for its supersonic airplane, Overture

Articles and talks

Works in Progress Issue #8 with articles from Stewart Brand, Jason Collins, and more
“Large changes in productivity are always widely shared” (Bryan Caplan)
Can exponential growth occur if productivity is the best draw from a probability distribution? A talk by Chad Jones (fairly technical)
A DNA origami rotary ratchet motor (Nature, h/t @s_r_constantin)

Quotes

Maps were originally created through the blood, sweat and tears of field agents
“Raise the social status of scientists”
The importance of skilled mechanics to early locomotives
Modern facilities added ~3x to the consumer value of housing
Childhood in the 19th century

Queries

What’s the best CO2 monitor? (@_brianpotter)

Threads

Charles Mann calls for industrial literacy in high school (@CharlesCMann). See my take on “industrial literacy” and the high school progress course I created
Some potential use-cases for crypto (@SBF_FTX)
Software + web tech recapitulating the institutions and ideals of the Enlightenment (@arbesman)
It turns out, nobody ever wanted to work (@paulisci)

Tweets & retweets

RIP Dee Hock, creator of Visa (@patrickc)
“The Roman Empire was not run out of offices” (Marc Andreessen on Conversations with Tyler)
The reason we don’t have fusion is we never decided it was a priority (@tobi)
Quantum Advantage in Learning from Experiments
by Dennis Towne — 2022-07-27T15:49:25.864Z
Link: https://ai.googleblog.com/2022/06/quantum-advantage-in-learning-from.html

For certain types of problems, quantum computing can provide an exponential speedup for ML algorithms. This is now being tested on actual QC hardware.

This strikes me as particularly concerning for hard take-off scenarios. It's a possible way to train dramatically larger AI systems faster, and it's likely to be in use for real problems within the next two decades.
Levels of Pluralism
by adamShimi — 2022-07-27T09:35:32.458Z

This post is part of the work done at Conjecture.

Preparing the Pluralism Question

When do we want all our research eggs in the same paradigm basket?

Although most people don't go as far as the extreme paradigmatism of

Thomas Kuhn

in

The Structure of Scientific Revolutions

, which only allows one paradigm at time (for a given science), the preference for less rather than more options is still pervasive. In the ideal, a convergence to one, even if it's not always feasible. After all, there's only one correct answer, right?

Putting that debatable question aside, I've become more and more convinced that pluralism, the pursuit of multiple lines of research in parallel, is far more prevalent and integral to the process of science than Kuhn's naive paradigmatism. This realization has emerged from studying History and Philosophy of Science, especially outside of physics which for many reasons is quite an unrepresentative science.[1]

But one crucial preliminary point is that pluralism can appear at multiple levels. And the value of pluralism also depends on the level at which it is applied.

So this post proposes a decomposition of the activity of research into four levels, and introduces the corresponding pluralism at each level. Although the point here is not (yet) to argue for pluralism, I offer some examples of pluralistic successes, as well as arguments for the epistemic circumstances where pluralism seems the most valuable. I also finish by proposing a geometric model for when each level of pluralism makes sense, based around considering bits of evidence as objects in high-dimensional space.

The four levels of pluralism I discuss are:

Individual pluralism: pluralism of the methods, ideas, and analogies used by a single researcher or a single research tradition.
Approach pluralism: pluralism of approaches to the same operationalization of the problem.
Operationalization pluralism: pluralism in the way that the problem itself is operationalized.
Problem pluralism: pluralism in the problem itself.

Thanks to Andrea Motti for feedback on a draft of this post.

Simplifying Assumption: Focus on Epistemic Circumstances

When investigating under which circumstances some

epistemic strategy

applies, there are many confusing and complicating factors coming from psychology and sociology. Taking pluralism as an example, the following non-exhaustive list comes to mind:

How

...
Human trials for the Marburg vaccine: funding opportunity?
by americanwalrus — 2022-07-27T05:53:49.570Z
Link: https://www.independent.co.uk/news/health/marburg-virus-vaccine-ghana-cases-latest-b2130777.html

According to the Independent, scientists at Oxford have developed a potential vaccine for Marburg. However, they have been unable to run human trials due to lack of funding.

Are any institutional or high net worth funders in the EA community looking at this opportunity? In the event that the current Marburg outbreak gets out of control, a few weeks saved on vaccine approval could save thousands of lives.
“Fanatical” Longtermists: Why is Pascal’s Wager wrong?
by Yitz — 2022-07-27T04:16:55.792Z

I’ve recently heard a number of people arguing for “fanaticism“ when it comes to longtermism. Basically, if a cause area has even a minuscule probability of positively affecting the long-term future of humanity (and thus influencing an effectively unbounded number of lives), we should fund/support that cause even at the expense of near-term projects with high probability of success. If this is so, I have trouble seeing why Pascal’s Wager (or the even less probable Pascal’s Mugging) shouldn’t hold. I know most people (even religious people) don’t believe Pascal’s argument is valid, but most arguments against it I’ve read would seem to also exclude low-probability longtermist causes from being valid. What am I missing here?
Unifying Bargaining Notions (2/2)
by Diffractor — 2022-07-27T03:40:30.524Z

Alright, time for the payoff, unifying everything discussed in the previous post. This post is a lot more mathematically dense, you might want to digest it in more than one sitting.

Imaginary Prices, Tradeoffs, and Utilitarianism

Harsanyi's Utilitarianism Theorem can be summarized as "if a bunch of agents have their own personal utility functions Ui, and you want to aggregate them into a collective utility function U with the property that everyone agreeing that option x is better than option y (ie, Ui(x)≥Ui(y) for all i) implies U(x)≥U(y), then that collective utility function must be of the form b+∑i∈IaiUi for some number b and nonnegative numbers ai."

Basically, if you want to aggregate utility functions, the only sane way to do so is to give everyone importance weights, and do a weighted sum of everyone's individual utility functions.

Closely related to this is a result that says that any point on the Pareto Frontier of a game can be post-hoc interpreted as the result of maximizing a collective utility function. This related result is one where it's very important for the reader to understand the actual proof, because the proof gives you a way of reverse-engineering "how much everyone matters to the social utility function" from the outcome alone.

First up, draw all the outcomes, and the utilities that both players assign to them, and the convex hull will be the "feasible set" F, since we have access to randomization. Pick some Pareto frontier point u1,u2...un (although the drawn image is for only two players)

Use the Hahn-Banach separation theorem to create a linear function ϕ:Rn→R such that ϕ(u1,u2...un)≥ϕ(F). (such that is abbreviated s.t. from here on out) Or put another way, u1,u2...un is one of the points in the feasible set F that maximizes the linear function ϕ you created. In the image, the lines are the level sets of the linear function, the set of all points where ϕ(x1,x2...xn)=c.

That linear function ϕ:Rn→R can be written as (x1,x2...xn)↦a1x1+a2x2+...+anxn. Bam, those coefficients are the utility weights you need. u1,u2...un is a point that maximizes the function ϕ, and the function ϕ is implementing "take this particular weighted sum of the utilities of the players...
AGI ruin scenarios are likely (and disjunctive)
by So8res — 2022-07-27T03:21:57.615Z

Note: As usual, Rob Bensinger helped me with editing. I recently discussed this model with Alex Lintz, who might soon post his own take on it (edit: here).

Some people seem to be under the impression that I believe AGI ruin is a

small and narrow

target to hit. This is not so. My belief is that most of the outcome space is full of AGI ruin, and that avoiding it is what requires navigating a treacherous and narrow course.

So, to be clear, here is a very rough model of why I think AGI ruin is likely. (>90% likely in our lifetimes.)[1]

My real models are more subtle, take into account more factors, and are less articulate. But people keep coming to me saying "it sounds to me like you think humanity will somehow manage to walk a tightrope, traverse an obstacle course, and thread a needle in order to somehow hit the narrow target of catastrophe, and I don't understand how you're so confident about this". (Even after reading Eliezer's

AGI Ruin

post—which I predominantly agree with, and which has a very disjunctive character.)

Hopefully this sort of toy model will at least give you some vague flavor of where I’m coming from.

Simplified Nate-model

The short version of my model is this: from the current position on the game board, a lot of things need to go right, if we are to survive this.

In somewhat more detail, the following things need to go right:

The world’s overall state needs to be such that AI can be deployed to make things good. A non-exhaustive list of things that need to go well for this to happen follows:

The world needs to admit of an AGI deployment strategy (compatible with realistic alignable-capabilities levels for early systems) that prevents the world from being destroyed if executed.
At least one such strategy needs to be known and accepted by a leading organization.
Somehow, at least one leading organization needs to have enough time to nail down AGI, nail down alignable AGI, actually build+align their system, and deploy their system to help.

This very likely means that there needs to either be only one organization capable of building AGI for several years, or all the AGI-capable organizations need to be very cautious and friendly and deliberately avoid exerting too much pressure upon each other.

It needs to be the case that no local or global governing powers flail around (either prior to AGI, or during AGI development) in

...
Technocracy and the Space Age
by jasoncrawford — 2022-07-26T23:14:07.308Z
Link: https://rootsofprogress.org/technocracy-and-the-space-age

Earlier I proposed a hypothesis that the 1930s–60s in the US were characterized by the attempt to achieve progress through top-down control by a technical elite.

The 1997 preface to Walter McDougall’s Pulitzer-winning The Heavens and the Earth: A Political History of the Space Age adds some evidence for this. McDougall laments NASA’s failed promises and the lost potential of space technology, and he ties this in to the broader theme of failures of centralized federal programs:

From today’s vantage point the Space Age may well be defined as an era of hubris. Not only did it become obvious in the 1960s and 1970s that “planned invention of the future” through federal mobilization of technology and brainpower was failing everywhere from Vietnam to our inner cities, but that it even failed in the arena for which it had seemed ideally suited: space technology.

What were the promises?

In the years following Sputnik I, experts assured congressional committees that by the year 2000 the United States and the Soviet Union would have lunar colonies and laser-armed spaceships in orbit. The film 2001: A Space Odyssey (1968) depicted Hilton hotels on the Moon and a manned mission to Jupiter (January 12, 1992, was the supercomputer Hal’s birthday in the film). In the late 1960s, NASA promoters imagined reusable spacecraft ascending and descending like angels on Jacob’s ladder, permanent space stations, and human missions to Mars—all within a decade. In the 1970s, visionaries looked forward to using the Space Shuttle to launch into orbit huge solar panels that would beam unlimited, nonpolluting energy to earth, hydroponic farming in space to feed the earth’s exploding population, and systems to control terrestrial weather for civilian or military purposes. In the 1980s, the space station project was revived (to be completed again “within a decade”), the Strategic Defense Initiative was to put laser-beam weapons in orbit to shoot down missiles and make nuclear weapons obsolete, and the space telescope was to unlock the last secrets of the universe. By 1990, a manned mission to Mars by the year 2010 was on the president’s wish list, and research had begun on an aerospace plane (the “Orient Express”) to whisk passengers across the Pacific in an hour and land like an airplane in Asia.

None of it came to pass.

McDougall describes the decline of NASA’s budget and the loss of its talent ...
«Boundaries», Part 1: a key missing concept from utility theory
by Andrew_Critch — 2022-07-26T23:03:55.941Z

This post has been recorded as part of the LessWrong Curated Podcast, and can be listened to on Spotify, Apple Podcasts, and Libsyn.

This is Part 1 of my «Boundaries» Sequence on LessWrong.

Summary: «Boundaries» are a missing concept from the axioms of game theory and bargaining theory, which might help pin down certain features of multi-agent rationality (this post), and have broader implications for effective altruism discourse and x-risk (future posts).

1. Boundaries (of living systems)

Epistemic status: me describing what I mean.

With the exception of some relatively recent and isolated pockets of research on embedded agency (e.g., Orseau & Ring, 2012; Garrabrant & Demsky, 2018), most attempts at formal descriptions of living rational agents — especially utility-theoretic descriptions — are missing the idea that living systems require and maintain boundaries.

When I say boundary, I don't just mean an arbitrary constraint or social norm. I mean something that could also be called a membrane in a generalized sense, i.e., a layer of stuff-of-some-kind that physically or cognitively separates a living system from its environment, that 'carves reality at the joints' in a way that isn't an entirely subjective judgement of the living system itself. Here are some examples that I hope will convey my meaning:

a cell membrane (separates the inside of a cell from the outside);
a person's skin (separates the inside of their body from the outside);
a fence around a family's yard (separates the family's place of living-together from neighbors and others);
a digital firewall around a local area network (separates the LAN and its users from the rest of the internet);
a sustained disassociation of social groups (separates the two groups from each other)
a national border (separates a state from neighboring states or international waters).

Figure 1: Cell membranes, skin, fences, firewalls, group divisions, and state borders as living system boundaries.

Comparison to Cartesian Boundaries.

For those who'd like a comparison to 'Cartesian boundaries', as in Scott Garrabrant's Cartesian Frames work, I think what I mean here is almost exactly the same concept. The main differences are these:

(life-focus) I want to focus on boundaries of things that might naturally be called "living systems" but that might not broadly be considered "agents", such as a human being

...
Incoherence of unbounded selfishness
by emmab — 2022-07-26T22:27:21.757Z

Unbounded agentic selfishness isn't possible unless you can unboundedly index on and retrocausally compress "self". Otherwise future "you" is just this animal walking around, and how do you specify them in particular for good things to happen to? In practice, evil people do this by cancer like "make good things happen to people with my skin color".

If you believe humans are "not coherent agents" i.e. what values someone ends up with is underdetermined, then why would you believe that acquisition of ability to index on values as self is coacquired with the underdetermined acquisition of those values? Like, that would be inconsistently postulating both having and lacking "inner alignment".

(Note that Lesswrongers don't think about cybernetics much like "but how does the thing reference itself", if this style of argument seems unusual to you)
«Boundaries» Sequence (Index Post)
by Andrew_Critch — 2022-07-26T19:12:36.368Z

In an upcoming short sequence of posts, I'm going to try to describe a causal pathway from

a key missing idea in the utility-theoretic foundations of game theory, leading to
some problems I think I see in effective altruism discourse, leading further to
gaps in some approaches to AI alignment, and finally,
implications for existential risk.

By default, I'll write one post for each of the above points, since they have different epistemic statuses and can be debated separately. Posts 1 and 3 will be somewhat technical and research-oriented, and cross-posted to the alignment forum, whereas 2 and 4 will be non-technical and community-oriented, and cross-posted to the EA forum. After that there might be more posts in the sequence, depending on the ensuing conversation. In any case I'll try to keep this index post updated with the full sequences.

Here goes!

ETA Oct 31, 2022: There is now a proper sequence index for this sequence :)
→ «Boundaries» Sequence
Active Inference as a formalisation of instrumental convergence
by Roman Leventov — 2022-07-26T17:55:58.309Z
Link: https://direct.mit.edu/books/oa-monograph/5299/Active-InferenceThe-Free-Energy-Principle-in-Mind

The goal of this post is mainly to increase the exposure of the AI alignment community to Active Inference theory, which seems to be highly relevant to the problem but is seldom mentioned on the forum.

This post links to a freely available book about Active Inference, published this year. For alignment researchers, the most relevant chapters will be 1, 3, and 10.

Active Inference as a formalisation of instrumental convergence

Active Inference is a theory describing the behaviour of agents that want to counteract surprising, “entropic” hits from the environment via accurate prediction and/or placing themselves in a predictable (and preferred) environment.

Active Inference agents update their beliefs in response to observations (y), update the parameters and shapes of their models Q and P (which can be seen as a special case of updating beliefs), and act so that they minimise the expected free energy, G:

Where x are the hidden states of the world, ~x is a sequence or trajectory of the hidden states over some time period in the future (not specified), y are the agent’s observations, ~y is the sequence or trajectory of the agent’s expected observations in the future, P(xn+1|xn) is the agent’s generative model of the world’s dynamics (including themselves), P(y|x) is the agent’s generative model of the observations from the hidden states, π is an action plan (called policy in the Active Inference literature) that the agent considers (the agent chooses the plan that entails the minimal expected free energy), Q(~x) is the distribution of beliefs over the hidden states over a period of time in the future, C are agent’s preferences or prior beliefs.

Active Inference framework is agnostic about the time period over which the expected free energy is minimised. Intuitively, that should be the agent’s entire lifetime, potentially indefinite in the case of AI agents. The expected free energy for the indefinite time period diverges, but the agent can still follow a gradient on it by improving its capacity to plan accurately farther into the future and to execute its plans.

Therefore, we can equate instrumental convergence from the AI alignment discourse with agents minimising their expected free energy in the Active Inference framework.

Preferences in Active Inference

For biological agents,&nb...
NeurIPS ML Safety Workshop 2022
by Dan H — 2022-07-26T15:28:52.441Z
Link: https://neurips2022.mlsafety.org/

We're excited to announce the NeurIPS ML Safety workshop! To our knowledge it is the first workshop at a top ML conference to emphasize and explicitly discuss x-risks.

X-Risk Analysis Prizes

$100K in paper prizes will be awarded. There is $50K for best paper awards. There is also $50K in awards for discussing x-risk. This will be awarded to researchers who adequately explain how their work relates to AI x-risk. Analyses must engage with existing arguments for existential risks or strategies to reduce them.

What is the topic of the workshop?

Broadly, the focus of the workshop is on

ML Safety

, which is the umbrella term that refers to research in the following areas:

Robustness: designing systems to be resistant to adversaries.

Monitoring: detecting undesirable behavior and discovering unexpected model functionality.

This category contains interpretability and transparency research, which could be useful for understanding the goals/thought processes of advanced AI systems. It also includes anomaly detection, which has been useful for detecting proxy gaming. It also includes Trojans research, which involves identifying whether a deep neural network will suddenly change behavior if certain unknown conditions are met.

Alignment: building models that represent and safely optimize hard-to-specify human values.

This also includes preventing agents from pursuing unintended instrumental subgoals and designing them to be corrigible.

Systemic Safety: using ML to address broader governance risks related to how ML systems are handled or deployed. Examples include ML for cyberdefense, ML for improving epistemics, and cooperative AI.

How do academic workshops work?

The majority of AI research is published at conferences. These conferences support independently run workshops for research sub-areas. Researchers submit papers to workshops, and if their work is accepted, they are given the opportunity to present it to other participants. For background on the ML research community and its dynamics, see A Bird's Eye View of the ML Field.

Background

A broad overview of these research areas is in

Unsolved Problems in ML Safety

.

For a discussion of how these problems impact x-risk, please see Open Problems in AI X-Risk.