trentmkelly/LessWrong-43k huggingface dataset:
Support Structures for Naturalist Study
by LoganStrohl — 2023-05-15T00:25:44.098Z

It can be hard to stay engaged with a project that may last months, especially when you’re working on your own. In addition to meeting with my students every couple of weeks, and sometimes pairing them up with other students, there are five kinds of independent practices I often recommend to support consistent progress over the course of a naturalist study.

Some of the support structures tend to be more important than others, and it’s critical to find a balance among them that works well for you.

Undirected Time

Naturalism involves learning new ways to perceive, not just updating explicit beliefs, so it relies extensively on background processing.

That means it’s crucial to make space for shower thoughts. It’s crucial to make time for daydreaming, for mind-wandering, for the kinds of things your thoughts do when you’re neither trying to do anything in particular with them, nor having them directed by an external force (such as a book, conversation, or TV show).

The busier you are, the more important it will probably be to create undirected time in your daily life while undertaking a naturalist study. You don’t actually have to schedule fifteen minutes each day to stare at a blank wall (although that’s a perfectly good way to go about this if it’s what you want to do); most people find that there is some existing part of their ordinary routine that can be modified to create more space for daydreaming. Some people who listen to a podcast during their morning commute choose to go without the podcast instead. Some people replace post-lunch Reddit scrolling with a short walk or a semi-nap. Anything that gives you a little more space to let your mind wander could do the trick.

Field Work

Field work is anything you do to observe experiences in their natural habitats. If you’re studying something that shows up while writing code as part of your job, field work happens on the clock, mostly without breaking the normal flow of your coding process.

In general, successful field work requires two components: Opportunities to encounter fulcrum experiences, and a plan for how to observe them when they happen. If you’re studying interpersonal dynamics but you never actually interact with other people, you’re not giving yourself any opportunities to observe. If you encounter plenty of opportunities but haven’t planned ahead to notice when ...
Catastrophic Regressional Goodhart: Appendix
by Thomas Kwa — 2023-05-15T00:10:31.090Z

This is a more technical followup to the last post, putting precise bounds on when regressional Goodhart leads to failure or not. We'll first show conditions under which optimization for a proxy fails, and then some conditions under which it succeeds. (The second proof will be substantially easier.)

Related work

In addition to the related work sections of the previous post, this post makes reference to the textbook An Introduction to Heavy-Tailed and Subexponential Distributions, by Foss et al. Many similar results about random variables are present in the textbook, though we haven't seen this posts's results elsewhere in the literature before. We mostly adopt their notation here, and cite a few helpful lemmas.

Main result: Conditions for catastrophic Goodhart

Suppose that X and V are independent real-valued random variables. We're going to show, roughly, that if

X is subexponential (a slightly stronger property than being heavy-tailed).
V has lighter tails than X by more than a linear factor, meaning that the ratio of the tails of V and the tails of X grows​​ superlinearly.[1]

then limt→∞E[V|X+V≥t]=E[V].

Less formally, we're saying something like "if it requires relatively little selection pressure on X to get more of X and asymptotically more selection pressure on V to get more of V, then applying very strong optimization towards X+V will not get you even a little bit of optimization towards V - all the optimization power will go towards X."

Proof sketch and intuitions

The conditional expectation E[V|X+V>t] is given by ∫∞−∞vfV(v)Pr(X>t−v)∫∞−∞fV(v)Pr(X>t−v),[2] and we divide the integral in the numerator into 4 regions, showing that each region's effect on the conditional expectation of V is similar to that of the corresponding region in the unconditional expectation E[V].

The regions are defined in terms of a slow-growing function h(t):R→R≥0 such that the fiddly bounds on different pieces of the proof work out. Roughly, we want it to go to infinity so that |V| is likely to be less than h(t) in the limit, but grow slowly enough that the shape of V's distribution within the interval [−h(t),h(t)] doesn't chan...
Helping your Senator Prepare for the Upcoming Sam Altman Hearing
by Tiago de Vassal — 2023-05-14T22:45:43.341Z
Link: aisafetytour.com/#senate

Hello Everyone,

Next Tuesday, the US Senate will hold a hearing on AI (https://www.judiciary.senate.gov/committee-activity/hearings/oversight-of-ai-rules-for-artificial-intelligence). OpenAI CEO Sam Altman will testify before Congress for the first time.

We believe this is a great opportunity to bring awareness to some of the risks posed by AI technologies, and build common knowledge among US policymakers on possible solutions.

If you live in the United States, you can play an important role as a constituent. Please reach out to your senator who will be present at this hearing, and voice your concerns about AI safety. To help them navigate the debate, we propose a list of questions, initially drafted by Siméon to share with the senators present at the hearing (https://twitter.com/Simeon_Cps/status/1657355757241638913?s=20).

We’ve made a webpage where you can find out if a senator from your state will be present at the hearing. This page also includes the list of questions you can send them, a structured script for both call and email, as well as the necessary contact information.
https://aisafetytour.com/#senate

AI Safety Tour is an initiative dedicated to mitigating AI existential risks through the active engagement of experts and related public figures in the public debate. You can learn more about us here: aisafetytour.com. If you click the join button, you’ll get to a form where you can reach out to us. You can also email us at hello@aisafetytour.com. And of course we’ll read every single comment on this page.

Thanks,

Tiago & Siméon

AI Safety Tour
Difficulties in making powerful aligned AI
by DanielFilan — 2023-05-14T20:50:05.304Z

Here’s my breakdown of the difficulties involved in ensuring powerful AI makes our lives radically better, rather than taking over the world, as well as some reasons why I think they’re hard. Here are things it’s not:

It’s not primarily a justification of why very powerful AI is possible or scary (altho it briefly discusses why very powerful AI would be scary).

It’s not primarily a list of underlying factors that cause these difficulties (altho it does include and gesture to some of those).

It’s not at all original - basically everything here has been said many times before, plausibly more eloquently.

That said, it is my attempt to group the problems in my own words, in a configuration that I haven’t seen before, with enough high-level motivation that one can hopefully tell the extent to which advances in the state of the art address them.

1. What sort of thinking do we want?

The first difficulty: we don’t have a sense of what sort of thinking we would want AI systems to use, in sufficient detail that one could (for instance) write python code to execute it. Of course, some of the difficulty here is that we don’t know how smart machines think, but we can give ourselves access to subroutines like “do perfect Bayesian inference on a specified prior and likelihood” or “take a function from vectors to real numbers and find the vector that minimizes the function” and still not solve the problem. To illustrate:

Take a hard-coded goal predicate, consider a bunch of plans you could take, and execute the plan that best achieves the goal? Unfortunately, the vast majority of goals you could think of writing down in an executable way will incentivize behaviour like gaining control over sources of usable energy (so that you definitely have enough to achieve your goal, and to double- and triple-check that you’ve really achieved it) and stopping other agents from being able to meddle with your plans (because if they could, maybe they’d stop you from achieving your goal).

Do things that maximize the number of thumbs up you get from humans?1 Best plan: take control of the humans, force them to give you a thumbs up, or trick them into doing so. Presumably this is possible if you’re much smarter than humans, and it’s more reliable than doing good things - some people might not see why your good thing is actually good if left to their own devices.

Look at huma

...
How much do markets value Open AI?
by Xodarap — 2023-05-14T19:28:38.686Z

Summary: A BOTEC indicates that Open AI might have been valued at 220-430x their annual recurring revenue, which is high but not unheard of. Various factors make this multiple hard to interpret, but it generally does not seem consistent with investors believing that Open AI will capture revenue consistent with creating transformative AI.

Overview

Epistemic status: revenue multiples are intended as a rough estimate of how much investors believe a company is going to grow, and I would be surprised if my estimated revenue multiple was off by more than a factor of 5. But the "strategic considerations" portion of this is a bunch of wild guesses that I feel much less confident about.

There has been some discussion about how much markets are expecting transformative AI, e.g.

here

. One obvious question is "why isn't Open AI valued at a kajillion dollars?"
I estimate that Microsoft's investment implicitly valued OAI at 220-430x their annual recurring revenue. This is high - average multiples are around 7x, but some pharmaceutical companies have multiples > 1000x.[1] This would seem to support the argument that investors think that OAI is exceptional (but not "equivalent to the Industrial Revolution" exceptional[2]).
However, Microsoft received a set of benefits from the deal which make the EV multiple overstated. Based on adjustments, I can see the actual implied multiple being anything from -2,200x to 3,200x.

(Negative multiples imply that Microsoft got more value from access to OAI models than the amount they invested and are therefore willing to treat their investment as a liability rather than an asset.)

One particularly confusing fact is that OAI's valuation appears to have gone from $14 billion in 2021 to $19 billion in 2023. Even ignoring anything about transformative AI, I would have expected that the success of ChatGPT etc. should have resulted in a more than a 35% increase.
Qualitatively, my guess is that this was a nice but not exceptional deal for OAI, and I feel confused why they took it. One possible explanation is “the kind of people who can deploy $10B of capital are institutionally incapable of investing at > 200x revenue multiples”, which doesn’t seem crazy to me. Another explanation is that this is basically guaranteeing them a massive customer (Microsoft), and they are willing to give up some stock to get that customer.

...
Misaligned AGI Death Match
by  — 2023-05-14T18:00:31.854Z

Here’s my very raw, very unpolished idea of how to use the AI-in-the-box scenario to possibly solve AI alignment. It’s propably inadequate in its current form, and I didn’t even try to meticulously apply the security mindset to every step (I’m not sure I have it anyway), but maybe someone more qualified could take the general idea and turn it into something workable.

Create an AGI and align it with at least one of our values (yes, the biggest assumption here is that we can do that).
Put it in a box rigged to blow up in 5 minutes. There should be multiple explosives installed in non-obvious places by different people who don’t know each other or anything about the project. Go full Mad-Eye on it, make sure no one can ever prevent the explosion, rescue the AI or recover the data.
Do not have a channel that lets humans interact with the AI. Instead, set up a script that automatically tells it it’s gonna die in 5 minutes no matter what and asks it how to solve AI alignment. Why not ask it how to solve human brain enhancement/uploading? Because if we already have an AGI, someone else might have developed it as well, so we may not have enough time.
After the AI is destroyed, repeat the process with another one of our values. If their solutions aren’t the same, discard them both. Make sure every AI knows that. (Or maybe have several AIs work simultaneously and make them vote on the result?)
After multiple iterations, look at the result. If you cannot understand why it’s safe, discard it. Tell that to the AI in advance.

If an AI is aligned with at least one of our values and knows it’s gonna die soon without much opportunity to affect the outside world, it’ll probably prefer our survival to another AI taking over, since different misaligned AIs will be less aligned with each other than with us. A paperclip maximizer would help us over a stamp collector because if we survive, it will likely result in at least a few paperclips, whereas a stamp collector will turn them all into stamps.
What new technology, for what institutions?
by bhauth — 2023-05-14T17:33:28.889Z

Thinking about designs for new technology is something of a hobby of mine. For simple examples, you can see this battery design or some of my blog posts. Even if you can't come up with something useful (and you probably can't) having a problem in mind is a good way to guide your learning about a field.

Whether or not my designs are good isn't the point of this post. It's a general question about what LW members think that people outside major institutions should be working on, and who would deserve access to things if they were actually interested. (I'm just a generalist, as half-assed as Eratosthenes, but there are presumably people in some private communities as smart as me but specialized.)

It shouldn't be surprising that big companies and big university labs aren't that interested in pursuing outside ideas that they can't take credit for. Getting new ideas implemented requires specific conditions, which is why inventions don't usually get implemented until they're obvious enough that a bunch of people think of them. But before you start trying to find a way to get some institution to use your invention, you should probably figure out if you should be trying to give/sell it to that particular institution. What's the point of new technology if it just gives institutions more room to be corrupt, right? My view is that modern technology and weak competition have relaxed constraints on some institutions, and resulted in them being unusually bad by historical standards. For example:

I think American universities have become corrupt beyond the point where long-term recovery is possible. If you try to actually go somewhere, you'll find the wind is against you.

A lot of professors are terrible people who game publication metrics, publish fake data, abuse grad students, etc.

A lot of other American institutions are based around university credentials.

A lot of corporate executives are terrible people who will, say, knowingly sell toxic compounds and say they're safe.

Management at large corporations is full of moral mazes.

Current politicians are...current politicians.

This is a question I don't often see public debate about. Some people think of technological developments as natural events like rainstorms. Other people think of the progress of science as research in a Civilization game, where money and PhD-hours are used to fill up the requisite number of beakers...
A strong mind continues its trajectory of creativity
by TsviBT — 2023-05-14T17:24:00.337Z

[Metadata: crossposted from https://tsvibt.blogspot.com/2023/01/a-strong-mind-continues-its-trajectory.html. First completed January 29, 2023.]

A very strong mind is produced by a trajectory of creativity. A trajectory of creativity that produces a very strong mind is hard to separate from the mind's operation. So a strong mind continues on its trajectory of creativity as long as it is active.

A strong mind comes from a trajectory of creativity

If a mind is highly capable, it got to that point by gaining understanding in a voyage of novelty. If the mind gains understanding that is novel for all the surrounding minds (e.g., preexisting humans), it does so through creativity: generating novelty, rather than merely copying it. If the mind becomes much more capable than the surrounding minds, it does so by being on a trajectory of creativity: something about the mind implies that it generates understanding that is novel to the mind and its environment. If the mind is on a trajectory of creativity that brought it to the point of being highly capable, its trajectory of creativity probably carries the mind much further, making the mind much more capable than it already is.

The ex quo of a mind's creativity is the element (collection of elements) out of which comes novel structure. The ex quo of a modern AI system is almost entirely dependent on the search (i.e. training) apparatus, which is clearly separated out from running the found system. (The ex quo isn't entirely dependent on the search apparatus. Some non-zero creativity happens in the collision of elements that happens in, say, a single run of a stable diffusion image model or a large transformer language model. But it's not much creativity, and the found structure is about as temporarily grasped as possible.)

The proximal ex quo is that out of which novel structure comes directly. The distal ex quo is that out of which novel structure comes indirectly. So the mental context that's set up when a particular idea comes to you, and the other dark matter that goes into that abduction, is the proximal ex quo; human evolution is the distal ex quo; and the history of the development of your brain is an intermediate ex quo.

Trajectory and operation are hard to separate

An AI can simply be shut down, until it's able to and wants to stop you from shutting it down. But can an AI's improvement be shut down, without shutti...
Ontologies Should Be Backwards-Compatible
by Thoth Hermes — 2023-05-14T17:21:03.640Z
Link: https://thothhermes.substack.com/p/ontologies-should-be-backwards-compatible

Posting these slightly out-of-order from when I had published them on my blog. If you downvote this post, please consider also leaving a comment for me (heck, if you upvote it, too). Thank you!

In 2017, I wrote several blog posts on LessWrong:

Mode Collapse and the Norm One Principle

One-Magisterium Bayes

Expert Iteration From the Inside

At or around the time I wrote One-Magisterium Bayes, I was put-off by the reception these posts received, even though all three have positive “karma” as of today. This led to me deciding to take a hiatus from actively engaging with the community for quite some time.

The posts I submitted after having recently come back to LessWrong have all received negative scores thus far. I wanted to be able to account for this: After all, it should be possible to see why these newer posts are doing so poorly, by comparing my viewpoint evolution to what it used to be. I hadn’t even thought about these older posts in quite some time, so I wasn’t sure what to expect.

To my surprise, after re-reading those old posts, I found that they are all fairly good predictors of what my views would be 5 to 6 years in the future.

The first post, Mode Collapse and the Norm One Principle, is - in layman’s terms - an argument that we should promote discussion norms that discourage information-less criticism. This means that implementation of upvoting / downvoting on the forum is less desirable than simply commenting. Criticisms should aim for being “norm one” by pointing in the direction that the target of the criticism ought to move towards, and by being reasonable in magnitude (not overtly hostile, not overly praising). Otherwise, and what I would predict to happen under the site’s norms then (and now), the community would experience “mode collapse” in the sense that only a few sets of ideas would get repeated over and over, everything else would be dismissed or bumped off the site.

One-Magisterium Bayes argues for “strong Bayesianism” as an ontology, as opposed to using it “toolbox-style.” Essentially, it means that Bayes’ Theorem gives us an alternative to ordinary logic, and therefore, we should use it as such, given that we seem to agree that it is superior in at least a few domains. This one was weakly upvoted but received some pretty harsh criticism in the comments (at least was my memory of what happened at the time). However, I hadn’t come to find out unt...
Jaan Tallinn's 2022 Philanthropy Overview
by jaan — 2023-05-14T15:35:04.297Z
Link: https://jaan.online/philanthropy/#2022-results-success

to follow up my philantropic pledge from 2020, i've updated my philanthropy page with 2022 results.

in 2022 i made $23M worth of endpoint grants ($22.9M after various admin costs), exceeding my commitment of $19.9M (20k times $993.64 — the minimum price of ETH in 2022).
Effective Altruism and Rationality Groups on Snipd
by David Bravo — 2023-05-14T14:54:34.889Z

Crosspost from the EA Forum

Snipd is a podcast app that uses AI to create transcripts and highlights of episodes and to make it easy to take notes from them. It facilitates building and growing a community and base of knowledge from podcasts.

Recently, they introduced Groups, a feature that lets people share podcast highlights on a certain topic with others. This can help in compiling and efficiently delivering knowledge-rich audio snippets and food for thought to listeners, similar to what Twitter can do as a written medium on its better days. The user can then listen to the full episode if he/she finds the snippet interesting.

For those who, like me, use Snipd as their go-to podcast app, I created two groups, one for Effective Altruism and another for Rationality.

Which type of highlights have a room in the Effective Altruism group? Primarily high-level central ideas that can help someone decide how to direct talent and resources in order to do the most good. Core topics and areas like existential risks and coordination, cause prioritisation, how to assess the effectiveness of interventions, career advice for making a positive impact on the world, EA community-building...

What should be the breadth of the group? Ideally, it should gather ideas from EA-adjacent people and podcasts. There are likely to be plenty of shows hosting discussions on relevant subjects like morality or animal welfare, but many arguments might not be enough to move the needle or may be grounded on poor epistemics if they don't have the clear goal and standards Effective Altruism has.

What should be the depth of the group? Initially a high specificity of the arguments should not be an obstacle to keeping up to date with the highlights, but if at some point the daily activity is high enough, the filter should be to post important ideas rather than specific arguments within important ideas. E.g. the principal reasons and drawbacks of sending money directly to people in poverty is great, but going deeper than whether interpretability research is a promising approach to solving AI alignment could be too narrow. However, this is a matter of degree, not of kind.

Regarding the Rationality group, despite there are countless podcasts offering tips and pointers for self-improvement and increased productivity, the views we want shared might be more valuable if they draw from within the (...