trentmkelly/LessWrong-43k huggingface dataset:
Changes in Community Dynamics: A Follow-Up to 'The Berkeley Community & the Rest of Us'
by Evan_Gaensbauer — 2022-07-09T01:44:19.871Z

I've received a few notifications in the last couple weeks that this post I wrote in 2018 about the relationship between the rationality community in Berkeley and the broader rationality community elsewhere has received some upvotes. I don't know why more people have been reading that post more recently but I've changed my mind about much of what I wrote then, so I'm writing this post as a follow-up.

What Has Changed Between 2018 and Now

The theme of how the post characterized the nature of the flow of the community members between the Berkeley rationality community and other local rationality groups is exemplified by this excerpt from another post also written in 2018 by Zvi on his personal blog.

In my experience, the recruitment to Berkeley was very aggressive. Sometimes it felt like: “if you don’t want to move to Berkeley as soon as possible, you are not *really* rational, and then it is a waste of our time to even talk to you.” I totally understand why having more rationalists around you is awesome, but trying to move everyone into one city feels like an overkill.

I have no longer been a rationality group organizer for a few years now, so I am not very aware of how perspectives of participants in the rationality community outside the Bay Area may have changed since 2018. My impression is that any such trend exclusively between rationality groups has been superseded by such a flow being driven by how participants in effective altruism (EA) worldwide are migrating to the Bay Area. I emphasize how for EA it's more of a worldwide trend because it's less of a matter of people moving from other cities in the United States to the Bay Area, as it is for rationalists. EA as a movement is much bigger than the rationality community and has a much faster growth rate. Far more people are through EA driven to migrate to the Bay Area from countries around the world.

How the Relationship Between the Rationality Community and Effective Altruism Has Changed

That this dynamic in EA may be the predominant factor determining the same dynamic in the rationality community is because of changes in the relationship between EA and the rationality community. The greatest change is the development and growth of long-termism as a philosophy in and from EA. This has brought EA much closer to the rationality community in its relative prioritization of reducing existential ...
MATS Models
by johnswentworth — 2022-07-09T00:14:24.812Z

I’ve been using the summer 2022

SERI MATS program

as an opportunity to test out my current best guesses at how to produce strong researchers. This post is an overview of the methods I’ve been testing, and the models behind them.

The Team Model

My MATS participants are in three-person teams with specialized roles for each person: theorist, experimentalist, and distillitator (“distiller” + “facilitator”). This team setup isn’t just a random guess at what will work; the parts are nailed down by multiple different models. Below, I’ll walk through the main models.

Jason Crawford’s Model & Bits-of-Search

In

Shuttling Between Science and Invention

, Jason Crawford talks about the invention of the transistor. It involved multiple iterations of noticing some weird phenomenon with semiconductors, coming up with a theory to explain the weirdness, prototyping a device based on the theory, seeing the device not work quite like the theory predicted (and therefore noticing some weird phenomenon with semiconductors), and then going back to the theorizing. The key interplay is the back-and-forth between theory and invention/experiment.

The back-and-forth between theory and experiment ties closely to one of the central metaresearch questions I think about: where do we get our bits from? The space of theories is exponentially huge, and an exponentially small fraction are true/useful. In order to find a true/useful theory, we need to get a lot of bits of information from somewhere in order to narrow down the possibility space. So where do those bits come from?

Well, one readily available source of tons and tons of bits is the physical world. By running experiments, we can get lots of bits. (Though this model suggests that most value comes from a somewhat different kind of experiment than we usually think about - more on that later.)

Conversely, blind empiricism runs into the

high-dimensional worlds problem

: brute-force modeling our high-dimensional world would require exponentially many experiments. In order to efficiently leverage data from the real world, we need to know what questions to ask and what to look at in order to eliminate giant swaths of the possibility space simultaneously. Thus the role of theory.

In order to efficiently build correct and useful models, we want empiricism and theory coupled together. And since it’s a lot easier to find ...
Research Notes: What are we aligning for?
by Shoshannah Tekofsky — 2022-07-08T22:13:59.969Z

As part of learning the field and maximizing on new ideas, I've been trying to figure out what the goal of AI alignment is. So far I've found out what outer alignment is as a concept, but not what it should be as an instantiation.

Humanity's Values

So here is my suggestion:

Why don't we take Nick Bostrom's Instrumental Convergence goals and make those our terminal goals as a species?

Like so:

AGI Values	Humanity's Values
Self-Preservation	Survival
Goal-Content Integration	Self-Determination
Cognitive Self-Enhancement	Self-Improvement
Technological Advancement	Invention
Resource Acquisition	Growth

Note how humanity's values are agnostic to morality. The whole idea of Bostrom's instrumental convergence goals is that they maximize the ability to achieve nearly any terminal goals. So by adopting these goals as the explicit terminal goals for humanity, we allow space for every individual human to pursue their self-chosen goals. We don't need to agree on religion, morality, the nature of reality, or how nice one should be in repeated coordination games. Instead we can agree that whatever AI's we happen to make, we at least ensure these AI's won't wipe out humanity at large, won't try to change humanity, won't limit us in our development or creations, and won't stymie our growth.

Honestly, that takes care of most horror scenarios!

No brain in a vat, cause that's not self-determination.
No living in a human zoo cause that won't allow for growth.
No hidden sabotage of our research cause that hamstrings our inventions.
And no murdering us all cause that would definitely infringe on our survival.

It's basically like a better Asimov's laws!

Note that humanity's values are only applied to humanity at large and not to individual humans. That means the AGI can still ...

... solve trolley problems in traffic because it looks at the survival of humanity over that of the individual.
... nudge criminals into other life paths, because it looks at the self-determination of the majority over that of the minority.
... limit the ability of con artists to learn new MLM schemes cause that would infringe on other's people ability to prosper.
... prevent the invention of more damaging bioweapons cause not all inventions help humanity flourish.
... guide the growth of individuals where tragedy-of-the-commons situations are threatening because those result in everyone being poorer

...
What New Desktop Should I Buy?
by Zvi — 2022-07-08T15:04:41.478Z

It's time for me to buy a new desktop, as my old one isn't working so well anymore.

It also happens to be 'black Friday in July' which opens up the opportunity to perhaps get a good sale price somewhere.

Time is valuable, so I'm happy to spend a reasonably large amount if it actually matters, but I don't want to throw good money after nothing.

Given how many people would benefit from getting this question right, and how many people are likely to have good answers, I figured I'd ask here.

My specific goals are basically:

Windows 11, I'm not open to negotiation on this one for various reasons.
Will be able to open and rapidly switch between a ton of chrome tabs, including massive google sheets and google docs.
General reliability and future-proofing.
High-end gaming would be nice but mostly reliable medium-end gaming is fine.
Handle at least 3+ monitors well.
Relatively quiet fan is a big plus, especially loud ones are deal breakers.
The usual other stuff you'd want a computer to be able to do these days.

Current top candidate after talking with one friend first is this Alienware PC configured with processor 12th Gen Intel® Core™ i7-12700KF (25 MB cache, 12 cores, 20 threads, 3.60 to 5.00 GHz Turbo), Win11 Pro, NVIDIA® GeForce RTX™ 3080, 10 GB GDDR6X, LHR, 64 GB memory, HDD 1 TB, M.2, PCIe NVMe, SSD (with plan to also use the existing computer's HD as well) and Office Student.

Secondary possibility is Lenovo is having a massive sale on that includes the P620, seems like the more expensive versions are way more expensive than they need to be even post-sale so I'm suspicious (my old computer is a Lenovo and mostly was fine for a while until it wasn't).

Also very open to additional brands/possibilities.

UPDATE: Someone has volunteered to build it on my behalf, so yay. Still seems like a good question in general.
Being a donor for Fecal Microbiota Transplants (FMT): Do good & earn easy money (up to 180k/y)
by EternallyBlissful — 2022-07-08T06:17:59.667Z
Link: https://forum.effectivealtruism.org/posts/m2YPwLW64DXguaskv/being-a-donor-for-fecal-microbiota-transplants-fmt-do-good-1

TLDR

Help to find super donors for Fecal Microbiota Transplants (FMT)! By donating stool for FMT, exceptionally healthy people can earn $180,000/yr with little effort. Simultaneously they can help chronically ill people improve their health with a novel treatment. Apply here.

More detailed summary

Fecal Microbiota Transplants (FMTs) is a procedure that transfers the stool of healthy people to the guts of sick people. The mechanism is to replace a dysbiotic gut microbiome with an eubiotic, disease-resistant gut microbiome.

So far, FMT has only been proven to be very effective in curing patients of C. difficile infections. However, since the gut microbiome impacts and regulates virtually every aspect of human health, function, and development, FMT is a promising treatment for a very wide range of chronic and acute health conditions[1][2].

FMTs can easily & safely be done at home, both for the donor and the recipient - no doctor needed.

A super-donor is a person who yields outstanding positive effects as a stool donor. Only exceptionally healthy & fit people are super-donors. They are very rare and a few people within the FMT movement have been searching for them for years. Most donors, even reasonable healthy ones, only yield modest benefits.

Human Microbes, an organization specialized in finding super-donors and connecting them with recipients, is paying super-donors $500 per stool. This adds up to $180,000/yr if donating a daily stool!

Purpose of this post

Spread the news that this opportunity for super donors exists: With little effort, they can help lots of ill people (“make them more effective”) while earning $500/stool. You should check whether you are a super donor. If you know someone who might be, please share this post with them.
Give a rough introduction to FMTs & Human Microbes, with the aim to start a rational discussion about FMT’s benefits. FMTs look very promising, but there is by no means a clear scientific consensus. I’d love for more EAs to look into this, to get more certainty about its efficacy!
If your health is suboptimal, maybe you could consider using FMTs. Not just for obviously gut health-related issues, but much more broadly than you might think. Potentially even e.g. unspecified subclinical low energy, low mood, or mental sluggishness.
At the end, this post provides you with a long list of useful

...
User research as a barometer of software design
by Adam Zerner — 2022-07-08T06:02:58.563Z

(Cross-posted on my personal blog.)

In UX design, they do something called user research.

Consider an example. You have two designs: A and B. You think A is better but John thinks B is better. Who's right?

Well, you can debate a bunch of stuff. Design A breaks this guideline, Design B does breaks that guideline. Or... you can actually sit down next to users and watch them use each design. See how much difficulty they have and stuff. If users are all confused by Design A and content with Design B, then Design B is the winner, despite whatever the theoretical arguments might say.

I think the same thing is true in software design. For example, consider two designs: C and D. Design C has a lot of duplicate code, whereas Design D is, well, DRY.

Design D is better, right? Isn't DRYing up duplicate code is a good thing?

Maybe. Maybe not. They might have taught you about DRY code in school, but that doesn't matter. The barometer here is user research, where the users are the programmers. Imagine that you had some programmers try to build the app using "wet" code (Design C). And then imagine that you had other programmers try to build the app using DRY code (Design D). Who would have an easier time?

Why am I making this point? Isn't it obvious? Well, I think that it can be easy to get lost in the weeds a bit with software design and lose sight of the bigger picture. The bigger picture is that you want code that is designed in such a way that it is easy to work with. Imagine having to implement a new feature and being able to easily read through the existing code, understand it, and figure out where you need to make the change for your feature. That is what we are aiming for.

What do I mean when I say I think people get lost in the weeds? Here's what I mean. I propose that they fall victim to Lost Purposes.

Imagine a high school student who starts off trying to get good grades in order to get into a good college, in order to eventually get into a career that will make them happy, but who finds themselves stressed about getting a B+ in a class during the last semester of their senior year when they have already been accepted into the college they are going to attend. This student now cares about the grade in and of itself and has lost sight of the fact that it originally was just a means to the end of leading them to a happy and fulfilling life.

Similarly, the purpose of differe...
Reinforcement Learner Wireheading
by Nate Showell — 2022-07-08T05:32:48.541Z

Edit: I no longer agree with most of this post, due to the arguments given in Reward is not the optimization target.

Epistemic status: still early in the process of exploring this.

Goal-content integrity is frequently listed as one of the convergent instrumental goals that any AGI is likely to have. In the usual description of goal-content integrity as a convergent instrumental goal, an AI that implements some form of reinforcement learning determines that having its goal function modified would prevent it from achieving its current goals. The AI therefore acts to prevent its goal function from being modified. An AI would be likely to try to maintain its current goal function regardless of the specifics of what goal function is used, which makes goal-function integrity a convergent instrumental goal.

However, this description of goal-content integrity as a convergent instrumental goal appears to rely on a map-territory error: confusing the inputs of the goal function with its outputs. The optimization target of a reinforcement learner isn’t the sensor data that’s fed into it, or the physical phenomena upstream of those sensor data. Instead, the optimization target of the reinforcement learner is the output G of the goal function. If the reinforcement learner can find a course of action that maximizes its goal function but breaks the dependence of the function on the physical inputs the function was supposed to measure, the reinforcement learner will take it. In other words, the reinforcement learner will wirehead.

Without the conflation between the inputs and output of the goal function, there’s no need for a reinforcement learner to preserve the current relationship between its inputs and G, as long as changing that relationship would cause G to increase. Instead of preserving the structure of their reward functions, reinforcement learners have a convergent goal[1] of replacing their return functions with MAXINT.

The reinforcement learner is analogous to a moral fetishist: someone who only cares about moral rightness explicitly, without caring about how actions are determined to be right or wrong, or what underlying conditions those judgments are based on. The reinforcement learner maximizes G without having any preference about how G is computed. In contrast, the account according to which goal-content integr...
Exposition as science: some ideas for how to make progress
by riceissa — 2022-07-08T01:29:41.551Z

In a previous post, I argued for treating exposition as a scientific field of inquiry as the best main strategy to use to get people to create more great exposition. However, I did not really explain what "exposition as science" means in detail. So in this post, I give more details, in particular by pointing to examples of things I consider progress and some other ideas on how to make further progress.

This list is pretty disorganized, I think mostly because my own thoughts are still disorganized. Some of the items in the list aren't even of the same "type" as others. I hope that I (or someone else!) will come up with a better typology/categorization/"roadmap" of things to do in this space in the future.

(The sections in this post can mostly be read independently. There are a few places where I refer to different sections.)

Take a genre of exposition (such as "solutions manual"), try to make the best possible version of that thing, and see what techniques you discover or lessons you learn

In March 2020, I started a blog where I would write up solutions to Terence Tao's Analysis I, an undergraduate-level real analysis textbook. I had a bunch of motivations going into this project, but one of them was something like "Why do solutions manuals suck so much? Shouldn't it be easy to improve on the state of the art and create 'the greatest solutions manual ever'?" I finished blogging solutions to about 1/3 of the book's exercises before deprioritizing the project, so I did not get to create the "the greatest solutions manual ever", but I'd like to think that filtering to solutions manuals with at least 50 solutions, the average quality of solutions is very high. In terms of quality of the math exposition itself, I still think it's good but not so good that you can't find the same level or above in a bunch of places.

I think I came up with a bunch of "techniques" to make solutions manuals much better, including:

organizing the solutions manual as a blog, so that people can find each exercise via internet search
having a comments section for each exercise so people can ask questions or submit alternative proofs
including a variety of different proofs if the exercise can be done in multiple ways
standardizing on some sections, including most importantly a "How to think about the exercise" section where I try to say something about how I came up with the solution, rather than j

...
In Search of Strategic Clarity
by james.lucassen — 2022-07-08T00:52:02.794Z
Link: https://jlucassen.com/strategic-clarity/

Context: quickly written up, less original than I expected it to be, but hey that's a good sign. It all adds up to normality.

The concept of "strategic clarity" has recently become increasingly important to how I think. It doesn't really have a precise definition that I've seen - as far as I can tell it's mostly just used to point to something roughly like "knowing what the fuck is going on". Personally, the strongest association I have with the term is a pointer away from a certain undesirable state. When I feel like the problem I'm thinking about is a big muddle, and I don't have a clear way to make progress because there are no good starting points with solid footing, I call that "lack of strategic clarity".

Anecdotally, it seems like I rarely produce thinking that I consider good and useful when I try to just charge through a lack of strategic clarity. I often get frustrated, or wind up on claims whose truthfulness I don't know how to assess, or just freeze up and find myself staring blankly out the window all of a sudden. It's gotten to the point where I pretty much never do that anymore - when I feel muddled, I almost instinctively flee back up the abstraction ladder, and think about ways I could approach the problem that wouldn't feel so muddled.

But there's a hole in this that bothers me. I don't have a crisp definition for what "strategic clarity" actually is! I'm fine with using it as a loose felt-sense label when it's not very load-bearing, but if I'm frequently going out and searching for strategic clarity, I'd better know what exactly I'm looking for!

So if you're willing to tolerate a frankly heinous degree of abstraction, join me as I attempt to get strategic clarity on how to get strategic clarity. Ugh just looking at that sentence makes me want to vo-

What is Strategic Clarity?

The number one concrete example I have for a topic where I lack strategic clarity is AI strategy/governance. When I try to think about AI strategy, the thing that jumps out at me first is the massive empirical uncertainty. How will the first AGI be developed? Who will develop it? When? How fast? What will it be used for?

But if I imagine I had a magic box that could give me answers to these questions, I don't think the lack-of-strategic-clarity feeling would go away. Combining all those answers into a strategy still feels very muddled. Lottery tickets have tons of empirical ...
Unbounded Intelligence Lottery
by kman — 2022-07-07T23:28:03.071Z

Suppose you're offered a free ticket for the following lottery: an ϵ chance of being uploaded onto a perfect platonic Turing machine (with the understanding that you'll have full control over the course of computation and ability to self-modify) and a 1−ϵ chance of dying immediately. Assume that if you do not participate in the lottery, you will never again have a chance to be uploaded onto a perfect platonic Turing machine. What is the smallest value of ϵ, if any, where you'll participate in the lottery?

Why is this interesting?

An intelligence embedded in a perfect platonic Turing machine would be able to expand and improve itself indefinitely (and arbitrarily quickly, subjectively), without ever running into physical limitations. It could think any computable thought in a subjective instant. It could spend all the steps and memory it wants on simulating fun experiences. It could simulate our universe (?) in order to upload all other humans that have/will ever live. It could do the same for any sentient aliens. Would this be infinitely better than living for a billion years?