trentmkelly/LessWrong-43k huggingface dataset:
Cost, Not Sacrifice
by Joe Rogero — 2024-11-20T21:32:26.281Z
Link: https://subatomicarticles.com/cost-not-sacrifice/

In a recent bonus episode of the Bayesian Conspiracy podcast, Eneasz Brodski shared a thought experiment that caused no small amount of anguish. In the hypothetical, some eccentric but trustworthy entity is offering to give you an escalating amount of money for your fingers, starting at $10,000 for the first one and increasing 10x per finger up to $10 trillion for all of them.[1] On encountering this thought experiment, Eneasz felt (not without justification) that he mostly valued his manual dexterity more than wealth. Then, two acquaintances pointed out that one could use the $10 trillion to do a lot of good, and Eneasz proceeded to feel terrible about his decision.

I had several responses to this episode, but today I'm going to focus on one of them: the difference between cost and sacrifice.

How Ayn Rand Made Me a Better Altruist

But first, a personal anecdote. I was raised Catholic, and like the good Catholic boy that I was, I once viewed altruism through the lens of personal sacrifice. For the uninitiated, Catholic doctrine places a strong emphasis on this notion of sacrifice - an act of self-abnegation which places The Good firmly above one's own wants or needs. I felt obligated to help others because it was the Right Thing to Do, and I accepted that being a Good Person meant making personal sacrifices for the good of others, regardless of my own feelings. I divided my options into "selfish" and "selfless" categories, and felt guilty when choosing the former. Even as I grew older and my faith in Catholicism began to wane, this sense of moral duty persisted. It was a source of considerable burden and struggle, for me, made worse by the fact that the associated cultural baggage was so deeply ingrained as to be largely invisible to me.

Then, in a fittingly kabbalistic manner, Atlas Shrugged flipped my world upside down.[2]

Ayn Rand, you see, did not believe in sacrifice. In her philosophy, the only real moral duty is the duty to oneself and one's own principles. She happened to hold a great many other convictions about what those principles ought to be, some of which I now dispute; but in this, I believe, she was wholly correct.

My teenage self, at least, found this perspective incredibly freeing. (Perhaps a bit too freeing, as I've always been the sort of person who enjoys being smugly right about things, and taking the word "selfish" ...
China Hawks are Manufacturing an AI Arms Race
by garrison — 2024-11-20T18:17:51.958Z
Link: https://garrisonlovely.substack.com/p/china-hawks-are-manufacturing-an

This is the full text of a post from "The Obsolete Newsletter," a Substack that I write about the intersection of capitalism, geopolitics, and artificial intelligence. I’m a freelance journalist and the author of a forthcoming book called Obsolete: Power, Profit, and the Race for Machine Superintelligence. Consider subscribing to stay up to date with my work.

An influential congressional commission is calling for a militarized race to build superintelligent AI based on threadbare evidence

The US-China AI rivalry is entering a dangerous new phase.

Earlier today, the US-China Economic and Security Review Commission (USCC) released its

annual report

, with the following as its top recommendation:

Congress establish and fund a Manhattan Project-like program dedicated to racing to and acquiring an Artificial General Intelligence (AGI) capability. AGI is generally defined as systems that are as good as or better than human capabilities across all cognitive domains and would surpass the sharpest human minds at every task.

As someone

observed

on X, it’s telling that they didn’t call it an “Apollo Project.”

One of the USCC Commissioners, Jacob Helberg,

tells

Reuters that “China is racing towards AGI ... It's critical that we take them extremely seriously.”

But is China actually racing towards AGI? Big, if true!

The report clocks in at a cool 793 pages with 344 endnotes. Despite this length, there are only a handful of mentions of AGI, and all of them are in the sections

recommending

that the US race to build it.

In other words, there is no evidence in the report to support Helberg’s claim that "China is racing towards AGI.”

Nonetheless, his quote goes unchallenged into the 300-word Reuters story, which will be read far more than the 800-page document. It has the added gravitas of coming from one of the commissioners behind such a gargantuan report.

I’m not asserting that China is definitively NOT rushing to build AGI. But if there were solid evidence behind Helberg’s claim, why didn’t it make it into the report?

Helberg has not replied to a request for comment.

As the report notes, the CCP has

long expressed

a desire to lead the world in AI development. But that’s not the same thing as deliberately trying to build AGI, which could have profoundly destabilizing effects, even if we had a surefire way of aligning such a system with its ...
Why I Think All The Species Of Significantly Debated Consciousness Are Conscious And Suffer Intensely
by Bentham's Bulldog — 2024-11-20T16:48:44.859Z

Crosspost of this on my blog.

1 My basic view

'Cause everybody cries
Everybody hurts sometimes

Sometimes everything is wrong

—Rem “Everybody hurts”

You didn’t think the era of absurdly long posts was over, did you?

Prior to the 2024 election, there were a series of swing states that pundits forecasted could have gone either way. All of them ended up going to Trump by a fairly wide margin, as predicted by some people (e.g.

Glenn

,

Keith Rabois

). None of the states that were solidly blue went to Trump, but all the states that pundits and prediction markets weren’t sure about went red.

I have a similar bet regarding consciousness.

I suspect that all of the species that are currently of significantly debated consciousness—call them swing state species—are conscious. This would include crabs, lobsters, fish, and most insects, but it wouldn’t include oysters or AI. There’s fairly widespread agreement that such beings aren’t conscious. Oysters are like California.

Furthermore, I suspect that they all suffer fairly intensely—they don’t just have very basic slightly negative experience. This would include fish (though they are reaching the point where their suffering is beyond doubt), crustaceans (like shrimp, crabs, and lobsters), and insects.

Here is a thesis that seems decently intrinsically probable which I’ll refer to as the ubiquitous pain thesis: feeling pain is fairly ubiquitous across the animal kingdom, such that all the species of significantly debated sentience are sentient. I think, aside from the simplicity, plausibility, and naturalness of the thesis, there are five reasons to think it’s right that don’t require reviewing any complex behavioral evidence.

The first reason is evolutionary: it’s beneficial for a fish or insect to feel pain for the same reason it’s beneficial for you to feel pain; it encourages them to avoid dangers. If we hadn’t observed any creatures, we should expect decent odds that many creatures feel pain because of the obvious evolutionary reasons supporting pain.

Second, these creatures act in most ways like they’re in pain. If an insect or shrimp is exposed to damage, they’re struggle with great effort and try to get away. Either they are in pain or they evolved a response that makes them behave like they’re in pain. But if a creature struggles and tries to get away, a natural inference is that they’re in pain. This is why we’d natu...
aspirational leadership
by dhruvmethi — 2024-11-20T16:07:43.507Z

Crossposted from my blog.
I’m going to first describe a leader’s responsibility to their team because I don’t think it’s possible to develop good principles for working and communicating effectively with individuals without understanding the broader environment that the leader-individual relationship exists within.

Leaders have the following responsibilities towards their teams:

Identify your team's purpose — leaders must know their team's role in the success of the broader organization. A clear purpose helps the team adopt habits and actions essential for success, growth, and survival.
Identify the resources available to the team — this essentially is just the skills, talents, and attitudes of the individuals on the team, but it also includes material resources like tools, software, etc. The key recognition here is that individuals are, for lack of better words, dynamic resources - they perform at an extremely variable level depending on the context that they’re placed within. This means that leaders must not only learn individual talents but also the contexts in which those talents manifest most fully for everyone on the team.
Figure out how your team can best leverage the resources available to fulfill its purpose — this involves building a set of structures for the team that gives the team its best chance to fulfill its purpose properly given its resources.

The biggest challenge is that all of these are so dynamic. Unless you’re in a large and bureaucratic organization, things change all the time - reorganizations happen, people can join or leave, priorities shift, and so, so much more. It’s a constant challenge for leaders to stay on top of these changes and adapt the structures that their teams operate within.

The leader-individual relationship is extremely tough to manage within this context, especially given that the leader’s responsibility to the team and the organization as a whole must come ahead of their responsibility to each individual on their team[1]. Despite this challenge, there is a set of principles leaders can adopt to not only help individuals navigate the changes and continue contributing effectively but also bring the best out of individuals and help them grow:

Make them feel seen
Help them understand what is happening
Provide meaning to their work

make them feel seen

There are two critical questions that every effective leader must help individu...
Zvi’s Thoughts on His 2nd Round of SFF
by Zvi — 2024-11-20T13:40:08.092Z

Previously: Long-Term Charities: Apply For SFF Funding, Zvi’s Thoughts on SFF

There are lots of great charitable giving opportunities out there right now.

I recently had the opportunity to be a recommender in the Survival and Flourishing Fund for the second time. As a recommender, you evaluate the charities that apply and decide how worthwhile you think it would be to donate to each of them according to Jaan Tallinn’s charitable goals, and this is used to help distribute millions in donations from Jaan Tallinn and others.

The first time that I served as a recommender in the Survival and Flourishing Fund (SFF) was back in 2021. I wrote in detail about my experiences then. At the time, I did not see many great opportunities, and was able to give out as much money as I found good places to do so.

How the world has changed in three years.

This time I found an embarrassment of riches. Application quality was consistently higher, there were more than twice as many applications, and essentially everyone is looking to scale their operations and their spending.

Thus, this year there will be two posts.

This post contrasts between this experience and my first experience in 2021.

The other post will be an extensive list of charities that I believe should be considered for future donations, based on everything I know, including the information I gathered at SFF – if and only if your priorities and views line up with what they offer.

It will be a purely positive post, in that if I don’t have sufficiently net helpful things to say about a given charity, or I believe they wouldn’t want to be listed, I simply won’t say anything. I’ve tried to already reach out to everyone involved, but: If your charity was in SFF this round, and you either would prefer not to be in the post or you have new information we should consider or share, please contact me this week.

This first post will contain a summary of the process and stand on its own, but centrally it is a delta of my experiences versus those in 2021.

Table of Contents

How the S-Process Works in 2024.

Quickly, There’s No Time.

The Speculation Grant Filter.

Hits Based Giving and Measuring Success.

Fair Compensation.

Carpe Diem.

Our Little Corner of the World.

Well Well Well, If It Isn’t the Consequences of My Own Actions.

A Man’s Reach Should Exceed His Grasp.

Conclusion.

How the S-Process Works in 2024

Note that the sp...
A Little Depth Goes a Long Way: the Expressive Power of Log-Depth Transformers
by Bogdan Ionut Cirstea — 2024-11-20T11:48:14.170Z
Link: https://openreview.net/forum?id=zDze7VtB5C

Authors: Anonymous (I'm not one of them).

Abstract:

Most analysis of transformer expressivity treats the depth (number of layers) of a model as a fixed constant, and analyzes the kinds of problems such models can solve across inputs of unbounded length. In practice, however, the context length of a trained transformer model is bounded. Thus, a more pragmatic question is: What kinds of computation can a transformer perform on inputs of bounded length? We formalize this by studying highly uniform transformers where the depth can grow minimally with context length. In this regime, we show that transformers with depth O(log⁡C) can, in fact, compute solutions to two important problems for inputs bounded by some max context length C, namely simulating finite automata, which relates to the ability to track state, and graph connectivity, which underlies multi-step reasoning. Notably, both of these problems have previously been proven to be asymptotically beyond the reach of fixed depth transformers under standard complexity conjectures, yet empirically transformer models can successfully track state and perform multi-hop reasoning on short contexts. Our novel analysis thus explains how transformer models may rely on depth to feasibly solve problems up to bounded context that they cannot solve over long contexts. It makes actionable suggestions for practitioners as to how to minimally scale the depth of a transformer to support reasoning over long contexts, and also argues for dynamically unrolling depth as a more effective way of adding compute compared to increasing model dimension or adding a short chain of thought.
What changes should happen in the HHS?
by ChristianKl — 2024-11-20T11:04:01.778Z

Currently, it looks like Robert F. Kennedy Jr. will become HHS secretary and change a lot of things at the HHS. There will likely be both good and bad changes.

It's a time when the ideas that lay around matter. What changes do you think should be made? What clever policy ideas should be adopted?
What are the good rationality films?
by Ben Pace — 2024-11-20T06:04:56.757Z

I run a weekly sequences-reading meetup with some friends, and I want to add a film-component, where we watch films that have some tie-in to what we've read.

I got to talking with friends about what good rationality films there are. We had some ideas but I wanted to turn it to LessWrong to find out.

So please, submit your rationalist films! Then we can watch and discuss them :-)

Here are the rules for the thread.

Each answer should have 1 film.
Each answer should explain how the film ties in to rationality.

Optional extra: List some essays in the sequences that the film connects to. Yes, non-sequences posts by other rationalists like Scott Alexander and Robin Hanson are allowed.

Spoilers

If you are including spoilers for the film, use spoiler tags! Put >! at the start of the paragraph to cover the text, and people can hover-over if they want to read it, like so:

This is hidden text!
Valence Need Not Be Bounded; Utility Need Not Synthesize
by Lorec — 2024-11-20T01:37:20.911Z

As I relayed in my last post, in "Theory of Games and Economic Behavior", von Neumann and Morgenstern defended the validity of using utility to predict economic outcomes despite the fact that cross-individual utility cannot apparently be measured or apprehended directly:

"The [ . . . ] difficulties of the notion of utility, and particularly of the attempts to describe it as a number, are well known [ . . . ] It is sometimes claimed in economic literature that discussions of the notions of utility and preference are altogether unnecessary, since these are purely verbal definitions with no empirically observable consequences, i.e., entirely tautological. It does not seem to us that these notions are qualitatively inferior to certain well established and indispensable notions in physics, like force, mass, charge, etc. That is, while they are in their immediate form merely definitions, they become subject to empirical control through the theories which are built upon them" [von Neumann and Morgenstern, 8-9]

"[W]e wish to describe the fundamental concept of individual preferences by the use of a rather far-reaching notion of utility. Many economists will feel that we are assuming far too much [ by purporting to treat utility synthetically ], and that our standpoint is a retrogression from the more cautious modern technique of 'indifference curves' [ . . . ]" [von Neumann and Morgenstern, 16]

And they construct a theoretical framework for doing so:

"[A] numerical utility is dependent upon the possibility of comparing differences in utilities. This may seem - and indeed is - a more far-reaching assumption than that of a mere ability to state [subjective] preferences. [ . . . ]

Let us for the moment accept the picture of an individual whose system of preferences is all-embracing and complete, i.e. who, for any two objects or rather any two imagined events, possesses a clear intuition of preference.

More precisely we expect him, for any two alternative events which are put before him as possibilities, to be able to tell which of the two he prefers.

It is a very natural extension of this picture to permit such an individual to compare not only events, but even combinations of events with stated probabilities.

By a combination of two events we mean this: Let the two events be denoted by B and C [which are] 50%-50%. Then the "combination" is the prospect of seeing B occur with ...
Value/Utility: A History
by Lorec — 2024-11-19T23:01:39.167Z

[ I am writing this post because, while many people on LessWrong know something about the history of value and utility, it's vanishingly rare that someone is confident enough that they understand the whole thing, that they feel they can speak authoritatively about what these concepts 'really mean'. This often blocks important discussions. I had a faint memory suggesting that the canonical history is too short and understandable to really warrant this, so I looked up the whole thing, and as far as I can tell, that's true.

[One relevant Wikipedia section; I'm basically trying to write this up more tersely and straightforwardly and with less left up to assumed shared reader background.]

As a sometime hobbyist historian, I do wince publishing something so simplified. I know my picture of what the important events were would radically shift if I knew a little more. However, I think LessWrong working off this painfully simplified consensus would be a night-and-day improvement.

This post contains extensive quotes/cribbing from "Do Dice Play God?" by the mathematician Ian Stewart. IMO it's a great book on probability theory, although not mathematically sophisticated or ideologically Bayesian, because it gives the surprising historical/cultural motivations for a comprehensive breadth of "accepted procedures" in statistics usually taken at face value. [ Word of caution: Stewart conspicuously leaves out such figures as ET Jaynes, and Karl Friston [in his chapter about the brain as a Bayesian dynamical system!] suggesting he isn't aware of everything. ] ]

I. Cardano

The history books say: everybody was confused about chance until Girolamo Cardano, an Italian algebraist and gambler who wrote "Book on Games of Chance" in 1564 [ it was not published until 1663, long after he'd died ]. "At first sight [most Roman dice] look like cubes, but nine tenths of them have rectangular faces, not square ones. They lack [ . . . ] symmetry [ . . . ], so some numbers would have turned up more frequently than others. [ . . . ] So why didn't Roman gamblers object when they were asked to play with biased dice? [ . . . ] a belief in fate, rather than physics, might be the explanation. If you thought your destiny was in the hands of the gods, then you'd win when they wanted you to win and lose when they didn't. The shape of the dice would be irrelevant." [Stewart 26]

Cardano "used dice to illust...
Why Don't We Just... Shoggoth+Face+Paraphraser?
by Daniel Kokotajlo — 2024-11-19T20:53:52.084Z

Daniel Kokotajlo

Here's a fairly concrete AGI safety proposal:

Default AGI design: Let's suppose we are starting with a pretrained LLM 'base model' and then we are going to do a ton of additional RL ('agency training') to turn it into a general-purpose autonomous agent. So, during training it'll do lots of CoT of 'reasoning' (think like how o1 does it) and then it'll output some text that the user or some external interface sees (e.g. typing into a browser, or a chat window), and then maybe it'll get some external input (the user's reply, etc.) and then the process repeats many times, and then some process evaluates overall performance (by looking at the entire trajectory as well as the final result) and doles out reinforcement.

Proposal part 1: Shoggoth/Face Distinction: Instead of having one model undergo agency training, we have two copies of the base model work together, specializing in different parts of the job, and undergo training together. Specifically we have the "shoggoth" copy responsible for generating all the 'reasoning' or 'internal' CoT, and then we have the "face" copy responsible for the 'actions' or 'external' outputs. So e.g. in a conversation with the user, the Shoggoth would see the prompt and output a bunch of reasoning token CoT; the Face would see the prompt + all that CoT and then output a message to pass to the user, and then the process would repeat.
Proposal part 2: Blind the Evaluation Process to the Internal Reasoning: Remember that reward process that evaluates overall performance? Hide from it all the outputs of the Shoggoth. Just show it the outputs from the Face.
Proposal part 3: Use Paraphraser: One more complication. The outputs of the Shoggoth? Paraphrase them. For example, after each line of reasoning CoT is complete, pause the Shoggoth and have a different specialized paraphraser model generate 10 syntactically distinct but semantically equivalent ways of saying the same thing. Then pick one at random and replace the original line of reasoning with it. Then boot up the Shoggoth again and have it continue the CoT.
How it all (hopefully) fits together: First, because of the paraphraser, the Shoggoth will not learn to speak in some optimized alien language, or even to speak in jargon or euphemisms -- because semantics not recognized by the paraphraser will be scrambled by it. So the Shoggoth will continue to 'think

...