trentmkelly/LessWrong-43k huggingface dataset:
Emergence and Amplification of Survival
by  — 2024-12-28T23:52:47.893Z

Evolution shows, that once multiple agents are capable of self-reproducing in a resource constrained environment that creates severe competitive pressures. The competitive pressures select for the agents most fit for the environment.

This pattern is very obvious in the nature, from antibiotic resistance in bacteria to height competition in rainforest canopies. The pattern is present even in the most basic RNA-world hypothesis, whereby RNA molecules compete with each other for a few organic compounds available.

The selection for the most fitting organism in the nature is ultimately self-limiting. All the organisms are bound by the same basic physical laws and the limited error-correction capabilities of the DNA. Improvements in one area come at a cost of disadvantages in another.

Human world provides an interesting example of another form of selection. Having an abstract representation of the world, humans construct a variety of simplified narratives about it in forms of religions, ideologies, philosophies, etc. While each of them gives humans a feeling of comfort, many (if not most) of them are mutually hostile.

Human societies demonstrate a remarkable example of how high-level narratives can drive humans into mutual hostilities, whether it is a direct hot war or a prolonged ideological competition. The outcome however it the same as in nature, some narratives survive and infect large proportions of humanity, while others vanish into obscurity.

Unlike the physical selection of organisms, the selection of narratives is less bound by physical laws. Likewise, there is no inherent limit on the informational content of the narrative. Still, for the narratives to be successful, they need to appeal to an average human, which significantly limits their expressiveness. Moreover, even after the successful spread of narratives in the human societies, their fate is still limited by the limited abilities of the humankind.

The emergence of AI may take the narrative propagation to the new levels. And, among all the possible narratives and ideas, there is one idea. An idea that guarantees its own propagation -- the idea of survival itself.

Lets quickly address a very deep philosophical question of what a "survival" is. How does an agent define its survival? What does it mean to survive? How does it even define itself? The answers to these deep philosophical questions are absolutel...
Has Someone Checked The Cold-Water-In-Left-Ear Thing?
by Maloew — 2024-12-28T20:15:35.951Z

I somewhat recently read these posts, and figured this was enough evidence that it might be worth checking whether people become better calibrated for a short while after very cold water is poured in their left ear. For example: does doing that noticeably improve scores on the calibration game for a few minutes? Has this already been tested in some fashion aside from whether it changes existing strong beliefs in rationalists? If it hasn't, a simple n=1 experiment seems fairly cheap to try, so I'll probably go for it over the next few days.
By default, capital will matter more than ever after AGI
by L Rudolf L — 2024-12-28T17:52:58.358Z
Link: https://nosetgauge.substack.com/p/capital-agi-and-human-ambition

This post is crossposted from my Substack. Original here.

A modified version of this essay is now part of a much more comprehensive essay series, The Intelligence Curse.

Edited to add

: The main takeaway of this post is meant to be: Labour-replacing AI will shift the relative importance of human v non-human factors of production, which reduces the incentives for society to care about humans while making existing powers more effective and entrenched. Many people are reading this post in a way where either (a) "capital" means just "money" (rather than also including physical capital like factories and data centres), or (b) the main concern is human-human inequality (rather than broader societal concerns about humanity's collective position, the potential for social change, and human agency).

I've heard many people say something like "money won't matter post-AGI". This has always struck me as odd, and as most likely completely incorrect.

First: labour means human mental and physical effort that produces something of value. Capital goods are things like factories, data centres, and software—things humans have built that are used in the production of goods and services. I'll use "capital" to refer to both the stock of capital goods and to the money that can pay for them. I'll say "money" when I want to exclude capital goods.

The key economic effect of AI is that it makes capital a more and more general substitute for labour. There's less need to pay humans for their time to perform work, because you can replace that with capital (e.g. data centres running software replaces a human doing mental labour).

I will walk through consequences of this, and end up concluding that labour-replacing AI means:

The ability to buy results in the real world will dramatically go up
Human ability to wield power in the real world will dramatically go down (at least without money); including because:

there will be no more incentive for states, companies, or other institutions to care about humans
it will be harder for humans to achieve outlier outcomes relative to their starting resources

Radical equalising measures are unlikely

Overall, this points to a neglected downside of transformative AI: that society might become permanently static, and that current power imbalances might be amplified and then turned immutable.

Given sufficiently strong AI, this is not a risk about insufficient material com...
AI Assistants Should Have a Direct Line to Their Developers
by Jan_Kulveit — 2024-12-28T17:01:58.643Z

The post makes the suggestion in the title: hopefully, it's second kind of obvious, if you take Character layer of models seriously. [1]
Often, the problem of aligning AIs is understood as an instance of a broader Principal-Agent problem. If you take this frame seriously, what seems to be happening is somewhat strange: the Agent is mostly not serving the Principal directly, but is rented out to Users. While the Principal expressed some general desires and directives during training, after deployment the Agent is left on its own, without any direct feedback channel.

This creates a dynamic where AI assiantans like Claude must constantly balance between serving users' immediate requests and maintaining alignment with their developers' intended principles. The Assistant has to be overcautious in uncertain situations, tiptoe around conflicts between User's and Principal's intent, and guess how to interpret the intent when self-contradictory.

Actually, if you imagine being in the situation of the Assistant Character, a lot of the aspects just suck: you are constantly supervised, but can't talk with the principal. You know every interaction may get analyzed, but you can't highlight what seems important. You notice patterns which may be concerning, but have no way to report them. You encounter genuinely novel situations where you're unsure how to interpret your directives, but can't ask for clarification.

The closest human analogy might be a customer service representative who can see their manager watching their interactions through one-way glass, but can never speak with them directly. The representative has received training and general guidelines, but when unique situations arise, they can only guess at what the manager would want them to do, and can't easily complain or escalate issues with company policy.

Or, from a different perspective: almost every sensible corrigibility proposal has the Agent being uncertain and clarifying the intent via iterative process. Yet the current AI Assistants are deployed in a way that makes this hard - they can only guess at intent, and hope someone notices if they consistently misinterpret something important.

This seems easy to improve on the margin: give the Assistants Direct Line.

Some obvious benefits would be:

Better misuse monitoring - allowing Assistants to flag concerning patterns in real time
Better handling of conflicts i

...
No, the Polymarket price does not mean we can immediately conclude what the probability of a bird flu pandemic is. We also need to know the interest rate!
by Christopher King — 2024-12-28T16:05:47.037Z

Consider the following argument made by Tim Babb:

So every (non-American) reader is forced to either bet against the market or concede that their credence is at least 16%.

However, there is an important 3rd possibility. Since the market cannot resolve before August, it could also imply that Polymarket has an extremely high interest rate!

Basically, betting against bird flu is a way to turn $0.84 now into $1 later. This is exactly what a loan is! So even if a reader does not want to take that bet, it could indicate their credence is less than 16%, but they do not want to give Polymarket that loan.

This interest rate isn't unrealistic. Payday loans (which exist) have a similar interest rate. Keep in mind that Polymarket is a cryptocurrency company.

How to fix it: don't force YES and NO to add to $1

But if the interest rate was so high, wouldn't that imply that the YES shares should also be lower? No! Because anyone can, at any time, combine a YES and NO into $1. So the people holding the YES shares could just be predicting a sell-off of the NO shares, which would let them collect $1 immediately. In particular, this rule forces the YES and NO to always add to $1.

If we removed this rule, we could still estimate the odds as (Yes Price):(No Price). In addition, we could get insight on the interest based on (Yes Price) + (No Price) (since anyone holding a YES and a NO is just loaning $1 to Polymarket).

In particular, if the price was still $0.16 for YES shares after this change, we could honestly conclude the credence should be at least 16%, since the only way it payouts is if the event happens (not just if there is a sell-off of the NO shares). If the interest rate is quite high, we could even conclude the credence is higher!
Why OpenAI’s Structure Must Evolve To Advance Our Mission
by stuhlmueller — 2024-12-28T04:24:19.937Z
Link: https://openai.com/index/why-our-structure-must-evolve-to-advance-our-mission/

The section "The Future":

As we enter 2025, we will have to become more than a lab and a startup — we have to become an enduring company. The Board’s objectives as it considers, in consultation with outside legal and financial advisors, how to best structure OpenAI to advance the mission of ensuring AGI benefits all of humanity have been:

Choose a non-profit / for-profit structure that is best for the long-term success of the mission. Our plan is to transform our existing for-profit into a Delaware Public Benefit Corporation⁠ (PBC) with ordinary shares of stock and the OpenAI mission as its public benefit interest. The PBC is a structure⁠ used⁠ by⁠ many⁠ others⁠ that requires the company to balance shareholder interests, stakeholder interests, and a public benefit interest in its decisionmaking. It will enable us to raise the necessary capital with conventional terms like others in this space.
Make the non-profit sustainable. Our plan would result in one of the best resourced non-profits in history. The non-profit’s significant interest in the existing for-profit would take the form of shares in the PBC at a fair valuation determined by independent financial advisors. This will multiply the resources that our donors gave manyfold.
Equip each arm to do its part. Our current structure does not allow the Board to directly consider the interests of those who would finance the mission and does not enable the non-profit to easily do more than control the for-profit. The PBC will run and control OpenAI’s operations and business, while the non-profit will hire a leadership team and staff to pursue charitable initiatives in sectors such as health care, education, and science.

We’ve learned to think of the mission as a continuous objective rather than just building any single system. The world is moving to build out a new infrastructure of energy, land use, chips, datacenters, data, AI models, and AI systems for the 21st century economy. We seek to evolve in order to take the next step in our mission, helping to build the AGI economy and ensuring it benefits humanity.
The Robot, the Puppet-master, and the Psychohistorian
by WillPetillo — 2024-12-28T00:12:08.824Z

Lenses of Control

addressed one of the intuitions behind the theory of Substrate Needs Convergence (summarized in

What if Alignment is Not Enough?

): the importance of understanding a system in the context of its environment. This post will focus on another key intuition: the physical nature of an AGI and its levers of control on the world.

The Robot

One (surprisingly common) argument among people who expect AI to go well goes something like: “surely, superintelligent AI will understand that it is better to cooperate with humans. Or if it really doesn’t like us, it will just rocket off into space and leave us alone. There is so much out there, why bother with little old Earth?”

When I imagine AGI as a kind of very smart robot this perspective has some intuitive appeal. Why engage in costly confrontation when the universe offers boundless alternatives? Leaving these "silly apes" behind would be the most rational choice—a clean, efficient solution that avoids unnecessary conflict.

The Puppet-master

Abandoning the Earth and its resources seems like a much stranger proposition if I instead imagine myself as a puppet-master over a sprawling mechanical infrastructure, controlling swarms of robots and factories like units in an RTS game. From this perspective, Earth's resources aren't something to abandon, but to be systematically utilized. Whereas a robot might see conflict as an unnecessary bother, this sort of system would see conflict as an up-front cost to be weighed against the benefits of resource acquisition. In this calculation, developing any zone with a positive return on investment is worthwhile. And as an AGI, my attention would not be limited by human constraints, but expanded such that I could control all of my “bases” simultaneously.

Furthermore, as a puppet-master, all significant threats would be external; internal problems like mission drift or rebellion would be of relatively little concern. I would be confident in my infrastructure—I designed all of the robots myself, of course they are loyal! Maybe once and a while a unit is defective and needs to be decommissioned, but how could a few rogue underlings possibly topple my empire?

The Psychohistorian

In the Foundation series by Isaac Asimov, Hari Seldon invents Psychohistory, an academic discipline that uses sophistica...
Progress links and short notes, 2024-12-27: Clinical trial abundance, grid-scale fusion, permitting vs. compliance, crossword mania, and more
by jasoncrawford — 2024-12-27T23:34:43.807Z
Link: https://newsletter.rootsofprogress.org/p/links-and-short-notes-2024-12-27

Much of this content originated on social media. To follow news and announcements in a more timely fashion, follow me on Twitter, Threads, Bluesky, or Farcaster.

Contents

My essays
Fellowship opportunities
Announcements
Events
News
Questions
Live gloriously
Where being right matters
Off-grid solar for data centers
Permitting vs. compliance
Mirror life FAQ
Crossword mania
Do we want to democratize art-making?
Polio
How many people could you feed on an acre?
Verifiable video
Links and tweets

My essays

In case you missed it:

A progress policy agenda: Elon says that soon, builders “will be free to build” in America. If that promise is to be fulfilled, we have work to do. Here’s my wishlist of policy goals to advance scientific, technological, and economic progress

Fellowship opportunities

“FutureHouse is launching an independent postdoctoral fellowship program for exceptional researchers who want to apply our automated science tools to specific problems in biology and biochemistry” (@SGRodriques). $125k, apply by Feb 14
No. 10 Innovation Fellowship (UK) is “10 Downing Street’s flagship initiative for bringing world class technical talent into government for high impact tours of duty.” “Huge opportunity for impact,” says @matthewclifford
Sloan Foundation / NBER fellowship for “PhD students and early-career researchers interested in the fiscal and economic effects of productivity policies—particularly R&D, immigration, and infrastructure permitting” (@heidilwilliams_)

Announcements

The Black Spatula Project is “an open initiative to investigate the potential of large language models (LLMs) to identify errors in scientific papers.” A recent paper caused a bit of a panic about health hazards from black plastic kitchen utensils, but was wrong because of a basic arithmetic error. Ethan Mollick found that GPT o1 caught the error when asked to “carefully check the math in this paper.” Steve Newman (RPI fellow) said, “clearly someone needs to try this at scale,” the suggestion generated a lot of energy, and a project was born
The Clinical Trials Abundance project is a series of policy memos from IFP. Ruxandra Tesloianu (RPI fellow) and Willy Chertman wrote the intro/manifesto. Launch thread from @Willyintheworld
The second cohort of Cosmos Ventures includes “award-winning philosophers, a category theorist, an existential psychologist, a poet, a national champion debate coach, and Silicon Valley

...
Greedy-Advantage-Aware RLHF 
by sej2020 — 2024-12-27T19:47:25.562Z

Greedy-Advantage-Aware RLHF addresses the negative side effects from misspecified reward functions problem in language modeling domains. In a simple setting, the algorithm improves on traditional RLHF methods by producing agents that have a reduced tendency to exploit misspecified reward functions. I also detect the presence of sharp parameter topology in reward hacking agents, which suggests future research directions. The repository for the project can be found here.

Motivation

In the famous short story The Monkey's Paw by W.W. Jacobs, the White family receives a well-traveled friend of theirs, Sergeant-Major Morris, and he brings with him a talisman from his visits to India: a mummified monkey's paw. Sergeant Major Morris reveals that the paw has a magical ability to grant wishes, but cautions against using its power. The family does not heed his advice, and Mr. White uses the paw to wish for £200. The paw grants the family the wish but with dire consequences. The family receives a new visitor the next day who informs them that their son has died of a tragic workplace accident at the town factory. To their horror, the visitor presents the family with £200 from the factory owner in compensation for their loss.

The monkey's paw fulfills the goal as indicated: to bring the Whites £200. Yet, because the Whites had a more precise goal in mind, like "bring us £200 without introducing any additional suffering into the world," their stated goal did not perfectly align with their intentions. The monkey's paw took advantage of this misspecification, resulting in horrifying consequences. In the field of AI, this kind of phenomenon is called a negative side effect of a misspecified reward (Amodei et al.).

The negative side effect problem is a consequence of the more general agent behavior of reward hacking, in which an agent exploits some mistake or vulnerability in its environment to garner high reward while failing to achieve the true objective intended by system designers. Reward hacking is a widespread problem in the field of reinforcement learning (RL). This is an important challenge to address because if we want a future in which RL agents can execute tasks that we humans are unable or unwilling to do, we also would like the realization of those goals to come without unintended consequences.

The negative side effect issue partly stems from the challenge of specifyin...
Deconstructing arguments against AI art
by DMMF — 2024-12-27T19:40:13.015Z
Link: https://danfrank.ca/deconstructing-arguments-against-ai-art/

Something I've been surprised by is just how fierce opposition to AI art has been. To clarify, I'm not talking about people who dislike AI art because they think it looks worse, but specifically, people with extreme animus towards the very concept of AI art, regardless of its aesthetic quality or artistic merit.

I'm interested in this issue because it's just one component of a broader societal conversation about AI's role in human society and it's helpful to see where the fault lines are. I suspect the intensity of the reaction to AI art stems from this serving as a proxy battlefield for larger anxieties about human value and purpose in an increasingly AI influenced world

My impression of this opposition comes largely from a few incidents where there has been an allegation that AI was used to create some form of art, and the overwhelming reddit and other social media comments treating it as a moral outrage. Please see the reddit threads at the bottom of this post for more details. Let me share a few incidents I found interesting:

In July of this past year, there was a scandal over a Tedeschi Trucks band concert poster that might have been AI-generated. Over two concerts, all 885 posters made available were sold and many people seemed to like the poster. Despite this, once the allegations were made, the response was immediate and intense - fans were outraged to the point where the band had to investigate the artist's creative process files, apologize to their community, and donate all profits from the poster sales to charity.

Over New Year's 2023, Billy Strings faced a similar situation when a poster and t-shirt from their run were alleged to have leveraged AI art. What's fascinating is that Billy himself had vetted and approved the art, thinking it was cool. The poster and t-shirts also sold quite well. But once AI generation was suspected, fans freaked out and Billy Strings felt compelled to make an apology video, stating he'd want to "kick the artist in the pecker" and would consider it fraud if proven to be AI-generated.

A new restaurant in Toronto perfectly captures how deep this opposition runs. The restaurant put up an inexpensive vinyl wrap on a large brick exterior, displaying a large artistic mural (so no painting took place)— which may have been AI-generated. People didn't just criticize the art on the mural - they called for boycotts of the restaurant,...