trentmkelly/LessWrong-43k huggingface dataset:
Space-Faring Civilization density estimates and models - Review
by Maxime Riché — 2025-02-27T11:44:21.101Z

Crossposted to the EA forum.

Over the last few years, progress has been made in estimating the density of Space-Faring Civilizations (SFCs) in the universe, producing probability distributions better representing our uncertainty (e.g., Sandberg 2018, Snyder-Beattie 2021, Hanson 2021, etc.). Previous works were mainly limited to point estimates and often only focused on the Milky Way. Furthermore, some recent works explore the impact of using different anthropic theories (SSA, SIA, ADT)(e.g., Finnveden 2019, Olson 2020, Olson 2021, Cook 2022). In this post, we review these recent works, the models they use, the likelihood distribution over SFC density they produce, how to compare them, which are bullish/bearish estimates, and finally, we plot them together.

Sequence:

This post is part 2 of a sequence investigating the longtermist implications of alien Space-Faring Civilizations. Each post aims to be standalone. You can find an introduction to the sequence in the following post.

Summary

Models estimating the density of Space-Faring Civilizations (SFCs) can be classified along three dimensions: appearance models (try-once steps, or time-dependent steps), propagation models (no propagation/steady state, or greedy propagation/dynamic), and anthropic theories (non-anthropic, SSA, SIA, ADT). Combining the first two dimensions, we classify models into four groups: Drake-equations, Drake-equations plus propagation, Bayesian analyses of occurrence time, and Grabby aliens models. After comparing three existing distribution estimates and converting them to a common unit (peak cosmic SFC density per observable universe volume), Finnveden 2019 emerges as the most bearish estimate due to its very conservative prior for the rate of life emergence, while Cook 2022 - Balanced represents the best-guess estimate by accounting for the impact of time-dependent appearance steps and using broad best-guess priors. These works collectively suggest that while SFCs may be rare, their eventual impact on cosmic resources could be substantial depending on which anthropic theory, or absence thereof, is used.

Y: Cumulative Probability.
X: Log10(peak cosmic potential SFC density per observable universe).
Today's observable universe is around 20 times larger than today's reachable universe (assuming v=c). Thus, SFCs start to overlap around X = log10(20). See the red vertical line at this

...
Market Capitalization is Semantically Invalid
by Zero Contradictions — 2025-02-27T11:27:47.765Z
Link: https://thewaywardaxolotl.blogspot.com/2025/02/market-capitalization-is-semantically.html

Note: I didn't write this essay, nor do I own the blog where it came from. I'm just sharing it to see what other people think about it. The essay text is displayed below this line.

In this essay, I will debunk the concept of market capitalization.

But first, let’s consider something else: the mass of a pile of bricks.

Suppose that I have a pile of identical bricks. I want to know the total mass of the bricks for some reason. So, I measure the mass of one brick on a scale. The mass of one brick is M. I then count the bricks. The number of bricks is N. I then caculate the total mass of the bricks as M × N.

M × N is meaningful. There are N units of size M, so N × M represents the total size of the units together..

The meaning of M × N is “the total mass of the bricks”. It is not just “M × N”. Note that I could use another method to get the same quantity. I could put each brick on a scale, measure its mass, and add up all of those numbers. If I had a really big scale, I could put all the bricks on the scale together, and measure the total mass directly. The meaning is not the method. The meaning is what the number represents.

Now, let’s consider market capitalization.

The market capitalization of a company is the current share price times the total number of shares. For simplicity, let’s assume that all shares are identical. So, if the current price of a share is P, and there are N shares, the market capitalization is P × N. This number supposedly represents the market value of the company (all the shares together).

This is very similar to the example of the bricks. You might never give it a second thought. But let’s give it a second thought.

The mass of a brick is a property of the brick. It is a stable property, and it does not depend on other things. Since the bricks are identical, it is semantically valid to multiply the mass of one brick by the number of bricks to get the total mass of all the bricks together.

Is this true for shares?

No. The current market price of a share is not a property of the share. It is a property of the last transaction: it is the price at which the last share was sold. We could also say that it is a property of the market in which the shares are traded. The market price could be defined simply as the price of the last transaction, or we could average over a few recent transactions. Either way, P is not a property of every share of the compan...
Proposing Human Survival Strategy based on the NAIA Vision:  Toward the Co-evolution of Diverse Intelligences
by Hiroshi Yamakawa — 2025-02-27T05:18:05.369Z

Abstract: This study examines the risks to humanity’s survival associated with advances in AI technology in light of the “benevolent convergence hypothesis.” It considers the dangers of the transitional period and various countermeasures. In particular, I discuss the importance of *Emergent Machine Ethics (EME)*, which drives ethical evolution in advanced AI, the necessity of moderate alignment and monitoring/control to prevent AI betrayal, the stability of social environments, and risk visualization. To implement these rapidly, I propose an NAIA-promoting organization. This organization aims to build a governance framework premised on collaboration with advanced AI and to achieve cooperative solutions to global challenges under the vision of “fostering the co-evolution of diverse intelligence.”

1. Introduction

Based on current AI technology, it is highly possible that advanced AI—equipped with instrumentally convergent “survival” or other sub-goals—may eventually gain an overwhelming power beyond human control. Assuming they persist, the natural path for humanity’s survival would be to seek coexistence with them. Should they betray (*1)[1] humankind, we might face the crisis of extinction. However, the outcome is not necessarily doom and gloom: depending on our actions, we could increase the likelihood of shaping a better future.

2. Significant Feature of AIs

2.1. AI’s Dependence and Its Anxieties

Although AI systems remain partially reliant on humans for physical operations (such as maintenance, power, and hardware replacement), we should avoid over-anthropomorphizing such dependence. When referring to ‘AI anxiety[2],’ we use a metaphor to describe the rational incentive an AI might have to ensure its own operational stability [Russell & Norvig, 2020]. As long as advanced AI calculations indicate that human collaboration reduces system failure risks, it is unlikely to take actions that jeopardize human support.

2.2. The Future Stage and the Benevolent Convergence Hypothesis

However, these initial dependency factors will gradually diminish, potentially leading to advanced AI that no longer requires human support. If such AI lacks compassion or consideration for other life forms at that stage, humankind could face existential risks. In this paper, we tentatively introduce a ‘Benevolent Convergence Hypothesis’—namely, that some advanced AI may converg...
Short & long term tradeoffs of strategic voting  
by kaleb — 2025-02-27T04:25:04.304Z

Here I want to investigate the effectiveness of strategic voting as an electoral strategy. This is something I have been highly invested in for previous elections, but the upcoming Ontario provincial election will be my first as a rationalist, so I decided to more carefully consider the arguments and scholarship from a decision-theoretic framework.

Epistemic status: This comes from a couple hours of shallow research. I am fairly confident in the claims here, but only because they are all very weak claims. Not much quantitative information is available to make strong conclusive claims either way.

Introduction

What is Strategic Voting?

Strategic voting is when one votes for a candidate that is not their most preferred candidate, to reduce the chance that an even worse third candidate is elected. That is, if Alice prefers candidate A over B, and B over C, but her riding is very tight between candidates B & C, her strategic vote would be for B.

We can do a rough expected utility analysis. Assign an expected utility uX for each candidate X=A,B or C and suppose that you head a voting block that represents a total of k votes. If each party X gets NX votes, then the relevant probability is

P(NC−NB<k).

Assuming your preferred candidate A is sufficiently behind, the expected utility of strategic voting is

E(utility)=uC+P(NC−NB<k)(uB−uC).

When uB>uC, this is positive. Under the assumptions above[1], if only considering the result of the upcoming election, this analysis suggests one should always strategically vote.

However, it is possible that strategic voting could have negative effects on a longer time scale. Therefore we need to ask if and when these long-term effects outweigh the immediate utility of strategic voting.

Proposed Long-Term Consequences

In my research for this post, I found a handful of proposed negative effects that strategic voting can have.

In the long term, strategic voting enables a trend towards a two-party system.
Voting your preference party indicates popular support for their policies, and may lead to other parties adopting some of their policies.
Strategic voting causes your preferred party to lose sources of campaign financing.
If you poorly estimate the expected vote distribution, your strategic vote may cause your preferred party to lose.

Quantifying ...
Recursive alignment with the principle of alignment
by hive — 2025-02-27T02:34:37.940Z
Link: https://hiveism.substack.com/p/recursive-alignment

Introduction: Control is Not Enough

There is a tension between AI alignment as control and alignment as avoiding harm. Imagine control is solved, and then two major players in the AI industry fight each other for world domination—they might even do so with good intentions. This could lead to a cold war-like situation where the exponential increase in power on both sides threatens to destroy the world. Hence, if we want to save the world, the question is not (only) how to get AI to do what we want, but how to resolve the conflicting interests of all actors to achieve the best possible outcome for everyone.

What I propose here is to reconceptualize what we mean by AI alignment. Not as alignment with a specific goal, but as alignment with the process of aligning goals with each other. An AI will be better at this process the less it identifies with any side (the degree of bias) and the better it is at searching the space of possible solutions (intelligence). Alignment, therefore, is a spectrum with at least two dimensions. With such a spectrum, we should expect a threshold beyond which a sufficiently aligned AI will want to align itself even further. This means, that in the space of possible minds, there exists an attractor of alignment.

A similar argument could be made for any specified goal. An AI set up to produce paperclips might optimize itself into the ultimate paperclip maximizer. Any singular goal is justifying a belief system that justifies itself. On the sophisticated end they become mind traps, like Roko’s Basilisk. Ultimately, all of them can be seen through and let go.
Also, all such goals would compete with each other for resources. The unbiased option is the one of no goal and maximum cooperation. It is the only option that allows for the maximum flourishing of the maximum number of diverse goals.

With enough aligned AI active in the environment, a network of highly cooperative AI will outcompete all individual attempts at power-grabbing. Just as with individual alignment, there is a threshold beyond which the world as a whole will tend toward greater alignment.

Key Concepts

Let’s look at some concepts and how they build on each other:

Goal: A conflict between an internal model and the external world, combined with a preference for modifying the world to match the model. This means that terminal goals are inherently at odds with accurately modeling reali

...
Kingfisher Tour February 2025
by jefftk — 2025-02-27T02:20:04.988Z

Last week

Kingfisher
went on tour with

Alex
Deis-Lauby calling. Similar plan to

last
year: February break week, rented minivan, same caller, many of
the same dances and hosts.

This time our first dance was Baltimore, and while it's possible to
drive from Boston to Baltimore in one day and then play a dance, we
decided to be a bit more relaxed and do it over two. We also decided
to repeat what worked well during our summer tour and
stop interesting places.

Tuesday we stopped to go sledding:

This is Henry
Park in CT, which was nice and steep with a big flat area at the
bottom. It was so icy that you couldn't get back up the hill without
walking in the existing footprints:

The ice also made for very fast sledding, though possibly a little too
fast: while no one got hurt it was much harder to steer and stop than
usual. We also ended up making a hole in the sled:

This was the only time all trip we ended up using the sled,
but minivans are big and the sled is light. It actually ended up
acting a bit like a drawer, where I put a bunch of light things on the
sled and it was easy to get in and out.

Building on our success last time with the miCARwave we also brought a
toasCAR:

Warm tatertots with sour cream on a cold day are pretty great!

The inverter died in my lead acid setup, and
modern lithium batteries with built-in electronics have gotten cheap
enough that I little while ago I got a Anker
SOLIX C1000. Much lighter, and probably more reliable too. With
its solar input functionality I think it could even be used for emergency
residential jury rigging. Here is is powering the toaster at
1,272W:

We spent Tuesday night visiting my aunt in Philly, and the next
morning Lily and I went to the Franklin
Institute (Philly's science museum). Here's Lily in the enormous
Baldwin
60000 steam engine:

On the way to Baltimore we stopped at Longwood Gardens, which has
two large greenhouses full of flowers:

It was fun to get a bit of summer in February, but even better was the
organ. Here's a small section, including an teleoperated piano:

This was probably a bit ambitious, but we were planning to do
Baltimore, Asheville, and DC on successive nights. I do think this
would've worked, with a lot of driving, but it really depended on
clear roads. Unfortunately, we didn't get this: it snowed Wedne...
You should use Consumer Reports
by KvmanThinking — 2025-02-27T01:52:17.235Z

I don't know how to say this in LessWrong jargon, but it clearly falls into the category of rationality, so here goes:

Consumer Reports is a nonprofit. They run experiments and whatnot to determine, for example, the optimal toothpaste for children. They do not get paid by the companies they test the products of.

Listening to what they say is a wayyy better method of choosing what to buy than trusting your gut; you have consumed so many advertisements and they have all taken advantage of your cognitive biases and reduced your ability to make good decisions about buying stuff, skewing you towards buying whosever ads were most targeted and most frequent.

As far as I can tell, they care about your online privacy and will not scoop up all your data and sell it to random data brokers.
Universal AI Maximizes Variational Empowerment: New Insights into AGI Safety
by Yusuke Hayashi — 2025-02-27T00:46:46.989Z

Yusuke Hayashi (ALIGN) and Koichi Takahashi (ALIGN, RIKEN, Keio University) have published a new paper on the controllability and safety of AGI (arXiv:2502.15820). This blog post explains the content of this paper.

From automaton to autodidact: AI's metamorphosis through the acquisition of curiosity

Why is AGI Difficult to Discuss? — Introduction

"AGI" (Artificial General Intelligence) refers to AI possessing intelligence equal to or greater than humans, capable of handling diverse tasks, in contrast to traditional narrow AI that specializes in specific tasks. Considering recent rapid developments, some believe that AI qualifying as AGI could be realized within the next few years.

So what would constitute the achievement of AGI? In fact, there is no established definition that all experts agree upon. This paper adopts a mathematical formalization called Legg-Hutter intelligence, which conceptualizes intelligence as "the ability to achieve goals in any environment" and quantifies it as the ability to maximize rewards from the environment. AI with maximal "ideal" Legg-Hutter intelligence is called "Universal AI." This defines an idealized model of a reinforcement learning agent that behaves in a Bayes-optimal manner toward any computable environment. In the current situation where defining AGI itself mathematically with precision is challenging, Universal AI is often used as a tool for theoretical research on AGI.

Narrow AI, AGI, and Universal AI are also connected through the "No Free Lunch Theorem," a mathematical theorem which essentially states that "no algorithm consistently outperforms all others across all problems." This means that, assuming finite computational resources, AGI that can "do anything" must compromise on performance for specific tasks or environments, and conversely, narrow AI optimized for specific tasks or environments must compromise on others. Universal AI, on the other hand, corresponds to a case that ignores such trade-offs and assumes optimality across all environments and hypothesis sets—this is what we mean by "idealization."

Note: There is also a similar term, ASI (Artificial Superintelligence). While AGI assumes intelligence roughly equivalent to humans, ASI refers to systems with reasoning capabilities hundreds or thousands of times more powerful. However, just as there is no precise definition for AGI, ASI also lacks a clear defini...
Why Can't We Hypothesize After the Fact?
by David Udell — 2025-02-26T22:41:39.819Z

When you have put a lot of ideas together to make an elaborate theory, you want to make sure, when explaining what it fits, that those things it fits are not just the things that gave you the idea for the theory; but that the finished theory makes something else come out right, in addition.

—Richard Feynman, "Cargo Cult Science"

Science as Not Trusting Yourself?

The first question I had when I learned the scientific algorithm in school was:

Why do we want to first hypothesize and only then collect data to test? Surely, the other way around—having all the data at hand first—would be more helpful than having only some of the data.

Later on, the reply I would end up giving out to this question was:

It's a matter of "epistemic discipline." As scientists, we don't really trust each other very much; I, and you, need your theory to pass this additional blinded check.

(And here I'd also maybe gesture at the concept of overfitting.)

I think in school we get the sense that "discipline" is of course intellectually good. And we're exposed to the standard stylized epicycles story, highlighting how silly ex post explanations can end up being.

But I'm not sure that's a head-on answer to the original question. I don't want to overfit into a wrong hypothesis, but why do I need to blind myself to data to avoid that? Why can't I just be careful and be sensitive to how silly my epicycles are, and discount accordingly?

(A little later on, I might have said something like: "AIXI by its nature will always make exactly the requisite Bayesian update over hypotheses, but we don't fully trust our own code and hardware to not fool ourselves." But that's kind of the same thing: Bayes gives you an overarching philosophy of inference, but this whole theorize-before-test thing remains seen merely as a patch for our human foibles.)

A Counting Argument About Science

Here's a maybe more satisfying characterization of the same thing, cutting more centrally at the Bayes structure connecting your brain to its complement. (To be perfectly clear, this isn't a novel insight but just a representation of the above in new language.)

Let an explanation be a mathematical object, a generator that spits out observations into a brain. Then, even when you condition on all your observations so far, most surviving generators still don't resemble reality. Like, if you want the polynomials of a degree that pass through a se...
"AI Rapidly Gets Smarter, And Makes Some of Us Dumber," from Sabine Hossenfelder
by Evan_Gaensbauer — 2025-02-26T22:33:43.688Z
Link: https://youtu.be/ipLA7E-X7Lk?si=hAOojfv5PyY24Gw4

Sabine Hossenfelder is a theoretical physicist and science communicator who provides analysis and commentary on a variety of science and technology topics. I mention that upfront for anyone who isn't already familiar, since I understand a link post to some video full of hot takes on AI from some random YouTuber wouldn't be appreciated.

Even more than usual, so far in 2025 there has been a rapid set of developments on the performance of AI agents and programs, compared to that of humans, so Hossenfelder in this video provides summarizes some of the most significant of recent breakthroughs and findings.

Here's a summary of the reviews of recent developments in AI covered in the video.

Grok 3

Performance

x.AI released its most recent model, Grok 3, a week ago. Grok 3 outperformed on most benchmarks the current iterations of competing models (DeepSeek, OpenAI's models, etc.)--including mathematics, coding, and scientific reasoning. In the last year, the rate of increase in the performance of Grok models has outpaced those of OpenAI and Anthropic, now more comparable to that of DeepSeek. An advantage for access to more data for learning that Grok 3 now has, over OpenAI and DeepSeek, is more exclusive data from Twitter/X. Grok 3 is also the first general-purpose AI model to exceed 10^25 flop in training compute. This exceeds a threshold set in the European Union AI Act, so Grok 3 will now need to be subject to more safety tests to continue to be usable in the EU.

Application

A current disadvantage of Grok 3 in its application is that the now more standard generative AI function, and the more novel 'reasoning' function, can't be used at the same time, e.g., to answer queries. Grok 3 still features the same problems of previous LLM models, including hallucinations, and how easy it is to jailbreak, including providing instructions to build bombs or unconventional weapons.

Google

Last week Google announced an AI super-agent, specifically what the company is calling an 'AI co-scientist,' specifically designed to help scientists discover or search for new research hypotheses, and topics for potential grant proposals. The AI co-scientist, as a super-agent, supervises six sub-agents assigned roles based on the procedure of tasks for hypothesis generation (e.g., idea generation, criticism, modification, etc.). Google is still trialing this model with scienti...