trentmkelly/LessWrong-43k huggingface dataset:
AI may pursue goals
by Algon — 2025-05-28T09:30:03.439Z

Context: This is a linkpost for https://aisafety.info/questions/NM3J/5:-AI-may-pursue-goals

This is an article in the new intro to AI safety series from AISafety.info. We'd appreciate any feedback. The most up-to-date version of this article is on our website.

Suppose that, as argued previously, in the next few decades we’ll have superintelligent systems. What role will they play?

One way to imagine these systems is purely as powerful and versatile tools, similar to most current systems. They could take broad directions from humans about what actions to take or what questions to answer, and cleverly fill in the details.

But another way is as agents, operating autonomously in the world. They could have their own goals — some kinds of futures they seek out over other futures — and take whatever actions will most likely lead to those futures, adapting as circumstances change.

As long as AIs are tools, they can be used for good or ill, like all technologies. They can radically increase the scope of the problems humans can solve and create.

But it’s unlikely that they’ll remain only tools, because:

A good planning tool can easily be turned into an agent. Just tell it: “repeatedly come up with actions that would make goal X more likely, and execute those actions”. People keep building software frameworks for doing this, and to the extent they succeed, better tools will result in better agents.
When a planning tool is sufficiently more competent than humans, keeping humans in the loop to direct its activities will just get in the way. It will make the system less efficient (as we’re already seeing for some medical tasks), less profitable, and less able to compete.
There may be some tasks that highly intelligent agents can do and tools just can’t, like implementing hard new research programs.
Increasingly agent-like systems are already being created — see, for example, OpenAI’s Operator, which types, clicks, and scrolls in a web browser, and Anthropic’s “computer use” feature for its chatbot Claude.
Even if most AIs remain non-agentic, some people will create agents for various reasons of their own — to be the creator of a new species, or for the heck of it.

If we’re going to build AI systems that pursue goals, it would be good if those goals matched ours. It’s not clear if we’ll succeed at making that the case.

Related

What is an agent?

What is scaffold

...
The Best Way to Align an LLM: Is Inner Alignment Now a Solved Problem?
by RogerDearnaley — 2025-05-28T06:21:42.324Z

This is a link-post for a new paper I read: Safety Pretraining: Toward the Next Generation of Safe AI by Pratyush Maini, Sachin Goyal, et al.
For a couple of years I (and others) have been proposing an approach to alignment: what the authors of this recent paper name "safety pretraining". In a nutshell: that it's best to apply your alignment training as part of the standard pretraining process to produce a base model that is already aligned — simply pretrain it on data including a lot of clearly marked examples of aligned behavior (then prompt for it).

I've regarded this approach as a major advance ever since I read the seminal 2023 paper on the topic: Pretraining Language Models with Human Preferences by Tomasz Korbak et al., and I'm absolutely delighted to finally see someone else publish another paper on this approach — I'm only sad it has taken so long.

I highly encourage everyone interested in AI alignment to go read both of these papers (if you haven't already) — between them they strongly suggest that the authors have found a more effective way to align an AI: an alignment approach better than any that people are (as far as we know) currently using. I believe this is extremely important: I see it as major progress on alignment. So I think it directly reduces the p(DOOM) for the most critical current x-risk to our entire species.

For more detailed expositions of this approach and why I think it's an excellent idea, see my previous posts How to Control an LLM's Behavior (why my P(DOOM) went down), A "Bitter Lesson" Approach to Aligning AGI and ASI, and Why Aligning an LLM is Hard, and How to Make it Easier.

(I'm also delighted that the authors of the recent paper tested out some of the follow-on ideas I'd been proposing in those posts on Less Wrong. One was training the model to generate control-tag tokens that label portions of the text as good or bad behavior, and then for conditional generation altering the token generation process, leveraging these tokens, so as to induce the model to behave well not badly. Another was using synthetic data editing to modify problematic raw training examples by supplementing them with more moral or correct behavior or commentary. They elaborated on both of these, or independently reinvented them, and even confirmed that both of these appear to work about as well as I'd been hoping.)

Hence, in order to encourage peopl...
Spectral radii dimensionality reduction computed without gradient calculations
by Joseph Van Name — 2025-05-28T05:06:05.144Z

In this post, I shall describe a fitness function that can be locally maximized without gradient computations. This fitness function is my own. I initially developed this fitness function in order to evaluate block ciphers for cryptocurrency technologies, but I later found that this fitness function may be used to solve other problems such as the clique problem (which is NP-complete) in the average case and some natural language processing tasks as well. After describing algorithms for locally maximizing this fitness function, I conclude that such a fitness function is inherently interpretable and mathematical which is what we need for AI safety.

Let K denote either the field of real numbers, complex numbers, or the division ring of quaternions. Given (A1,…,Ar)∈Mn(K)r and 1≤d<n, the goal is to find a tuple (X1,…,Xr)∈Md(K)r most similar to (A1,…,Ar). In other words, we want to approximate the n×n-matrices (A1,…,Ar) with d×d-matrices.

Suppose that A1,…,Ar∈Mn(K),X1,…,Xr∈Md(K). Define a function Γ(A1,…,Ar;X1,…,Xr):Mn,d(K)→Mn,d(K) by setting

Γ(A1,…,Ar;X1,…,Xr)(X)=∑jAjXX∗j, and defineΦ(A1,…,Ar)=Γ(A1,…,Ar;A1,…,Ar).

If X is a matrix, then let ρ(X) denote the spectral radius of X.

Define the L2-spectral radius similarity between (A1,…,Ar) and (X1,…,Xr)

by setting ∥(A1,…,Ar)≃(X1,…,Xr)∥2

=ρ(Γ(A1,…,Ar;X1,…,Xr))ρ(Φ(A1,…,Ar))1/2ρ(Φ(X1,…,Xr))1/2.

The quantity ∥(A1,…,Ar)≃(X1,…,Xr)∥2 is always a real number in the interval [0,1] (the proof is a generalization of the Cauchy-Schwarz inequality).

If 1≤d<n, and A1,…,Ar∈Mn(K), then we say that (X1,…,Xr)∈Md(K)r is an L2,d-spectral radius dimensionality reduction (LSRDR) of (A1,…,Ar) if the similarity ∥(A1,…,Ar)≃(X1,…,Xr)∥2 is locally maximized.

One can produce an LSRDR of (A1,…,Ar) simply by locally maximizing (X1,…,Xr) using gradient ascent. But there is another way to obtain LSRDRs since they behave mathematically.

Let Z(K) denote the center of the algebra K. In particular,Z(R)=R,Z(C)=C,Z(H)=R.

Empirical observation: If (X1,…,Xr)∈Md(K)r is an LSRDR of (A1,…,Ar), then typically exists λ∈Z(K) along with matrices R,S with Xj=λRAjS for all j and w...
If you're not sure how to sort a list or grid—seriate it!
by gwern — 2025-05-28T03:54:13.810Z
Link: https://www.jstatsoft.org/article/download/v025i03/227

"Getting Things in Order: An Introduction to the R Package seriation", Hahsler et al 2008:

Seriation [or "ordination"], i.e., finding a suitable linear order for a set of objects given data and a loss or merit function, is a basic problem in data analysis. Caused by the problem's combinatorial nature, it is hard to solve for all but very small sets. Nevertheless, both exact solution methods and heuristics are available.

In this paper we present the package seriation which provides an infrastructure for seriation with R. The infrastructure comprises data structures to represent linear orders as permutation vectors, a wide array of seriation methods using a consistent interface, a method to calculate the value of various loss and merit functions, and several visualization techniques which build on seriation.

To illustrate how easily the package can be applied for a variety of applications, a comprehensive collection of examples is presented.

Have you ever wondered how to sort a list or a folder of files where no strict sorting comparison operator like 'newer than' is quite right? It turns out that it is perfectly possible to loosen the definition of 'sorting' to something more approximate like 'try to minimize how different each item is from the next one'; this approximate or generalized sorting is called 'seriation'. It is obscure (I had never heard of the term until a year ago or so), but highly useful: it works for everything from seriating Egyptian graves by rough burial time to organizing tag entries by topic. Since we now have neural embeddings for just about every modality there is, that means you can seriate anything.

What might you use it for? I use it on Gwern.net (background) to organize the 'similar links' recommendations in a way much smarter than the naive k-NN embedding distance retrieval approach. If you just sort them by 'distance', it is mostly meaningless and produces a jumble (see for example any algorithmic set of recommendations, like YouTube video lists - if I open a cat video, I see cat/anime/Touhou/cat/CS/music/cat/...); if you seriate them, however, suddenly you see clear clusters/topics emerge out of the chaos, and it's easier to skim the list. Because it works so well, and is so simple to implement (simple greedy distance minimization heuristic worked surprisingly well for my use-case, no need for TSP solvers), I initially called it...
Briefly analyzing the 10-year moratorium amendment
by RobertM — 2025-05-28T03:11:20.124Z

This is the result of a half-day research sprint on the recently-introduced amendment to institute a 10-year moratorium on state-level AI regulations in the current budget reconciliation bill, with a focus on figuring out "is it likely to survive the Byrd Rule".

It seems quite obviously in violation of the Byrd Rule because it violates at least 2 of the 6 tests[1]:

It has no budget effect, i.e. does not change outlays or revenues. (Probably this is not actually true; I think on net we would expect it to somewhat increase revenues via 2nd/3rd order effects, but in the relevant legal sense, it's not an explicit additional tax and it's not an additional explicit expenditure. Also, the CBO estimate for that entire section is -500m, which is the full sum of the $500m it appropriates to modernize and secure Federal information technology systems through the deployment of commercial artificial intelligence, the deployment of automation technologies, and the replacement of antiquated business systems in accordance with subsection (b))
Even if it did have a budget effect, it would be incidental to the non-budgetary component of the provision, which is clearly meant to be a "policy change".

There is a "Byrd bath" that reconciliation bills go through, directed by the Senate Parliamentarian, before the bill is taken up by the Senate. The Parliamentarian & Senate staff identify the sections that seem like obvious violations of the Byrd Rule and try to work with whoever drafted the relevant section of text to redraft or amend it so that it doesn't violate the Byrd Rule (if possible; sometimes they're just deleted wholesale). It's not clear to me what percentage of violations get scrubbed out at this step, but various LLMs think it's most of them.

If it does still somehow make it through that step, Senators can raise a "point of order" against subtitles, sections, lines, or even single words in the bill. Other Senators can motion to "waive" a point of order; these waiver motions require a 3/5 majority vote (60 Senators) to pass. Separately, the Senate chair (currently JD Vance) can rule against the point of order without another Senator initiating a motion to "waive" it. Historically, the chair has deferred to the Parliamentarian's advice about whether the text being objected to is extraneous or not; as far as I can tell there haven't bee...
Does Sort Really Fall Back to Disk?
by jefftk — 2025-05-28T01:20:04.399Z

The unix

sort command is clever: to sort very large
files it does a series of in-memory sorts, saving sorted chunks to
temporary files, and then does a merge sort on those chunks. Except
this often doesn't work anymore.

Here's what I see if I run man sort and look at the
documentation for --buffer-size:

use SIZE for main memory buffer

That's pretty terse! What does my Mac say?

Use size for the maximum size of the memory buffer. Size
modifiers %,b,K,M,G,T,P,E,Z,Y can be used. If a memory limit is
not explicitly specified, sort takes up to about 90% of available
memory. If the file size is too big to fit into the memory buffer,
the temporary disk files are used to perform the sorting.

Makes sense! But then the docs for --temporary-directory
say:

use DIR for temporaries, not $TMPDIR or
/tmp; multiple options specify multiple directories

And these days /tmp is often memory-backed, via tmpfs. This changed in
Fedora
18 (2013) and Ubuntu
24.10 (2024), and is changing in Debian 13
(in a month or two).

It seems to me that these days it would be better for
--temporary-directory to default to
/var/tmp, which is preserved
across reboots and so will generally be backed by disk even on
systems that use tmpfs for /tmp. In the meantime,
sort --temporary-directory /var/tmp will do the trick.
Shift Resources to Advocacy Now (Post 4 of 7 on AI Governance)
by Mass_Driver — 2025-05-28T01:19:27.307Z

In my previous post in this series, I estimated that we have 3 researchers for every advocate working on US AI governance, and I argued that this ratio is backwards. When allocating staff, you almost always want to have more people working on the more central activity. I argued that in the case of AI policy, the central activity is advocacy, not research, because the core problem to be solved is fixing the bad private incentives faced by AI developers.

As I explained, the problem with these incentives is less that they’re poorly understood, and more that they require significant political effort to overturn. As a result, we’ll need to shift significant resources from research (which helps us understand problems better) to advocacy (which helps us change bad incentives).

In this post, I want to explain why it’s appropriate for us to shift these resources now, rather than waiting until some future date to scale up our advocacy.

The arguments for why we should wait until later to start advocating can be divided into three broad categories: (a) we aren’t yet sure that any of our policies will be robustly helpful, (b) we haven’t yet learned how to advocate successfully, and (c) we don’t yet have enough senior politicians in the movement to staff a large advocacy effort. What these objections have in common is that they seem to justify an emphasis on research for the next few years – if we’re missing knowledge or experience that we would need in order to win at advocacy, then maybe we should bide our time while we gather more resources.

In my opinion, none of these objections are well-founded. We should be very confident that our best policies offer net-positive value, we should be very confident that advocating for those policies will increase their chance of passage, and we should be confident that we can solve the challenge of recruiting effective advocates with willpower, flexibility, and funding.

Moreover, even if we were not self-confident, our chance of success will not be significantly improved by waiting or by doing more academic research. On the contrary, we face enormous time pressure to get better incentives in place before AI developers push us past the point of no return, so we need to start moving AI regulations forward now, using whatever policy tools and policy advocates are currently available.

OUR BEST POLICIES OFFER POSITIVE EXPECTED VALU

...
Colonialism in space: Does a collection of minds have exactly two attractors?
by StanislavKrym — 2025-05-27T23:35:52.221Z

Depending on the estimates of parameters, the Drake equation produces drastically different amounts of contactable civilisations in the Milky Way. Some estimates imply that the reason is the extreme rarity of life, while others suggest that we just happened to be the first civilisation in the galaxy[1] that is likely to reach other systems in the foreseeable future.

The diameter of the Milky Way is less than 105 light years. Meanwhile, even currently existing proposals like the fission-fragment rocket are estimated to allow transportation at speeds at least 0.05 times the speed of light. If an AI tried to colonize the entire galaxy, it would need at most[2] 107 years, which is, apparently, at least 2.5 OOM less than the length of the age[3] when sapient life might appear in a stellar system in the Milky Way.

Therefore, many encountered planets with life will contain non-sapient life. But a primitive alien civilisation or a planet which might generate a sapient alien lifeform would, as I argued here, have some kind of rights to their system and to their part of the space. But the lifeform's fate depends only on the will of the discoverers.

Suppose that in systems that are likely to be reached by currently primitive aliens races of good explorers establish only outposts that consume a tiny amount of resources and protect the system, while races of evil explorers gather most resources and use them for the evil explorers' goals.[4] The equilibrium between good explorers and evil ones is likely to be unstable, and the third option doesn't seem to exist.

An additional implication is the following. Since there doesn't seem to exist a third option, then does any collection of minds converge to one of the two attractors? Since humanity is unlikely to return to the colonialistic attractor, how can humans increase the chance that the AI created by good actors[5] will also be aligned to the anti-colonialistic attractor? What about the chance that an AI aligned to said attractor won't destroy humanity?

^

Humans might also be the first civilisation in the spacetime cone from where a civilisation armed with advanced tech can reach the Earth. If this is true, then humans (or human-created AIs) could encounter the aliens (or their AIs) after they both began the space colonization. But then the two sides of a potential

...
What are the best arguments you've seen for the Litany of Gendlin?
by flowerfeatherfocus — 2025-05-27T21:19:21.398Z

That is, that learning the truth will never leave you worse off, or that useful fictions don't exist.
I'm hoping to see it defended in they way I frequently see it used, not just as a prior, but as a principle that admits basically no exceptions.
What We Learned from Briefing 70+ Lawmakers on the Threat from AI
by leticiagarcia — 2025-05-27T18:23:55.938Z
Link: https://substack.com/home/post/p-164221773

Between late 2024 and mid-May 2025, I briefed over 70 cross-party UK parliamentarians. Just over one-third were MPs, a similar share were members of the House of Lords, and just under one-third came from devolved legislatures — the Scottish Parliament, the Senedd, and the Northern Ireland Assembly. I also held eight additional meetings attended exclusively by parliamentary staffers. While I delivered some briefings alone, most were led by two members of our team.

I did this as part of my work as a Policy Advisor with ControlAI, where we aim to build common knowledge of AI risks through clear, honest, and direct engagement with parliamentarians about both the challenges and potential solutions. To succeed at scale in managing AI risk, it is important to continue to build this common knowledge. For this reason, I have decided to share what I have learned over the past few months publicly, in the hope that it will help other individuals and organisations in taking action.

In this post, we cover: (i) how parliamentarians typically receive our AI risk briefings; (ii) practical outreach tips; (iii) effective leverage points for discussing AI risks; (iv) recommendations for crafting a compelling pitch; (v) common challenges we've encountered; (vi) key considerations for successful meetings; and (vii) recommended books and media articles that we’ve found helpful.

(i) Overall reception of our briefings

Very few parliamentarians are up to date on AI and AI risk: Around 80–85% of parliamentarians were only somewhat familiar with AI, with their engagement largely limited to occasional use of large language models (LLMs) like ChatGPT for basic tasks (e.g., getting assistance with writing a speech). Their staff were slightly more familiar with AI, but few were well-versed in the broader conversation surrounding it.

Capacity is the main limiting factor: MPs typically have 3–5 staffers, many of whom focus primarily on constituency work. Members of devolved legislatures usually have 2–4 staffers, while Peers often have even less support – some have no dedicated staff at all.

As a result, there is rarely anyone on these teams who can dedicate significant time to researching AI. Except for a few staffers with a personal interest in AI, most staffers we spoke to had little or no familiarity with it. While most of those we spoke to expressed a desire to learn more, the...
My script for organizing OBNYC meetups
by Orioth — 2025-05-27T18:14:31.674Z

(I wrote this about a year ago, so parts of it might be out of date; I'm not much involved with OBNYC anymore.)

Context

Overcoming Bias NYC, the rationalist meetup in New York City, has for the last several years had a system of rotating month captains, who each volunteer to take responsibility for scheduling meetups in a given month. Before we adopted this system (in 2017, I think), there was already a tradition of having a meetup every Tuesday, but this ran into some difficulties:

During some periods, there'd be diffusion of responsibility and sometimes the weekly meetup would fail to happen, since no-one took it upon themself to organize one (or there'd be a meetup but without a topic, which tended to result in worse meetups than those with a pre-announced topic)

During other periods, one de-facto leader would do almost all the work of making sure the meetups happened, which resulted in the organizer being potentially overburdened, and the meetups being overly reliant on that one person.

The month captain system initially worked quite well, in my opinion, and is still going, though lately it has been faltering due to a lack of people willing to volunteer for the role.

I've been Month Captain a few times in the early days of the position, and once recently. The rest of this post will consist of a description of what I do in that role.

This post is descriptive, not prescriptive

This post is not intended as a claim regarding whether this is a good way to approach about the role of OBNYC Month Captain, nor a claim as to whether being OBNYC Month Captain is a worthwhile activity. It's just a description of what I did, not of what I think people should do.

Goal

My primary goal as Month Captain is to cause there to be a meetup every week, (unless there's a very good reason to skip a week). They're on Tuesday by default, but changing that is fine if there's a reason for it.

Ideally, they should be

Good

About a rationality-related topic

Announced at least three days beforehand

But announcing things more than a week in advance can confuse people

I aim to announce Tuesday meetups on the Friday before, but this is allowing some margin for error

Posting a reminder about a day before the meetup is also helpful, or at least seems to increase attendance

Also, if anything comes up that someone in OBNYC should clearly handle, but there isn't anybody in partic...