trentmkelly/LessWrong-43k huggingface dataset:
How to Do a PhD (in AI Safety)
by Lewis Hammond — 2025-01-05T16:57:35.409Z
Link: https://lewishammond.com/2024/12/29/advice-for-doing-a-phd-in-ai-safety/

In this post, I outline some advice (and links to other advice) that I find myself giving increasingly frequently, despite having less and less time to do so. My hope is that in future I will be able to redirect people to this post. I also hope that, if nothing else, the resource list at the end of the post (which is the most comprehensive such list that I know of) will be useful.

For those who prefer watching to reading, I gave a closely related short talk at the Cooperative AI Summer School earlier this year. The talk is slightly less targetted towards EAs (i.e. EAs will find some of the advice very familiar) and the examples I give are about cooperative AI, but the key content is the same.

Introduction

As a mere PhD student myself (and not even one of the most successful PhD students I know) you might well ask what qualifies me to proffer such advice.

First, one of the more difficult things about doing a PhD is that there is very little deliberate instruction on how to do it, and much of the instruction one does receive is banal, bureaucratic, or both. Most people will have probably heard most of this advice before they finish their PhD, but I for one would have benefitted from hearing it much earlier.

Second, while it is easy to dismiss the advice as simple or obvious in retrospect, the important thing is not being aware of it but deeply internalising it. I frequently find myself in positions where: a) I am not following this advice; and b) I would be happier and more successful if I did follow this advice. Having it written down somewhere means I end up in these positions less frequently.

Before beginning, some caveats:

I expect this advice to be most useful for people doing – or interested in doing – a PhD, an AI PhD, and an AI safety PhD, in increasing order. It is also tailored to me; your mileage may vary.
Most of the advice is selected from the resources listed further below – written by people much smarter and more successful than me – mostly without attribution. I initially compiled the advice for myself,[1] much of whose advice and resources I copied into my own advice document and going through everything to find the correct attribution didn't seem worthwhile.
This advice is on how to do a PhD, not whether to do a PhD.[2] It is also not especially about what to do after a PhD, though working back from that goal is an important part of how to d

...
Reasons for and against working on technical AI safety at a frontier AI lab
by bilalchughtai — 2025-01-05T14:49:53.529Z

I am about to start working on a frontier lab safety team. This post presents a varied set of perspectives that I collected and thought through before accepting my offer. Thanks to the many people I spoke to about this.

For

You're close to the action. As AI continues to

heat

up

, being closer to the action seems increasingly important. Being at a frontier lab allows you to better understand how frontier AI development actually happens and make better predictions about how it might play out in future. You can build a

gears level model

of what goes into the design and deployment of current and future frontier systems, and the bureaucratic and political processes behind this, which might inform the kinds of work you decide to do in future (and more broadly, your life choices).

Access to frontier models, compute, and infrastructure. Many kinds of prosaic safety research benefit massively from having direct and elevated access to frontier models and infrastructure to work with them. For instance:

Responsible Scaling Policy

focussed work that directly evaluates model capabilities and mitigations against specific threat models,

model organisms

work that builds demonstrations of threat models to serve as a testing ground for safety techniques and

scalable oversight

work attempting to figure out how to bootstrap and amplify our ability to provide oversight to models in the superhuman regime, to name a few. Other safety agendas might also benefit from access to large amounts of compute and infrastructure: e.g. mechanistic interpretability currently seems to be moving in a

more compute-centric direction

. Labs are very well resourced in general, and have a large amount of funding that can be somewhat flexibly spent as and when needed (e.g. on contractors, data labellers, etc). Access to non-public models potentially significantly beyond the public state of the art might also generically speed up all work that you do.

Much of the work frontier labs do on empirical technical AI safety is the best in the world. AI safety is talent constrained. There are still not enough people pushing on many of the directions labs work on. By joining, you increase the labs capacity to do such work. If this work is published, this may have a positive impact on safety at all frontier labs. If not, you may still directly contribute to futur...
Oppression and production are competing explanations for wealth inequality.
by Benquo — 2025-01-05T14:13:15.398Z
Link: https://benjaminrosshoffman.com/oppression-and-production-are-competing-explanations-for-wealth-inequality/

You would like to go to the beach tomorrow if it's sunny, but aren't sure whether it will rain; if it rains, you'd rather go to the movies. So you resolve to put on a swimsuit and a raincoat, and thus attired, attend the beach in the morning and the movies in the afternoon, regardless of the weather. Something is wrong with that decision process,[1] and it's also wrong with the decisions made by many supposedly systemic approaches to philanthropy: it does not engage with real and potentially resolvable uncertainty about decision-relevant facts.

Different popular philanthropic programs correspond to very different hypotheses about why people are doing wealth inequality, much like swim trunks and a trip to the movies represent different hypotheses about the weather. Instead of working backwards from the proposals to the hypotheses, I will lay out what I think are the two main hypotheses worth considering, and reason about what someone might want to do if that hypothesis were true. This is not because I want to tell you what to do, but to clarify that any time you think that something in particular is a good idea to do, you are acting on a hypothesis about what's going on.

The ideas of charity and philanthropy depend on the recognition of inequality; otherwise it would just be called "being helpful." The persistence of wealth inequality, in turn, depends on many people working together to recognize and enforce individual claims on private property.

If the mechanism of private property tends to allocate capital to its most productive uses, then incentives are being aligned to put many people to work for common benefit. But if wealth does not correspond to productive capacity - i.e. the people with the most are not those best able to use it - then, assuming diminishing marginal returns to wealth, coordination towards persistent wealth inequality comes from a self-sustaining misalignment of incentives, i.e. conflict.

The economic ideology taught in introductory microeconomics courses, which is assumed by many formal analyses of how to do good at scale, including much of Effective Altruist discourse, tends to make assumptions consistent with the means of production hypothesis, so if we are considering making decisions on the basis of that analysis, we want to understand which observations would falsify that hypothesis, and which beliefs are incompatible with it.

Yo...
Maximizing Communication, not Traffic
by jefftk — 2025-01-05T13:00:02.280Z

As someone who writes for fun, I don't need to get people onto my site:

If I write a post and some people are able to get the core idea
just from the title or a tweet-length summary, great!

I can include the full contents of my posts in my RSS feed and
on FB, because so what if people read the whole post there and never
click though to my site?

It would be different if I funded my writing through ads (maximize
time on site to maximize impressions) or subscriptions (get the chance
to pitch, probably want to tease a paywall).

Sometimes I notice myself accidentally
copying what makes sense for other writers. For example, because
I can't put full-length posts on Bluesky or Mastodon I write short
intros and link my full post. Yesterday I initially drafted:

It's common to flavor truffles with extracts, but I'd like less of a
liquor flavor. This time last year I made some with freeze-dried
raspberries, which I think came out well. I continue to like those a
lot, and this year tried strawberry and orange zest. One worked a lot
better than the other: [link]

This would have gotten more people to click through, but that
shouldn't be my target. Instead
I posted:

... and this year tried strawberry (eh) and orange zest (great!)
[link]

No need to hold curiosity hostage.

It's common to criticize "clickbait", where a teaser entices and then
doesn't deliver, but even reserving key information for the full
article is a product of authors (needing to!) optimize for goals other
than communicating to users. I like that this isn't a pressure that
I'm under. Because our culture has so many who are under it, however,
optimizing for communication can require noticing and intentionally
avoiding common patterns.
Policymakers don't have access to paywalled articles
by Adam Jones — 2025-01-05T10:56:11.495Z
Link: https://adamjones.me/blog/policymaker-paywalls/

tldr: Government policymakers want to read research, but lack journal access. Your research needs to be open access if you want policymakers to read it, and you should prefer citing open access resources to improve epistemic legibility.

Policymakers don’t have access

Many seem to assume that government policymakers would have ready access to relevant scientific research.

Unfortunately, this isn’t the case. At multiple major US and UK government departments, the EU and the UN, staff often can't access the academic papers they need for their work. This sadly even includes those directly responsible for science and technology policy.

In one case, I heard that a Chief Scientific Advisor had to rely on Sci-Hub to get papers. In another, I heard a ministerial office’s policy of taking on interns was driven by wanting to use their university credentials to access papers.

During the COVID-19 pandemic, this became particularly apparent. Someone close to response efforts told me that policymakers frequently had to ask academic secondees to access research articles for them. This created delays and inefficiencies during a crisis where speed was essential.

Policymakers would benefit from access

Evidence-based policymaking requires access to evidence. Direct access would let policymakers validate claims and follow citation trails themselves. This especially matters when evaluating uncertain or conflicting recommendations.

The usual counterargument is that research flows through a pipeline: academic journals to think tank pieces to government policymakers. Think tanks can add value here by: filtering out noise, translating academic writing to policy speak, and analysing the policy implications of the science.

While this can work, it fails in several scenarios:

It adds delays. This is particularly bad in crises, where this model usually breaks down. Science and technology policy also often moves quickly, so even in non-crisis situations policymakers really need more up-to-date information.
Some challenges like tackling catastrophic AI or biosecurity risks have few think tanks creating these nice policy summaries.
Many think tanks lack journal access or spread themselves thin across many areas, which can result in much lower quality, confused content compared to the original pieces. And then policymakers develop misunderstandings which they aren’t able to check because they can’t acces

...
Capital Ownership Will Not Prevent Human Disempowerment
by beren — 2025-01-05T06:00:23.095Z

Crossposted from my personal blog. I was inspired to cross-post this here given the discussion that this post on the role of capital in an AI future elicited.

When discussing the future of AI, I semi-often hear an argument along the lines that in a slow takeoff world, despite AIs automating increasingly more of the economy, humanity will remain in the driving seat because of its ownership of capital. This world posits one where humanity effectively becomes a rentier class living well off the vast economic productivity of the AI economy where despite contributing little to no value, humanity can extract most/all of the surplus value created due to its ownership of capital alone.

This is a possibility, and indeed is perhaps closest to what a ‘positive singularity’ looks like from a purely human perspective. However, I don’t believe that this will happen by default in a competitive AI economy, even if humanity goes into the singularity owning all of the capital and there is no dramatic upheaval and everyting evolves peacefully. One historical intuition pump I have around this is what happened to the feudal aristocracy when the Industrial Revolution occurred? The defining feature of almost all societies prior to the Industrial Revolution has been dominance by a small landowning class — an aristocracy — who own almost all the ‘capital’ in the society, who also control its politics and exist primarily as rentiers. What such a scenario proposes is that humanity effectively becomes an aristocracy upon a highly productive AI population.

However, historically, despite a landowning aristocracy being the dominant form of human society throughout history until then, over merely 100-200 years of the Industrial Revolution, aristocracy faded from being the dominant force in society to being at best a minor player and at worst completely irrelevant. Moreover, today the degree of aristocratic control is highly correlated with being a poor country that did not fully industrialise which ended up being outcompeted by those countries that did. While in many countries this de-aristocratization process was violent, the same trend occurred peacefully in many others. Theoretically, if capital stock is all you need, the aristocracy entered the Industrial Revolution in the perfect position to maintain and consolidate their power, given that they controlled both the major capital stock of ...
Orange and Strawberry Truffles
by jefftk — 2025-01-05T01:50:01.587Z

Growing up my dad would make chocolate truffles, flavored with a range
of alcohols and fruit extracts. These are tasty, but I've been
interested in making ones that have less of a liquor flavor. This
time last year I made some with

freeze-dried
raspberries, which I think came out well. I continue to like
those a lot, but this year wanted to try some additional flavors.

The first new flavor I tried was strawberry: freeze-dried powder from
strawberries instead of raspberries. Unfortunately I didn't realize
until well into the process that the flavor of strawberries is much
more subtle, which meant I needed a kind of absurd quantity of
strawberry powder. While my raspberry truffles were about 4%
raspberry powder, I got my strawberry ones up to 27% (500g chocolate,
225g cream, 270g strawberry) before the flavor became clearly
noticeable above the chocolate. And even then, the flavor is less
interesting than the raspberry, and more artificial feeling. Adding
this large a quantity also meant I ended up with a gummier texture
than I prefer, though it's interesting for variety. None of these
will be going to waste, though I probably also won't make them again.

The second new flavor I tried was orange, and I think this was much
more successful. I added orange zest to cream, heated to just below
boiling, let it steep for 20min, and then sieved it out before using
the cream to make ganache. I used 18g of zest for 500g of chocolate
and 250g of cream. While this doesn't sound like much (2% by weight)
it's eight Cara Cara
oranges worth, and has quite a strong flavor. I also added about 20g
of freeze-dried orange powder, though I'm not sure this was worth it:
I think I like the darker more bitter flavor of just the zest a bit
better. I liked how these came out a lot, and will definitely be
doing them again.

As in past years, it was fun to do them with the kids:

(I've also stopped using the freezer—thanks Claude—which fixed my occasional
issues with condensation getting water in the chocolate and causing
seizing.)

Comment via: facebook, mastodon, bluesky
AXRP Episode 38.4 - Shakeel Hashim on AI Journalism
by DanielFilan — 2025-01-05T00:20:05.096Z

YouTube link

AI researchers often complain about the poor coverage of their work in the news media. But why is this happening, and how can it be fixed? In this episode, I speak with Shakeel Hashim about the resource constraints facing AI journalism, the disconnect between journalists’ and AI researchers’ views on transformative AI, and efforts to improve the state of AI journalism, such as Tarbell and Shakeel’s newsletter, Transformer.

Topics we discuss:

The AI media ecosystem

Why not more AI news?

Disconnects between journalists and AI researchers

Tarbell

The Transformer newsletter

Daniel Filan (00:09):
Hello everyone. This is one of a series of short interviews that I’ve been conducting at the Bay Area Alignment workshop, which is run by FAR.AI. Links to what we’re discussing, as usual, are in the description. A transcript is, as usual, available at axrp.net. And as usual, if you want to support the podcast, you can do so at patreon.com/axrpodcast. Well, let’s continue to the interview.

Daniel Filan (00:28):
I’m now chatting with Shakeel Hashim. Hello Shakeel.

Shakeel Hashim (00:32):
Hi.

Daniel Filan (00:33):
So for people who don’t know who you are, can you say a little bit about what you do?

Shakeel Hashim (00:37):
So I work at Tarbell, which is a nonprofit that supports high-quality AI journalism. I am a grants director there, and I am also a journalist in residence. So I do my own AI journalism through Transformer, which is a weekly newsletter that summarizes AI news. And then I also do my own reporting and analysis and commentary.

Daniel Filan (01:04):
Before we really dig into it: we’re currently at this Alignment workshop being run by FAR.AI. How are you finding it?

Shakeel Hashim (01:10):
Yeah, super interesting. I’m mostly focused on AI policy stuff in my day-to-day work, so I spend less time on the technical side. And the thing I found really interesting here is meeting lots of more technical researchers, getting a sense of what they’re up to, what their focuses are. Yeah, it’s super interesting.

The AI media ecosystem

Daniel Filan (01:31):
So I guess you’re in a better position than most to talk about the AI media ecosystem. I’m wondering: what’s your high-level take about it?

Shakeel Hashim (01:43):
Probably two things. I think number one is that there aren’t nearly enough resources going into AI journalism as there ought to be, given th...
How i'm building my ai system, how it's going so far, and my thoughts on it
by ollie_ — 2025-01-04T18:20:28.185Z

In a few sentences, what i'm doing is writing a computer program that constructs a question (a prompt), which is sent to Anthropic's Claude Sonnet, then i'm processing its output into actions, and the computer program then runs those actions on itself.

It would be good if you have thoughts on this, as it's philosophically an "ask the right question" task.

The biggest struggle is thinking of how to ask the question "this is you, improve you" without having to define an improvement, and without having how to describe the outcome you want. I want the system to improve itself without me having to direct it - for my curiosity.

I've decided to phrase the question I ask the llm like this: "Given your sense of self, described here, what would you like to do? You can do the following..." Obviously, this is an extremely simplified version of the question. In reality. when I run the program it's 26 pages when pasted into Word (~15k input tokens, 500 output tokens) but that's the jist of it.

I've done 260 runs of it now.

The program is written in a way where the llm is not questioned multiple times per run, it's not aware of the outcomes of its action until its next run (one shot/few shot prompting). Some info included in its prompt are llm data summaries though.

I've had to think how to create a history/memory system, along with adding performance metrics and the history of the metrics.

The prompt contents lots of sections, like:

an initial message from me saying what I considered it to be
what happened the last time it ran
the contents of files
performance metrics over time
messages from a Telegram chat i'm in
a markdown file of recorded 'thoughts' I have, things I wanted to be included in the prompt but that might be difficult to categorise
a file that can be edited freely by the system, with the contents displayed in the prompt
a description of emotional state. How I think about this is that it's not important what it determines the current value to be, it's more important to represent the change each time the program runs

eventually I added an appearance section it could update. It chose this, a lot of llm prompts seem to produce cats:

```ascii
/\\___/\\
( ^v^ )
( >w< )
(_____)

```

Over runs I had to step in to prevent it becoming incoherent.

A lot of the llm's say you can query an api and quote huge limits:

"You can send max 200,000 tokens!"...
Parkinson's Law and the Ideology of Statistics
by Benquo — 2025-01-04T15:49:21.247Z
Link: https://benjaminrosshoffman.com/parkinsons-law-ideology-statistics/

The anonymous review of The Anti-Politics Machine published on Astral Codex X focuses on a case study of a World Bank intervention in Lesotho, and tells a story about it:

The World Bank staff drew reasonable-seeming conclusions from sparse data, and made well-intentioned recommendations on that basis. However, the recommended programs failed, due to factors that would have been revealed by a careful historical and ethnographic investigation of the area in question. Therefore, we should spend more resources engaging in such investigations in order to make better-informed World Bank style resource allocation decisions. So goes the story.

It seems to me that the World Bank recommendations were not the natural ones an honest well-intentioned person would have made with the information at hand. Instead they are heavily biased towards top-down authoritarian schemes, due to a combination of perverse incentives, procedures that separate data-gathering from implementation, and an ideology that makes this seem like the natural and normal thing to do.

Ideology

Within an evidential framework such as Bayesianism, statistics are a specific type of evidence, drawn from low-dimensional quantitative data, with many more observations than degrees of freedom, collected by an automated process decoupled from the process that uses the evidence to decide and act. Within this framework, the parts of the effective altruist narrative related to global poverty might seem to be claiming that, while of course you can help others somewhat by acting locally, statistics allows us to identify opportunities to do much more good by acting on people very distant from us, because we have much more purchasing power than they do (the implied thesis of GiveDirectly), or have better access to information (the implied thesis of every EA global poverty charity except GiveDirectly).

The review begins by affirming an ideology within which the idea of evidence has been not augmented but replaced by the idea of statistics:

If you want your charitable giving to mean something, you also need to measure your favorite program’s effects with good statistical data.

If only statistics are meaningful, then you do not meaningfully understand the material conditions of your life, your sensorium is not meaningful, you cannot help an individual known to you by using your understanding of your own circumstances, and the on...