trentmkelly/LessWrong-43k huggingface dataset:
AI Constitutions are a tool to reduce societal scale risk
by Sammy Martin — 2024-07-25T11:18:17.826Z

This work was funded by Polaris Ventures

As AI systems become more integrated into society, we face potential societal-scale risks that current regulations fail to address. These risks include cooperation failures, structural failures from opaque decision-making, and AI-enabled totalitarian control. We propose enhancing LLM-based AI Constitutions and Model Specifications to mitigate these risks by implementing specific behaviours aimed at improving AI systems' epistemology, decision support capabilities, and cooperative intelligence. This approach offers a practical, near-term intervention to shape AI behaviour positively. We call on AI developers, policymakers, and researchers to consider and implement improvements along these lines, as well as for more research into testing Constitution/Model Spec improvements, setting a foundation for more responsible AI development that reduces long-term societal risks.

TL:DR - see the section on Principles which explains what improvements we think should be made to AI constitutions in detail.

Introduction

There is reason to believe that in the near future, autonomous, LLM based AI systems, while not necessarily surpassing human intelligence in all domains, will be widely deployed throughout society. We anticipate a world where AI will be making some decisions on our behalf, following complex plans, advising on decision-making and negotiation, and presenting conclusions without human oversight at every step. While this is already happening to some degree in low-stakes settings, we must prepare for its expansion into high-stakes domains (e.g. politics, the military), and do our best to anticipate the systemic, societal scale risks that might result and act to prevent them. Most of the important work on reducing societal-scale risk will, by their very nature, have to involve policy changes, for example to ensure that there are humans in the loop on important decisions, but there are some technical interventions which we have identified that can help.

We believe that by acting now to improve the epistemology (especially on moral or political questions), decision support capabilities and cooperative intelligence of LLM based AI systems, we can mitigate near-term risks and also set important precedents for future AI development. We aim to do this by proposing enhancements to AI Constitutions or Model Specifications. If adopted, we...
Determining the power of investors over Frontier AI Labs is strategically important to reduce x-risk
by Lucie Philippon — 2024-07-25T01:12:20.518Z

Produced as part of the ML Alignment & Theory Scholars Program - Summer 2024 Cohort

This post presents the basic story behind my current research direction at MATS. I'm looking for feedback on whether the basic premises hold up, and which research questions should I prioritize.

The role of investors in fueling the AGI race is unclear

Most Frontier AI Labs acknowledge the extreme risks of AGI development, invest in safety, and seem to care at least somewhat about making sure their systems don't destroy the world.

However, Frontier Labs currently appear to be in a race to build AGI, which most fear will lead Labs to make risky bets and not take adequate safety measures to avoid catastrophic outcomes, in order to have a chance of winning the race. A common explanation is that Labs are for-profit companies, which prevents them from putting safety first, because investors would force them to put profit first and continue to rapidly develop capabilities even though executives know the risks.

But do investors actually have so much influence over companies that they can force them to build dangerous models against their best judgement?

There are strong economic arguments for why companies that prioritize profit are rewarded in the long run, both through selection effects (profit-seeking companies are less likely to go bankrupt) and through capital allocation (non-profit-seeking companies don't receive investment or don't even get started). However, these mechanisms only work on average over long periods of time, and in the short term investors regularly fail to keep the companies they invest in from making stupid decisions and going bankrupt (e.g. FTX).

So I think it is unclear how much power investors have over Frontier Labs' overall strategy for building AGI, or over specific short-term decisions like whether to train a particular dangerous model. I expect their effective power to vary by company, by type of decision and by timescale. Their power will depend on specific corporate structures, executives, boards, internal cultures, ownership types and applicable corporate laws.

Identifying the power of investors over Labs can inform AI governance strategy

I expect the details of investors' influence over Labs' decisions to be relevant to several strategic considerations:

Whether widespread investor awareness of the extreme dangers of AGI development would change the

...
FLI is hiring across Comms and Ops
by beisenpress — 2024-07-25T00:06:59.433Z

Future of Life Institute (FLI) is hiring multiple roles across communications and operations:

US Communications Manager (Full time)
Operations Associate (Full time)
Video Producer (Part-time contractor)
Content Writer and Researcher (Part-time contractor)

All roles are remote. Get applications in by August 4th!

About the Future of Life Institute

Founded in 2014, FLI is an independent non-profit working to steer transformative technology towards benefitting life and away from extreme large-scale risks. Our work includes grantmaking, educational outreach, and policy engagement.

Our work has been featured in The Washington Post, Politico, Vox, Forbes, The Guardian, the BBC, and Wired.

Some of our achievements include:

Pause Giant AI Experiments, an open letter calling for a 6 month pause on the training of AI systems more powerful than GPT-4. The letter has been signed by more than 30,000 people, including Yoshua Bengio, Stuart Russell, Elon Musk, Steve Wozniak, Yuval Noah Harari, and Andrew Yang.
The Asilomar AI Principles, one of the earliest and most influential sets of AI governance principles.
Slaughterbots, a viral video campaign raising awareness about the dangers of lethal autonomous weapons.
The Future of Life Award, which retrospectively awards unsung heroes who made the world a better place. Past winners include individuals who prevented nuclear wars, helped to eradicate smallpox, and solved the ozone crisis.
Worldbuild.ai, which imagines flourishing futures with strong AI and works out how to get there.

FLI is a largely virtual organization, with a team of 20 distributed across Europe and the US. We have offices in Campbell, California and in Brussels, Belgium, and meet in person as a full team twice a year.
A framework for thinking about AI power-seeking
by Joe Carlsmith — 2024-07-24T22:41:01.685Z

This post lays out a framework I’m currently using for thinking about when AI systems will seek power in problematic ways. I think this framework adds useful structure to the too-often-left-amorphous “instrumental convergence thesis,” and that it helps us recast the classic argument for existential risk from misaligned AI in a revealing way. In particular, I suggest, this recasting highlights how much classic analyses of AI risk load on the assumption that the AIs in question are powerful enough to take over the world very easily, via a wide variety of paths. If we relax this assumption, I suggest, the strategic trade-offs that an AI faces, in choosing whether or not to engage in some form of problematic power-seeking, become substantially more complex.

Prerequisites for rational takeover-seeking

For simplicity, I’ll focus here on the most extreme type of problematic AI power-seeking – namely, an AI or set of AIs actively trying to take over the world (“takeover-seeking”). But the framework I outline will generally apply to other, more moderate forms of problematic power-seeking as well – e.g., interfering with shut-down, interfering with goal-modification, seeking to self-exfiltrate, seeking to self-improve, more moderate forms of resource/control-seeking, deceiving/manipulating humans, acting to support some other AI’s problematic power-seeking, etc.[1] Just substitute in one of those forms of power-seeking for “takeover” in what follows.

I’m going to assume that in order to count as “trying to take over the world,” or to participate in a takeover, an AI system needs to be actively choosing a plan partly in virtue of predicting that this plan will conduce towards takeover.[2] And I’m also going to assume that this is a rational choice from the AI’s perspective.[3] This means that the AI’s attempt at takeover-seeking needs to have, from the AI’s perspective, at least some realistic chance of success – and I’ll assume, as well, that this perspective is at least decently well-calibrated. We can relax these assumptions if we’d like – but I think that the paradigmatic concern about AI power-seeking should be happy to grant them.

What’s required for this kind of rational takeover-seeking? I think about the prerequisites in three categories:

Agential prerequisites – that is, necessary structural features of an AI’s capacity for planning i

...
Llama Llama-3-405B?
by Zvi — 2024-07-24T19:40:07.565Z

It’s here. The horse has left the barn. Llama-3.1-405B, and also Llama-3.1-70B and Llama-3.1-8B, have been released, and are now open weights.

Early indications are that these are very good models. They were likely the best open weight models of their respective sizes at time of release.

Zuckerberg claims that open weights models are now competitive with closed models. Yann LeCun says ‘performance is on par with the best closed models.’ This is closer to true than in the past, and as corporate hype I will essentially allow it, but it looks like this is not yet fully true.

Llama-3.1-405B not as good as GPT-4o or Claude Sonnet. Certainly Llama-3.1-70B is not as good as the similarly sized Claude Sonnet. If you are going to straight up use an API or chat interface, there seems to be little reason to use Llama.

That is a preliminary result. It is still early, and there has been relatively little feedback. But what feedback I have seen is consistent on this.

Prediction markets are modestly more optimistic. This market still has it 29% to be the #1 model on Arena, which seems unlikely given Meta’s own results. Another market has it 74% to beat GPT-4-Turbo-2024-04-09, which currently is in 5th position. That is a big chance for it to land in a narrow window between 1257 and 1287. This market affirms that directly on tiny volume.

Such open models like Llama-3.1-405B are of course still useful even if a chatbot user would have better options. There are cost advantages, privacy advantages and freedom of action advantages to not going through OpenAI or Anthropic or Google.

In particular, if you want to distill or fine-tune a new model, and especially if you want to fully own the results, Llama-3-405B is here to help you, and Llama-3-70B and 8B are here as potential jumping off points. I expect this to be the main practical effect this time around.

If you want to do other things that you can’t do with the closed options? Well, technically you can’t do most of them under Meta’s conditions either, but there is no reason to expect that will stop people, especially those overseas including in China. For some of these uses that’s a good thing. Others, not as good.

Zuckerberg also used the moment to offer a standard issue open source manifesto, in which he abandons any sense of balance and goes all-in, which he affirmed in a softball interview with Rowan Cheung.

On the safety f...
AI Safety Memes Wiki
by plex — 2024-07-24T18:53:04.977Z
Link: https://aisafety.info/questions/MEME/AI-Safety-Memes-Wiki

Extensive collection of memes compiled by Victor Li and other contributors on AI Safety Info, mostly using memes by AI Notkilleveryoneism Memes.

Memes can, at their best, convey key points in a sticky and easily sharable form. Having a index seems potentially quite helpful, and also is a fun resource, so we've adopted it along with other living documents like the AI Safety Videos index. If you know of or want to build a resource which would be a good fit, adding it to aisafety.info is as simple as transferring the Google doc.

Come read many more at AI Safety Info, or contribute ones we're missing on the Google Doc!
Research Discussion on PSCA with Claude Sonnet 3.5
by Robert Kralisch — 2024-07-24T16:53:41.857Z

A while ago, I published a sort of conceptual intro to a cognitive architecture I am working on, the "Prop-room and Stage Cognitive Architecture" (or PSCA for short).
I recently had a somewhat stimulating research chat about it with Claude and wanted to try out sharing it, as it is a potentially easy way for me to share more of my writing and thinking through a dialogue format. On that note, I am also happy to do LW dialogues with anyone who'd be interested!
I didn't try particularly hard to prompt Claude into high performance/interestingness mode, but some of my responses are intended to guide the conversation into that direction. So, if you are confused about why I am telling it something (or tell something in a particular way), chances are this is related.
Here it is:

Robert(1):

Hey Claude, I would like to discuss a novel cognitive architecture design with you today. I'll just share the overview document with you first and you can tell me your initial thoughts if that is alright with you. [Here I linked the text from the PSCA post]

Claude(1):

Certainly, I'd be happy to discuss this novel cognitive architecture design with you. I've read through the overview document you've shared, which details the Prop-room and Stage Cognitive Architecture (PSCA). This is an intriguing and complex proposal for an AI system. Here are my initial thoughts:

Conceptual Framework: The PSCA is an ambitious attempt to create a neuro-symbolic architecture that mimics certain aspects of human cognition, particularly the brain's ability to simulate and focus on relevant aspects of the current scene.
Key Components:

The Stage: A hypergraph representing the current scene with relevance-adjusted resolution.
The Prop-room: A larger, more static hypergraph containing the agent's world model.

Innovative Aspects:

The use of hypergraphs for representing both the current scene and the world model.
The concept of relevance-adjusted resolution for elements in the scene.
The integration of constraint satisfaction for decision-making and action selection.

Potential Strengths:

Flexibility in handling both micro-actions and more abstract planning.
Potential for interpretability, especially in the Stage component.
The architecture's approach to learning and concept formation seems promising for developing natural abstractions.

Challenges:

Implementing effective learning mechanisms at multiple levels simultaneous

...
Reading More Each Day: A Simple $35 Tool
by aysajan — 2024-07-24T13:54:04.290Z

Life is full of clever little hacks. Sometimes the most impactful ones are also the simplest. I recently stumbled upon a basic tool that has remarkably boosted my daily reading time. I'm excited to share this game-changing discovery.

Last Thanksgiving, during a visit to my cousin-in-law's cozy home in Houston, I noticed something peculiar. As we lounged in the dining room enjoying our delicious food, her 11-year old son was periodically reading a book. But what really caught my eye wasn't just his admirable commitment to literature, but rather the mechanism facilitating his reading: a compact, portable book stand.

Not only did this book stand accompany his meals, but it seemed to be his constant companion throughout the day. Two things struck me: first, the sheer volume of reading he was able to consume daily, and second, the ubiquity of the book stand in his routine.

Upon returning home, I decided to try integrating a similar system into my own life. A quick search online led me to find this lightweight, adjustable aluminum stand on Amazon (anything similar would work) for only $35. It seemed a small price to pay for the potential productivity and enjoyment gains.

Setting it up on my kitchen counter, I began my experiment. The transformation was almost immediate. The stand enabled me to:

1. Read while standing. This not only improved my posture but also kept me active.

2. Maintain an ergonomic eye level. Gone were the days of craning my neck downward, causing unnecessary strain.

3. Grab snippets of reading time. Those 'few minutes here and there' began to add up, be it post-work relaxation, pre-bed wind-downs, or coffee breaks. Whenever there was a moment to spare, my book called from its stand.

Within a week, the difference was clear. My earlier seated reading sessions often felt cumbersome, but this newfound approach was refreshing and more accommodating to my daily routine.

Seeing the undeniable benefits, I introduced another book stand to my office space. For those without the luxury of a standing desk, this tool is a game-changer. I found an added advantage in maintaining proper posture even in the office environment. Plus, skimming through lines with my index finger—a technique to increase reading speed—felt more efficient and focused.

Fast forward six months, and my reading volume has increased exponentially. The modest $35 investment has paid rich divid...
You should go to ML conferences
by Jan_Kulveit — 2024-07-24T11:47:52.214Z

This is a second kind of obvious point to make, but if you are interested in AI, AI safety, or cognition in general, it is likely worth going to top ML conferences, such as NeurIPS, ICML or ICLR. In this post I cover some reasons why, and some anecdotal stories.

1. Parts of AI alignment and safety are now completely mainstream

Looking at the "Best paper awards" at ICML, you'll find these safety-relevant or alignment-relevant papers:

Stealing part of a production language model by Carlini et al.

Probabilistic Inference in Language Models via Twisted Sequential Monte Carlo

by Zhao et al.
Debating with More Persuasive LLMs Leads to More Truthful Answers by Khan et al.

Genie: Generative Interactive Environments

Bruce et al.

which amounts to about one-third (!). "Because of safety concerns" is part of the motivation for hundreds of papers.

While the signal-to-noise ratio is even worse than on LessWrong, in total, the amount you can learn is higher - my personal guess is there is maybe 2-3x as much prosaic AI safety relevant work at conferences than what you get by just following LessWrong, Alignment Forum and safety-oriented communication channels.

2. Conferences are an efficient way how to screen general ML research without spending a lot of time on X

Almost all papers are presented in the form of posters. In case of a big conference, this usually means many thousands of posters presented in huge poster sessions.
My routine for engaging with this firehose of papers:

For each session, read all the titles. Usually, this prunes it by a factor of ten (i.e. from 600 papers to 60).
Read the abstracts. Prune it to things which I haven't noticed before and seem relevant. For me, this is usually by a factor of ~3-5.
Visit the posters. Posters with paper authors present are actually a highly efficient way how to digest research:

Sometimes, you suspect there is some assumption or choice hidden somewhere making the result approximately irrelevant - just asking can often resolve this in a matter of tens of seconds.
Posters themselves don't undergo peer review which makes the communication more honest, with less hedging.
Usually authors of a paper know significantly more about the problem than what's in the paper, and you can learn more about negative results, obstacles, or directions people are excited about.

Clear disadvantage of conferences is the time lag; by the ti...
The last era of human mistakes
by owencb — 2024-07-24T09:58:42.116Z
Link: https://strangecities.substack.com/p/the-last-era-of-human-mistakes

Suppose we had to take moves in a high-stakes chess game, with thousands of lives at stake. We wouldn't just find a good chess player and ask them to play carefully. We would consult a computer. It would be deeply irresponsible to do otherwise. Computers are better than humans at chess, and more reliable.

We'd probably still keep some good chess players in the loop, to try to catch possible computer error. (Similarly we still have pilots for planes, even though the autopilot is often safer.) But by consulting the computer we'd remove the opportunity for humans to make a certain type of high stakes mistake.

A lot of the high stakes decisions people make today don't look like chess, or flying a plane. They happen in domains where computers are much worse than humans.

But that's a contingent fact about our technology level. If we had sufficiently good AI systems, they could catch and prevent significant human errors in whichever domains we wanted them to.

In such a world, I think that they would come to be employed for just about all suitable and important decisions. If some actors didn’t take advice from AI systems, I would expect them to lose power over time to actors who did. And if public institutions were making consequential decisions, I expect that it would (eventually) be seen as deeply irresponsible not to consult computers.

In this world, humans could still be responsible for taking decisions (with advice). And humans might keep closer to sole responsibility for some decisions. Perhaps deciding what, ultimately, is valued. And many less consequential decisions, but still potentially large at the scale of an individual’s life (such as who to marry, where to live, or whether to have children), might be deliberately kept under human control[1].

Such a world might still collapse. It might face external challenges which were just too difficult. But it would not fail because of anything we would parse as foolish errors.

In many ways I’m not so interested in that era. It feels out of reach. Not that we won’t get there, but that there’s no prospect for us to help the people of that era to navigate it better.

My attention is drawn, instead, to the period before it. This is a time when AI will (I expect) be advancing rapidly. Important decisions may be made in a hurry. And while automation-of-advice will be on the up, it seems like wildly unprecedented...