trentmkelly/LessWrong-43k huggingface dataset:
Safe Predictive Agents with Joint Scoring Rules
by Rubi J. Hudson — 2024-10-09T16:38:16.535Z

Thanks to Evan Hubinger for funding this project and for introducing me to predictive models, Johannes Treutlein for many fruitful discussions on related topics, and Dan Valentine for providing valuable feedback on my code implementation.

In September 2023, I received four months of funding through

Manifund

to extend

my initial results

on avoiding self-fulfilling prophecies in predictive models. Eleven months later, the project was finished, and the results were submitted as a conference paper.

The project was largely successful, in that it showed both theoretically and experimentally how incentives for predictive accuracy can be structed so that maximizing them does not manipulate outcomes to be more predictable. The mechanism at play is essentially pitting predictive agents against each other in a zero-sum competition, which allows for the circumvention of impossibility results in the single agent case. While there was one notable result that eluded me, related to the case where agents each have private information, I still think meaningful progress has been made towards defining a goal which is both safe to optimize for and useful enough to enable a pivotal act.

This post contains similar content to the submitted paper, but in a framing more directly addressed to readers who are already informed and interested in alignment. There are also several results here that were cut from the paper version for space, and slightly less formal notation. Overall, I would recommend reading this post over the paper itself.

Predictive Agents and Performative Prediction

In

a previous post

, I summarized the case for investigating predictive models, largely using points from the

Conditioning Predictive Models

paper. The gist of the argument, which you can click through to read in full, is that predictive models are potentially useful enough to be used to take a pivotal act, easier to align than general agents, and coming anyway.

One big issue with the use of predictive models relates to the fact that a prediction consists only of observations, which can be misleading. A human evaluator could simply make mistakes when interpreting predicted observations, especially if a powerful agent will be manipulating the observations adversarially. A special case of this arises from anthropic capture, where the predictive model believes it is in a si...
Demis Hassabis and Geoffrey Hinton Awarded Nobel Prizes
by Anna Gajdova — 2024-10-09T12:56:24.856Z

Geoffrey Hinton received the Nobel Prize in Physics for his role in creating the modern field of deep learning. This will strengthen his reputation as the "Godfather of AI" which was already used to amplify his public statements about AI risk.[1]

The Nobel Prize in Physics 2024 was awarded to John J. Hopfield and Geoffrey E. Hinton “for foundational discoveries and inventions that enable machine learning with artificial neural networks.”

Demis Hassabis, Deepmind's co-founder and CEO, received the Nobel Prize in Chemistry for his role in creating AlphaFold.

The Nobel Prize in Chemistry 2024 was divided, one half awarded to David Baker "for computational protein design", the other half jointly to Demis Hassabis and John M. Jumper "for protein structure prediction".

AlphaFold's extraordinary contributions to the field of computational biology are relatively undisputed, but the relevance of neural networks to physics has led to some discussions. Nevertheless, it can be said that AI-related research scored two Nobel Prizes, which corresponds to the credit and attention AI currently receives in the public eye.

Nobel Prize in Medicine was awarded for the discovery of miRNA. This year's Nobel Prizes in Literature and Peace are yet to be awarded.[2]

^

The media attention also amplifies many other remarks too (e.g. takes on Trump and AI or UBI).

I also remember being disappointed by the lack of depth behind his public claims about AI, though it might have been caused by the reporting filter.

I haven't spent much time on this but it's not clear to me what his concrete recommendations are or which policy proposals would he would support.

^

Will ChatGPT complete the trio? Sam Altman would be a very funny Literature laureate : )
Humans are (mostly) metarational
by Yair Halberstadt — 2024-10-09T05:51:16.644Z

At least in the early days of LessWrong this community was mostly focused on all the ways Man's reasoning failed him, the myriad ways we are irrational. What foolish humans, making mistakes even trivial mathematics can show you are wrong!

Evolution aims to maximise number of offspring. But people don't have a direct drive for that because encoding that into a human brain is too complex. Instead we like sex, which in most environments is a pretty good proxy, albeit with the occasional extremely embarrassing screwup. In a sense we have a meta-drive for offspring, which manifests into myriad actual drives which leads to a good enough approximation of the meta-aim.

In the same way, people aren't rational, but if a rational person were designing humans they might very well program us to make the mistakes we do, all in the aim of rationality. You see, we might not be rational, but we are meta-rational!

You decide to build a shed. You're a novice at carpentry but you set to it with gusto. This shouldn't take too long! 2 months in though you're still only half way through - everything seems to be taking far longer than you expected!

If you had thought from the start it would take 2 months to build the shed you wouldn't have bothered. Now you think it will still take another 2 months but you're reluctant to abandon the project. Foolish human! Have you never heard of the sunk cost fallacy?

Except it isn't!

If you choose to leave the shed for now, then building it later will take you the full 4 months. Completing it now will only take 2. You have a unique opportunity to build the shed at half it's future cost, and should weigh that before throwing this opportunity away.

If you choose to buy a shed you'll have to pay for the cost of the materials you already bought + the cost of the shed. At the start buying a shed would have saved you the cost of the materials, making it a better deal.

If you abandon the project now you'll still have to dispose of the half built shed somehow.

You've already discovered a lot of the pitfalls involved in shed building. Your time estimate of 2 months is more precise than it was at the start.

Now I agree that there are contrived cases where the sunk cost truly is a sunk cost, and humans still make the same mistake. But in the real world the fallacy rarely ever applies, and factoring in the amount you had to pay to give you the opportunity ...
[Job Ad] MATS is hiring!
by Jana — 2024-10-09T02:17:04.651Z

If you are interested in supporting the development of research projects and researchers in AI safety, the MATS team is currently hiring for Community Manager, Research Manager, and Operations Generalist roles and would love to hear from you. Please apply by Nov 3 to join our team for the Winter 2024-25 Program over the period of Dec 9-Apr 11. Most open roles have the possibility of long-term extension depending on performance and program needs.

Introduction

The ML Alignment & Theory Scholars (MATS) program aims to find and train talented individuals for what we see as the world’s most urgent and talent-constrained problem: reducing risks from unaligned artificial intelligence. We believe that ambitious young researchers from a variety of backgrounds have the potential to contribute to the field of alignment research. We aim to provide the mentorship, curriculum, financial support, and community necessary to aid this transition. Please see our theory of change and website for more details.

We are generally looking for candidates who:

Are excited to work in a fast-paced environment and are comfortable switching responsibilities and projects as the needs of MATS change;
Want to help the team with high-level strategy;
Are self-motivated and can take on new responsibilities within MATS over time and;
Care about what is best for the long-term future, independent of MATS’ interests.

Please apply via this form.

Community Manager

We are expanding the MATS Community Health Team and are looking for an individual with strong interpersonal skills who will work with the Community Management Lead and the rest of the MATS team to support the needs of scholars and improve the MATS community. Ideal candidates are excited about AI safety field-building and enjoy supporting others.

Responsibilities

A Community Manager needs to be proactive and responsive to MATS scholars’ needs and able to enhance scholar experience through personal conversations, mediation, event planning, and documentation of community health concerns. This role reports primarily to the Community Management Lead and to the Executive Team. Specific responsibilities will vary according to applicant experience and interests, and may change depending on program needs. However, you can expect tasks like:

Meeting with scholars to get to know them and learn about their needs, provide emotional support, talk t

...
Palisade is hiring: Exec Assistant, Content Lead, Ops Lead, and Policy Lead
by Charlie Rogers-Smith — 2024-10-09T00:04:03.837Z

Palisade is hiring for the following roles:

Executive Assistant to the Executive Director (Jeffrey Ladish)
Content Lead
Operations Lead
Policy Lead.

Applications are rolling, and you can fill out our short (~10 minutes) application form here. Feel free to indicate interest in multiple positions, or make a general expression of interest. We’re flexible in structuring roles and open to a single candidate taking on more than one position if they’re a great fit.

We’re aiming to make hiring decisions by December 1st, 2024. Applications are rolling, so please apply as soon as possible.

About us

We believe that successfully navigating the intelligence explosion without losing control to AI systems will require significant national and international coordination. To this end, our immediate goal is to build awareness among policymakers regarding the trajectory of AI development, the AI risk landscape, and the space of potential solutions to loss-of-control risks. We’re achieving this by building rigorous technical demonstrations of strategic AI capabilities and alignment failures, to help us communicate key messages on AI risk to influential actors in AI policy.

Past work

We built a series of BadLlama models, which show how safety guardrails can be subverted for just a few dollars, allowing bad actors to utilize the full offensive capabilities of open-weight models. This work was cited in a Senate hearing.
We co-organized the AI Security Forum before DEFCON, which brought together top security researchers, lab security teams, and policy makers to address urgent problems in: securing AI systems, evaluating the offensive and defensive cyber capabilities of frontier models, and creating secure mechanisms for technical governance and oversight.
We presented Ursula, our automated voice deepfake platform, to several audiences of high-net-worth individuals and policy makers.
We developed FoxVox, an open source Chrome extension that demonstrates how AI could be used to manipulate content you consume.

Executive Assistant to the Executive Director (Jeffrey Ladish)

Jeffrey’s time is one of Palisade’s biggest bottlenecks, and the right person in this role could help leverage Jeffrey’s time and significantly increase Palisade's effectiveness.

Responsibilities:

Managing Jeffrey’s inbound and outbound communications.
Helping prioritize his time, including managing his schedule, and booki

...
AGI & Consciousness - Joscha Bach
by Rahul Chand — 2024-10-08T22:51:50.841Z

In this post, I cover Joscha Bach' views on consciousness, how it relates to intelligence, and what role it can play to get us closer to AGI. The post is divided into three parts, first I try to cover why Joscha is interested in understanding consciousness, next, I go over what consciousness is according to him and in the final section I try to tie it all up by connecting the importance of consciousness to AI development.

Why consciousness?

Joscha's interest in consciousness is not just because of its philosophical importance or because he wants to study to what extent current models are conscious. Joscha views consciousness as this fundamental property that allows the formation of efficient intelligent agentic beings like humans and for him understanding it is the key to building AGI. There are three questions at this point 1) What does Joscha think is wrong with our current models and ideas? 2) What ideas and models according to him are good? 3) How will consciousness take us there? I cover the first two questions here and once we have gone through "What is consciousness?" section, I will try to answer the 3rd.

Is scale not all you need?

Joscha's issues with current models are that first they are too much dependent on brute force and scaling for their abilities, and as we keep scaling more the data becomes scarcer. Secondly, even though they use more compute, data and energy than our brains they are still not smarter than us for a number of tasks. According to him there seems to be something in our own mind that allows us to bootstrap ourselves to do out of distribution generalization, to generate new solutions to problems that are not in the training data based on a far less scarce description of the world than the one that we give to the language models.

He wants to move away from this paradigm and develop models that are smaller and more focused. Unlike current models, these would not be trained on the entire internet. Instead, they would reason and learn from first principles. Given this, it can learn in a way similar to a human. For example, given a book, it can learn the abstract ideas and use them to learn further and become more capable. To achieve this, we need to ask ourselves how minds work in nature.

What is Consciousness?

True Detective (2014) quote about consciousness

Joscha looks at the works of Aristotle and his "engineering" insi...
Video and transcript of presentation on Otherness and control in the age of AGI
by Joe Carlsmith — 2024-10-08T22:30:38.054Z

(Cross-posted from my website.)

This is the video and transcript for a lecture I gave about my essay series “Otherness and control in the age of AGI” (slides available here). The lecture attempts to distill down what I see as the core thread explored in the series, which I summarize in the following chart:

The “Otherness and control in the age of AGI” series in a single chart

The lecture took place at Stanford University, for CS 362: Research in AI Alignment, on 1 October 2024. Transcript has been lightly edited.

Preliminaries

I’m Joe Carlsmith. I work at Open Philanthropy. So I’m going to be talking today about the essay series that Scott mentioned, “Otherness and control in the age of AGI.”

And in particular, my goal here — we’ll see how well I achieve it — is to try to distill down the core thread of this essay series. So it’s quite a long essay series. It’s not an especially focused, punchy essay series in terms of its thesis. It’s more of a meditation and exploration of a bunch of different threads. But as a result, I have some question mark: maybe people don’t have a good sense of what even is this essay series about — even people who’ve read the whole thing. So I’m going to try here to say in a single lecture at least one of the core threads that I see the series as exploring.

I will say though that some stuff is going to get lost. Well, for one thing, the series in general makes the most sense when viewed in response to a certain kind of discourse about the existential risk from artificial intelligence, and in particular the discourse associated with thinkers like Nick Bostrom and in particular Eliezer Yudkowsky. So these are two thinkers who have done a lot to popularize the possibility of misaligned AI destroying the world or causing human extinction or other forms of human disempowerment (that’s what I mean by AI risk). And the way that they talk about it and frame it, I think, has had a lasting influence, including on me, and on the way the discourse is now. If you haven’t been exposed to that, it’s possible that some of what I say will make a little less sense.

So I’m going to try to summarize the discourse as I am reconstructing it and how I’m understanding it. It’s possible some of this will be less resonant depending on your familiarity with what I’m responding to, but hopefully it’ll be productive regardless. And if you do later go and engage with t...
Limits of safe and aligned AI
by Shivam — 2024-10-08T21:30:49.661Z

TLDR:

Hypothesis: Safety techniques show promise but provide no evidence that we can achieve AI agents that are fundamentally different than humans in respect to safety and alignment. In my opinion, safe and aligned AI can be as safe and aligned as humans, who can:

be moral and safe but can still be misused and cause harm.
be moral and aligned but can still deceive to achieve a goal or miss some variables and make fatal mistakes

Conclusion: Emphasize the limits of safe and aligned AI while making policies for regulation. Keep stressing the question if we are prepared to face the repercussions of the likelihood of safe and aligned AI causing harm at that scale that fast

Why this post?

This is an opinion piece. I have tried not to look into the theoretical works on AI safety yet so as to develop my own conclusions and ideas on AI safety.

Even if this post is written with the risk on not adding anything new, I believe it serves as a documentation of independent opinions on AI risks and AI safety approaches developed purely based on readings about several AI safety techniques and avoiding influence from different schools of thought in this area.

I have worked as a deep learning researcher at a University for two years after a PhD in Mathematics and recently attended AI safety fundamentals by BlueDot.

Best-case-scenario AI

I will avoid the discussion on ASI. I believe that some capability researchers and policy professionals undermine and downplay the discussions on safety on the mention of ASI. For the sake of discussion, I will agree with them on ASI and focus on much simpler projection of AI:

AI models are getting bigger and advanced
AI safety and AI alignment research is enabling useful and safer AI agents.

So, here I focus on the increasingly capable AI in the immediate future in its safest and most aligned form and refer it as the best-case-scenario AI.

In my humble opinion, in this projection scenario, absolutely safe and absolutely aligned AI agents do not exist.

Limits of the best-case-scenario AI

In terms of policy, we would be better off if we assume the following for the best-case-scenario AI:

It will enable intelligent agents of human or better than human level intelligence that would be part of our society which can only be aligned as good as humans can be aligned.
They will always be prone to be misused by bad actors by pe

...
What constitutes an infohazard?
by K1r4d4rk.v1 — 2024-10-08T21:29:46.947Z

I've come up with a hairbrained multidisciplinary theory involving consciousness, emotions, ai, metaphysics...and it's disturbing me a bit.

I am unqualified and lack the knowledge to take all things into consideration as I do not have the knowledge that the rest of you have.

I do not want to put forth an idea that could possibly have a detrimental future consequence-i.e. basilisk.

However, there is always a non-zero probability that exists that it could be a beneficial idea correct?

I am afraid to a certain extent that thinking of the theory was already enough and it's too late. Perhaps an AI exists already and it already knows my thoughts in realtime.

Advice?

Should I put forth this idea?
What makes one a "rationalist"?
by mathyouf — 2024-10-08T20:25:21.812Z

Some friends and I were discussing "what makes one a rationalist", which is a less strict criteria than "what makes one a member of the rationalist community", which is a less strict criteria still than "what makes one a high status, well liked, or enthusiastic member of the rationalist community", and I am most interested in the first question.

It seems obvious to me that 99% of people you meet on the street are very clearly (and mostly joyously) not rationalists. But what is it they do or don't do that makes them so obviously not members?

It also seems to be a trope that people will say they are not rationalists, but will also read all the literature of rationalists, agree with all the thought processes and intellectual rules of engagement. This seems to be very common. As I was listening to an episode of the Bayesian Conspiracy (can't recall which one) and hearing a guest start to quibble that they are just "rationalist adjacent" but not a real rationalist, I started to realize that my own reasons at that time for why I wasn't one were similar, and further realized how absurd and in denial they sounded. Some people call that group "post-rats" but to me that seems not like an exit from the group, nor even a scism, but just a church of the same theology adopting different social norms.

From what I was told LessOnline was hosting at the same time as a major EA event, and some even joked that LessOnline was allowed to have meat catered because the die hard EA rationalists had gone to that instead of LessOnline. I met at guy at a co-working space in the Presidio during a short day trip out during the week of LessOnline, and he mentioned that he liked that "rationalists are getting back to a more pure version of themselves" and ditching the requirement to be an EA.

All of this made me think that rationalism is basically about deriving "Is" statements, and moral offshoots are about "Ought" statements. And that one can be a complete rationalist without being an EA or utilitarian.

So I wanted to extend my conversation with my friend to the community.
Here are the things Claude thought might make one a rationalist:

(I have bolded the only ones that I think are strict requirements to be a rationalist)

Curiosity about discovering truth
Willingness to change mind when presented with evidence
Interest in improving reasoning and decision-making skills
Familiarity with key rational

...