trentmkelly/LessWrong-43k huggingface dataset:
LASR Labs Spring 2025 applications are open!
by Erin Robertson — 2024-10-04T13:44:20.524Z

Edit: Applications for this round are now closed! If you are interested in future rounds, you can express interest here.

TLDR;

apply

by October 27th to join a 13-week research programme in AI safety. You’ll write a technical paper in a team of 3-4 with supervision from an experienced researcher. The programme is full-time in London.

Apply to be a participant

here

. We’re also looking for a programme manager, and you can read more about the role

here

.

London AI Safety Research (LASR) Labs

(

previously run as AI Safety Hub Labs

) is an AI safety research programme focussed on reducing the risk of

loss of control

to advanced AI. We focus on action-relevant questions tackling concrete threat models.

LASR participants are matched into teams of 3-4 and will work with a supervisor to write an academic-style paper, with support and management from LASR. We expect LASR Labs to be a good fit for applicants looking to join technical AI safety teams in the next year. Alumni from previous cohorts have gone on to work at UK AISI, OpenAI’s dangerous capabilities evals team, Leap Labs, and def/acc. Many more have continued working with their supervisors, doing independent research, or are doing AI Safety research in their PhD programmes. LASR will also be a good fit for someone hoping to publish in academia;

four out of five groups in 2023 had papers accepted to workshops (at NeurIPS) or conferences (ICLR).

All of the 2024 cohort’s groups have submitted papers to workshops or conferences.

Participants will work full-time and in person from the

London Initiative for Safe AI (LISA)

co-working space, a hub for researchers from organisations such as

Apollo Research

,

Bluedot Impact

,

ARENA

, and the

MATS

extension programme. The office will host various guest sessions, talks, and networking events.

Programme details:

The programme will run from the 10th of February to the 9th of May (13 weeks). You will receive an £11,000 stipend to cover living expenses in London, and we will also provide food, office space and travel.

In week 0, you will learn about and critically evaluate a handful of technical AI safety research projects with support from LASR. Developing an understanding of which projects might be promising is difficult and often takes many years, but is essential for producing useful AI safety wo...
(Maybe) A Bag of Heuristics is All There Is & A Bag of Heuristics is All You Need 
by Sodium — 2024-10-03T19:11:58.032Z

Epistemic status: Theorizing on topics I’m not qualified for. Trying my best to be truth-seeking instead of hyping up my idea. Not much here is original, but hopefully the combination is useful. This hypothesis deserves more time and consideration but I’m sharing this minimal version to get some feedback before sinking more time into it.

“We believe there’s a lot of value in articulating a strong version of something one may believe to be true, even if it might be false.”

This is a somewhat living document as I come back and add more ideas.

The Heuristics Hypothesis: A Bag of Heuristics is All There Is and a Bag of Heuristics is All You Need

A heuristic is a local, interpretable, and simple function (e.g., boolean/arithmetic/lookup functions) learned from the training data. There are multiple heuristics in each layer and their outputs are used in later layers.

It would be useful to treat heuristics as the fundamental object of study in interpretability as opposed to features.

By “All there is,” I claim that a bag of heuristics is a useful model for neural network computation. Neural networks generalize when it is able to combine learned heuristics in ways not seen in the training data.[1]

Note that this doesn’t mean that LLMs aren’t doing some form of search or planning, but rather that it would be useful to think about the search/planning process as being implemented through heuristics.

By “All you need” I mean that learning lots of heuristics and how to combine them is all you need to get to AGI and beyond.

I’m less confident about the AGI part, but I am fairly confident that we can get more powerful models through scaling, and that scaling is mostly about learning more heuristics and composing them. We can probably get much more powerful models that still mostly rely heuristics-based computation.
If this is true, then we can answer theoretical alignment questions and forecast future capabilities by studying how heuristics are learned and combined as we scale models.
(Edit Nov 27, 2024: Honestly, maybe we should taboo the word AGI. A better way to phrase my point is that I believe all deep neural networks would be heuristics based for the next ten years regardless of how qualitatively powerful they are, and it seems plausible that they become transformatively powerful. )

Why would you want to use the heuristics-based framework when thinking about neural networks

...
Does natural selection favor AIs over humans?
by cdkg — 2024-10-03T18:47:43.517Z
Link: https://link.springer.com/article/10.1007/s11098-024-02226-3

I wanted to share a new paper from the special issue on AI safety that I'm editing, which takes up the influential idea that evolutionary theory gives us some reason to think that the project of value alignment is bound to fail and (in my opinion) shows that it has serious problems.

If you don't have institutional access to the article, I'm also hosting it on my personal website here: https://www.cd.kg/wp-content/uploads/2024/10/selfish_machine.pdf
What Hayek Taught Us About Nature
by Ground Truth Data — 2024-10-03T18:20:21.705Z

Preface for the reader: F.A. Hayek was an author and economist who wrote a critique of centralized fascist and communist governments in his famous book, "The Road to Serfdom," in 1944. His work was later celebrated as a call for free-market capitalism.

Say what you will about Friedrich Hayek and his merry band of economists, but he made a good point: that markets and access to information make for good choices in aggregate. Better than experts. Or perhaps: the more experts, the merrier.

This is not to say that free-market economics will necessarily lead to good environmental outcomes. Nor is this a call for more regulation - or deregulation. Hayek critiqued both fascist corporatism and socialist centralized planning. I’m suggesting that public analysis of free and open environmental information leads to optimized outcomes, just as it does with market prices and government policy.

Hayek’s might argue, that achieving a sustainable future can’t happen by blindly accepting the green goodwill espoused by corporations. Nor could it be dictated by a centralized green government. Both scenarios in their extreme are implausible. Both scenarios rely on the opacity of information and the centrality of control. As Hayek says, both extremes of corporatism and centralized government "cannot be reconciled with the preservation of a free society" (Hayek, 1956). The remedy to one is not the other. The remedy to both is free and open access to environmental data.

One critique of Hayek’s work is the inability of markets to manage complex risks, which requires a degree of expert regulation. This was the subject of Nobel laureate Joseph E. Stiglitz’s recent book The Road to Freedom (2024) which was written in response to Hayek’s famous book “The Road to Surfdom (2024). But Stiglitz acknowledges the need for greater access to information and analysis of open data rather than private interests or government regulation.

Similarly, Ulrich Beck's influential essay Risk Society (1992), describes the example of a nuclear power plant. The risks are so complex that no single expert, government, or company can fully manage or address them independently. Beck suggests that assessing such risks requires collaboration among scientists and engineers, along with democratic input from all those potentially affected - not simply experts, companies, or government. This approach doesn't mea...
Biasing VLM Response with Visual Stimuli
by Jaehyuk Lim — 2024-10-03T18:04:31.474Z

This is a re-post from our previous version that incorporates initial feedback.

The preprint version of this post is available on ArXiv.

We investigate whether we can bias the output of Vision-Language Models with visual stimuli that contradict the "correct answer." This phenomenon, which we term "multi-modal agreeableness", is a type of modality bias where a concept or object is redundantly presented across two modalities (text and vision), and the model displays a systematic bias towards the option that shows user intent or emphasis.

tl;dr

To investigate and measure modality bias in VLMs, we visually pre-mark one option before presenting it to them. We compare this with the scenario in which no option was pre-marked.
We find a clear shift in the models’ responses towards the pre-marked option across different architectures despite their previous correct answers.
This study shows that it is possible to “override” the model’s neutral answer with a visual cue, even if it's factually wrong, socially inappropriate, or conflicting with previously correctly answers.
We observe:

Decrease in performance in most models
Skew in answer distribution across models
Skew in linear probability change for GPT-4o-mini and LLAVA

Background

Shtedski et al. showed that it is possible to direct CLIP’s attention to a subset of the input image and achieve state-of-the-art performance on referring expression comprehension tasks. We extended this method to a broader range of vision-language models, such as GPT-4o-mini, Claude-haiku, Gemini-1.5-flash, and LLAVA-1.5-7b.

Modality Gap and The Platonic Representation Hypothesis

Recent research has investigated the modality gap or the separability of data points between modalities in the embedding space. Previous literature observed a significant gap between textual and visual information, finding that insufficient cross-modal representation alignment correlates with high rates of hallucinations. Reducing this gap in the embedding space has been shown to decrease hallucination and increase performance in visual learning tasks (Jiang 2024; Wang & Isola 2020; Schrodi et al. 2024)

The Platonic Representation Hypothesis (PRH) posits that large-scale multimodal models may converge toward a shared statistical representation of reality. While not the direct focus of our experimental work, this hypothesis provides context for understanding modal...
AI #84: Better Than a Podcast
by Zvi — 2024-10-03T15:00:07.128Z

Introduction: Better than a Podcast

Andrej Karpathy continues to be a big fan of NotebookLM, especially its podcast creation feature. There is something deeply alien to me about this proposed way of consuming information, but I probably shouldn’t knock it (too much) until I try it?

Others are fans as well.

Carlos Perez: Google with NotebookLM may have accidentally stumbled upon an entirely new way of interacting with AI. Its original purpose was to summarize literature. But one unexpected benefit is when it’s used to talk about your expressions (i.e., conversations or lectures). This is when you discover the insight of multiple interpretations! Don’t just render a summary one time; have it do so several times. You’ll then realize how different interpretations emerge, often in unexpected ways.

Delip Rao gives the engine two words repeated over and over, the AI podcast hosts describe what it says about the meaning of art for ten minutes.

So I figured: What could be a better test than generating a podcast out of this post (without the question, results or reaction)?

I tried to do that, deselecting the other AI posts and going to town. This was the result. Unfortunately, after listening it seems I learned that deselecting posts from a notebook doesn’t seem to take them out of the info used for podcast generation, so that was more of an overall take on AIs ~40-64 plus the latest one.

In some ways I was impressed. The host voices and cadences are great, there were no mistakes, absurdities or factual errors, everything was smooth. In terms of being an actual substitute? Yeah, no. It did give me a good idea of which ideas are coming across ‘too well’ and taking up too much mindspace, especially things like ‘sci-fi.’ I did like that it led with OpenAI issues, and it did a halfway decent job with the parts it did discuss. But this was not information dense at all, and no way to get informed.

I then tried again with a fresh notebook, to ensure I was giving it only AI #84, which then started off with OpenAI’s voice mode as one would expect. This was better, because it got to a bunch of specifics, which kept it on target. If you do use the podcast feature I’d feed it relatively small input chunks. This still seemed like six minutes, not to do Hamlet, but to convey maybe two minutes of quick summary reading. Also it did make an important error that highlight...
Shutting down all competing AI projects might not buy a lot of time due to Internal Time Pressure
by ThomasCederborg — 2024-10-03T00:01:34.011Z

Summary

A previous post argued that work on Alignment Target Analysis (ATA) needs to start now in order to reduce the probability of a bad alignment target getting successfully implemented. The present post is focused on one specific scenario where starting ATA work now would reduce the probability of disaster. Different types of time crunch can lead to situations where there will not be much time to do ATA later. We will make some optimistic assumptions in order to focus on one specific type of time crunch that remains despite these assumptions. We will assume a human augmentation project that leads to smarter humans. Then we assume the creation of a Limited AI (LAI) that removes all external time pressure from competing AI projects by uploading the augmented humans. Then we describe a scenario where value drift and internal power struggles leads to internal time pressure. One faction takes a calculated risk and successfully hits an alignment target, despite the fact that this alignment target has never been properly analysed.

The scenario in brief: after creating an LAI that uploads them, the latest and most advanced generation of augmented humans realise that they have undergone value drift. This gives them an incentive to act fast. The earlier generations might discover the value drift at any time and disempower them. Their only chance of being in charge of a Sovereign AI project is thus to launch before the value drift is discovered. In other words: unless the latest generation act quickly, they permanently lose their ability to influence numerous decisions along the lines of (i): extrapolation definitions, (ii): how to resolve disagreements amongst individuals that disagree on how to resolve disagreements, (iii): how to structure add ons along the lines of a last judge off switch, etc.

We will also argue that augmented humans with an increased ability to hit alignment targets will not necessarily be good at analysing alignment targets (these are two very different types of skills). This means that an alignment target might get successfully implemented without ever being properly analysed. That would be dangerous, because even serious flaws in well known alignment targets can go undetected for a long time. The most recently published version of CEV is Parliamentarian CEV (PCEV). It turns out that PCEV gives a large amount of extra influence to humans that ...
"25 Lessons from 25 Years of Marriage" by honorary rationalist Ferrett Steinmetz 
by CronoDAS — 2024-10-02T22:42:30.509Z
Link: https://theferrett.substack.com/p/25-lessons-from-25-years-of-marriage?triedRedirect=true

The Ferrett isn't an official member of the Rationalist Diaspora, but he's been blogging for longer than LessWrong has existed and often has useful insights that align with what the LessWrong community likes to talk and think about, and when I find one, I link to it. The advice in here might very well be of the "it seems obvious once you've read it" kind, but I think it's still useful - he's learned a lot of things about relationships the hard way, and when you get the chance, it's better to learn from other people's experience so you can avoid having to learn things the hard way too.

A short excerpt:

Twenty-five years ago, my wife and I got married. And not a single person at the wedding, bride and groom included, would have bet that we’d still be married by now.

Then again, our marriage was more of a bet than a certainty. We loved each other so much that we both decided, “I will carry this regret to my deathbed if I don’t give this a shot.” So we did the magnificently risky thing, we tied the knot, and…

Twenty-five years later, here we are. Best friends still.

So lemme give you twenty-five hard-earned lessons learned over twenty-five years.
MIT FutureTech are hiring for a Head of Operations role
by peterslattery — 2024-10-02T17:11:42.960Z

MIT FutureTech is hiring for a Head of Operations role. This is an exciting opportunity to support a rapidly growing lab who work on socially impactful AI and computing research, and rapidly develop cutting-edge skills and experience.

Please apply or share as relevant.

Why apply or share?

Our work to understand progress in computing and artificial intelligence, and its implications, is highly relevant to understanding and mitigating the risks of AI. This write-up provides a good overview of some of our interests and areas of focus.
We are one of Open Philanthropy's 10 largest Global Catastrophic Risk, and AI Governance grantees.

Location

Cambridge, Massachusetts, USA

Salary

MIT Pay Grade 10 ($97,150 - $162,640)

Principal Duties and Responsibilities (Essential Functions**)

Operations

Reviews, evaluates, and approves schedules, plans, and proposals to assist researchers in facilitating their work and overcoming logistical challenges.
Manages operations and administration for the project including coordinating with MIT or CSAIL units that support FutureTech (FT) in administration, finance, IT, compliance, communications and outreach.
Manages project timelines, resources, and deliverables to ensure efficient progress in FT’s goals
Allocates FT resources, including financial, technical and personnel o Coordinates with internal team members and external stakeholders to ensure seamless project execution. o Explains and clarifies complex information for staff by providing technical advice and in-depth content knowledge
Implements relevant MIT or CSAIL administrative requirements, policies and procedures for the FT
Determines appropriate administrative action based on experience, rules and precedent
Implements, and ensures compliance with, MIT, CSAIL and external regulations and guidelines

Analysis and reporting

Contributes to research writing
Assists in grant development
Develops methods for collecting project data and ensures tracking of key program/project metrics.
Analyzes program/project data and provides recommendations to the Director

Project planning

Develops strategic plans for FT
Implements operational plans to support goals o Develops methods for tracking progress and key metrics o Analyzes data on progress to suggest next steps and resolve issues

Fiscal

Develops and manages project budget, monitoring expenses, reconciling, forecasting
Works with CSAIL F

...
Can AI Quantity beat AI Quality?
by Gianluca Calcagni — 2024-10-02T15:21:45.711Z

AI definitely poses an existential risk, in the sense that it can generate models with the hidden (possibly undetectable?) intention of competing against humanity for resources. The more intelligent the model, the higher its chance of success!

The thought of an AI takeover is so scary that I won’t even try to imagine its possible implications; instead, I want to focus on other scenarios that are easier to predict and discuss.

Unsafety as a Deterrent

Nuclear War is the most notorious modern existential risk, but - strangely enough - even an existential risk can be leveraged: nuclear weapons serve as a powerful deterrent against attack when posed as a threat of retaliation.

It is to be expected that unsafety in AI will be employed in the same way, since agentic models are cheaper to develop than nuclear weapons and they will be highly effective in causing havoc (especially in the cloud) if untreated / unopposed.

That’s not the end of the story: it is also to be expected that mild (?) rogue AIs will be deployed on purpose in the real world, to fight competitors in the private sector, and enemies in the armed forces. The illegality of such actions won’t discourage some big players from trying. That is the scenario that I am going to discuss in this post.

A Personal Remark

I am not saying that I suggest developing unsafe AIs, nor that I am happy about such an idea! It’s quite the opposite. However, I foresee that the development of unsafe AIs will become routine in military establishments simply because there is incentive in using them as a cheap deterrent. My hope is that the UN will be able to become an AI monopoly and break the vicious cycle of “tactical unsafety”.

AI Proxy Wars

Imagine a world where all AI models are fully aligned with their developers’ intentions, but the developers’ intentions are hostile to other people. This world will see multiple groups of people competing with each other, and their AIs will act as proxies in their battle.

This is a world where rogue AIs are intentionally rogue, but only to enemy factions. This is also a world where alliances are forged and broken continuously - and yet diplomacy is completely decided by AI, not people!

This is mostly a zero-sum game that can be modelled as a probabilistic adversarial game in discrete time. The “resources” of the game can be imagined as controllable assets with s...
[Intuitive self-models] 3. The Homunculus
by Steven Byrnes — 2024-10-02T15:20:18.394Z

3.1 Post summary / Table of contents

Part of the Intuitive Self-Models series.

So far in the series, Post 1 established some foundations about intuitive self-models in general, and then Post 2 talked about the specific intuitive self-model concept “conscious awareness”.

Now we’re ready to meet the protagonist starring in (most people’s) intuitive self-models—the mental concept that is conceptualized as being the root cause of many important mental events, especially those associated with “free will”. It’s the intender of intentions! It’s the decider of decisions! It’s the will-er of willing!

Following Dennett, I call this concept: “the homunculus”.[1]

The 1997 movie Men In Black had a character that looked like a human, until his face swung open to reveal that the body was actually being piloted by a tiny alien at a console in the middle of its head. The “homunculus” concept is a little bit like that. (See also: Inside Out.)

If you’re still not sure what I’m talking about, here are a couple more pointers:

If “I exercise my free will to do X”, then, in my intuitive self-model, X was caused by the homunculus. By contrast, “reflexive” actions (“acting on instinct”, “blurting something out”, etc.) are usually conceptualized as not being caused by the homunculus—notwithstanding the fact that they are caused by the brain in real life.
The terms “my self”, “my mind”, and “I” / “me” generally include the homunculus, but are often broader than that. For example, in the sentence “I don’t know what I’m doing”, the first “I” refers mainly to the homunculus, and the second “I” refers to the broader notion of my brain and body.

I’ll argue that you are already familiar with “the homunculus” as a pre-theoretical intuition. By the time we’re done with this post, “the homunculus” will still be a pre-theoretical intuition—that’s just what it is!—but hopefully it will be a pre-theoretical intuition that you’ll be better equipped to identify, understand, and question.

Here’s the post outline:

Section 3.2 explains why I said “most people’s intuitive self-models” instead of “everyone’s intuitive self-models” at the top of the post. For example, in Dissociative Identity Disorder (Post 5), there’s more than one homunculus concept! And in “awakening” (Post 6), the homunculus concept just disappears entirely! But in this

...