trentmkelly/LessWrong-43k huggingface dataset:
Searching for phenomenal consciousness in LLMs: Perceptual reality monitoring and introspective confidence
by EuanMcLean — 2024-10-29T12:16:18.448Z

To update our credence on whether or not LLMs are conscious, we can ask how many of the Butlin/Long indicator properties for phenomenal consciousness are satisfied by LLMs. To start this program, I zoomed in on an indicator property that is required for consciousness under higher-order theory, nicknamed “HOT-2”: Metacognitive monitoring distinguishing reliable perceptual representations from noise. Do today’s LLMs have this property, or at least have the prerequisites such that future LLMs may develop it?

In this post, I’ll describe my first-pass attempt at answering this question. I surveyed the literature on LLM capabilities relevant to HOT-2, namely LLM metacognition, confidence calibration and introspection. There is evidence that LLMs have at least rudimentary versions of each capability. Still, the extent to which the exact results of the experiments translate to properties relevant to HOT-2 requires further clarification. I propose a new experiment that more directly probes the properties necessary for HOT-2.

This was a useful exercise for testing how smoothly one can translate a Butlin/Long indicator property into a concrete empirical signature one can look for in an LLM. The main lesson was that more conceptual work is needed on what words like introspection or metacognition mean when applied to LLMs. Overall, there is plenty of tractable work to be done in this program, and I encourage anyone interested in consciousness in LLMs to consider picking up one of the threads I introduce below.

This work was carried out during the PIBBSS research fellowship 2024, under the mentorship of Patrick Butlin. Thanks to Patrick and the PIBBSS team for all the help and support 🙏 and thanks to Guillaume Corlouer for feedback on a draft.

Summary

Might current or future LLMs have phenomenal consciousness? Is there any way we can test such a question empirically? I think yes. While we don’t yet know what consciousness is, there are many well-established theories that constitute a better guess than randomly guessing. Butlin & Long 2023 studied a number of neuroscientific computational functionalist theories of consciousness and extracted from these a list of “indicator properties” one could look for in AI systems that would indicate consciousness under each theory. The more indicator properties a system has, the more likely we should consider it to ...
AI #87: Staying in Character
by Zvi — 2024-10-29T07:10:08.212Z

The big news of the week was the release of a new version of Claude Sonnet 3.5, complete with its ability (for now only through the API) to outright use your computer, if you let it. It’s too early to tell how big an upgrade this is otherwise. ChatGPT got some interface tweaks that, while minor, are rather nice, as well.

OpenAI, while losing its Senior Advisor for AGI Readiness, is also in in midst of its attempted transition to a B-corp. The negotiations about who gets what share of that are heating up, so I also wrote about that as The Mask Comes Off: At What Price? My conclusion is that the deal as currently floated would be one of the largest thefts in history, out of the nonprofit, largely on behalf of Microsoft.

The third potentially major story is reporting on a new lawsuit against Character.ai, in the wake of a 14-year-old user’s suicide. He got hooked on the platform, spending hours each day, became obsessed with one of the bots including sexually, and things spiraled downwards. What happened? And could this spark a major reaction?

Table of Contents

Top story, in its own post: Claude Sonnet 3.5.1 and Haiku 3.5.

Also this week: The Mask Comes Off: At What Price? on OpenAI becoming a B-corp.

Language Models Offer Mundane Utility. How about some classical liberalism?

Language Models Don’t Offer Mundane Utility. That’s not a tree, that’s my house.

Deepfaketown and Botpocalypse Soon. The art of bot detection, still super doable.

Character.ai and a Suicide. A 14 year old dies after getting hooked on character.ai.

Who and What to Blame? And what can we do to stop it from happening again?

They Took Our Jobs. The experts report they are very concerned.

Get Involved. Post doc in the swamp, contest for long context window usage.

Introducing. ChatGPT and NotebookLM upgrades, MidJourney image editor.

In Other AI News. Another week, another AI startup from an ex-OpenAI exec.

The Mask Comes Off. Tensions between Microsoft and OpenAI. Also see here.

Another One Bites the Dust. Senior Advisor for AGI Readiness leaves OpenAI.

Wouldn’t You Prefer a Nice Game of Chess. Questions about chess transformers.

Quiet Speculations. Life comes at you fast.

The Quest for Sane Regulations. OpenAI tries to pull a fast one.

The Week in Audio. Demis Hassabis, Nate Silver, Larry Summers and many more.

Rhetorical Innovation. Citi

...
A path to human autonomy
by Nathan Helm-Burger — 2024-10-29T03:02:42.475Z

"Each one of us, and also us as the current implementation of humanity are going to be replaced. Persistence in current form is impossible. It's impossible in biology; every species will either die out or it will change and adapt, in which case it is again not the same species. So the next question is once you've given up the idea that you can stay exactly as you are, what would you like to be replaced by?"

Michael Levin [1]

But if the technological Singularity can happen, it will. Even if all the governments of the world were to understand the "threat" and be in deadly fear of it, progress toward the goal would continue. In fiction, there have been stories of laws passed forbidding the construction of "a machine in the form of the mind of man". In fact, the competitive advantage -- economic, military, even artistic -- of every advance in automation is so compelling that passing laws, or having customs, that forbid such things merely assures that someone else will get them first.

Vernor Vinge, Singularity

A Path to Human Autonomy

A future with human empowerment, or even survival, is not a given. I argue that there is a narrow path through the unprecedented existential risks we face. If successful, we need not relinquish the reins of the future. This path requires challenging our assumptions about what it means to be human. We must open our hearts to diversity, more than ever before.

In this essay I attempt to lay out a coherent [2] plan for humanity to address the radical changes ahead. Many of the other plans recently published are incoherent, by which I mean that they neglect key strategic details which make their proposed plans unworkable or assume a particular way that certain future events will resolve. In striving to make this plan coherent, I aim to address what I see as a worst case scenario with above 1% likelihood. Namely, that AI progress and Biotech progress continue accelerating, that there is no sigmoid plateau of these techs. Sometime within the next five years we see a combination of: AI capable at nearly all computer tasks at above average human level, AI becoming competent at multi-step agentic tasks, AI sufficiently capable to initiate a recursive self-improvement process, substantial algorithmic advances which bring down the cost of creating an AI agent, AI capable of controlling robotic actuators to...
D&D.Sci Coliseum: Arena of Data Evaluation and Ruleset
by aphyer — 2024-10-29T01:21:03.075Z

This is a follow-up to last week's D&D.Sci scenario: if you intend to play that, and haven't done so yet, you should do so now before spoiling yourself.

There is a web interactive here you can use to test your answer, and generation code available here if you're interested, or you can read on for the ruleset and scores.

RULESET

A character's class/race/level/items boil down to giving them scores in two separate stats: Speed and Power.

Stat Calculation

Each parameter of a character has an effect on their stats:

Class: Your class provides base stats as follows:

Knight: 2 Speed/12 Power
Warrior: 4 Speed/10 Power
Ranger: 6 Speed/8 Power
Monk: 8 Speed/6 Power
Fencer: 10 Speed/4 Power
Ninja: 12 Speed/2 Power

Race: Your race modifies stats as follows:

Dwarf: -3 Speed, +3 Power
Human: No Effect
Elf: +3 Speed, -3 Power

Level: Each level provides +1 Speed and +1 Power.
Items: Boots of Speed +X grant +X Speed, Gauntlets of Power +Y grant +Y Power.

So, for example, the following characters all have Speed 10 and Power 10:

A Level 3 Elf Warrior (Speed 4 + 3 + 3, Power 10 - 3 + 3)
A Level 3 Dwarf Fencer (Speed 10 - 3 + 3, Power 4 + 3 + 3)
A Level 2 Human Ranger with Boots of Speed +2 (Speed 6 + 2 + 2, Power 8 + 2)
A Level 2 Human Monk with Gauntlets of Power +2 (Speed 8 + 2, Power 6 + 2 + 2)
A Level 1 Elf Knight with Boots of Speed +4 (Speed 2 + 3 + 1 + 4, Power 12 - 3 + 1)
A Level 1 Dwarf Ninja with Gauntlets of Power +4 (Speed 12 - 3 + 1, Power 2 + 3 + 1 + 4)

These characters are, from a mechanical perspective, all completely identical. They will behave the same in every possible matchup.

Particular congratulations are due here to simon, who managed to figure out this whole mapping within 24 hours (actually posting an essentially-complete solution before the scenario even made it to the frontpage).

Combat

When two characters fight, each one calculates a Combat Score:

In the brutal one-on-ones of Arena combat, which can end within seconds, being just a bit faster than your opponent is vital. Whoever's Speed is higher wins initiative and gets the first strike, which counts for +8 to Combat Score.
Power is also very useful, but it doesn't have a sharp cutoff the way Speed does. Each character adds their Power to their Combat Score.
Each player rolls 1d6 and adds that to their Combat Score.[1]
Whoever ends up higher wins. In the case of a tie, the winner is decided at random.

So, ...
Gwern: Why So Few Matt Levines?
by kave — 2024-10-29T01:07:27.564Z
Link: https://gwern.net/matt-levine

Matt Levine is the most well-known newslettrist (“Money Stuff”) in the financial industry, having blogged or written since 2011, finding his niche in popularization after stints in Wall Street & law. His commentary is influential, people leak to him, he sometimes interviews major figures (notoriously, Sam Bankman-Fried) or recounts inside information, and a number of phrases like his “laws of insider trading” (specifically, how not to) have gained currency to the point where readers can now do much of the work of sourcing an issue for him.

He is read by hundreds of thousands of readers (including myself)—everyone from shoeshine boy to billionaire. The size of his audience is respectable, but perhaps its most remarkable feature is that many of those readers have nothing to do with the financial industry. Though his newsletter is officially a Bloomberg News newsletter which he simply writes, many of his readers will visit Bloomberg solely for him, and indeed, might have little idea who or what a Bloomberg is. Nevertheless, readers loyally tune in for each installment every few days to learn about arcane financial instruments they have never heard of before, and (except for Levine) never will again.

One might ask (and indeed, a billionaire once did), “where are the other Matt Levines?” or “who are the Matt levines of other fields?” That is, where are the Matt Levines of, say, chemistry or drug development[1], who explain & popularize other major industries which are vital to modern life, directly or indirectly appear in the news often, and yet people are widely ignorant of it, and deeply misunderstand its fundamental dynamics? Why don’t we have a Matt Levine for every industry? Where is the Levine of, I don’t know, petroleum refining or fracking, of shipping containers? Are we just in need of a good list of recommendations? Or could we just set up a prize to coax out some potential Levines in other industries?

Much more at the link

^

While I enjoy Derek Lowe, the extent to which his posts are inside-baseball and do not repeat themes, or only repeat many years apart, emphasize the contrast with Levine
October 2024 Progress in Guaranteed Safe AI
by Quinn — 2024-10-28T23:34:51.689Z
Link: https://gsai.substack.com/p/october-2024-progress-in-guaranteed

Safeguarded AI TA 1.4 funding call

A way of verifying quantitative properties of program pairs

The lean zulip aggregates recent ICLR submissions.

If rust's compiler wasn't strong enough already...

verified controller synthesis for heterogeneous dynamical systems

A dafny coding benchmark
5 homegrown EA projects, seeking small donors
by Austin Chen — 2024-10-28T23:24:25.745Z

What do I mean by “homegrown”? These projects are:

Local: Creators have a good track record in the EA or AI Safety community
Modest: The amount requested is not large; $5k would be meaningful
Overlooked: Not already backed by large institutional funders like OpenPhil

If you’re a small donor or earn to give, consider giving to projects like these:

1. Feature-length documentary on SB 1047

By Michael Trazzi — $16k raised of $55k

Michael has already recorded interviews with the main characters of SB 1047: sponsors like Scott Wiener and Dan Hendrycks, proponents like Zvi Mowshowitz and Holly Elmore, and opponents like Dean Ball and Timothy B Lee. Now he needs the funding to turn it into a 1-hour feature documentary. This is a rare chance to sponsor a high-quality video narrative, and share it beyond our existing ecosystem. I’ve personally donated $10k towards this, and expect Michael to be able to effectively use much more.

More info & donate here:

https://manifund.org/projects/finishing-the-sb-1047-documentary-in-6-weeks

2. Overview of AI Safety in 2024

By Gavin Leech — $8k of $17.6k raised

Gavin Leech is a forecaster, researcher and founder of

Arb

; he’s proposing to re-rerun a 2023 survey of AI Safety. The landscape shifts pretty quickly, so I’d love to see what’s changed since last year.

As I was writing this, regrantor Neel Nanda funded it to the minimum $8k ask! Neel adds:

I think collections like this add significant value to newcomers to the field, mostly by being a list of all areas worth maybe thinking about, and key links (rather than eg by providing a lot of takes on which areas are more or less important, unless the author has excellent taste). Gavin has convinced me that the previous post gets enough traffic for it be valuable to be kept up to date.

More info & donate here:

https://manifund.org/projects/shallow-review-of-ai-safety-2024

3. Podcast series on Effective Altruism’s values

By Elizabeth Van Nostrand — $1.3k raised of $2.6k

Elizabeth & Timothy’s initial podcast was very well received, drawing extensive, thoughtful comments

from a variety of folks

. I’d be excited to see them continue this series, especially if they bring in folks involved with steering the EA community (like Sarah Cheng, who has

extensively engaged

with their points)

More info & donate here:

https://manifund.org/projects/elizabeth-and-timothy-podcast-on-values-in-effectiv

...
How might we solve the alignment problem? (Part 1: Intro, summary, ontology)
by Joe Carlsmith — 2024-10-28T21:57:12.063Z

Introduction/summary

In my last post, I laid out my picture of what it would even be to solve the alignment problem. In this series of posts, I want to talk about how we might solve it.

To be clear: I don’t think that humans necessarily need to solve the whole problem – at least, not on our own. To the contrary, I think we should be trying hard to get help from sufficiently capable and trusted AIs.[1] And I think various paths that my last post would class as “avoiding” or “handling but not solving” the problem, rather than “solving it,” are important to consider too, especially in the near-term.[2]

I’ll discuss this more in future posts. Still, even if we ultimately need AIs to help us solve the problem, I expect it to be useful to have as direct a grip as we can, now, on what the full problem is, what makes it difficult, and what approaches are available for solving it. More on why I think this below.

A few other meta notes:

The content here is rough. I’m hoping, later, to revise it, along with some other work (including some of the future posts just mentioned), into something more polished and cohesive. But for various reasons, I wanted to get it out there on the earlier side.

Partly because I think we ultimately want to get lots of AI help on this problem, my aim here isn’t to defend one specific “solution.” Rather, my main aim is to improve our thinking about the space of solutions overall (though I do give various takes on how promising different approaches seem, and what’s required for them to work). Maybe I’ll try something more ambitious in future.[3] But I’m not actually sure what level of ambition in this respect is appropriate, from our current position. And regardless: this is where I’m at so far.

As often in discussions of aligning/controlling advanced AI systems: I think it’s plausible that various of the AIs at stake in this discussion would be moral patients by default; and if they are, many if not all of the interventions I discuss raise serious ethical concerns. I’m leaving discussion of this issue to future work. But I want to keep acknowledging it regardless.[4]

Summary of the series

The series is in four parts. Here’s a summary of the posts that have been released thus far (I’ll update it as more come out).

Part 1 -- Ontology

The first post (this one) lays out the general ontology I’ll use for thinking about approaches to...
Enhancing Mathematical Modeling with LLMs: Goals, Challenges, and Evaluations
by ozziegooen — 2024-10-28T21:44:42.352Z

Introduction

Motivation

At QURI, we've been investing in developing and analyzing mathematical models, recently focusing on Squiggle AI, a system that leverages LLMs to generate models using the Squiggle language. We believe LLMs hold substantial potential for improving reasoning and are keen to explore how best to integrate them with mathematical modeling.

While recent attempts to automate narrow forecasts using LLMs have shown promise, mathematical models can offer far greater expressiveness and power. We're particularly interested in LLMs' ability to generate complex forecasts and explanations through formal modeling. However, the lack of clear evaluations or benchmarks for these models poses a challenge. This essay aims to dissect the key aspects of mathematical models and suggest ways they can be automated and evaluated effectively.

Reflections / Epistemic Status

This post summarizes roughly a week of focused thinking on mathematical modeling and LLMs, building on my work with Squiggle AI. While I'm confident in many details of what we want from mathematical models, I have more uncertainty about the broader categorization framework presented here.

The intersection of LLMs and mathematical modeling is a niche topic in our community, and I expect low engagement with this piece. However, this niche status means this area is neglected. More attention here, even by casual readers, could go a long way.

I used LLMs for help writing this post. This has resulted in a dryer writing style, but I think it made things more understandable. If people suggest prompts for preferred writing styles, I'd be interested in those for future pieces.

Some Recent Related Posts

My Current Claims and Cruxes on LLM Forecasting & Epistemics
Squiggle AI: An Early Project at Automated Modeling
The RAIN Framework for Informational Effectiveness

The Need for a Nuanced Understanding of Mathematical Models

Stakeholders often have varying perspectives on the value of mathematical models:

Decision-makers may view them as adjustable tools for quick scenario analysis.
Analysts might see models as means to demonstrate reasoning processes.
Researchers could focus on computationally intensive models for precise parameter estimation.

These differing views highlight the necessity for a more nuanced understanding of what mathematical models can offer. Evaluating and optimizing ma...
AI & wisdom 3: AI effects on amortised optimisation
by L Rudolf L — 2024-10-28T21:08:56.604Z
Link: https://rudolf.website/wisdom3/

Written for the AI Impacts essay competition on the automation of wisdom and philosophy

Having waxed philosophical about wisdom and amortised optimisation, I now turn to the question of what AI will concretely do:

How will AI affect human mechanisms for amortised optimisation, such as cultural evolution & written knowledge?

How can AI help us avoid strategic errors? I consider two types:

Mistakes in planning for big things.

Having the wrong frame/ontology for a problem.

A note on amortised and direct optimisation in LLMs

Current LLMs are products of amortised optimisation: they're trained with vast compute on massive datasets to create a distilled function-approximator, are not known to do significant direct optimisation during inference, and do relatively little work during inference compared to training. GPT-4's training is rumoured to have required on the order of 1025 FLOPs, compared to about5 times1014 FLOPs to generate a token, so GPT-4 could generate 20 billion tokens with the compute it was trained on. (In comparison, humans could say only about 300k words in the working time it takes us to complete a 16-year education.)

So: are current LLMs the peak of wisdom over intelligence?

In the amortised v direct optimisation framework:

LLMs are products of amortised optimisation, and most of their strength does lie in doing tasks that are more like amortised optimisation: distilling and synthesising information from their past "experience", and being extremely knowledgeable. LLMs already play the role of a wise guru to many: any programmer these days will use LLMs like they might have used an especially-available senior programmer who knows the best established practice for every scenario. It is true that they seem worse at having useful world models than humans are, and their wisdom runs less deep. More than that, perhaps, their ability to seem wise is dented by their blandness, a result both of their pre-training to mimic average internet text, and the fine-tuning done on them. They also seem quite bad at epistemics; as discussed in the first part, some kind of good epistemics is often (but not always) required to be able to make good use of the products of amortised optimisation, even if you have them at hand. Currently, that has to be largely provided by the human who's interacting with the LLM.

The extent to which LLMs do direct optimisatio

...