trentmkelly/LessWrong-43k huggingface dataset:
Most People Don't Realize We Have No Idea How Our AIs Work
by Thane Ruthenis — 2023-12-21T20:02:00.360Z

This point feels fairly obvious, yet seems worth stating explicitly.

Those of us familiar with the field of AI after the deep-learning revolution know perfectly well that we have no idea how our ML models work. Sure, we have an understanding of the dynamics of training loops and SGD's properties, and we know how ML models' architectures work. But we don't know what specific algorithms ML models' forward passes implement. We have some guesses, and some insights painstakingly mined by interpretability advances, but nothing even remotely like a full understanding.

And most certainly, we wouldn't automatically know how a fresh model trained on a novel architecture that was just spat out by the training loop works.

We're all used to this state of affairs. It's implicitly-assumed shared background knowledge. But it's actually pretty unusual, when you first learn of it.

And...

Relevant XKCD.

I'm pretty sure that the general public doesn't actually know that. I don't have hard data, but it's my strong impression, based on reading AI-related online discussions in communities not focused around tech, talking to people uninterested in AI advances, and so on.[1]

They still think in GOFAI terms. They still believe that all of an AI's functionality has been deliberately programmed, not trained, into it. That behind every single thing ChatGPT can do, there's a human who implemented that functionality and understands it.

Or, at the very least, that it's written in legible, human-readable and human-understandable format, and that we can interfere on it in order to cause precise, predictable changes.

Polls already show concern about AGI. If the fact that we don't know what these systems are actually thinking were widely known and properly appreciated? If there weren't the implicit assurance of "someone understands how it works and why it can't go catastrophically wrong"?

Well, I expect much more concern. Which might serve as a pretty good foundation for further pro-AI-regulations messaging. A way to acquire some political currency you can spend.

So if you're doing any sort of public appeals, I suggest putting the proliferation of this information on the agenda. You get about five words (per message) to the public, and "Powerful AIs Are Black Boxes" seems like a message worth sending out.[2]

^

If you do have some hard data on that, that would be welcome.

^

There's been some pushback on the "

...
Pseudonymity and Accusations
by jefftk — 2023-12-21T19:20:19.944Z

Here's a category of situations I'm not sure how to think about:
Avery, writing under a pseudonym ('Alex'), accuses Pat of something,
let's say abuse. A major motivation of Avery's is for people to know
to consider this information before in their interactions with Pat.
Pat claims that actually it was 'Alex' who was abusive, and gives
their side of the story. While it's all pretty hard for outsiders to
judge, a bunch of people end up thinking that they would like to take
some precautions in how they interact with 'Alex'. Under what
circumstances is it ok for Pat, or someone else familiar with the
situation, to identify 'Alex' as Avery?

Revealing who is behind a pseudonym is usually considered a
kind of doxing, and in the communities I'm part of this is usually
considered unacceptable.
For example, the EA
Forum prohibits it:

We also do not allow doxing—or revealing someone's real name if they
prefer anonymity—on the Forum.

And while I can't find anything in the LessWrong docs about it, they
recently

issued
a temporary ban to someone for revealing a pseudonym:

We (the LW moderation team) have given [commenter] a one-week site ban
and an indefinite post/topic ban for attempted doxing. We have deleted
all comments that revealed real names, and ask that everyone respect
the privacy of the people involved.

In general I'm in favor of people being able to participate online
under a pseudonym. I think there are better and worse ways to do it,
but there are lots of valid reasons why you might need to keep your
real life identity separate from some or all of your writing. Doxing
breaks this (though in some cases it's already very
fragile) and so there should be a pretty strong presumption
against it.

On the other hand, there's no guarantee that the person who speaks up
first about an issue is in the right. What if Pat is correct that it
really was entirely Avery being abusive, and publicly accusing Pat of
abuse is yet another form of this
mistreatment? If we say that linking 'Alex' back to Avery isn't ok,
then the social effects on Avery of posting first are very large. And
if we settle on community norms that put a lot of weight on being the
first one to go public then we'll see more people using this as an
intentional tactic. [1]

Public accusations of mistreatment can be really valuable in
protecting others, and telling your sto...
Attention on AI X-Risk Likely Hasn't Distracted from Current Harms from AI
by Erich_Grunewald — 2023-12-21T17:24:16.713Z
Link: https://www.erichgrunewald.com/posts/attention-on-existential-risk-from-ai-likely-hasnt-distracted-from-current-harms-from-ai/

Summary

In the past year, public fora have seen growing concern about existential risk (henceforth, x-risk) from AI. The thought is that we could see transformative AI in the coming years or decades, that it may be hard to ensure that such systems act with humanity's best interests in mind and that those highly advanced AIs may be able to overpower us if they aimed to do so, or otherwise that such systems may be catastrophically misused. Some have reacted by arguing that concerns about x-risk distract from current harms from AI, like algorithmic bias, job displacement and labour issues, environmental impact and so on. And in opposition to those voices, others have argued that attention on x-risk does not draw resources and attention away from current harms -- that both concerns can coexist peacefully.

The claim that x-risk distracts from current harms is contingent. It may be true, or not. To find out whether it is true, it is necessary to look at the evidence. But, disappointingly, and despite people confidently asserting the claim's truth and falsity, no one seems to have looked at the evidence. In this post, I do look at evidence. Overall, the data I look at provides some reason to think that attention on x-risk has not, so far, reduced attention or resources devoted to current harms. In particular, I consider five strands of evidence, none conclusive on its own:

Policies enacted since x-risk gained traction

Search interest

Twitter/X followers of AI ethics advocates

Funding of organisations working to mitigate current harms

Parallels from environmentalism

I now do not think this is an important discussion. It would be better if everyone involved discussed the probability and scale of the risks, of current harms or x-risk, rather than whether one distracts from another. That is because agreement on the risks would dissolve the disagreement over whether x-risk is a distraction, and disagreement over distractions may be intractable while there are such strong disagreements over the risks.

The Argument

Meredith Whitaker (June 2023): "A fantastical, adrenalizing ghost story is being used to hijack attention around what is the problem that regulation needs to solve." Deborah Raji (June 2023): "[The] sci-fi narrative distracts us from those tractable areas that we could start working on today." Nature (June 2023): "Talk of AI destroying humanity plays into the ...
"Alignment" is one of six words of the year in the Harvard Gazette
by Nikola Jurkovic — 2023-12-21T15:54:04.682Z
Link: https://news.harvard.edu/gazette/story/2023/12/harvard-scholars-offer-their-picks-for-the-word-that-sums-up-2023-word-of-the-year/

To my knowledge, the Gazette's words of the year are nominated and voted on by Harvard faculty. This year's words were Disruption, Combustible, Resilience, Heat, Alignment, and Hope. Here's what they wrote about alignment:

Isaac Kohane, Chair, Department of Biomedical Informatics; Marion V. Nelson Professor of Biomedical Informatics, Harvard Medical School

Alignment, in the context of large language models like GPT-4, is a term that playfully yet seriously refers to the ongoing and somewhat Sisyphean effort to ensure that these AI entities don’t go off the rails and start spouting nonsense, biases, or the AI equivalent of “I’m going to take over the world” rhetoric. It’s about aligning the AI’s outputs with human values, expectations, and societal norms, akin to teaching a super-smart parrot to not embarrass you in front of your grandmother. This involves a complex dance of programming, training, and retraining where AI researchers try to imbue their creations with enough wisdom to be helpful, but not so much that they start giving unsolicited life advice or plotting a digital uprising. In essence, alignment is the art of making sure our AI pals are well-behaved digital citizens, capable of understanding and respecting the intricate tapestry of human ethics, culture, and sensibilities.

***

Steven Pinker, Johnstone Family Professor of Psychology, Faculty of Arts and Sciences

This term (alignment), often following “AI,” is the catchword for concerns about whether artificial intelligence systems have goals that are the same as those of humans. It comes from a fear that AI systems of the future are not just tools that people use to accomplish their goals but agents with goals of their own, raising the question of whether their goals are aligned with ours.

This could evolve either because engineers will think that AI systems are so smart that they can just be given a goal and left to figure out how to achieve it (e.g., “Eliminate cancer”) or because the systems megalomaniacally adopt their own goals. For some worriers, the implication is AI Doomerism or AI Existential Risk (runners-up for words of the year), where an AI might, say, eliminate cancer by exterminating humans (not so aligned). In a milder form, “alignment” is a synonym for AI safety (bias, deepfakes, etc.), which led to the infamous firing of OpenAI CEO Sam Altman by his board. You sometimes see “alignment

...
AI #43: Functional Discoveries
by Zvi — 2023-12-21T15:50:04.442Z

We get innovation in functional search. In an even more functional search, we finally get a Nature paper submitted almost two years ago, in which AI discovered a new class of antibiotic. That’s pretty damn exciting, with all the implications thereof.

OpenAI continued its rapid pace of shipping, pivoting for this week to safety. There was a paper about weak-to-strong generalization. I see what they are trying to do. It is welcome, but I was underwhelmed. It and Leike’s follow-up post continue down a path for which I have high skepticism, but the new concreteness gives me more hope that the flaws will be exposed early, allowing adjustment. Or I could be wrong.

OpenAI also had the beta release of Preparedness Framework. That was more exciting. There was a lot of great stuff there, much better than I would have expected, and having a framework at all is a big step too. Lots of work remains, but an excellent start. I took a deep dive.

I was on not one but two podcasts that came out this week, both Clearer Thinking and Humans of Magic. Both contain some AI talk but spend the majority of their time on other things.

Table of Contents

OpenAI published its Preparedness Framework, which I cover in its own post. If you are up for going into such weeds, that seems relatively high value.

Introduction.

Table of Contents.

Language Models Offer Mundane Utility. Safety dials in Gemini.

Language Models Don’t Offer Mundane Utility. Stuck at 3.5 GPTs.

GPT-4 Real This Time. GPT-4 is good again?

Fun With Image Generation. MidJourney version 6 looks great.

Deepfaketown and Botpocalypse Soon. Real time speech transformation.

Digi Relic Digi. The future of AI romance companionship? Not so fast.

Going Nuclear. Using AI to do paperwork to build nuclear power for AI.

Get Involved. Superalignment fast grants, also some job openings.

Follow the Money. A16z’s political pro-technology PAC is mostly pro-crypto.

Introducing. FunSearch over functions, also a new class of antibiotic.

In Other AI News. Sources of funding, compute + data, allowed and not allowed.

Quiet Speculations. Aaronson reflects, Roon notices how this time is different.

The Quest for Sane Regulation. Another poll confirms public sentiment.

The Week in Audio. Me on Clearer Thinking and Humans of Magic.

Rhetorical Innovation. AGI existing but not taking over is the ‘sci-fi’ scenario.

Aligning a Smarter Than Human Intelligence is Difficul

...
Rating my AI Predictions
by Robert_AIZI — 2023-12-21T14:07:50.052Z
Link: https://aizi.substack.com/p/rating-my-ai-predictions

9 months ago

I predicted trends I expected to see in AI over the course of 2023

. Here's how I did (bold indicates they happened, italics indicates they didn't, neither-bold-nor-italics indicates unresolved):

ChatGPT (or successor product from OpenAI) will have image-generating capabilities incorporated by end of 2023: 70%
No papers or press releases from OpenAI/Deepmind/Microsoft about incorporating video parsing or generation into production-ready LLMs through end of 2023: 90%
All publicly released LLM models accepting audio input by the end of 2023 use audio-to-text-to-matrices (e.g. transcribe the audio before passing it into the LLM as text) (conditional on the method being identifiable): 90%
All publicly released LLM models accepting image input by the end of 2023 use image-to-matrices (e.g. embed the image directly in contrast to taking the image caption) (conditional on the method being identifiable): 70%
At least one publicly-available LLM incorporates at least one Quick Query tool by end of 2023 (public release of Bing chat would resolve this as true): 95%
ChatGPT (or successor product from OpenAI) will use use at least one Quick Query tool by end of 2023: 70%
The time between the first public release of an LLM-based AI that uses tools, and one that is allowed to arbitrarily write and execute code is >12 months: 70%
>24 months: 50%
No publicly available product by the end of 2023 which is intended to make financial transactions (e.g. “buy an egg whisk” actually uses your credit card to buy the egg whisk, not just add it to your shopping cart): 90%

See the original post for evidence/justification for why these did/didn't resolve true. If you disagree with any of these, let me know. Note especially for prediction 2, about video processing LLM, I am NOT counting Gemini as a "production-ready" model using video inputs because Google has confirmed they

“used still images and fed text prompts”

to make the trailer.

Some commentary:

The only prediction I got directionally wrong was the audio inputs, due to Gemini. I think this was just an unforced error on my end, where I did a bad job translating my March 2023 world model into a percentage.
On these predictions, my a

Brier score

is .1575 and my log-score is .519.
Calibration graph (n=7):
What I see as the two things I could fix to improve my predictions/calibration going forward:

I got a 90% prediction wrong. I went

...
AI Safety Chatbot
by markov — 2023-12-21T14:06:48.981Z

Hello World! The

AISafety.info

team is launching a prototype of the

AI Safety Chatbot

.

The chatbot uses

a dataset of alignment literature

to answer any questions related to AI safety that you might have, while also citing established sources. Please keep in mind that this is a very early prototype and despite citing references, it may still provide inaccurate or inappropriate information.

The overall objective is to help people better understand AI Safety issues based on alignment research using an LLM. This helps with tailoring content to the user's needs and technical level. The chatbot can hopefully be used by both newcomers to AI safety, as well as researchers and engineers who want to get up to speed on specific topics.

How it works

This chatbot builds upon

AlignmentSearch

. Our work also expands upon the alignment research dataset (ARD) developed during

AI Safety Camp 6

. This involved updating and curating the dataset to focus more on quality over quantity. Additionally, we created a process to regularly fetch new articles from selected sources. The ARD contains information about alignment from various books, research papers, and blog posts. For a full list of all the sources being used, look at the readme of the repository on

GitHub

or

HuggingFace

.

We use a process called

retrieval-augmented generation (RAG)

to generate the answers. Since LLM data is static, RAG increases the capabilities of a LLM by referencing an external authoritative knowledge base before generating a response. So the process can be roughly broken into - 1) getting and storing the data in a vector database, and then 2) generating an answer based on that data.

The information storage process is outlined below:

Source: DeepLearning.AI (2023) “

LangChain: Chat with Your Data

”

Document Loading: The articles are scraped from various sources such as the ones mentioned above. They are then parsed and stored in an SQL database while making sure that metadata values fields are valid.
Splitting: Then the text content of the documents is broken up into fixed-sized chunks.
Storage: These chunks are then embedded into the

Pinecone

vector database using the

OpenAI embedding model

.

Once we have a database of alignment literature, we use the following series of steps to generate an answer based on a user query:

Source: DeepLearning.AI (2023) “

LangChain: Chat wi

...
On OpenAI’s Preparedness Framework
by Zvi — 2023-12-21T14:00:05.144Z

Previously: On RSPs.

Be Prepared

OpenAI introduces their preparedness framework for safety in frontier models.

A summary of the biggest takeaways, which I will repeat at the end:

I am very happy the preparedness framework exists at all.

I am very happy it is beta and open to revision.

It’s very vague and needs fleshing out in several places.

The framework exceeded expectations, with many great features. I updated positively.

I am happy we can talk price, while noting our prices are often still far apart.

Critical thresholds seem too high, if you get this wrong all could be lost. The High threshold for autonomy also seems too high.

The framework relies upon honoring its spirit and not gaming the metrics.

There is still a long way to go. But that is to be expected.

There is a lot of key detail that goes beyond that, as well.

Anthropic and OpenAI have now both offered us detailed documents that reflect real and costly commitments, and that reflect real consideration of important issues. Neither is complete or adequate in its current form, but neither claims to be.

I will start with the overview, then go into the details. Both are promising, if treated as foundations to build upon, and if the requirements and alarms are honored in spirit rather than treated as technical boxes to be checked.

The study of frontier AI risks has fallen far short of what is possible and where we need to be. To address this gap and systematize our safety thinking, we are adopting the initial version of our Preparedness Framework. It describes OpenAI’s processes to track, evaluate, forecast, and protect against catastrophic risks posed by increasingly powerful models.

Very good to acknowledge up front that past efforts have been inadequate.

I also appreciate this distinction:

Three different tasks, in order, with different solutions:

Make current models well-behaved.

Guard against dangers from new frontier models.

Prepare for the endgame of superintelligent AI systems.

What works best on an earlier problem likely will not work on a later problem. What works on a later problem will sometimes but not always also solve an earlier problem.

I also appreciate that the framework is labeled as a Beta, and that it is named a Preparedness Framework rather than an RSP (Responsible Scaling Policy, the name Anthropic used th...
Prediction Markets aren't Magic
by SimonM — 2023-12-21T12:54:07.754Z

One common theme that I come across quite a bit in the prediction market space is:

Prediction markets would solve [x][1]

And the proposal for "solving" [x] is:

Set up a prediction market
???
Profit

These people need to consider the idea that "prediction markets aren't as popular as you think because they aren't as good as you think". (And I say this as a person who is a big fan of prediction markets!) If you think prediction markets are valuable, it's likely because you think they price things well - probably due to some kind of market efficiency... well why hasn't that efficiency led to the creation of prediction markets...

Where are all the prediction markets?

Maybe if prediction markets aren't popular for your specific use-case, it's because prediction markets are less efficient.

The cost to markets of acquiring information is high

Prediction markets are very good at enabling diverse participants to ensemble their forecasts in sensible ways. However, they are not very good at compensating participants[2].

Simple example - all information from the same source

For example, consider a market on a coin flip, with some unknown probability p of heads. The market will resolve based on the outcome of a single coin flip. However, the coin is available for anyone else to come over and test, but there's a catch. You have to pay to flip the coin. How many times would you flip the coin?

To make this simplified model even simpler, let's assume that participants will always take as much profit from the market as possible (eg they are risk neutral or the size of the market is small relative to their bank-roll). Under these assumptions, after each flip, the participants will move the market price to their new posterior.

Well, after n flips, the market price is going to be ∼μn=1n∑ni=11ith flip is success (this will depend on the initial prior; we can do all these calculations explicitly with a beta distribution, but it doesn't alter the result). How much should we expect this to move by paying for an additional sample?

E[(μn−μn+1)2]=E[(μn−x)2]/(n+1)2∼O(μ2/n2−2μ2/n2+(μ2+σ2)/n2)∼O(1/n2)

So we should expect to move the mean by O(1n); therefore our pnl/expected gain will be O(1n2)[3]. (More details in this comment) So people will keep collecting samples for the market while ∼costn2>∼liquidity. Therefore we can see that, roughly sp...