trentmkelly/LessWrong-43k huggingface dataset:
Linkpost: We need another Expert Survey on Progress in AI, urgently
by David Mears — 2023-08-11T08:22:48.296Z
Link: https://open.substack.com/pub/philosophybear/p/we-need-another-expert-survey-on

Philosophy Bear writes:

One way to get a sense of when new technologies might exist is to ask people working in the area. With high stakes, but hard to predict, technologies like AI, this serves a special role. These surveys are often used to give existential risk researchers and policymakers a sense of how quickly things may happen. This last function is very important right now, as high level policy discussions about AI- and even existential risk- are happening around the world.

The last survey of machine learning experts on the future of AI happened in June-August 2022. On average, researchers predicted the arrival of artificial general intelligence (which the survey called HLMI- Human level machine intelligence) in 2059.

I think that, since then, there has been a bit of a sea change, and there are very strong reasons to commission another survey:

Palm540B, ChatGPT and GPT-4 have all came out since the survey.

There has seemingly been a shift in public views. Plausibly, this has affected experts as well for many reasons- most notably, a good chunk of the new information might have been new to experts as well (e.g. GPT-4). I don’t think we would have got these results from the public back in ‘22.

There has been an apparent sudden rise in expert concern over AI risk- in many cases due to concern over existential risk from AGI. See, for example, this statement on existential risk from AI signed by numerous prominent researchers in the area. It is very plausible this rise in expert concern is linked to shorter estimated timelines. Even if AI timelines haven’t shifted, assessing what proportion of experts broadly agree with the statement on existential risk would be valuable.

At the time of the last survey, Metaculus’s aggregate prediction for AGI (open to everyone) had just fallen from about 2054 to 2040 about a month before the survey, most likely as an update due to the launch of PALM540B- which was, at the time, a tremendous step forward. It’s plausible experts had not fully internalized this information by the time of the survey, unlike prediction markets which internalize new information quickly (sometimes too quickly). Since the survey finished Metaculus’s aggregate prediction has fallen another eight years, from 2040 to 2032- almost half the ‘remaining’ time on this model- most likely due to the launch of Chat-GPT and GPT-4. In total, since the st

...
What Does a Marginal Grant at LTFF Look Like? Funding Priorities and Grantmaking Thresholds at the Long-Term Future Fund
by Linch — 2023-08-11T03:59:51.757Z
Link: https://forum.effectivealtruism.org/posts/7RrjXQhGgAJiDLWYR/what-does-a-marginal-grant-at-ltff-look-like-funding

The Long-Term Future Fund (LTFF) makes small, targeted grants with the aim of improving the long-term trajectory of humanity. We are currently fundraising to cover our grantmaking budget for the next 6 months. We would like to give donors more insight into how we prioritize different projects, so they have a better sense of how we plan to spend their marginal dollar. Below, we’ve compiled fictional but representative grants to illustrate what sort of projects we might fund depending on how much we raise for the next 6 months, assuming we receive grant applications at a similar rate and quality to the recent past.

Our motivations for presenting this information are a) to provide transparency about how the LTFF works, and b) to move the EA and longtermist donor communities towards a more accurate understanding of what their donations are used for. Sometimes, when people donate to charities (EA or otherwise), they may wrongly assume that their donations go towards funding the average, or even more optimistically, the best work of those charities. However, it is usually more useful to consider the marginal impact for the world that additional dollars would buy. By offering illustrative examples of the sort of projects we might fund at different levels of funding, we hope to give potential donors a better sense of what their donations might buy, depending on how much funding has already been committed. We hope that this post will help improve the quality of thinking and discussions about charities in the EA and longtermist communities.

[...]

see the full post here
Will posting any thread on LW guarantee that a LLM will index all my content, and if questions people ask to the LLM after my name will surface up all my LW content?
by Alex K. Chen (parrot) — 2023-08-11T01:40:10.933Z

Eg a LLM like GPT4/GPT5 (they don't seem to capture all of it yet)? Would it capture all shortform posts and shortform questions?

If I want to reliably surface some rare content, would LW be of the best places to surface it?

And which LLM would it most likely come from?
AI Safety Concepts Writeup: WebGPT
by JustisMills — 2023-08-11T01:35:31.196Z

This is a crosspost of https://forum.effectivealtruism.org/posts/4ni3GBBzRRAgiksHT/ai-safety-concepts-writeup-webgpt.

Intro

Chris Patrick (a science writer) and I were awarded a grant by the Long Term Future Fund to interview AI safety researchers and condense their findings into something digestible for an educated layperson. Chris was the primary recipient - I helped with editing, content knowledge, and general support. We've noticed that there are lots of extremely introductory and broad AI safety articles, and lots of highly advanced blog posts that assume background knowledge, with somewhat of a gulf in between.

We interviewed two researchers. But we were committed not to publish anything without final signoff from the researchers, and one never got back to us. Assuming we don't hear back from this second researcher, we're only at liberty to share one of the pieces we made. Here it is with original formatting intact - I'll also reproduce the text next in this post, followed by a brief project postmortem and next steps.

AI Safety Concepts Writeup: WebGPT

Intro to AI Safety Concepts

WebGPT

The goal of AI safety researchers is to foretell and prevent potential negative outcomes associated with the growing intelligence and use of AI before it’s too late. Jacob Hilton, for example, helped develop a language model called WebGPT whose lies can be caught.

Citations allow checks of AI’s truthfulness

A Wikipedia-inspired language model shows where on the web it finds answers, which could help better align future AI to do what we want.

If AI systems aren’t trained to tell the truth, they might accidentally be trained to lie instead. And that could spell danger when models become as smart as – or smarter – than humans.

“We want to make sure they’re doing what we want, not saying false things or worse, deliberately trying to trick us,” said Jacob Hilton, a researcher at OpenAI at the time of this interview who is now at the Alignment Research Center working on the Eliciting Latent Knowledge agenda. In an effort to avoid this, he and his coauthors, Reiichiro Nakano, Suchir Balaji, and John Schulman, developed a language model with an extra add-on while Hilton was working at OpenAI.

Language models are trained on large bodies of text, mostly from the internet, to be able to predict text. With fine-tuning, AI developers have taught these language models how to an...
What is science?
by Adam Zerner — 2023-08-11T00:00:34.884Z

It seems like a very 101-level question that someone like me shouldn't have, but I am actually confused.

Is there a strict, black-and-white, approximately universally agreed upon definition? I don't get the impression that there is. People seem to use the term to mean different things.

Furthermore, I get the sense that -- I'm not sure how to say this -- it's a typicality thing. That science is something for which there are more and less typical examples of.
Three configurable prettyprinters
by philh — 2023-08-10T23:10:02.000Z

I gave a fifteen minute talk about this at Zurihac 2023. If you read this essay, I don't think there's much point in additionally watching the video.

I've been exploring a new-to-me approach to stringification. Except that right now it's three different approaches that broadly share a common goal. I call it/them pretty-gist.

The lowest-friction way to stringify things in Haskell is usually show. It gives the user close to zero ability to control how the thing is rendered.

The Show1 and Show2 classes give some control, but only in limited ways. Importantly, they only work on parameterized values; you could use Show2 to change how you render the keys of a Map Int String, but not of an IntMap String.

There are also things that give you some control over layout, but not much control over content. For example, aeson-pretty lets you pretty-print json data, and pretty-simple can pretty-print typical output from show. Both let you configure indent width, and pretty-simple additionally lets you choose some different layout styles (where to put newlines and parentheses) and stuff. But they both operate by a model where you produce a relatively simple data structure that they know how to deal with, they give you a few knobs and render it in full. (For aeson-pretty the data structure is JSON, for pretty-simple it's a custom type that they parse from show output and that's pretty good at reproducing it.)

Here are some use cases where I've wanted more control than I can easily get:

My test suites generate a complicated data structure. Most of the time, most of this data structure is irrelevant to the test failures and I don't want to see it.

A complicated data structure contains JSON values, and I want them rendered as JSON rather than as a Haskell data type. Or it contains floating-point numbers, and I want them rounded to 3dp with underscore separation (17_923.472). Or strings, which I want printed using C-style escapes rather than Haskell-style, and with unicode rendered.

A list might be infinite, and I only want to show the first ten elements. Or a tree might actually be cyclic, and I want to only show three levels deep.

pretty-gist aims to enable stuff like this. I call the rendered outputs it produces "gists" following Raku's use of that term, where I think the intention is something like "take a guess about what I'm likely to want to se...
Ilya Sutskever's thoughts on AI safety (July 2023): a transcript with my comments
by mishka — 2023-08-10T19:07:44.902Z

There has been a 25 min interview with Ilya conducted by Sven Strohband and released on July 17: https://www.youtube.com/watch?v=xym5f0XYlSc

This interview has a section dedicated to AI safety (7 min starting from 14:56). Ilya is now the co-lead of the OpenAI "superalignment" effort, and his thinking will likely be particularly influential in how this effort evolves.

What he is saying seems to be somewhat different from what is in the consensus OpenAI "superalignment" documents. It's compatible, but the emphasis is rather different. In particular, thinking about humans controlling or steering a superintelligent system is limited to an analogy of controlling a nuclear reactor to prevent a meltdown, and a more collaborative approach between humans and AIs seems to be emphasized instead.

(I am not sure when the interview has been recorded, but no earlier than July 6, since it mentions Introducing Superalignment.)

Here is my attempt at editing the YouTube transcript of that part of the conversation.
The truly interesting part starts at 20:07. He hopes that a collaboration with superintelligence could solve the issues of misuse (so, no, he is not aiming to make superintelligence alignable to arbitrary goals, designing the proper goals is likely to be a collaborative activity between humans and AIs). I've put some bold marks for emphasis.

15:03 Sven: it's worthwhile to also talk about
AI safety, and OpenAI has released the
document just recently where you're
one of the undersigners.
Sam has testified in front of
Congress.
What worries you most about AI
safety?

15:27 Ilya: Yeah I can talk about that.
So let's take a step back and talk about
the state of the world.
So you know, we've had this AI research
happening, and it was exciting, and now
you have the GPT models, and now you all
get to play with all the different chatbots
and assistance and, you know, Bard
and ChatGPT, and they say okay that's
pretty cool, it can do things;
and indeed they already are.
You can start perhaps worrying about the
implications of the tools that we have
today,
and I think that it is a very valid
thing to do,
but that's not where I
allocate my concern.

16:14 The place where things get really tricky
is when
you imagine fast forwarding some number
of years, a decade let's say,
how powerful will AI be?
Of course with this
incredible future power of AI which I
think will be difficult to imagin

...
Seeking Input to AI Safety Book for non-technical audience
by Darren McKee — 2023-08-10T17:58:29.618Z

TLDR: You are invited to make suggestions for ideas/information/framing for an upcoming AI safety book for a non-technical audience.
Context:
I'm still writing an accessible book about AI Safety/Risk for a non-technical audience to serve both the AI safety cause and the community. The book's intended audience is likely not people who read this forum but rather your friends, family, policy makers, and non-science people who are curious about the topic. I started last June, it is pretty far along, and I hope to have it available within the next three months (I received a LTFF grant last year to help it come into existence).
Briefly, the purpose of the book is to communicate that intelligence is really powerful, AI progress is happening fast and AI systems are becoming more intelligent/powerful, that advanced AI is a threat/risk to humanity because it may not be aligned with our values and may be uncontrollable, therefore we should act now to reduce the risk.
Opportunity:
You can present ideas, facts, framing, or anything else you think would be important in such a book in the comments below or send me a message.
If interested, you may still be wondering what I've already included. Broadly, as a heuristic, if your idea is very obvious, I'm probably already including it. But it could be still be useful for you to suggest it, so I can see that others think it is important.
If your idea is highly technical, I have likely chosen not to include it. But it could still be useful to suggest if it is a key consideration that can be made more accessible. I'm trying to open-minded but efficient with people's time.
I am also trying to minimize the occurrence of someone saying "I really wish he had mentioned X" after the book comes out. No promises of inclusion but at least your suggestions will be considered.
Finally, I'm more than happy to have people be more involved as getting feedback from a range of knowledgeable people is useful for a variety of reasons.
Evaluating GPT-4 Theory of Mind Capabilities
by gcmac — 2023-08-10T17:57:25.905Z

Note: This work is a close collaboration between myself and Nathan Labenz (@Nathan). Many thanks to him for starting this particular piece of research and letting me tag along as well to @JustisMills for help on navigating the submission process and giving feedback on the draft. If you're interested in running experiments yourself, check out our Replit instance or read the code on github.

Introduction

Theory of mind (ToM) is the ability to attribute mental states to ourselves and others; accurately predicting other people’s beliefs, intents, desires and emotions is key to navigating everyday life. Understanding whether LLMs have theory of mind is an interesting theoretical question. However the question of whether AI can accurately assess human state of mind has important practical implications too. Deception is a key component to many of the scenarios that concern the AI safety community, and theory of mind is a key component of deception.

In addition to being an important question, identifying the extent of ToM in LLMs provides a rich canvas against which to discuss capability evaluation in general. We’ve tried to include key takeaways in each section that are practical considerations when trying to test whether LLMs have a given capability. Each task is unique, but we think we’ve found some important things to consider, if you want to understand AI ability in both best case and worst case scenarios.

We begin by discussing the particular problem and data set we are trying to evaluate, TOMI. We evaluate GPT-4’s capability to answer these questions correctly using a prompt with more context and a chain of thought approach, and compare this approach to prior results. We then classify the errors the model made into several buckets and examine the patterns. Then, we discuss adjustments to the prompts that have interesting implications on the model’s performance. Finally, we discuss the variability of same-prompt output we observed using the GPT-4 API at temperature 0 and how that impacts our ability to measure the model’s capabilities.

The TOMI Dataset; zero shot vs. chain of thought performance

The TOMI data set, which comes from this Le et. al. paper in 2019, is a set of stories and questions that try to measure the answerer’s ability to understand the state of mind of different characters in each story. The questions are structured with two parts: a story (set of ...
Some alignment ideas
by SelonNerias — 2023-08-10T17:51:09.399Z

Epistemic status: untested ideas

Introduction

This post is going to describe a few ideas I have regarding AI alignment. It is going to be bad, but I think there are a few nuggets worth exploring. The ideas described in this post apply to any AI trained using a reward function directly designed by humans. With AIs where the reward function is not directly designed by humans, it might be nice if some of the concepts here were present and some (Noise and A knowledge cap) could potentially be added as a final step.

Terms (in the context of this paper)

Reward-function: A function which the AI uses to adjust its model.

Variable: In this context variables are measurements from the environment. The AI model uses variables to determine what to do, and the reward-function uses variables to determine how good the AI model did.

Model: A combination of a mathematical model of the world, and instructions on how to act within the world.

Noise: Randomness in a measurement or variable.

Tweak: A small adjustment (to the AI’s model)

Idea 1: Noise

Add random (quantum) noise to the reward-function, perhaps differing amounts/types of noise for different parts of the reward-function.

Why?

1.

Noise might make it so the AI doesn’t optimise its model for every statistically barely significant increase in its rewards. This because it might learn that small tweaks are unlikely to consistently yield it rewards and/or only yield small insignificant rewards.

Removing small tweaks might be beneficial. It might prevent the AI from overfitting the reward-function. For example, if a really small slightly unreliable thing such as giving every kid liquorice (or making as many paperclips as possible) is the most cost-effective thing the AI can do to increase the rewards it gets, the noise might make it less likely the AI will pick up on this.

This is useful because the AI and/or the reward-function might be wrong in identifying such small things as the most cost-effective thing to increase what we care about. Optimising for such small tweaks might be disastrous (turn everything into liquorice for kids or the classic paperclips). All things which only optimise for one reward-function variable could be disastrous, but optimising for something which would be rendered statistically insignificant by a little added noise would be more so than optimising for something which we think is overwhelmingly positive (m...
Self Supervised Learning (SSL)
by Varshul Gupta — 2023-08-10T17:43:36.469Z
Link: https://dubverseblack.substack.com/p/ba889fa6-29c5-4fd2-b8c3-6ab26d02d243

Self Supervised Learning (SSL)

"Unlocking Powerful Representations: The Frontier of Self-Supervised Learning"

JASKARAN SINGH

AUG 9, 2023

Share

With all that’s been happening in the AI/ML industry for the past few weeks, it is important we address the elephant in the room.

The Idea Behind SSL

SSL comes under the umbrella of Unsupervised Learning. One thing that worked for NNs is that they are able to fit a curated dataset with ease given they have labels to optimize for (Supervised Learning), but this dataset may not be large enough, instead, it would be very expensive or impossible to create such a dataset. Once NNs have good representations of the task-related work they can learn a new task even more rapidly when shown how to do it only once (Zero-shot).

But what if we can create synthetic Labels from the data?
they don’t need to be highly curated. They can be thought of as corrupting the data. we can use these labels to train NNs and in the process, these NNs learn a powerful representation (depending upon the data and compute) of the Data that can be fine-tuned with a handful of data to produce quality results.

How to SSL

The main goal of SSL is to generate labels out of the dataset itself, without much human effort. The basic idea is to corrupt the data or either do a lossy compression to predict the same data back from the Model. Let’s discuss some ways in which researchers have used this technique in different domains:

NLP

While training a Language Model you can do SSL in two ways:

MLM: Masking some words of the text (Corrupt) and letting the model train on getting these masked tokens correctly.

Research Paper

LM: given some context let the Model predict the rest of the text, one token at a time.

Research Paper

These approaches are part of contrastive training: in short, you train the model to differentiate between the different classes, during training you want the model to give a high probability to the ground truth and near zero probability to the rest of the labels.

Speech

Wav2Vec 2.0 model: The Idea is similar to MLM, Using CNN Layers to create quantized representations and then doing MLM on top of it.

Research Paper

Vision

VicReg: This training paradigm is based on regularization rather than a contrastive approach

Research Paper

The model is given two augmented version of the same image (SSL) and the distance between these embeddin...