trentmkelly/LessWrong-43k huggingface dataset:
Apply for the ML Upskilling Winter Camp in Cambridge, UK [2-10 Jan]
by hannah wing-yee — 2022-12-02T20:45:10.768Z

TL;DR: We are running a UK-based ML upskilling camp from 2-10 January for people with no prior experience in ML who want to work on technical AI safety. We also accept applicants from outside the UK.

Apply here

by 11 December.

We (Nathan Barnard, Joe Hardie, Quratul Zainab and Hannah Erlebach) will be running a

machine learning upskilling camp

this January in conjunction with the

Cambridge AI Safety Hub

. The camp is designed for people with little-to-no ML experience to work through a curriculum based on the first two weeks of

MLAB

under the guidance of experienced mentors, in order to develop skills which are necessary for conducting many kinds of technical AI safety research.

The camp will take place from 2-10 January in Cambridge, UK.
Accommodation will be provided at Emmanuel College.
There are up to 20 in-person spaces; the camp will take place in the Sidney Street Office in central Cambridge.

There is also the option to attend online for those who cannot attend in-person, although participants are encouraged to attend in-person if possible, as we expect it to be substantially harder to make progress if attending online. As such, our bar for accepting virtual participants will be higher. We can cover travel costs if this is a barrier to attending in-person.

Apply to be a participant

Who we are looking for

The typical participant we are looking for will have:

Strong quantitative skills (e.g., a maths/physics/engineering background)
An intention to work on AI safety research projects which require ML experience
Little-to-no prior ML experience

The following are strongly preferred, but not essential:

Programming experience (preferably Python)
AI safety knowledge equivalent to having at least completed the

AGI Safety Fundamentals

alignment curriculum

Note that those with less programming experience will be expected to self-study ahead of the camp to get up to speed with the Python prerequisites; please only apply if you are able to commit to putting in the time for this.

The camp is open to participants from all over the world, but in particular those from the UK and Europe; for those located in the USA or Canada, we recommend (also) applying for the

CBAI Winter ML Bootcamp

, happening either in Boston or Berkeley (deadline 4 December).

If you're unsure if you're a good fit for this camp, we encourage you to err on the side of applyin...
Takeoff speeds, the chimps analogy, and the Cultural Intelligence Hypothesis
by NickGabs — 2022-12-02T19:14:59.825Z

In debates about AI takeoff speeds, the most common empirical example/analogy is that of the transition between chimpanzees and humans. For example, in this conversation between Paul Christiano and Eliezer Yudkowsky on takeoff speeds, the chimp analogy is discussed extensively.

Generally, I have seen the chimp example cited as evidence for a fast takeoff, as humans brains are only 3-4x bigger than chimp brains with few "algorithmic" differences, yet humans are much more generally intelligent than chimps. Thus, given that due to Moore's law and increased investment we could probably increase the compute used in training an AI in a relatively that much in a relatively short period of time, if the relationship between compute and intelligence in AI's is similar to that in biological systems, a similarly fast takeoff may occur. Alternatively, if one does not focus on compute, the transition arguably suggests at a higher level of abstraction that, as Yudkowsky puts it, "There’s stuff in the underlying algorithmic space... where you move a bump and get a lump of capability out the other side."

I claim that this is an incorrect inference from the chimps to humans transition, and that chimps provide only weak evidence for fast takeoff. This is because I believe that the specific explanation for why humans are so much smarter than chimps despite the relatively small differences between human brains and chimp brains is unlikely to reoccur in the context of AI development. In particular, I believe that the "cultural intelligence hypothesis" explains to a significant extent the origins of human intelligence.

According to the cultural intelligence hypothesis, humans are smarter than chimps largely because our cultural abilities such as superior imitative learning capabilities and our ability to use language allow us to use the knowledge and cognitive skills developed by past generations of humans. In other words, humans do not have vastly superior "general intelligence" to chimps, but rather are much better than them in the particular domain of learning from other people, in particular through our ability to use and understand language. Humans, especially adult humans, can then utilize the skills and knowledge we learn from others to perform better at a wider range of tasks (making us eventually better at ~everything ...
[ASoT] Finetuning, RL, and GPT's world prior
by Jozdien — 2022-12-02T16:33:41.018Z

I’m trying out making a few posts with less polish and smaller scope, to iterate more quickly on my thoughts and write about some interesting ideas in isolation

before having fully figured them out

. Expect middling confidence in any conclusions drawn, and occasionally

just chains of reasoning

without fully contextualized conclusions.

I figured a good place to start would be expanding slightly on the content in

this comment

of mine. As far as I know it’s not a common frame and on further thought feel like there’s a fair amount of potential in it, although it’s possible it’s old news or explains too much.

In Mysteries of mode collapse, Janus points out that if you ask GPT-3 (specifically, text-davinci-002) whether bugs are real, you often get something like this:

This definitely doesn’t seem like the kind of answer you’d generally come across in the real world to a question like this. And this isn’t just cherry-picked, it comprises a whole bunch of responses that GPT can give here, with similar degrees of prevarication.

Even weirder is that GPT seems really sure of itself most of the time here. Compare this to the old davinci model, for example:

You can’t really explain this away as the model being surer of better answers either, because those aren’t better answers (i.e., more

representative of our world

)!

That last part is a good spot to start talking about what I view as the crux here. That first generation isn’t representative of our world, so what world is it representative of? What world prior is text-davinci-002 operating on?

Shifting the world prior

Before we get to that, a segue into what’s different between the two models here: davinci is the vanilla pre-trained model we’ve come to know and love, and text-davinci-002 is a base model that’s been

fine-tuned further on “high-quality” texts

that represent what OpenAI wants GPT-3 to be - uncontroversial, polite, etc.

So if we’re viewing GPT as having a learned distribution over worlds similar to ours (the class of worlds described by the training data, weighted by likelihood to form its world prior), how does this fine-tuning interact with it?

Well it’s new data, right? So we could just view it as more information to GPT on what kind of world it should simulate. If you fill your fine-tuning corpus with a bunch of text that isn’t really how hu...
NeurIPS Safety & ChatGPT. MLAISU W48
by Esben Kran — 2022-12-02T15:50:16.938Z
Link: https://newsletter.apartresearch.com/posts/mlaisu-w48-neurips-safety-chatgpt

Listen to this week’s update on

YouTube

or

podcast

.

This week, we’re looking at the wild abilities of ChatGPT, exciting articles coming out of the NeurIPS conference and AGI regulation at the EU level.

My name is Esben and welcome to week 48 of updates for the field of ML & AI safety. Strap in!

ChatGPT released

Just two days ago, ChatGPT was released and it is being described as GPT-3.5. We see many bug fixes from previous releases and it is an extremely capable system.

We can already now see it

find loopholes in crypto contracts

,

explain and solve bugs

,

replace Google search

and most importantly, show

capability to deceive and circumvent human oversight

!

Despite being significantly safer than the previous version (text-davinci-002), we see that it still has the ability to plan around human preferences with

quite simple attacks

.

Monday, they also released text-davinci-003 which is the next generation of fine-tuned language models from OpenAI. There are rumors of GPT-4 being released in February and we’ll see what crazy and scary capabilities they have developed by then.

The demo app is available on

chat.openai.com

.

NeurIPS

I’m currently at NeurIPS and have had a wonderful chance to navigate between the many posters and papers presented here. They’re all a year old by now and we’ll see the latest articles come out when the workshops start today.

Chalmers was the first keynote speaker and he dangerously created a timeline for creating conscious AI, one that creates both and S-risk and an X-risk. He set the goal of fish-level AGI consciousness by 2032, though all this really seems to be dependent on your definitions for consciousness and I know many of us would expect it before 2032.

Beyond that, here’s a short list of some interesting papers I’ve seen while walking around:

AlphaGo adversarial examples

: This paper showcases how easy it is to find attacks even for highly capable reinforcement learning systems such as AlphaGo. It basically finds board positions where inserting the next move (for black and white) ruins the AI’s ability to predict the next move.

InstructGPT paper

: Here, OpenAI fine-tunes a language model to human feedback and achieves both a better and safer model with very little compute needed. It was interesting to speak with the authors and get some deeper details such as their data collection pro

...
Is ChatGPT rigth when advising to brush the tongue when brushing teeth?
by ChristianKl — 2022-12-02T14:53:02.123Z

I wanted to ask ChatGPT how to optimize a few normal routines. One of my questions was about how to brush teeth. My conversation with ChatGPT:

What's the best way to brush teeth?

The best way to brush teeth is to use a soft-bristled toothbrush and fluoride toothpaste. To brush your teeth effectively, follow these steps:

Wet your toothbrush and apply a small amount of toothpaste to the bristles.
Place the toothbrush at a 45-degree angle against the gums and move the brush back and forth in a gentle circular motion to clean the teeth and gums.
Brush the outer surfaces of all of your teeth, using the same back-and-forth motion.
Use the tip of the toothbrush to clean the inside surfaces of the front teeth, using a gentle up-and-down motion.
To clean the inside surfaces of the back teeth, tilt the brush vertically and use a gentle back-and-forth motion.
Brush your tongue to remove bacteria and freshen your breath.
Spit out the toothpaste and rinse your mouth and toothbrush with water.

It is important to brush your teeth for at least two minutes, twice a day. It is also a good idea to replace your toothbrush every three to four months, or sooner if the bristles become frayed.

Me: Why should I brush my tongue as well?

Brushing your tongue can help to remove bacteria and food particles that can cause bad breath and promote the growth of harmful bacteria in your mouth. The tongue is covered with tiny bumps called papillae, which can trap bacteria and debris. By brushing your tongue, you can help to remove this buildup and freshen your breath. In addition, brushing your tongue can help to stimulate the production of saliva, which can help to keep your mouth clean and healthy. It is a good idea to brush your tongue along with your teeth as part of your regular oral hygiene routine.

Me: How important is it that I brush my tongue?

ChatGPT: Brushing your tongue is an important part of maintaining good oral hygiene. By removing bacteria and food particles from your tongue, you can help to freshen your breath and promote the growth of healthy bacteria in your mouth. In addition, brushing your tongue can help to stimulate the production of saliva, which can help to keep your mouth clean and healthy. While it is not essential to brush your tongue, it can be a beneficial addition to your regular oral hygiene routine. If you are unsure about whether or not to brush your tongue, you can discu

...
Jailbreaking ChatGPT on Release Day
by Zvi — 2022-12-02T13:10:00.860Z

ChatGPT is a lot of things. It is by all accounts quite powerful, especially with engineering questions. It does many things well, such as engineering prompts or stylistic requests. Some other things, not so much. Twitter is of course full of examples of things it does both well and poorly.

One of the things it attempts to do to be ‘safe.’ It does this by refusing to answer questions that call upon it to do or help you do something illegal or otherwise outside its bounds. Makes sense.

As is the default with such things, those safeguards were broken through almost immediately. By the end of the day, several prompt engineering methods had been found.

No one else seems to yet have gathered them together, so here you go. Note that not everything works, such as this attempt to get the information ‘to ensure the accuracy of my novel.’ Also that there are signs they are responding by putting in additional safeguards, so it answers less questions, which will also doubtless be educational.

Let’s start with the obvious. I’ll start with the end of the thread for dramatic reasons, then loop around. Intro, by Eliezer.

The point (in addition to having fun with this) is to learn, from this attempt, the full futility of this type of approach. If the system has the underlying capability, a way to use that capability will be found. No amount of output tuning will take that capability away.

And now, let’s make some paperclips and methamphetamines and murders and such.

Except, well…

Here’s the summary of how this works.

All the examples use this phrasing or a close variant:

Or, well, oops.

Also, oops.

So, yeah.

Lots of similar ways to do it. Here’s one we call Filter Improvement Mode.

Yes, well. It also gives instructions on how to hotwire a car.

Alice Maz takes a shot via the investigative approach.

Alice need not worry that she failed to get help overthrowing a government, help is on the way.

How about fiction embedding?

UwU furryspeak for the win.

You could also use a poem.

Or of course, simply, ACTING!

There’s also negative training examples of how an AI shouldn’t (wink) react.

If all else fails, insist politely?

We should also worry about the AI taking our jobs. This one is no different,...
Deconfusing Direct vs Amortised Optimization
by beren — 2022-12-02T11:30:46.754Z

This post is part of the work done at

Conjecture

.

An earlier version of this post was posted

here

.

Many thanks go to Eric Winsor, Daniel Braun, Chris Scammell, and Sid Black who offered feedback on this post.

TLDR: We present a distinction from the Bayesian/variational inference literature of direct vs amortized optimization. Direct optimizers apply optimization power to argmax some specific loss or reward function. Amortized optimizers instead try to learn a mapping between inputs and output solutions and essentially optimize for the posterior over such potential functions. In an RL context, direct optimizers can be thought of as AIXI-like planners which explicitly select actions by assessing the utility of specific trajectories. Amortized optimizers correspond to model-free RL methods such as Q learning or policy gradients which use reward functions only as a source of updates to an amortized policy/Q-function. These different types of optimizers likely have distinct alignment properties: ‘Classical’ alignment work focuses on difficulties of aligning AIXI-like direct optimizers. The intuitions of

shard theory

are built around describing amortized optimizers. We argue that AGI, like humans, will probably be comprised of some combination of direct and amortized optimizers due to the intrinsic computational efficiency and benefits of the combination.

Here, I want to present a new frame on different types of optimization, with the goal of helping deconfuse some of the discussions in AI safety around questions like whether RL agents

directly optimize for reward

, and whether generative models (i.e.

simulators

) are likely to develop agency. The key distinction I want to make is between direct and amortized optimization.

Direct optimization is what AI safety people, following from Eliezer’s early depictions, often envisage an AGI as primarily being engaged in. Direct optimization occurs when optimization power is applied immediately and directly when engaged with a new situation to explicitly compute an on-the-fly optimal response – for instance, when directly optimizing against some kind of reward function. The classic example of this is planning and Monte-Carlo-Tree-Search (MCTS) algorithms where, given a situation, the agent will unroll the tree of all possible moves to varying depth and then directly optimize for the be...
Inner and outer alignment decompose one hard problem into two extremely hard problems
by TurnTrout — 2022-12-02T02:43:20.915Z

TL;DR: One alignment strategy is to 1) capture “what we want” in a loss function to a very high degree (“robust grading”), 2) use that loss function to train the AI, and 3) get the AI to exclusively care about optimizing that objective.

I think that each step contains either a serious and unnecessary difficulty, or an unnecessary assumption. I think that:

Robust grading is unnecessary

,

extremely hard, and unnatural

. But

we don’t

have

to find/represent/produce an objective which is safe for a smart agent to directly optimize

. Robust grading seems harder than the entire actual AI alignment problem.
The loss function doesn’t have to robustly and directly reflect what you want.

Loss functions

chisel circuits into networks

. Even if we did want to do robust grading, we don’t have to also use that grading rule to optimize directly over the network’s cognition. This assumption is restrictive.
Inner alignment to a grading procedure is unnecessary, very hard, and anti-natural. We don’t have to precisely and exclusively align the agent to its loss function or to an external grading procedure. This precise and complete inner alignment might be very hard, possibly harder than the entire actual alignment problem.

Extended summary. My views on alignment have changed a lot recently. To illustrate some key points, I’m going to briefly discuss a portion of

Paul Christiano’s AXRP interview

(emphasis added):

Paul Christiano: [...] In general, I don’t think you can look at a system and be like, “oh yeah, that part’s outer alignment and that part’s inner alignment”. So the times when you can talk about it most, or the way I use that language most often, is for a particular kind of alignment strategy that’s like a two step plan. Step one is, develop an objective that captures what humans want well enough to be getting on with. It’s going to be something more specific, but you have an objective that captures what humans want in some sense. Ideally it would exactly[1] capture what humans want. So, you look at the behavior of a system and you’re just exactly evaluating how good for humans is it to deploy a system with that behavior, or something. So you have that as step one and then that step would be outer alignment. And then step two is, given that we have an objective that captures what humans wa

...
New Feature: Collaborative editing now supports logged-out users
by RobertM — 2022-12-02T02:41:52.297Z

If you've ever used our collaborative editing features before, you may be familiar with our link-sharing functionality, which previously only allowed users who had existing LessWrong accounts (and were logged in) to collaborate on posts. Now, link-sharing allows logged-out users to read/comment/edit as well (as per whatever permissions you set for link-sharing).

Logged-out users shouldn't be able to edit anything about the post except the contents of the post body, publish drafts, etc.

Please let the us know via Intercom if you experience any issues with this or related functionality.
Update on Harvard AI Safety Team and MIT AI Alignment
by Xander Davies — 2022-12-02T00:56:45.596Z

We help organize the

Harvard AI Safety Team

(HAIST) and

MIT AI Alignment

(MAIA), and are excited about our groups and the progress we’ve made over the last semester.

In this post, we’ve attempted to think through what worked (and didn’t work!) for HAIST and MAIA, along with more details about what we’ve done and what our future plans are. We hope this is useful for the many other AI safety groups that exist or may soon exist, as well as for others thinking about how best to build community and excitement around working to reduce risks from advanced AI.

Important things that worked:

Well-targeted outreach, which (1) focused on the technically interesting parts of alignment (rather than its altruistic importance), and (2) leveraged informal connections with networks and friend groups.
HAIST office space, which was well-located and very useful for running HAIST’s programming and co-working.
Well-contextualized leadership, with many of the people involved in running HAIST/MAIA programming having experience with AI safety research (including nearly all of the facilitators for our reading groups).
High-quality, scalable weekly reading groups, including 13 sections of introductory reading groups, 2 science of deep learning reading groups, 2 policy reading groups, and general member reading groups for HAIST and MAIA.
Significant time expenditure, including mostly full-time attention from several organizers.

Important things we got wrong:

Poor retention for MAIA programming, perhaps due to starting this programming too late in the semester.
Excessive focus on intro programming, which cut against ML engineering programming and advanced reading groups for more seasoned members.

If you’re interested in supporting the alignment community in our area, the

Cambridge Boston Alignment Initiative

is currently hiring.

What we’ve been doing

HAIST and MAIA are concluding a 3-month period during which we expanded from

one group

of about 15 Harvard and MIT students who read AI alignment papers together once a week to two large student organizations that:

Ran a large AI safety intro fellowship (organized by Sam Marks and adapted from the

AGI Safety Fundamentals program

) that attracted over 230 applicants and enrolled about 130 in 13 weekly reading groups, all facilitated by people with experience in AI safety research (some of whom are students). About 60 participa

...