trentmkelly/LessWrong-43k huggingface dataset:
A few thoughts on my self-study for alignment research
by Thomas Kehrenberg — 2022-12-30T22:05:58.859Z

In June, I received a grant from the LTFF for a 6-months period of self-study aimed at mastering the necessary background for AI alignment research. The following is advice I would give to people who are attempting something similar. I have tried to keep it short.

Basic advice

You’ll naturally want to maximize “insights per minute” when choosing what to read. But, don’t expect it to be obvious what the most impactful reading material is! It often takes actual focused thought to figure this out.

One shortcut is to just ask yourself what you are really curious about; based on the idea that your curiosity should track “value of information” to some degree, so it can’t be that wrong to follow your curiosity, but also, working through a textbook takes quite a bit of mental energy, so having natural curiosity to power through your study is very helpful.

If you don’t already have something you’re curious about, you can try the following technique to try to figure out what to read:

First, list all the things you could potentially read.

This step includes looking at recommendation lists from other people. (See below for two possible lists.)

For each thing on the list, write down how you feel about maybe reading that.

Be honest with yourself.
Try to think of concrete reasons that are shaping your judgment.

Then, look back over the list.

Hopefully, it should be easier to decide now what to read.

This was helpful to me, which doesn’t necessarily mean it’s helpful for you, but it’s maybe something to try.

Advice specifically for AI alignment

The above should hold for any topic; the following advice is for AI alignment research study specifically.

I think you basically can’t go wrong with reading all of (or maybe, the to-you-most-interesting 80% of) the AI alignment articles on Arbital. I found this to be the most effective way to rapidly acquire a basic understanding of the difficulty.
In terms of fundamental math, I just picked topics that sounded interesting to me from the MIRI research guide and John Wentworth’s study guide.[1]
It’s probably also a good idea to read some concrete results from alignment research, if only to inform you about what kind of math is required. I think Risks from Learned Optimization in Advanced Machine Learning Systems is one good option. I don’t know of a good list of other results.

Concrete reading recommendations

The following are recommendations for concre...
Christmas Microscopy
by jefftk — 2022-12-30T21:10:01.937Z

I got the kids a

cheap little
microscope for Christmas that attaches to your computer. While
we've played with microscopes before, it's much more fun having one
where everyone can see at the same time. Here are some things we saw:

A pine needle that we had just broken, with sap leaking out:

The horsehair of a violin bow:

The carbon fiber 'hair' of my Incredibow:

A cloth bag, showing how it's woven and looking a lot like a waffle:

A section of the bag where Julia had painted a number in white,
showing how the paint is on the surface:

A crayon box, where you can see that the apparently solid colors are
actually made with dots of a few different colors:

A fleece glove:

A paper bag, showing the fibers:

My wedding ring, looking very scratched despite feeling smooth to the
touch. You can see the reflection of the eight LEDs on the microscope:

The hook side of velcro, showing that the hooks are loops that have
been cut:

The hard part of any present like this is remembering to use it when
the opportunity comes up again. Next time the kids are curious about
something I think would show up well I'll get it out.

Comment via: facebook, mastodon
What "upside" of AI?
by False Name — 2022-12-30T20:58:49.165Z

Consider, first, what humanity would want some aligned AGI to do, were it obtained? Set aside details, and let's suppose, in the main, we want (not instruct, just want) AGI to "make our lives better." (Too, AGI could "make us more productive," but if what is produced is bad, let us suppose we do not desire it).

But now a problem: AI in general is meant to do what we cannot do - else we would have done it, already, without AI - but to "make things better", that we could very readily have done, already. Therefore, since the "upside" of AI is to make things better as we could not, but we could in fact have made things better, only some impulse in humanity held off improvement - it follows, there is no upside of AI, at least, none we couldn't have procured for ourselves; whatever we would get of AI is no "upside".

That we could have done better follows from considering counterfactuals of a civilization which takes AI alignment seriously, for instance. And we certainly had many opportunities to create such a civilization. The famed "Forty acres and a mule," duly provided; a propounded and enacted "One Big Union," from an unsuppressed IWW; validated efforts at corrections reform on Norfolk Island by Maconochie, or ratification of the Equal Rights Amendment, or widespread adoption of the teachings of Mo Zi, of Emma Goldman - or practical use of Hero's steam engine, or the survival of Archimedes, Abel, and Galois, or -

We've had plenty of opportunities to "make things better". The AI, then, will rather be engaged on a salvage operation, not a mission of optimization. Hence it will have first to un-do - and any such need carries the danger of removing what we might value (perhaps we value most what makes us less than best). Recall too, since we are observably incapable of making ourselves nearer these counterfactuals, nearer some "best", then AI on the current machine learning paradigm, mimicking our data, so mimicking us, therefore is apt to become likewise what is incapable of making things "the best". With no predominating tendency toward that - what tendency has it to any "better"?

And since, in fact, perhaps the "worse angels of our nature" propelled us away from what would have made things better, then, the AI entrained to our patterns by our data - why should not it manifest such worse, or the absolute worst, angels of us, to destroy us, if it is...
Evidence on recursive self-improvement from current ML
by beren — 2022-12-30T20:53:22.462Z

The goal of this post is to start a collaborative analysis. Please post papers or links or other evidence I have missed in comments. I want to get some of Lesswrong's collective mind focused on this question.

A core component of the classical case for AI risk is the potential for AGI models to recursively self-improve (RSI) and hence dramatically increase in capabilities once some threshold of intelligence is passed. Specifically, it is argued that an only somewhat superhuman AGI will rapidly be able to bootstrap itself into an exceptionally powerful superintelligence able to take over the world and impose its values on us against all opposition.

A lot of the debates on fast vs slow takeoffs hinge on the feasibility and dynamics of the process of RSI, as well as do many potential counters to AI risk. If strong takeoff is inevitable, then strategies like boxing, impact regularization, myopia, and human-in-the-loop auditing and interpretability and so on are intrinsically doomed, since the AGI will simply become too powerful too quickly and can thus break out of any box or outwit any human operators. In such world, iterative development of AGI safety techniques is doomed since the first AGI we build will immediately explode in capabilities. To survive in such a world, we need to design a foolproof solution to alignment before building the first AGIs.

On the other hand, if RSI is difficult and slow, then there is a chance that these techniques will succeed at containing any proto-AGIs that we build and prevent them from bootstrapping themselves to superintelligence. This doesn't necessarily mean that AI safety is solved, just that the world will likely look much more like a slow takeoff where various AGIs start participating in the economy and slowly accruing power or giving it to various human actors if alignment is at least partially solved. It also means that humanity as a whole have a lot of shots at alignment so that we can use iterative techniques to discover failures empirically and then correct them.

There is also a world where strong RSI is possible and relatively easy, but that containment techniques like boxing are powerful enough to have a good chance of containing near-future proto-AGIs. In this case, we end up in a few-shot alignment world where we can contain and experiment upon AGIs for a small number of experiments or for some length of...
Is ChatGPT TAI? 
by Amal  — 2022-12-30T19:44:50.508Z

Do you count ChatGPT and/or other similar systems (e.g. codex) as transformative AI? I would think so since it has a very fast adoption (million users in 5 days) and is apparently sufficiently more useful on some queries than Google that they had to declare "code red" - a possible threat to their core business. If not, why and where do you see the line it would need to cross to be TAI?
My thoughts on OpenAI's alignment plan
by Orpheus16 — 2022-12-30T19:33:15.019Z

Epistemic Status: This is my first attempt at writing up my thoughts on an alignment plan. I spent about a week on it.

I’m grateful to Olivia Jimenez, Thomas Larsen, and Nicholas Dupuis for feedback.

A few months ago, OpenAI released its

plan for alignment

. More recently, Jan Leike (one of the authors of the original post) released a

blog post

about the plan, and Eliezer & Nate encouraged readers to

write up their thoughts

.

In this post, I cover some thoughts I have about the OpenAI plan. This is a long post, and I’ve divided it into a few sections. Each section gets increasingly more specific and detailed. If you only have ~5 minutes, I suggest reading section 1 and skimming section 2.

The three sections:

An overview of the plan and some of my high-level takes (here)
Some things I like about the plan, some concerns, and some open questions (here)
Specific responses to claims in OpenAI’s post and Jan’s post (here)

Section 1: High-level takes

Summary of OpenAI’s Alignment Plan

As I understand it, OpenAI’s plan involves using reinforcement learning from human feedback and recursive reward modeling to build AI systems that are better than humans at alignment research. Open AI’s plan is not aiming for a full solution to alignment (that scales indefinitely or that could work on a superintelligent system). Rather, the plan is intended to (a) build systems that are better at alignment research than humans (AI assistants), (b) use these AI assistants to accelerate alignment research, and (c) use these systems to build/align more powerful AI assistants.

Six things that need to happen for the plan to work

For the plan to work, I think it needs to get through the following 6 steps:

OpenAI builds LLMs that can help us with alignment research
OpenAI uses those models primarily to help us with alignment research, and we slow down/stop capabilities research when necessary
OpenAI has a way to evaluate the alignment strategies proposed by the LLM
OpenAI has a way to determine when it’s OK to scale up to more powerful AI assistants (and OpenAI has policies in place that prevent people from scaling up before it’s OK)
Once OpenAI has a highly powerful and aligned system, they do something with it that gets us out of the acute risk period
Once we are out of the acute risk period, OpenAI has a plan for how to use transformative AI in ways that allow h

...
Beyond Rewards and Values: A Non-dualistic Approach to Universal Intelligence
by Akira Pyinya — 2022-12-30T19:05:24.664Z

(This is my first post on Lesswrong, including some thoughts on topics I have read in this year. Please leave your comment on them. Thanks.)

Abstract

Building an AI system that aligns with human values is believed to be a two-step process: first design a value function or learn human value using value learning methods, then maximize those values using rational agents such as AIXI agents. In order to integrate this into one step, we analyze the dualistic assumptions of AIXI, and define a new universal intelligence model that can align with human preferences or specific environments, called Algorithmic Common Intelligence (ACI), which can behave the same way as examples. ACI does not have to employ rewards or value functions, but directly learns and updates hypothetical policies from experience using Solomonoff induction, while making actions according to the probability of every hypothesis. We argue that the rational agency model is a subset of ACI, and the coevolution of ACI and humans provides a pathway to AI alignment.

1. AIXI as a dualistic model

Dualistic agent and embedded agent

In most agent-based intelligence models, the agents are cleanly separated from their environments, but agents in the real world are a part of their environments. Demski and Garrabrant termed the former dualistic agents, and the later embedded agents.

A dualistic agent acts like it’s playing a video game, interacts with the game only through well-defined input and output channels, such as the screen and the controller. The agent doesn't have any opportunity for self-modification. It’s immortal, and its reward circuit is not vulnerable to being hacked.

On the contrary, an embedded agent is a part of the universe. There is no clear boundary between the agent and the environment. An embedded agent may improve itself, but might also modify its original goals in undesirable ways, like “directly tamping its reward circuit to get rewards in a conventional way”, which was called self-rewarding or wireheading. It may even drop an anvil on its head to see what happens. Those possible behaviors are not within the scope of traditional rational agency models which assume the agents are dualistic.

In order to build a new theoretic framework for embedded agents, let’s investigate traditional models in search of when the dualistic assu...
10 Years of LessWrong
by SebastianG  — 2022-12-30T17:15:17.498Z

[I appreciate that less wrong has a very strong norm against navel gazing. Let's keep it that way. The purpose of this post is merely to reflect upon and highlight valuable tools and mental habits in our toolkit.]

The rationalsphere propelled me to where I am today by giving me tools that I wouldn’t otherwise have developed. The concept handles that we have developed are numerous and useful, but here I am only talking about the underlying habits that dug up good ore for the wordsmiths.

It was a typical path starting in 2012. Harry Potter and the Methods of Rationality led to LessWrong, led to Slate Star Codex. Combined they fed into game theory, from game theory economics, and from economics all worked towards a new way of thinking. On a different branch I went from Slate Star Codex to Tetlock and forecasting and participating in some studies and tournaments (I can’t beat the market). On a different branch, I have followed the AI literature from mostly MIRI and Paul Christiano and John Wentworth. And along with the AI DLC in my mind has come “AI development watching” and AI learning systems. This has been great, and it has been standard, and I am sure many others have done the same.

But it was also an atypical path. I was a classicist and philosopher, bouncing around on the back of a bus in Italy between archeological sites with my 17 inch laptop and 12 tabs of HPMOR and two tabs of LW reading on each journey. Not yet having any knowledge of either calculus or discrete math, nor any inkling of basic coding or incentive structures, I was a youngling in the art of rationality. But I was well on my way in the great books tradition.

I read HPMOR and the Sequences angrily. Many of the ideas I found refreshing and so on target, and many more I found blasted, awful, un-nuanced, and wrong. Many ideas about human nature and philosophy of language, logic, and science I wrestled with time and time again - always coming back for more. (Some are still wrong, mind you).

I had rejected psychology sophomore year of college on the grounds that several studies in our textbook obviously didn’t show what they claimed to show, and with that rejection of psychology, I rejected the idea of the quantification of human behavior. But reverse stupidity is not intelligence. So it took about three years before I could be salvaged from that position. It took scores of late night argument...
Chatbots as a Publication Format
by derek shiller — 2022-12-30T14:11:21.015Z

Overview: Recent developments in AI will change the world in all sorts of
ways. It is likely to revolutionize academic research. This is one
proposal regarding how AI might be used to improve the way that we communicate
arguments (such as in philosophy). I stand by behind the
thought that there are problems with present publication formats that
technology could solve, but this particular solution is highly speculative
and not super well thought through. Main ideas are bolded.

Problems with the linear paradigm

Current serious argumentative work (such as in philosophy) primarily takes
the form of written papers and books, and less commonly lectures and blog
posts. Theorists might explain their view in an interactive way in
personal contexts, such as at a departmental tea time, but what they
send elsewhere, the way that most people might
interact with their work, is linear.

Long-form linear argumentative writing is essentially an ancient Greek
technology. It hasn't been much improved since. Authors present ideas in
sequence, attempting to walk the reader through them in an order that is
conducive to understanding. Readers mostly read articles
and books straight through. They may occasionally skip around, especially
if they are trying to extract details about the argument. However, linear
writing isn't particularly suited for this.

Linear presentations have a few issues:

The amount of content in a written work is limited by space and the
reader's patience. Articles generally consist of less than a few dozen
pages. Books may be much longer, but still aren't long enough to capture
all of the author's views. Often, complex assumptions and arguments must
be glossed over or left out. Authors sometimes publish the same ideas
multiple times to serve the needs of different audiences.

Authors must present their ideas with at most only a few levels of
detail for readers in a single work. (E.g. the abstract, an
introductory summary, and the full argument.) Readers cannot easily get
access to all and only the level of detail of the arguments they want.
They can skim parts they aren't interested in, but they can't get more
information about arbitrary sections.

Authors have to decide how much
background material to include, and what level of sophistication to pitch it to.
Many books and articles are written for a professional audience and are
inaccessible by those without

...
Human sexuality as an interesting case study of alignment
by beren — 2022-12-30T13:37:20.176Z

This is cross-posted from my personal blog

Epistemic status: mostly interesting questions.

Here I want to bring attention to what I think is an extremely impressive case of evolution's ability to 'align' humans in the wild: the development of human sexuality.

Reasons why this is an interesting thing to study from the lens of alignment, and why it is a highly non-trivial accomplishment:

1.) Evolution has been very successful here: almost all humans end up wanting to have sex and typically with opposite-gender partners in a way that would result in children (and hence IGF) in the evolutionary environment.

2.) Sexuality, unlike many other drives such as hunger and thirst, is not something built into the brain from the beginning. Instead there is a sudden 'on switch' around puberty. What happens in the brain during this time? How does evolution exert such fine-grained control of brain development so long (decades) after birth?

3.) It is mostly independent of initial training data before puberty -- i.e. evolution can ignore a decade of data input and representation learning, which it cannot control, during a time period when the brain is undergoing extremely large changes, and still finds a way to instill a new drive highly reliably.

4.) It seems to occur mostly without RL. People start wanting to have sex before they have actually had sex. If sexuality developed by some RL mechanism, it would look like you go around doing your normal things, then at some point you have sex, and realize it is highly rewarding, and you slightly update your behaviours and/or values to get more sex or to want more sex. This is not what happens in humans. Instead, humans often want to start having sex before they have had it, and even before they really know what sex is[1].

5.) Evolution has solved some variant of the pointers problem to get humans assigning high value to both a previously unknown and mostly non-represented state (i.e. you don't usually have a well-represented sex concept before puberty), as well as also translating this desire to specific other agents in the world -- such as crushes, hot people etc. This is done, presumably, in an entirely genetically mediated way without requiring specific experience.

6.) Sexuality is, usually, a very strong drive which has a large influence over behaviour and long term goals. If we could create an alignment drive as strong in our AGI...
The Twitter Files: Covid Edition
by Zvi — 2022-12-30T13:30:01.073Z

At long last, this week we got the Covid Edition of The Twitter Files.

I focus here only on this edition, and not focusing on Musk or the Twitter Files more generally. Is there a there there?

Seems like the answer is, yes, but the there that was there seems not so out there?

I’ll walk through the thread.

Go on.

I am shocked that governments express preferences. Go on.

I sympathize. The first rule of panic buying is you do not talk about panic buying. It is the job of The New York Times, let’s say, to have the headline be “Widespread Panic Buying Across Nation” when there is widespread panic buying across the nation. You got to let journalists journalist. We can also realize that doing this is Not Helping. It is individually rational and socially harmful to panic buy, it is valuable information and big news that there is panic buying happening, and the government has a clear interest in hiding the existence of panic buying.

That does not justify censorship or labeling true information ‘misinformation’ because we prefer people not know. But I understand. This is kind of like shouting fire in a crowded theater, which a lot of people keep saying is illegal (but which, actually, is totally legal if there is an actual fire.)

Certainly if an individual speaker decided it was better not to mention something like panic buying on the internet, that is often a reasonable or even wise decision. Asking others politely to consider not mentioning or amplifying this news more than necessary also seems reasonable. I certainly understand, especially if they did not do this with other later issues that did not involve threats to public order.

The thread then skips directly to the Biden administration.

This speaks well of Twitter. A common dunk on The Twitter Files has been ‘oh look at these records that show a reasonable company acting completely reasonably.’ That is not a dunk. It is good to know how such things worked, including when actions were indeed reasonable or even close to ideal. That, too, is useful information. That we are given info when it makes Twitter look good, not only when it makes Twitter look bad, is to the credit of the whole project.

I really don’t know what anyone was expecting. There’s no way to deal with the volumes involved without bots, and the bots are going to make mistakes, more mistake...