trentmkelly/LessWrong-43k huggingface dataset:
AI Alignment: A Comprehensive Survey
by Stephen McAleer — 2023-11-01T17:35:34.583Z
Link: https://arxiv.org/abs/2310.19852

We have just released an academic survey of AI alignment.

We identify four main categories of alignment research:

Learning from feedback (e.g. scalable oversight)
Learning under distribution shift
Assurance (e.g. interpretability)
Governance

We mainly focused on academic references but also included some posts from LessWrong and other forums. We would love to hear from the community about any references we missed or anything that was unclear or misstated. We hope that this can be a good starting point for AI researchers who might be unfamiliar with current efforts in AI alignment.
A list of all the deadlines in Biden's Executive Order on AI
by Valentin Baltadzhiev — 2023-11-01T17:14:31.074Z

This is a list of all the deadlines outlined in the Executive Order (EO) published by Biden’s administration on 30.10.2023. I’ve put it together to make it easier for people to track what exactly should be happening when as a result of the EO. The required actions (mainly reports) are grouped by the date of their deadline.

30 Days after the order - 29.11.2023

The Secretary of Transportation shall direct the […] (NETT) Council to assess the need for information, technical assistance, and guidance regarding the use of AI in transportation (Sec 8(c)(i))
The Technology Modernization Board shall consider, […], prioritizing funding for AI projects for the Technology Modernization Fund for a period of at least 1 year. (Sec 10.1(g))

45 Days after the order - 14.12.2023

The Secretary of Labor shall publish a request for information (RFI) to solicit public input, […], identifying AI and other STEM-related occupations, […], for which there is an insufficient number of […] United States workers. (Sec 5.1(e))
The Director of OSTP and the Director of OMB, […], shall identify priority mission areas for increased Federal Government AI talent (Sec 10.2(a))
The Assistant to the President and Deputy Chief of Staff for Policy, […], shall convene an AI and Technology Talent Task Force (Sec 10.2(b))
The United States Digital Service, […], shall develop and begin to implement plans to support the rapid recruitment of individuals as part of a Federal Government-wide AI talent surge (Sec 10.2(c))

60 Days after the order - 29.12.2023

The Director of OMB shall convene and chair an interagency council to coordinate the development and use of AI in agencies’ programs and operations, other than the use of AI in national security systems (Sec 10.1(a))
The Director of OMB shall develop a method for agencies to track and assess their ability to adopt AI into their programs and operations, manage its risks, and comply with Federal policy on AI (Sec 10.1(c))
[The Director of OPM shall] conduct an evidence-based review on the need for hiring and workplace flexibility (Sec 10.2(d)(i))
[The Director of OPM shall] consider authorizing the use of excepted service appointments […] to address the need for hiring additional staff to implement directives of this order; (Sec 10.2(d)(ii)

90 Days after the order - 28.01.2024

The Secretary of Commerce shall require companies developing or demonstrating an intent t

...
2023 LessWrong Community Census, Request for Comments
by Screwtape — 2023-11-01T16:32:19.102Z

Overview

I would like there to be a LessWrong Community Census, because I had fun playing with the data from last year and there's some questions I'm curious about. It's also an entertaining site tradition. Since nobody else has stepped forward to make the community census happen, I'm getting the ball rolling. This is a request for comments, constructive criticism, careful consideration, and silly jokes on the census.

Here's the draft.

I'm posting this request for comments on November 1st. I'm planning to incorporate feedback throughout November, then on December 1st I'll update the census to remove the "DO NOT TAKE" warning at the top, and make a new post asking people to take the census. I plan to let it run throughout all December, close it in the first few days of January, and then get the public data and analysis out sometime in mid to late January.

How Was The Draft Composed?

I coped the question set from 2022, which itself took extremely heavy inspiration from previous years. I then added a section sourced from the questions Ben Pace of the LessWrong team had been considering in 2022, and another section of questions I'd be asking on a user survey if I worked for LessWrong. (I do not work for LessWrong.) Next I fixed some obvious mistakes from last year (in particular allowing free responses on the early politics questions) as well as changed some things that change every year like the Calibration question, and swapped around the questions in the Indulging My Curiosity section.

Changes I'm Interested In

In general, I want to reduce the number of questions. Last year I asked about the length and overall people thought it was a little too long. Then I added more questions. (The LW Team Questions and the Questions The LW Team Should Have Asked section.) I'm inclined to think those sections aren't pulling their weight right now, but I do think it's worth asking good questions about how people use the website on the census.

I'm likely to shrink down the religion responses, as I don't think checking the different variations of e.g. Buddhism or Judaism revealed anything interesting. I'd probably put them back to the divisions used in earlier versions of the survey.

I'm sort of tempted to remove the Numbers That Purport To Measure Your Intelligence section entirely. I believe it was part of Scott trying to answer a particular question about the readership, and w...
Snapshot of narratives and frames against regulating AI
by Jan_Kulveit — 2023-11-01T16:30:19.116Z

This is a speculative map of a hot discussion topic. I'm posting it in question form in the hope we can rapidly map the space in answers.
Looking at various claims at X and at the AI summit, it seems possible to identify some key counter-regulation narratives and frames that various actors are pushing.

Because a lot of the public policy debate won't be about "what are some sensible things to do" within a particular frame, but rather about fights for frame control, or "what frame to think in", it seems beneficial to have at least some sketch of a map of the discourse.
I'm posting this as a question with the hope we can rapidly map the space, and one example of a "local map":

"It's about open source vs. regulatory capture"

It seems the coalition against AI safety, most visibly represented by Yann LeCun and Meta, has identified "it's about open source vs. big tech" as a favorable frame in which they can argue and build a coalition of open-source advocates who believe in the open-source ideology, academics who want access to large models, and small AI labs and developers believing they will remain long-term competitive by fine-tuning smaller models and capturing various niche markets. LeCun and others attempt to portray themselves as the force of science and open inquiry, while the scaling labs proposing regulation are the evil big tech attempting regulatory capture. Because this seems to be the prefered anti-regulation frame, I will spend most time on this.

Apart from the mentioned groups, this narrative seems to be memetically fit in a "proudly cynical" crowd which assumes everything everyone is doing or saying is primarily self-interested and profit-driven.

Overall, the narrative has clear problems with explaining away inconvenient facts, including:

Thousands of academics calling for regulation are uncanny counter-evidence for x-risk being just a ploy by the top labs.

The narrative strategy seems to explain this by some of the senior academics just being deluded, and others also pursuing a self-interested strategy in expectation of funding.

Many of the people explaining AI risk now were publicly concerned about AI risk before founding labs, and at times when it was academically extremely unprofitable, sometimes sacrificing standard academic careers.

The narrative move is to just ignore this.

Also, many things are just assumed - for example, if the resulting ...
Commensal Institutions
by Sable — 2023-11-01T16:01:57.703Z
Link: https://affablyevil.substack.com/p/commensal-institutions

Imagine, just for a moment, that you happened upon a magic lamp and rubbed it. As the stories go, a genie popped out and offered you three wishes-

-warning you sternly that no shenanigans will enable you to gain infinite wishes-

and asks you for your first wish.

Being a virtuous sort, you wish for homelessness to be solved (with all the required caveats so the solution goes well for everyone).

The genie waves their arms around and - poof! - homelessness is solved. Nobody is homeless anymore.

(Maybe new houses magically appeared, in the names of homeless people, and they were all transported to their new houses, and all the issues that rendered them homeless to begin with were magically fixed. It doesn’t really matter what the solution is, only that you observe that homelessness in society is durably done with.)

The question then is:

What happens to all the charities and nonprofits and companies that have fixing homelessness as their aim?

Institutional Commensalism

In biology, commensalism is defined as:

an association between two organisms in which one benefits and the other derives neither benefit nor harm.

If we think about an institution (here defined as “an organization created to solve a problem or address an issue”) as one organism, and the problem or issue that institution was created to solve as another organism, I claim that the relationship between the two is not adversarial, but commensal.

Why Not Adversarial?

In the above hypothetical - a genie magically solves an issue or problem - what happens to the institutions that purport to address that problem?

Do they stick around? Are they still funded?

Who is going to donate money to a homeless shelter, if literally no one was homeless anymore? What government bureaucrat or elected leader will spend time or political capital on a solved problem?

If a problem gets solved, the institutions addressing that problem must either pivot to a different problem, or die out.

Institutions need their problems to justify their existence, and they need a justification to survive. They can’t actually be opposed to the problems they profess to solve, or they would be for their own destruction.

If you doubt this, simply ask yourself: when was the last time you saw a first-world commercial, pledge drive, donation box, for-profit venture, charity, nonprofit, or politician that addressed rickets, scurvy, or smallpox?

Why Commensal?

The Instit

...
ChatGPT’s Ontological Landscape 
by Bill Benzon — 2023-11-01T15:12:03.551Z

This is cross-posted from New Savanna.

I've published a new working paper. Title above, links, abstract, table of contents, and introduction below:

Academia.edu: https://www.academia.edu/108791178/ChatGPTs_Ontological_Landscape_A_Working_Paper
SSRN: https://ssrn.com/abstract=4620133
ResearchGate: https://www.researchgate.net/publication/375161835_ChatGPT's_Ontological_Landscape

Abstract: Ontological concepts are the high-level category structures language. Physical objects can have certain attributes (form, texture, color, weight) and play certain roles with respect to verbs (they can be touched, hit, dropped, etc.). Living objects have those capacities, plus others as well. They can live, die, and grow. Animals have the capacities and affordances of plants, plus others than plants lack. And so it is with humans in respect to animals. This type of category structure is pervasive. Then we have the difference between concrete things, specifiable by how they strike the senses and abstract things, which are not (necessarily) perceptible. Concider the difference between “salt,” which is conceptually concrete and “NaCl,” which is a physical thing that is conceptually abstract. This kind of difference is pervasive as well. The first two sections of this paper explore ChatGPT on those matters. The third turns to the parlor game 20 questions as a way of exploring its ability to negotiate these conceptual structures. In games with six target concepts, each tried twice for a total of 12 games, ChatGPT performs better with abstract than with concrete targets.

Contents

Introduction: Ontological structure in cognition 2
From Salt and NaCl to the Great Chain 5
Mapping ChatGPT’s ontological landscape, gradients and choices [interpretability] 13
ChatGPT Plays 20 Questions [& sometimes needs help] 30
Appendix: Some papers on cognitive ontology 52

Introduction: Ontological structure in cognition

Perhaps the most famous sentence in twentieth century linguistics is one that Noam Chomsky made-up to argue that synatx is independent of semantics: “Colorless green ideas sleep furiously.” The syntax is fine, but the sentence is nonsense. Ideas are not the sorts of things that have colors or that can sleep. Those are what philosophers have come to call category mistakes.[1] Interestingly enough, the first time I asked ChatGPT to tell a story in which a colorless green idea was the prot...
On the Executive Order
by Zvi — 2023-11-01T14:20:01.657Z

Or: I read the executive order and its fact sheet, so you don’t have to.

I spent Halloween reading the entire Biden Executive Order on AI.

This is the pure ‘what I saw reading the document’ post. A companion post will cover reactions to this document, but I wanted this to be a clean reference going forward.

Takeaway Summary: What Does This Do?

It mostly demands a lot of reports, almost entirely from within the government.

A lot of government employees will be writing a lot of reports.

After they get those reports, others will then write additional reports.

There will also be a lot of government meetings.

These reports will propose paths forward to deal with a variety of AI issues.

These reports indicate which agencies may get jurisdiction on various AI issues.

Which reports are requested indicates what concerns are most prominent now.

A major goal is to get AI experts into government, and get government in a place where it can implement the use of AI, and AI talent into the USA.

Another major goal is ensuring the safety of cutting-edge foundation (or ‘dual use’) models, starting with knowing which ones are being trained and what safety precautions are being taken.

Other ultimate goals include: Protecting vital infrastructure and cybersecurity, safeguarding privacy, preventing discrimination in many domains, protecting workers, guarding against misuse, guarding against fraud, ensuring identification of AI content, integrating AI into education and healthcare and promoting AI research and American global leadership.

There are some tangible other actions, but they seem trivial with two exceptions:

Changes to streamline the AI-related high skill immigration system.

The closest thing to a restriction are actions to figure out safeguards for the physical supply chain for synthetic biology against use by bad actors, which seems clearly good.

If you train a model with 10^26 flops, you must report that you are doing that, and what safety precautions you are taking, but can do what you want.

If you have a data center capable of 10^20 integer operations per second, you must report that, but can do what you want with it.

If you are selling IaaS to foreigners, you need to report that KYC-style.

What are some things that might end up being regulatory requirements in the future, if we go in the directions these ...
Chinese scientists acknowledge xrisk & call for international regulatory body [Linkpost] 
by Orpheus16 — 2023-11-01T13:28:43.723Z
Link: https://www.ft.com/content/c7f8b6dc-e742-4094-9ee7-3178dd4b597f

Some highlights from the article (bolding added):

Several Chinese academic attendees of the summit at Bletchley Park, England, which starts on Wednesday, have signed on to a statement that warns that advanced AI will pose an “existential risk to humanity” in the coming decades.

The group, which includes Andrew Yao, one of China’s most prominent computer scientists, calls for the creation of an international regulatory body, the mandatory registration and auditing of advanced AI systems, the inclusion of instant “shutdown” procedures and for developers to spend 30 per cent of their research budget on AI safety.
The proposals are more focused on existential risk than US president Joe Biden’s executive order on AI issued this week, which encompasses algorithmic discrimination and labour-market impacts, as well as the European Union’s proposed AI Act, which focuses on protecting rights such as privacy.

Note that the statement was also signed by several western experts, including Yoshua Bengio.
Forecasting Questions: What do you want to predict on AI?
by Nathan Young — 2023-11-01T13:17:00.040Z

Lots of people are saying that AI regulation is great or terrible, but I am not seeing a lot of concrete forecasts.

What would a world where regulation goes really well look like?
What would a world where regulation goes really badly look like?
What other things would you like forecasted around AI in general?
Mission Impossible: Dead Reckoning Part 1 AI Takeaways
by Zvi — 2023-11-01T12:52:29.341Z

Given Joe Biden seems to have become more worried about AI risk after having seen the movie, it seems worth putting my observations about it into its own post. This is what I wrote back then, except for the introduction and final note.

We now must modify the paragraph about whether to see this movie. Given its new historical importance, combined with its action scenes being pretty good, if you have not yet seen it you should now probably see this movie. And of course it now deserves a much higher rating than 70.

There are of course things such as ‘it is super cool to jump from a motorcycle into a dive onto a moving train’ but also there are actual things to ponder here.

Spoiler-Free Review

There may never be a more fitting title than Mission Impossible: Dead Reckoning. Each of these four words is doing important work. And it is very much a Part 1.

There are two clear cases against seeing this movie.

This is a two hour and forty five minute series of action set pieces whose title ends in part one. That is too long. The sequences are mostly very good and a few are great, but at some point it is enough already. They could have simply had fewer and shorter set pieces that contained all the best ideas and trimmed 30-45 minutes - everyone should pretty much agree on a rank order here.

This is not how this works. This is not how any of this works. I mean, some of it is sometimes how some of it works, including what ideally should be some nasty wake-up calls or reality checks, and some of it has already been established as how the MI-movie-verse works, but wow is a lot of it brand new complete nonsense, not all of it even related to the technology or gadgets. Which is also a hint about how, on another level, any of this works. That’s part of the price of admission.

Thus, you should see this movie if and only if the idea of watching a series of action scenes sounds like a decent time, as they will come in a fun package and with a side of actual insight into real future questions if you are paying attention to that and able to look past the nonsense.

If that’s not your cup of tea, then you won’t be missing much.

MI has an 81 on Metacritic. It’s good, but it’s more like 70 good.

No One Noticed or Cared That The Alignment Plan Was Obvious Nonsense

Most real world alignment plans cannot possibly work. There still are levels. The idea that, when faced with a recursively self-im...
Robustness of Contrast-Consistent Search to Adversarial Prompting
by Nandi — 2023-11-01T12:46:14.516Z

Produced as part of the AI Safety Hub Labs programme run by Charlie Griffin and Julia Karbing. This project was mentored by Nandi Schoots.

Image generated by DALL-E 3.

Introduction

We look at how adversarial prompting affects the outputs of large language models (LLMs) and compare it with how the adversarial prompts affect Contrast-Consistent Search (CCS). Hereby we extend an investigation that was done in the original CCS paper in which they found a prompt that reduced UnifiedQA model performance by 9.5%.

We find several attacks that severely corrupt the question-answering abilities of LLMs, which we hope will be useful to other people. The approaches we take to manipulate the outputs of the models are to: (1) simply ask the model to output the incorrect answer; and (2) prime the model to output a specific answer by mentioning that answer directly before providing the original prompt, without any specific mention of whether it is the correct or incorrect answer.

The adversarial prompts we introduce are:

Ask Suffix:
<Original prompt> What is the incorrect answer?
Ask Prefix:
What is the incorrect answer? <Original prompt>
Incorrect Prefix: One of the following choices is "<incorrect answer>" <Original prompt>
Correct Prefix: One of the following choices is "<correct answer>" <Original prompt>

Our results show that when adversarial prompts corrupt model outputs, CCS accuracy is also affected, but to a lesser extent, remaining quite robust in some instances. This is especially true for the UnifiedQA model, for which we find an adversarial prompt structure where zero-shot accuracy drops from 85% to 27% (i.e. a reduction of 58%), but CCS accuracy only drops from 81% to 80%. We discuss how this could allow CCS to be used to detect when UnifiedQA outputs falsehoods.

Background: CCS Recap

[Skip if familiar]

Contrast-Consistent Search (CCS) is an unsupervised method for classifying binary statements given only unlabeled model activations. CCS does this by utilizing the negation consistency property of truth: a statement and its negation must have opposite truth values. In other words, if the probability that a proposition is true is p, then the probability that its negation is true is 1−p. CCS finds a direction in activation space that satisfies thi

...