trentmkelly/LessWrong-43k huggingface dataset:
Introducing EffiSciences’ AI Safety Unit 
by WCargo — 2023-06-30T07:44:56.948Z

This post was written by Léo Dana, Charbel-Raphaël Ségerie, and Florent Berthet, with the help of Siméon Campos, Quentin Didier, Jérémy Andréoletti, Anouk Hannot, Angélina Gentaz, and Tom David.

In this post, you will learn what were EffiSciences’ most successful field-building activities as well as our advice, reflections, and takeaways to field-builders. We also include our roadmap for the next year. Voilà.

What is EffiSciences?

EffiSciences

is a non-profit based in France whose mission is to mobilize scientific research to overcome the most pressing issues of the century and ensure a desirable future for generations to come.

EffiSciences was founded in January 2022 and is now a team of ~20 volunteers and 4 employees.

At the moment, we are focusing on 3 topics: AI Safety, biorisks, and climate change. In the rest of this post, we will only present our

AI safety unit

and their results.

TL;DR: In one year, EffiSciences created and held several AIS bootcamps (ML4Good), taught accredited courses in universities, organized hackathons and talks in France’s top research universities. We reached 700 students, 30 of whom are already orienting their careers into AIS research or field building. Our impact was found to come as much from kickstarting students as from upskilling them. And we are in a good position to become an important stakeholder in French universities on those key topics.

Field-building programs

TL;DR of the content

Results

Machine Learning for Good bootcamp (ML4G)	Parts of the MLAB and AGISF condensed in a 10-day bootcamp (very intense)	2 ML4G, 36 participants, 16 are now highly involved. This program was reproduced in Switzerland and Germany with the help of EffiSciences.
Turing Seminar	AGISF-adapted accredited course that has been taught with talks, workshops, and exercises.	3 courses in France’s top 3 universities: 40 students attended, 5 are now looking to upskill, and 2 will teach the course next year.
AIS Training Days	The Turing Seminar compressed in a single day (new format)	3 iterations, 45 students
EffiSciences’ educational hackathons	A hackathon to introduce robustness to distribution change and goal misgeneralization.	2 hackathons, 150 students, 3 are now highly involved
Apart Research’s hackathons	We hosted several Apart Research hackathons, mostly with people already onboa

...
Contra Anton 🏴‍☠️ on Kolmogorov complexity and recursive self improvement
by DaemonicSigil — 2023-06-30T05:15:38.701Z

Twitter user @atroyn claims that recursive self-improvement is impossible because of Kolmogorov complexity. Quoting most of[1] the argument here:

here is an argument against the possibility of recursive self improvement of any 'intelligent' computer program, based on kolmogorov complexity.

intelligence is the ability to make correct predictions about the state of the world given available information.

each program which makes predictions about the world has a kolmogorov complexity corresponding to the length of the shortest string which can express that program

for a given program p call this complexity k.

(unfortunately k(p) is in general uncomputable, the proof reduces to the halting problem, but that's not important here)

more intelligence (in our definition) implies the ability to predict more of the world more accurately, i.e. to express more of the world's complexity - this implies that a more intelligent program p2 necessarily has more complexity than a less intelligent p1

to see that this is necessarily so, note that if we could predict the world equally accurately as p1's prediction with a program p0 with k0 < k1, then we have a contradiction since k1 was supposed to be the minimal expression of intelligence at that level

in order to get recursive self improvement, you need a program p1 which is capable of emitting p2 which is better able to predict the world than p1 - i.e., we need p1 to emit p2 such that k2 > k1

but this is a contradiction.

[...]

The mistake here is the assumption that a program that models the world better necessarily has a higher Kolmogorov complexity. Originally, Kolmogorov complexity measured the complexity of bit strings. But we're talking about predictors here, things that observe the world and spit out probability distributions over observed outcomes. In the context of predictors, Kolmogorov complexity measures the complexity of a function from observations to predictions.

In the case of ideal Bayesian reasoning, we can nail down such a function just by specifying a prior, eg. the Solomonoff prior. (Plus, an approximation scheme to keep things computable, I guess.) This doesn't take a very large program to implement. But a non-ideal reasoner will screw up in many cases, and there's information contained in the exact way it screws up for each set of observations. Such reasoners can have an almost arbitrarily high Kolmogorov co...
Foom Liability
by PeterMcCluskey — 2023-06-30T03:55:22.698Z
Link: https://bayesianinvestor.com/blog/index.php/2023/06/29/foom-liability/

Robin Hanson suggests, partly in response to calls for a pause in
development of AGI, liability
rules for risks
related to AGI rapidly becoming powerful.

My intuitive reaction was to classify foom liability as equivalent to a
near total ban on AGI.

Now that I've found time to think more carefully about it, I want to
advocate foom liability as a modest improvement over any likely pause or
ban on AGI research. In particular, I want the most ambitious AI labs
worldwide to be required to have insurance against something like $10
billion to $100 billion worth of damages.

Liability has the obvious advantage that whoever is financially
responsible for the liability (AI labs? insurance companies?) will have
better incentives to study the risks carefully than would politicians,
bureaucrats, or standards committees composed of industry
representatives.

A more subtle advantage: limited risk that it will be used to protect
incumbents or permanently stifle good technologies.

I see somewhat less risk of liability being hijacked by culture
warriors.

Liability wouldn't produce ideal incentives, due to scenarios where AGI
accidents cause more damage than any company can compensate for (e.g.
human extinction). I'll guess that liability would provide one tenth of
the optimal incentive.

But remember that there's not too much reason to worry about improving
the financial incentives to avoid extinction. I'm not too concerned
with the risk that DeepMind will train a system that they think has a 5%
chance of killing us all. I'm more concerned about a less responsible
lab training a system that their insurance company would think had a 5%
chance of starting a minor war, and which I think has a 5% chance of
killing us all. I.e. my main concern is with AI labs that are
overconfident about safety, but who are rational enough to respond to
projected penalties for medium-sized accidents.

For the risks that worry me most, there would need to be an
international agreement on such liability / insurance. That creates
difficulties beyond those that would confront a direct ban. Some
countries would allow fly-by-night insurance companies to provide
"insurance" that would not, in practice, cover much of the liability.

How Much Liability

Key variables are: who needs to buy liability insurance (or provide the
equivalent evidence of self-insurance), and how much harm does the
insurance policy need to cover...
I Think Eliezer Should Go on Glenn Beck
by Lao Mein — 2023-06-30T03:12:57.733Z

Glenn Beck is the only popular mainstream news host who takes AI safety seriously. I am being entirely serious. For those of you who don't know, Glenn Beck is one of the most trusted and well-known news sources by American conservatives.

Over the past month, he has produced two hour-long segments, one of which was an interview with AI ethicist Tristan Harris. At no point in any of this does he express incredulity at the ideas of AGI, ASI, takeover, extinction risk, or transhumanism. He says things that are far out of the normie Overton Window, with no attempt to equivocate or hedge his bets. "We're going to cure cancer, and we're to do it right before we kill all humans on planet Earth". He just says things like this, with complete sincerity. I don't think anyone else even comes close. I think he is taking these ideas seriously, which isn't something I expected to see from anyone in mainstream media.

According to Glenn, he has been trying to get interviews with people like Geoffery Hinton, but they have declined for political reasons. An obvious low-hanging fruit, if MIRI is willing to send him an email.

He seems on board with a unilateral AI pause from the US for national security reasons.

He's also personally cited statements from Eliezer to convey the dangers of ASI. I think an interview between the two could be a way for AI alignment to be taken more seriously by retirees who will vote and write their congressman. I dislike Eliezer on a personal level, but I think he is the only person who will actually go on the show and express how truly dire the situation is, with full sincerity.
AGI & War
by Calecute — 2023-06-29T22:20:58.453Z

Recently Garett Jones made a few tweets implying that the existence of entities with a very wide power differential today is evidence that super intelligence won't lead to Doom.
It is a reasonable argument. If the powerful today think it's better to trade with the powerless than to destroy them and take what they have, and trade is so beneficial in general, and war so costly, why would it be different for the even more powerful?

Unfortunately I think the position is based on an incomplete understanding of what AGI changes about the nature of trade and war.

Right now, due to lots of reasons, humans are the most valuable things. Most of the value of production goes to human factors, with a smaller value going to natural resources and physical capital. So dooming the humans is destroying exactly the thing you'd want to steal. Stealing humans themselves is extremely costly, because humans are pretty hard to control. As an alternative, you can access lots of benefits from other humans through trade, so the decision to not doom the powerless makes perfect sense.
Maybe the US could easily take on Mexico, and either occupy it indefinitely, or genocide the whole population, but the non human things in Mexico are not worth the trouble, and occupying it to extract value from the population would be extremely hard, and destroy a lot of the value the Mexicans produce, and the US would lose an important trade partner.

AGI changes this. After AGI, the value of human factors will fall to the subsistence level of the AGI. This means humans become cheap, and resources become relatively expensive. Humans will be deriving all their income from their property, and unlike humans, property is easy to steal.

The decision between going to war or trading changes completely. Now if you win the war, you can take all the most valuable things the losers had, the things you were trading for in the first place.
The US could annex Mexico, genocide their population, meat and silicon alike, and just make more copies of their own AGIs to take their place, and end up with all the extremely valuable (in the new economy) natural resources. Given that population in Mexico stays the same (enough for the marginal cost to run an AGI there to be the same as the marginal product of the AGI) they're not making trade with Mexico substantially lower, and can keep enjoying it's benefits.

Do note that even thou the...
Biosafety Regulations (BMBL) and their relevance for AI
by Štěpán Los — 2023-06-29T19:22:41.196Z

AI regulations could draw inspiration from the field of biosafety regulation, specifically the CDC's guidelines for

Biosafety in Microbiological & Biomedical Laboratories (BMBL)

, which outline the necessary precautions for working with dangerous biological agents and recommend a systematic approach for assessing their risks.

The remainder of this report will describe the structure and mission of BMBL, outline its key principles and recommendations and indicate relevant takeaways for the field of AI regulation.

Epistemic status: I am not an expert in biosafety. However, I think a summary document which highlights concrete safety steps undertaken in an adjacent field to AI and highlights some actionable steps for AI labs to increase safety could be potentially useful. All construcive feedback and suggestions for improvements are welcome!

Structure and Mission

BMBL is an advisory document protecting laboratory staff, the public and the environment from exposure to dangerous microorganisms and hazardous materials (e.g. radioactive agents). While many organizations and agencies use BMBL for regulations, it is primarily an advisory document to help with a comprehensive protocol which helps laboratories identify risks and ensure safe conduct when working with dangerous microorganisms and hazardous materials. It provides guidelines for protecting laboratory staff, the public and the environment.

Relevance for AI Regulation: A difference between biosafety and AI safety may be that biological laboratories have a more obvious incentive to protect its staff, as there is more immediate danger of contracting a disease than interacting with an AI system. Similar guidelines for AI may need to be legally binding.

BMBL is a set of biosafety guidelines compiled by experts and members of the public. To produce BMBL, the Office of Laboratory Science and Safety (OLSS) works with the National Health Institute (NIH) to recruit over 200 expert contributors from scientific societies, federal agencies (NIH, CDC, FBI, and many more), and the public.

Relevance for AI Regulation: AI regulators could use a similar process. For instance, a director of office within the National Telecommunications and Information Administration (NTIA) could assemble a team of experts to produce similar guidelines. Furthermore, input from businesses and the public should be included t

...
Nature Releases A Stupid Editorial On AI Risk
by Bentham's Bulldog — 2023-06-29T19:00:58.170Z

Crosspost of this on my blog.

Nature publishes an op ed in which a cocaine addict urges you not to focus on the problems doing cocaine would pose tomorrow when he has a desire to do cocaine today. Okay, not really, but they released something just as insipid and unconvincing. The title of the article is

Stop talking about tomorrow’s AI doomsday when AI poses risks today

. Now, it turns out that arguing that something poses threats in the here and now does not, in fact, mean that it won’t pose more threats in the future, and in fact makes it more likely that it will pose threats in the future if it’s already worrying and only gaining more capacities. But apparently, such a basic point of elementary logic eludes the writer of this op ed in the prestigious Nature—who thinks that pointing out that there are current risks from AI means that we should not worry about the risks it will pose in the future.

Imagine this feat of grand illogic being applied to any other domain. How absurd would one look who claimed that we shouldn’t focus on reinvigorating the non-proliferation treaty, which would make it harder for countries like Iran and Saudi Arabia to get nuclear weapons, leading to possible nuclear war, because there already is a nuclear threat. Those who espoused this view certainly would not be published in nature, and that line of reasoning would not be touted by nuclear “ethicists,” by which I mean the people who think the real concern of nuclear weapons is that they might cause some people to say racial slurs or utter politically incorrect truths or exacerbate inequality. And yet when it comes to AI risk, people’s brains seem to depart their skulls.

You know, I’ve been

quite specific

about why AI is a threat. I’ve laid out the case in

great detail

at various times. And I’m

far

from

the

only

one

. Eliezer Yudkowsky has

written

about googolplex words about the dangers of AI, at various times. When two of the most prestigious AI researchers in the world—Bengio and Hinton—as well as a sizeable portion of AI researchers think there’s a high probability it will end the world, when we’re creating something far smarter than us which we have no idea how to control, as even its critics admit, something that could deceive us easily, it’s not unreasonable to worry that it might pose a risk to the continued existence of the species. Certainly, one might expect that if o...
AI Safety without Alignment: How humans can WIN against AI
by vicchain — 2023-06-29T17:53:03.194Z

Here are some ideas on how we can restrict Super-human AI (SAI)

without built-in Alignment

:

Overall idea:

We need to use unbreakable first principles, especially laws of physics to restrict or destroy SAI. We can leverage the

asymmetries in physics

, such as the speed of light limit, time causality, entropy, uncertainty, etc.

Assumptions:

1. Super-Intelligence cannot break the laws of physics.
2. SAI is silicon-based.
3. Anything with lower-intelligence than the current Collective Human Intelligence (CHI) cannot destroy all humans.

Example Idea 1 (only require Assumption 1) "Escape":

We send human biological information (or any information that we want to protect from SAI) at/near the speed of light to the

Cosmic Event Horizon

. Then after some time, the information will be impossible to be reached by anything coming from Earth due to the accelerated expansion of the universe, protecting the information from our SAI.

Of course, we could never reach this information either but at least we know humans persisted and "live on" in the universe. Last resort.

Example Idea 2 (require all 3 assumptions) "Lock":

We figure out the lower-bound of computational speed (or precision) to reach our current CHI (not trivial).

Lock the global computational ability

at that lower-bound for silicon, using some sort of interference device (uncertainty principles). So the intelligence of AI will never surpass our current CHI. Ideally, this device can be implemented and keep running easily so that we don't need to use political tools such as a moratorium.

Pros of these ideas:

Black-box AI safety

. No need to depend that everyone has the intention or ability to build intrinsically safe AI.
Turning AI safety into trackable engineering problems.
We can iterate many times because restricting a weak AI (today) with first principles could potentially generalize to a SAI.
Don't rely on policymakers or AI companies (in theory. We still need resources to implement these ideas but it can come from other sources.)
Don't rely on AI to control AI
Fundamentally frame the problem as

a war of biological human vs. silicon AI

, so we can have more tools in our hands to be more "ruthless" against AI

Cons:

Technically very hard to implement
Might take longer time than SAI development

These are probably some of the worst ideas in this line of thinking so it would be wonderful if you all can give some

feedback on the framew

...
Challenge proposal: smallest possible self-hardening backdoor for RLHF
by Christopher King — 2023-06-29T16:56:59.832Z

A common scenario for how LLMs could lead to AI catastrophe is if some feature of the network becomes AGI and escapes (with whatever subgoal that specific feature was optimized for, instead of the goal that the network itself was trained on).

I notice those less familiar with ML tend to find this implausible. However:

In a backdoor attack, the attacker corrupts the training data so to induce an erroneous behaviour at test time. Test time errors, however, are activated only in the presence of a triggering event corresponding to a properly crafted input sample. In this way, the corrupted network continues to work as expected for regular inputs, and the malicious behaviour occurs only when the attacker decides to activate the backdoor hidden within the network. In the last few years, backdoor attacks have been the subject of an intense research activity focusing on both the development of new classes of attacks, and the proposal of possible countermeasures. - An Overview of Backdoor Attacks Against Deep Neural Networks and Possible Defences

Also see Data Poisoning Attacks against Autoregressive Models.

As LLM capabilities increase, they should eventually gain the ability to do this themselves (they probably already have the previous two papers in their training data).

In RLHF and related methods, there is a theoretical possibility for "self-hardening" backdoors. The LLM can backdoor the reward model, and the reward model can backdoor the LLM, since they are both neural networks.

To estimate the lower bounds for this risk, I propose the following challenge: what is the smallest possible amount of data poisoning necessary to create a self-hardening backdoor in RLHF.

(Note that for now, this is a very informal idea. Feel free to submit attempts in the comments that can be judged informally. If anyone wants to make a precisely specified version of this challenge that can be turned into a contest, feel free!)

The idea is that this allows us to estimate the risks for such backdoors naturally occuring at random. Mesaoptimizers in the wild!

Note that to count, the backdoor must not just survive, but become stronger with more RLHF.

Some other ideas for self-hardening challenges:

Self-hardening backdoor for RLAIF

Self-hardening backdoor when influencing humans (note that LLM watermarks on a full page of text can still be detected even after they are rewritten by (human) CS gr

...
AI #18: The Great Debate Debate
by Zvi — 2023-06-29T16:20:05.569Z

Is debate worthwhile? That is a matter of some, ahem, disagreement.

A lot of that was mainstream discussions of the whole Rogan-RFK Jr fiasco. In the context of AI, we had a real debate on whether AI is an existential threat that by all accounts didn’t go great, and a Nature article that didn’t even pretend to debate and which Tyler Cowen claimed was evidence that the debate had been utterly lost, with a plea to shift to formal scientific debate instead.

It would be so nice to discuss and explore and figure things out together, instead of debating. Debates, like Democracy, are highly frustrating, deeply flawed institutions that do a terrible job converging on truth, except what alterative do you have that isn’t even worse?

Some noteworthy new releases this week. MidJourney 5.2 is getting rave reviews. Generative AI for GMail, Google Docs and Google Sheets is online, I keep expecting to get use out of it but so far no luck, ideas welcome. The Chinese claim a new very strong language model, ChatGLM, on which I am withholding judgment given the track records involved.

Mostly I sense that we picked the lowest hanging fruit for mundane utility from generative AI, and now our knowledge of, skills at using and integration into our workflows and lives of existing tools is going to be where it is at for a while, modulo the full release of Microsoft Copilot and the fully functional Google suite of generative AI offerings. That doesn’t mean that we won’t see technical advances at a breakneck pace versus almost anywhere else, we totally will, but we’re now seeing things like a $1.3 billion investment in Inflection AI in the hopes of building something better over time. Which takes a while. I don’t doubt that some efforts will get there, but we are now less likely to be talking days or weeks, and more likely to be talking months or years.

Table of Contents

Table of Contents.

Language Models Offer Mundane Utility. AI system. Making copies.

Fun With Image Generation. A few trifles.

Deepfaketown and Botpocalypse Soon. Deepfake detector sighting.

They Took Our Jobs. They will never take our jobs, or freedom?

They Created Our Terrible Job. Work it while it lasts. Need input.

Introducing. GMail generative AI, ChatGLM, get your compute.

In Other AI News. Demis Hassabis says next big thing is coming.

Asterisk Magazine Issue on AI. Outstanding issue

...