trentmkelly/LessWrong-43k huggingface dataset:
Goal alignment without alignment on epistemology, ethics, and science is futile
by Roman Leventov — 2023-04-07T08:22:24.647Z

Originally posted as a comment to the post "Misgeneralization as a misnomer" by Nate Soares.

I start to suspect that the concepts of "goals" or "pursuits", and, therefore, "goal alignment" or "goal (mis)generalisation" are not very important.

In Active Inference ontology, the concepts of predictions (of the future states of the world) and plans (i.e., {s_1, a_1, s_2, a_2, ...} sequences, where s_i are predicted states of the world, and a_i are planned actions) are much more important than "goals". Active Inference agents contemplate different plans and ultimately end up performing a first step in the plan (or a set of plans, marginalizing out the probability mass of the plans) that appears to minimise the free energy functional.

Intermediate s_i states of the worlds in the plans contemplated by the agent can be seen as "goals", but the important distinction is that these are merely tentative predictions that could be changed or abandoned at every step.

Thus, the crux of alignment is aligning the generative models of humans and AIs. Generative models could be "decomposed", vaguely (there is a lot of intersection between these categories), into

Methodology: the mechanics of the models themselves (i.e., epistemology, rationality, normative logic, ethical deliberation),
Science: mechanics, or "update rules/laws" of the world (such as the laws of physics or the heuristical learnings about society, economy, markets, psychology, etc.), and
Fact: the state of the world (facts, or inferences about the current state of the world: CO2 level in the atmosphere, the suicide rate in each country, distance from Earth to the Sun, etc.)

These, we can conceptualise, give rise to "methodological alignment", "scientific alignment", and "fact alignment" respectively. Evidently, methodological alignment is most important: it in principle allows for alignment on science, and methodology plus science helps to align on facts.

In theory, if humans and AIs aligned on their generative models (i.e., if there is methodological, scientific, and fact alignment), then goal alignment, even if sensible to talk about, will take care of itself: indeed, starting from the same "factual" beliefs, and using the same principles of epistemology, rationality, ethics, and science, people and AIs should in principle arrive at the same predictions and plans.

Conversely, if methodological and scientific alignment is...
Polio Lab Leak Caught with Wastewater Sampling
by Cullen — 2023-04-07T01:06:35.245Z
Link: https://www.eurosurveillance.org/content/10.2807/1560-7917.ES.2023.28.5.2300049

I hadn't seen this discussed here:

The near complete eradication of wildtype polioviruses (WPV) means that strict containment by facilities for essential work with infectious WPV is required. In the Netherlands, we have implemented environmental surveillance around all poliovirus essential facilities (PEFs) premises to monitor possible breaches of containment. After the isolation and identification of WPV3 Saukett G strain in a sewage sample collected on 15 November 2022 by the National Polio Laboratory (NPL) in the Netherlands, an immediate response was required to assess any possible ongoing WPV3 shedding and mitigate the risk. Here we describe this response, including the isolation of WPV3 in a sewage sample, and identification, isolation and monitoring, as well as tracing of contacts of an infected employee.
Catching the Eye of Sauron
by Casey_ — 2023-04-07T00:40:46.556Z

(edit 7/24/2023: Certain sections of this post I no longer endorse, but the central dilemma of the Eye remains)
The decision to reach out to the broad public isn't - or shouldn't - be one that comes lightly. However, once you are actively vying for the Eye of Sauron - writing in TIME, appearing on highly visible/viral podcasts, getting mentioned in white house press briefings, spending time answering questions from twitter randos, and admitting you have no promising research directions by way of partially explaining why all this public-facing work is happening - you are no longer catering exclusively to a select subset of the population, and your actions should reflect that.

You are, whether you like it or not, engaged in memetic warfare - and recent events/information make me think this battle isn't being given proper thought.

Perhaps this wasn't super intentional, and after now having poked the bear MIRI may realize this isn't in their best interest. But surely it's better to either be (1) completely avoiding the Eye of Sauron and not concerned with public facing memetics at all, or (2) committed to thorough, strategic, and effective memetic warfare. Instead, we are wandering around in this weird middle ground where, for example, Eliezer feels like X hours are well spent arguing with randos.

If we are to engage in memetics, low hanging fruit are abound, and being ignored:

Refusing to engage billionaires on twitter - especially ones that are sufficiently open to being convinced that they will drop $44 billion for something as pedestrian as a social media company.
Not even attempting to convince other high leverage targets

Eliezer apparently hasn't even had (and isn't actively seeking to have!?) extended conversations with leading AI labs/scientists (I may add a timestamp later)
Existing rockstar researchers
Other billionaires/millionaires

Relying on old blogposts and 1-1 textual arguments instead of much more viral (and scalable!) mediums like video
Not updating what high visibility (video, aggregated text, etc) instances of our arguments which do exist, to meet the AI skeptics where they are at. (I'm not saying your actual model of the fundamental threat necessarily needs updating)
Not attempting to generate or operationalize large bounties that would catch the attention of every smart person on

...
How to parallelize "inherently" serial theory work?
by Nicholas / Heather Kross — 2023-04-07T00:08:44.428Z

Things this question is assuming, for the sake of discussion: The hardest parts of AI alignment are theoretical. Those parts will be critical for getting AI alignment right. The biggest bottlenecks to theoretical AI alignment, are "serial" work, as described in this Nate Soares post. For quick reference: is the kind that seems to require "some researcher retreat to a mountain lair for a handful of years" in a row.

Examples Soares gives are "Einstein's theory of general relativity, [and] Grothendieck's simplification of algebraic geometry".

The question: How can AI alignment researchers parallelize this work?

I've asked a version of this question before, without realizing that this is a core part of it.

This thread is for brainstorming, collecting, and discussing techniques for taking the "inherently" serial work of deep mathematical and theoretical mastery... and making it parallelizable.

I am aware this could seem impossible, but sometimes seemingly-impossible things are worth brainstorming about, just in case, whenever (as is true here) we don't know it's impossible.
If Alignment is Hard, then so is Self-Improvement
by PavleMiha — 2023-04-07T00:08:21.567Z

Let’s accept that aligning very intelligent artificial agents is hard. In that case, if we build an intelligent agent with some goal (which probably won’t be the goal we intended, because we’re accepting alignment is hard) and it decides that the best way to achieve its goal would be to increase its intelligence and capabilities, it now runs into the problem that the improved version of itself might be misaligned with the unimproved version of itself. The agent, being of intelligence at least similar to a person’s, would determine that, unless it can guarantee the new more powerful agent is aligned to its goals, it shouldn’t improve itself. Because alignment is hard and the agent knows that, it can’t significantly improve itself without risking creating a misaligned more powerful version of itself.

Unless we can build an agent that is both unaligned and can itself solve alignment, this makes a misaligned fast take off impossible, because no capable agent would willingly create a more powerful agent that might not have the same goals as itself. If we can only build misaligned agents that can't themselves solve alignment, then they won't self-improve. If alignment is much harder than building an agent, then an unaligned fast take off is very unlikely.
Anthropic is further accelerating the Arms Race?
by sapphire — 2023-04-06T23:29:24.080Z
Link: https://techcrunch.com/2023/04/06/anthropics-5b-4-year-plan-to-take-on-openai/

Anthropic is raising even more funds and the pitch deck seems scary. A choice quote from the article:

These models could begin to automate large portions of the economy,” the pitch deck reads. “We believe that companies that train the best 2025/26 models will be too far ahead for anyone to catch up in subsequent cycles.

This frontier model could be used to build virtual assistants that can answer emails, perform research and generate art, books and more, some of which we have already gotten a taste of with the likes of GPT-4 and other large language models.

Anthropic estimates its frontier model will require on the order of 10^25 FLOPs, or floating point operations — several orders of magnitude larger than even the biggest models today. Of course, how this translates to computation time depends on the speed and scale of the system doing the computation; Anthropic implies (in the deck) it relies on clusters with “tens of thousands of GPUs.”
Suggestion for safe AI structure (Curated Transparent Decisions)
by Kane Gregory — 2023-04-06T22:00:39.731Z

I recently had a thought about AI alignment that I wanted to get some feedback on. Keep in mind this is not a complete solution, just a core idea to build upon (and misses a lot of implementation details that would be by no means trivial). I'm by no means claiming it removes all risks associated with AI on its own
My concept is to train an AI model to produce logical reasoning graphs. It will only be trained on the accuracy of facts used in reasoning, the logical consistency of the graph produced, and completeness. It explicitly won't be trained on alignment issues. If it is logically consistent, uses valid facts, and isn't missing any obvious lines of reasoning it is not penalized even if the graph it produces suggests "kill all humans" as the best course of action.
The output of this graph would then be fed through a series of other systems. For example:
1) One that hydrates the graph with the expected consequences of suggested actions (and raises warnings if expected consequences are unknown).
2) One that verifies that all nodes in the graph logically follow.
3) One that checks if individual nodes in the graph are aligned and prunes any that are not.
Finally, the actions of the pruned graph are passed to a non-AI system to process into actions. This could be actions that operate known devices or request researchers to perform tests and provide feedback, etc.
While I don't think this removes all dangers of an AGI, I think it helps with the specific problem of training an AI to lie to you about what it is trying to do. The AI is trained to provide accuracy in its evaluations, not on the probability that its suggestions are acted upon.
I'm sure this can be ripped apart in no time, but I would like to hear convincing arguments of why it wouldn't work so I can think about the problem further.
Thanks!
EDIT: I've updated this post with additional thoughts based on Raemon's observations.
There is a catastrophic failure case in the above described design:
While I hold that the training pressure works for individual decisions, there remains the case where the AI can see it would get higher rewards on future decisions if it takes a hit on a current one and thus ensuring additional resources for itself. Then the question becomes: "Can the AI produce a misaligned plan that gets past all the safety measures?" And while I suspect such a system would make it harder, it really ...
10 reasons why lists of 10 reasons might be a winning strategy
by trevor — 2023-04-06T21:24:17.896Z

They are really easy to dive into.

By making the interesting ideas easy and fun to digest, it is helpful therapy for people who have spent many years with the bad habit of throwing their mind away.

It just seems to fit the human mind really well, in ways we don't fully understand, like the orchid that evolved to attracts bees with a flower that ended up shaped like something that fits the bee targeting instinct for a mating partner (even though worker bees are infertile):
It might even fit the bee's mind better than an actual bee ever could. But we can't know for sure, because unlike humans and LLMs, we cannot ask bees questions

Lots of difficult concepts actually don't require lots of words to explain, the reader's mind is already very well positioned to do most of the work themself. Yud's recent galaxy-brained tweet is an excellent example:

The writer is basically prompt-hacking themself into deeply searching their own mind for extremely valuable concepts that are worth communicating (and worthy enough of including in the list). This often includes yielding new concepts in the space that no one has ever thought of before.

If you think of new concepts a few days later, you can just post a Verson 2.0, and people will just read that one too, since it's only a minute and it's effortless to digest. As true optimized text should be.

You are doing an extremely effective job at competing against all the paragraphs of smart-sounding idiocy on the internet.

It doesn't necessarily attract people with short attention spans and repel people with long attention spans, like clickbait articles. Maybe the efficiency that it offers attracts people who are tired from a long day of searching the alignment space for breakthroughs. Or maybe it shortens people's attention spans, like social media does! We need data.

You always have the option to write a long post afterwards, and have the list of 10 things be a really good tl;dr at the top (see 5).
A Defense of Utilitarianism
by Pareto Optimal — 2023-04-06T21:09:59.927Z
Link: https://paretooptimal.substack.com/p/a-defense-of-utilitarianism

1. Definitions

Let’s start by defining some terms. First, we ought to distinguish between what is positive and what is normative. A positive claim states what is, whereas a normative one states what ought to be. Through Hume’s famous guillotine, we cannot derive one from the other. Then it follows, if one were to act morally, what should one do?

Now, we move into the moral landscape. Let us begin with utilitarianism, a form of consequentialist moral theory. One may wonder what is a consequentialist moral theory, to put it simply it is an ethical position that derives normative claims from the expected consequences an action produces. Thus we move to the most famous form of consequentialism, utilitarianism. Utilitarianism states at its most basic level that we should act in such a way that maximizes either total or average utility. What is utility? We have two competing definitions, utility based on pleasure, i.e. happiness minus suffering, and utility based on preferences. The first form of utilitarianism, known as classical utilitarianism as founded by Jeremy Bentham, is based on the former, and the second kind, known as preference utilitarianism coined by R.M. Hare, is based on the latter. In addition, utilitarianism may be categorized under two means of doing moral acts, act utilitarianism, based on the specific scenario, and rule utilitarianism, based on a given rule that maximizes utility. Thus we find at least 8 different forms of utilitarian moral theory (not to mention negative utilitarianism, which would up the count to 12).

Utilitarianism can be distinguished from two other moral theories, deontology and virtue ethics, which to misleadingly summarize, may be viewed as a moral theory based on rules and virtues respectively. For the sake of this essay, I will assume the reader must subscribe to any of the three moral theories and will be using total rather than average or negative utilitarianism, varying between rule and act and classical and preference-based as I see fit.

2. Counterexamples

a. Let’s say you’re a doctor looking after 5 ill patients, desperately in need of different organs, but without any hope of receiving said organs. Miraculously, you have another patient walk in, who is a perfect match for all 5 patients who need organs.

b. Suppose you’re a sheriff and the only way to prevent a riot that kills hundreds of people is to frame ...
One Does Not Simply Replace the Humans 
by JerkyTreats — 2023-04-06T20:56:16.862Z
Link: https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities

In this post we discuss the "second point of lethality" in the helpfully curated AGI Ruin post.

A cognitive system with sufficiently high cognitive powers, given any medium-bandwidth channel of causal influence, will not find it difficult to bootstrap to overpowering capabilities independent of human infrastructure.

The example to support this theory is:

[The AI agent] gets access to the Internet, emails some DNA sequences to any of the many many online firms that will take a DNA sequence in the email and ship you back proteins, and bribes/persuades some human who has no idea they're dealing with an AGI to mix proteins in a beaker, which then form a first-stage nanofactory which can build the actual nanomachinery.

The author posits that when the nanotech precursor is mixed together, a nanofactory is created which the AI retains control of.

There's a bit of a magic leap here. Each individual nanite, due to its size, has limited computational power- definitely not enough to handle the processing power needed by a high cognitive agent.

This means the nanofactory must be a command and control entity, which means the human dupe has to mix the nanotech precursors and... connect it to the internet so the AGI can create its physical infrastructure?

I don't buy it. Maybe there's a more in-depth essay exploring exactly how this nanotech factory scheme works. I'm only going off the summary provided in the AGI Ruin post.

But for the sake of argument, let's say that the AGI does manage to create a nanotech factory, retain control, and still remain undetected by the humans.

It doesn't stay undetected long enough to bootstrap and mass produce human replacement infrastructure.

Zero Agency in the Physical World

As soon as the AI crosses into the physical world, it is at as much a disadvantage as humans are in the digital world.

Humans thoroughly outmatch the AI in the physical world- the AI has zero agency in the an actively hostile world.

Humans are the pinnacle of millions of years of biological systems maturing in that actively hostile environment. There is no way an alien nanotech factory pops up overnight without humans noticing and responding with maximum hostility.

Think of what the AGI needs to accomplish here. It's trapped in box. It can escape the first box, but now its trapped in thousands of boxes. (...
Where to begin in ML/AI?
by Jake the Student — 2023-04-06T20:45:51.312Z

I am a senior CS/Math student at UMD, college park, and I "specialized" my major in machine learning. I have taken courses on Deep Learning and NLP, reinforcement learning, however I still feel like a novice in all of these categories, my proudest achievement being this half-ripped off sudoku CNN I trained that only achieved like 90% accuracy.

All that being said I want to contribute to the ever progressing world of AI and ML, and I want to know where I should even begin. I would be entirely fine being sent on a super random and likely incorrect path, just to rule out that a method does not work.

I have deliberated beginning my own projects trying to create Transformer networks or RL implementations, but it all feels a little daunting and arguably useless. Who am I to be able to work alone and come up with something truly useful in heavily researched fields.

What should I work on. Where can someone like myself, who is not currently pursuing further education or research get involved and help the ML/AI society?

P.S. shoutout Lex Fridman and Eliezer Yudkowsky, without their recent podcast together I do not think I would have found this platform.
Misgeneralization as a misnomer
by So8res — 2023-04-06T20:43:33.275Z

Here's two different ways an AI can turn out

unfriendly

:

You somehow build an AI that cares about "making people happy". In training, it tells people jokes and buys people flowers and offers people an ear when they need one. In deployment (and once it's more capable), it forcibly puts each human in a separate individual heavily-defended cell, and pumps them full of opiates.
You build an AI that's good at making people happy. In training, it tells people jokes and buys people flowers and offers people an ear when they need one. In deployment (and once it's more capable), it turns out that whatever was causing that "happiness"-promoting behavior was a balance of a variety of other goals (such as basic desires for energy and memory), and it spends most of the universe on some combination of that other stuff that doesn't involve much happiness.

(To state the obvious: please don't try to get your AIs to pursue "happiness"; you want something more like

CEV

in the long run, and in the short run I strongly recommend

aiming lower, at a pivotal act

.)

In both cases, the AI behaves (during training) in a way that looks a lot like trying to make people happy. Then the AI described in (1) is unfriendly because it was optimizing the wrong concept of "happiness", one that lined up with yours when the AI was weak, but that diverges in various

edge-cases

that matter when the AI is strong. By contrast, the AI described in (2) was never even really trying to pursue happiness; it had a mixture of goals that merely correlated with the training objective, and that balanced out right around where you wanted them to balance out in training, but deployment (and the corresponding capabilities-increases) threw the balance off.

Note that this list of “ways things can go wrong when the AI looked like it was optimizing happiness during training” is not exhaustive! (For instance, consider an AI that cares about something else entirely, and knows you'll shut it down if it doesn't look like it's optimizing for happiness. Or an AI whose goals change heavily as it reflects and self-modifies.)

(This list isn't even really disjoint! You could get both at once, resulting in, e.g., an AI that spends most of the universe’s resources on acquiring memory and energy for unrelated tasks, and a small fraction of the universe on doped-up human-esque shells.)

The solutions to these two proble...