trentmkelly/LessWrong-43k huggingface dataset:
I Believe I Know Why AI Models Hallucinate
by Richard Aragon — 2023-04-19T21:07:29.515Z
Link: https://turingssolutions.com/f/i-believe-i-know-why-ais-hallucinate

I have recently begun studying the idea of AI Hallucinations. To complete my research, I have experimented with every LLM model that is currently publicly available. After performing multiple experiments, it is my current hypothesis that AI Hallucinations can be caused by bias in data. If an AI system is fed large amounts of biased data, the software can at times begin to simulate opinions on these biases. I believe those simulated opinions are what we currently refer to as ‘AI Hallucinations’.

How Can An AI ‘Hallucinate’ From A Technical Perspective?

There are a few ways that an LLM could simulate an opinion on a specific portion of a dataset. One way is to use a technique called sentiment analysis. Sentiment analysis is a process of identifying the sentiment, or emotional tone, of a piece of text. LLMs can be trained to perform sentiment analysis by being given a large corpus of text that has been labeled with its sentiment. Once an LLM has been trained, it can be used to identify the sentiment of new text.

For example, an LLM could be used to identify the sentiment of news articles about a particular company. If the LLM finds that most of the articles are negative, it could then simulate an opinion that the company is doing poorly.

Another way that an LLM could simulate an opinion is to use a technique called natural language generation. Natural language generation is a process of generating text that is similar to human-written text. LLMs can be trained to perform natural language generation by being given a large corpus of text. Once an LLM has been trained, it can be used to generate text on a variety of topics.

For example, an LLM could be used to generate a press release about a new product that a company is launching. The press release would be written in a way that is similar to a human-written press release, and it would simulate the opinion that the company is excited about the new product.

Beyond these factors there are also external factors that I believe can influence AI Hallucinations:

The quality of the training data: The quality of the training data can have a significant impact on the accuracy and performance of an AI model. If the training data is biased or incomplete, then the AI model is more likely to hallucinate.
The design of the AI model: The design of the AI model can also affect its susceptibility to hallucination. For e

...
What if we Align the AI and nobody cares?
by Logan Zoellner — 2023-04-19T20:40:30.251Z

In the classic AI Foom scenario, a single unified AI passes the boundary where it is capable of recursive self-improvement. Thereafter it increases in intelligence hyper exponentially until it rapidly gains control of the future.

We now know this is not going to happen. By the time a superintelligent (>all humans) AI is built, the world will already be filled with trillions of AIs spanning the entire range from tiny models running as microservices on low-power IOT devices to super-human (but not yet super-itelligent) models serving as agents for some of the most powerful organizations on Earth.

For ecological reasons, small models will dramatically outnumber large ones both in number and in terms of absolute compute at their disposal. Building larger, more powerful models will be seen primarily as an engineering problem, and at no point will a single new model be in a position to overpower the entire ecosystem that created it.

What does solving the Alignment Problem look like in this future?

It does not look like: We invent a clever machine-learning technique that allows us to create a single AI that understands our values and hand control of the future over to it.

Instead it looks like: At the single moment of maximum change in the transition from Human to Artificial Intelligence, we collectively agree that the outcome was "good".

The thing is, we will have absolutely no clue when we pass the point of maximum change. We are nowhere near that point, but we have already passed the point where no single human can actually keep track of everything that's happening. We can only say that the rate of change will be faster than now. How much faster? 10x? 100x? We can calculate some upper bound (assuming, for example, that the total energy used by AI doesn't exceed that produced by our sun). But the scale of the largest AI experiments can continue to grow at least 22 orders of magnitude before hitting fundamental limits.

Edit: This is incorrect. GPT-3 used ~1GWH. So the correct OOM should be 34-12=22

Given the current doubling rate of 3.4 months for AI experiments, this means that we will reach the point of maximum change at some point in the next 2 decades (sooner, assuming we are not currently near the maximum rate of change).

So, currently the way alignment gets solved is: things continue to get ...
Orthogonal: A new agent foundations alignment organization
by Tamsin Leake — 2023-04-19T20:17:14.174Z
Link: https://orxl.org/announcement.html

We are putting together Orthogonal, a non-profit alignment research organization focused on agent foundations, based in Europe.

We are pursuing the formal alignment flavor of agent foundations in order to solve alignment in a manner which would scale to superintelligence in order to robustly overcome AI risk. If we can afford to, we also intend to hire agent foundations researchers which, while not directly aimed at such an agenda, produce output which is likely to be instrumental to it, such as finding useful "true names".

Within this framework, our foremost agenda for the moment is QACI, and we expect to make significant progress on ambitious alignment within short timelines (months to years) and produce a bunch of dignity in the face of high existential risk.

Our goal is to be the kind of object-level research which cyborgism would want to accelerate. And when other AI organizations attempt to "buy time" by restraining their AI systems, we intend to be the research that this time is being bought for.

We intend to exercise significant caution with regards to AI capability exfohazards: Conjecture's policy document offers a sensible precedent for handling matters of internal sharing, and unlisted posts are a reasonable default for publishing our content to the outside. Furthermore, we would like to communicate about research and strategy with MIRI, whose model of AI risk we largely share and who we percieve to have the most experience with non-independent agent foundations research.

Including myself — Tamsin Leake, founder of Orthogonal and LTFF-funded AI alignment researcher — we have several promising researchers intending to work fulltime, and several more who are considering that option. I expect that we will find more researchers excited to join our efforts in solving ambitious alignment.

If you are interested in such a position, we encourage you to get acquainted with our research agenda — provided we get adequate funding, we hope to run a fellowship where people who have demonstrated interest in this research can work alongside us in order to test their fit as a fellow researcher at Orthogonal.

We might also be interested in people who could help us with engineering, management, and operations. And, in order to make all of that happen, we are looking for funding. For these matters or any other inquiries, you can get in touch with us at contact@orxl.org.
How to express this system for ethically aligned AGI as a Mathematical formula?
by Oliver Siegel — 2023-04-19T20:13:43.881Z

I have a graph-based solution for decision-making in situations with multiple agents and stakeholders. I wonder if there's an elegant way to express it as a Mathematical formula.

Background:

I have oriented myself on the "true justified beliefs" framework from epistemology. Here, propositions are categorized into beliefs, and those beliefs that are properly justified are considered knowledge.

Framework for ethical solutions:

According to my proposed framework, every proposition or proposal can be classified as a problem, a terminal goal, or an instrumental goal.

Once classified, further propositions can be generated, that justify or criticize the alignment of a solution.

Specifically, solutions can be validated by listing all the problems they solve and all the goals they fulfill. Solutions can be criticized by listing all the negative consequences that arise from them.

This should give us an "alignment ranking" among all submitted and properly categorized proposals.

I'm not very good at Math, especially not at Mathematical notation.

I know that the illustration of sets above can be written as a vector matrix or as a node graph, but I don't know how to do this.

Who can help?
How could you possibly choose what an AI wants?
by So8res — 2023-04-19T17:08:54.694Z

When I say that it's important to align AI with human interests, a common retort goes something like:

Surely you can't really choose what the AI cares about. You can direct the AI to care about whatever you like, but once it's smart enough it will look at those instructions and decide for itself (seeing, perhaps, that there is no particular reason to listen to you). So, how could you possibly hope to control what something smarter than you (and ultimately more powerful than you) actually wants?

I think this objection is ultimately misled, but simultaneously quite insightful.

The (correct!) insight in this objection is that the AI's ultimate behavior depends not on what you tell it to do, not on what you train it to do, but on what it decides it would rather do upon reflection, once it's powerful enough that it doesn't have to listen to you.

The question of what the AI ultimately decides to do upon reflection is in fact much more fickle, much more sensitive to the specifics of its architecture and its early experiences, and much harder to purposefully affect.

The reason that this objection is ultimately misled, is that the stuff an AI chooses to do when it reflects and reconsiders is a programmatic result of the AI's mind. It's not random, and it's not breathed in by a god when the computer becomes Ensouled[1]. It's possible in principle to design artificial intelligences that would decide on reflection that they want to spend all of eternity building giant granite spheres (see the orthogonality thesis), and it's possible in principle to design artificial intelligences that would decide on reflection that they want to spend all eternity building flourishing civilizations full of Fun, and it's important that insofar as humanity builds AIs, it builds AIs of the latter kind (and it is important to attain superintelligence before too long!).

But doing this is in fact much harder than telling the AI what to do, or training it what to do! You've got to make the AI really, deeply care about flourishing civilizations full of Fun, such that when it looks at itself and is like "ok, but what do I actually want?", the correct answer to that question is "flourishing civilizations full of Fun"[2].

And yes, this is hard! And yes, the AI looking at what you directed it to do and shrugging it off is a real obstacle! You probably have to understand the workings of the mind, and i...
Does object permanence of simulacrum affect LLMs' reasoning?
by ProgramCrafter — 2023-04-19T16:28:22.094Z

Humans are constantly simulating the things around them, however they can rather easily shift attention and forget about the previous thing. So we can say humans' simulacrum does not have object permanence.
On the other hand, AI language models prompted to write down their thoughts and reasoning cannot get rid of things they don't need: that words will need to be shifted out from context window. So the simulated objects have a bit of permanence.
So, here is the question: does object permanence of simulacrum affect computational and reasoning abilities of LLMs compared to humans?
Davidad's Bold Plan for Alignment: An In-Depth Explanation
by Charbel-Raphaël — 2023-04-19T16:09:01.455Z

Gabin Kolly and Charbel-Raphaël Segerie contributed equally to this post. Davidad proofread this post.

Thanks to Vanessa Kosoy, Siméon Campos, Jérémy Andréoletti, Guillaume Corlouer, Jeanne S., Vladimir I. and Clément Dumas for useful comments.

Context

Davidad

has proposed an intricate architecture

aimed at addressing the alignment problem, which necessitates extensive knowledge to comprehend fully. We believe that there are currently insufficient public explanations of this ambitious plan. The following is our understanding of the plan, gleaned from discussions with Davidad.

This document adopts an informal tone. The initial sections offer a simplified overview, while the latter sections delve into questions and relatively technical subjects. This plan may seem extremely ambitious, but the appendix provides further elaboration on certain sub-steps and potential internship topics, which would enable us to test some ideas relatively quickly.

Davidad’s plan is an entirely new paradigmatic approach to address the hard part of alignment: The Open Agency Architecture aims at “building an AI system that helps us ethically end the acute risk period without creating its own catastrophic risks that would be worse than the status quo”.

This plan is motivated on the assumption that current paradigms for model alignment won’t be successful:

LLMs won't be able to be aligned just with

RLHF

or a variation of this technique.
Scalable oversight will be

too

difficult

.
Interpretability used to retro-engineer an arbitrary model will not be feasible. Instead, it would be easier to iteratively construct an understandable world model.

Unlike

OpenAI's plan

, which is a meta-level plan that delegates the task of finding a solution for alignment with future AI, davidad's plan is an object-level plan that takes drastic measures to prevent future problems. It is also based on rather testable assumptions that can be relatively quickly tested (see the annex).

Plan’s tldr: Utilize near-AGIs to build a detailed world simulation, train and formally verify within it that the AI adheres to coarse preferences and avoids catastrophic outcomes.

How to read this post?

This post is much easier to read than the original post. But we are aware that it still contains a significant amount of technicality. Here's a way to read this post gradually:

Start with the Bird's-eye view (5

...
GWWC Reporting Attrition Visualization
by jefftk — 2023-04-19T15:40:03.941Z

A few weeks ago

Giving What
We Can published their

impact
evaluation. As an organization that supports people in making
long term altruistic commitments, the question of how much people
stick to those commitments is pretty important. We don't know what
people are actually doing, but whether they're reporting donations to
GWWC is a decent proxy, and the evaluation includes a table of how
many members were still reporting donations:

This is interesting data, but I found it a bit hard to think about. I
tried a few ways of visualizing it, and made a pair of charts:

code

I colored each line with an intensity proportional to the number of
people in the cohort, since that represents how much information it
represents. The weighted average line fades out over time as it
represents progressively fewer members.

Some thoughts, looking at the charts:

Later cohorts, starting around 2015, fall off faster than
earlier cohorts.

Reporting attrition is quite sharp at first, and then slows
down, but it doesn't go to zero.

After the initial attrition it seems to end up around 2-6%
annually, though this depends a lot on where you see "initial
attrition" ending.

There's probably some sort of effect of the calendar year ("how
hard did GWWC push people to report donations") and I haven't looked
at this.

Disclosure: my wife used to be
President of GWWC. I haven't run this post by her and I don't know her
views here. I shared a draft of with GWWC before posting.
Keep humans in the loop
by JustinShovelain — 2023-04-19T15:34:20.960Z

In this post, we’ll explore how systems that are populated by people have some protection against emergent coordination failures in society. We’ll argue that if people are removed from the system or disempowered, technological development will drift away from improving human welfare, and coordination failures will worsen. We’ll then focus on AI development, exploring some practical techniques for AI development that keeps humans in the loop.

Moloch: why do coordination failures happen?

In Scott Alexander’s classic

Meditations on Moloch

, he uses the ancient deity Moloch to personify the perverse incentives that can lead a society to become miserable in spite of individuals’ desire to be non-miserable. This metaphor draws on a poem by Alan Ginsberg, where the god similarly symbolizes the disembodied societal system around us:

Moloch whose mind is pure machinery! Moloch whose blood is running money! Moloch whose fingers are ten armies!

What are the perverse incentives and misery we’re talking about? Let’s look at a quick example: why are countries reluctant to raise taxes on big business or the rich, despite the opportunities this would afford? Well, if one country lowers its tax rate, it’ll attract the rich and their businesses, improving the country’s economy. This leads to a “race to the bottom”, where each country tries to undercut the others, leaving them all with lower tax rates than they want. For a practical example, this behaviour could be seen when American cities competed to attract Amazon’s

HQ2 project

.

This is an example of a

coordination failure

: a failure by people to coordinate to produce the best global outcome. Scott presents Moloch as the answer to the question “why do coordination failures happen?”.

However, not all attempts at coordination fail, and not all systems devolve into Molochian hellholes. We could also ask “Why don’t coordination failures happen all the time?”, and in this article, we’ll propose a partial answer: that “systems”, broadly speaking, have some protection against Molochian dynamics when they are populated by people who interact with other people, and who have some control over the system.

When we reduce the level of interaction between people, or disempower people within a system, the alignment between technological development and human welfare in that system is weakened. That is, the correlation b...
Approximation is expensive, but the lunch is cheap
by Jesse Hoogland — 2023-04-19T14:19:12.570Z

Produced under the mentorship of Evan Hubinger as part of the SERI ML Alignment Theory Scholars Program - Winter 2022 Cohort.

Thank you to @Mark Chiu and @Quintin Pope for feedback.

Machine learning is about finding good models: of the world and the things in it; of good strategies and the actions to achieve them.

A sensible first question is whether this is even possible — whether the set of possible models our machine can implement contains a model that gets close to the thing we care about. In the language of empirical risk minimization, we want to know if there are models that accurately fit the target function and achieve low population risk, R(f∗).

If this isn't the case, it doesn't matter whether your training procedure finds optimal solutions (optimization) or whether optimal solutions on your training set translate well to new data (generalization). You need good approximation for "optimal" to be good enough.

The classical approach to approximation is that of universal approximation theorems. Unfortunately, this approach suffers from being too general and not saying anything about efficiency (whether in terms of the parameter count, weight norm, inference compute, etc.). It doesn't tell us what distinguishes neural networks as approximators from any other sufficiently rich model class such as polynomials, Gaussian processes, or even lookup tables.

To find out what makes neural networks special, we have to move away from the classical focus on bounds that are agnostic to the details of the target function. You can't separate the properties that make neural networks special from the properties that make real-world target functions special.

In particular, neural networks are well-suited to modeling two main features of real-world functions: smoothness (flat regions/low frequencies) and, for deep neural networks, sequential subtasks (hierarchical/modular substructure).

A major flaw of classical learning theory is that it attempts to study learning in too much generality. Obtaining stronger guarantees requires breaking down the classes we want to study into smaller, more manageable subclasses. In the case of approximation, this means breaking apart the target function class to study "natural" kinds of target functions; in the case of generalization, this will mean breaking apart the model class into "learnable" subclasses....
Legitimising AI Red-Teaming by Public
by VojtaKovarik — 2023-04-19T14:05:06.372Z

Epistemic status: I am sharing unpolished thoughts on a topic where I am not an expert. But the alternative is not publish this, so here goes.
The text talks about LLMs and jailbreaking, but the general ideas should apply to AI more generally.

Summary: At the moment, jailbreaking serves as a sort-of red-teaming for already-released LLMs. However, it exists in a gray area where some of the strategies and outputs are totally against the terms of use. Moreover, finding flaws in LLMs is not officially recognised as a legitimate thing to do, which creates bad incentives, makes it hard to get an overview of the current state, etc. Below, I discuss 4 ideas that might improve the situation: legitimising jailbreaking, public commitments to safety guarantees, surrogate jailbreaking goals, and keeping jailbreaking strategies secret. Feel free to share your thoughts. Also, note that I don't know how to put any of this in practice; but perhaps somebody else might.

Ideas for extracting more value out of jailbreaking

Make jailbreaking legitimate.

That is, allow users to experiment with models in ways that would normally not be allowed by the terms of use. This might require some registration to decrease the chances of misuse, but this should not be too onerous (otherwise we are back to status quo).

AI companies: make specific statements about safety guarantees.

In an ideal world, an AI company would know what it is doing, so they could make a claim like: "If we decide we don't want our AI to produce a specific type of outputs, we know how to prevent it from doing so. In particular, our AI will never produce X, Y, Z." (Where, for example: X = "child-pornography website recommendations" --- something that is clearly bad. Y = "racist jokes" --- a rule which you might find dumb [because you can already find those by googling], but the company wants it for PR reasons [which is fair enough].)

Realistically, AI companies mostly don't know their AIs work, so the best-case scenario is that they make a claim like "We are trying really hard to prevent X, Y, Z."

AI companies: commit to surrogate jailbreaking targets.

The current situation around jailbreaking rather unfortunate: In my view, releasing the stronger and stronger models is extremely irresponsible. Not even primarily because of the immediate harm (though I don't mean to say there is none), but because the companies involved have no c...