trentmkelly/LessWrong-43k huggingface dataset:
New paper on aligning AI with human values
by ryan.lowe — 2024-03-30T23:39:20.288Z

We’ve just published a paper on a new way to align language models with human values. We wanted to post it here to get more feedback from folk who have thought deeply about alignment.

I'm pretty excited about it. In the past, I worked on RLHF, InstructGPT, and GPT-4 alignment (though one could make the claim that this isn't "real alignment research"). In the last year, I've found myself gravitating more towards the question of "what do we align to?". It turns out that this is extremely hard, but I think the set of ideas in this paper are some of the best I've come across.

I also think "what do we align to?" is directly relevant to longer-term alignment research / X-risk. We have a section about this in the paper, and I'd love to hear from people who disagree.

You can find the paper here: https://meaningalignment.org/values-and-alignment-paper. Below I've pasted the abstract, in addition to the section of our discussion where we we relate the paper more explicitly to traditional alignment research.

Fire away!

Abstract

There is an emerging consensus that we need to align AI systems with human values (Gabriel, 2020; Ji et al., 2024), but there is very little work on what that means and how we actually do it. We split the problem of “aligning to human values” into three parts: first, eliciting values from people; second, reconciling those values into an alignment target for training ML models; and third, actually training the model. In this paper, we focus on the first two parts, and ask the question: what are “good” ways to synthesize diverse human inputs about values into a target for aligning language models? To answer this question, we first define a set of 6 criteria that we believe must be satisfied for an alignment target to shape model behavior in accordance with human values. We then propose a process for eliciting and reconciling values called Moral Graph Elicitation (MGE), which uses a large language model to interview participants about their values in particular contexts; our approach is inspired by the philosophy of values advanced by Taylor (1977), Chang (2004a), and others. We trial MGE with a representative sample of 500 Americans, on 3 intentionally divisive prompts (e.g. advice about abortion). Our results demonstrate that MGE is promising for improving model alignment across all 6 criteria. For example, almost all participants (89.1%) f...
Cryptocurrency taxation in the UK
by philh — 2024-03-30T13:10:07.424Z

Mostly out of curiosity, I've been looking into how cryptocurrency is taxed in the UK. It's not easy to get what I consider to be a full answer, but here's my current understanding, as far as I felt like looking into it. HMRC's internal cryptoassets manual is available but I didn't feel like reading it all, and some of it seems out of date (e.g. page CRYPTO22110 seems to have been written while Ethereum was in the process of transitioning from proof-of-work to proof-of-stake). I also have no particular reason to trust or distrust the non-government sources I use here. I am not any form of accountant and it would be surprising if I don't get anything wrong.

My impression is HMRC tends to be pretty tolerant of people making good faith mistakes? In that if they audit you and you underpaid, they'll make you pay what you owe but you won't get in any other trouble. Maybe they'd consider "I followed the advice of some blogger who explicitly said he wasn't an accountant" to be a good faith mistake? I dunno, but if you follow my advice and get audited, I'd love to hear what the outcome is.

After I published, reddit user ec265 pointed me at another article that seems more thorough than this one. I wouldn't have bothered writing this if I'd found that sooner. I didn't spot anywhere where it disagrees with me, which is good.

Capital gains tax

Very loosely speaking, capital gains is when you buy something, wait a bit, and then sell it for a different price than you bought it for. You have an allowance which in 2023-24 is £6,000, so you only pay on any gains you have above that. The rate is 10% or 20% depending on your income.

But with crypto, you might buy on multiple occasions, then sell only some of what you bought. Which specific coins did you sell? There's no fact of the matter.1 But the law has an opinion.

Crypto works like stocks here. For stocks HMRC explains how it works in a document titled HS283 Shares and Capital Gains Tax (2023), and there's also manual page CRYPTO22200 which agrees.

The rule is that when you sell coins in a particular currency, you sell them in the following order:

Any coins you bought that day;

Any coins you bought in the following 30 days;

Any coins you bought previously, averaged together as if you'd bought them all for the same price.

The "30 following days" thing is called the "bed and breakfasting" rule, and the point is to avoi...
Is there a "critical threshold" for LLM scaling laws?
by Logan Zoellner — 2024-03-30T12:23:11.938Z

Lots of important phenomena have a critical threshold. In nuclear weapons, a certain number of neutrons are produced by each fission event and some of those trigger more events. If the number of events triggered is slightly more, the result grows exponential. If slightly less, much less happens.

Similarly in Quantum Computing. Current computers struggle with quantum noise, which causes the superposition to break down over time. However, if we can keep the error rate low enough, it should be possible to use error-correcting codes to do arbitrarily complicated calculations.

When trying to extend LLMs to difficult multi-step problems, I often feel like I'm dealing with a similar phenomena. For example, if asking an LLM to write a novel, it will follow the plot of of the novel for a while and then spontaneously jump to a different story. It feels like the "amount of information" passed from one state to enough is not-quite-enough to keep the story going indefinitely. LLM Agents struggle with similar problems where they seem to work for a while, but after a while they get stuck in a loop or lose their train of thought.

It seems like there are two ways this behavior could change as we scale up LLMs:

LLMs get gradually better as we increase their capabilities (they go from being able write 1 page to writing 2 to writing 3...)
There is some "critical size" threshold above which agents are able to self-improve without limit and suddenly we go from writing pages to writing entire encyclopedias.

Does anyone know of good evidence for/against either of these cases? (the strongest evidence in favor of 1 seems to be "that's how it's gone so far")
Metascience of the Vesuvius Challenge
by Maxwell Tabarrok — 2024-03-30T12:02:38.978Z
Link: https://www.maximum-progress.com/p/metascience-of-the-vesuvius-challenge

The

Vesuvius Challenge

is a million+ dollar contest to read 2,000 year old text from charcoal-papyri using particle accelerators and machine learning. The scrolls come from the ancient villa town of Herculaneum, nearby Pompeii, which was similarly buried and preserved by the eruption of Mt. Vesuvius. The prize fund comes from tech entrepreneurs and investors

Nat Friedman

,

Daniel Gross

, and several other donors.

From this

In the 9 months after the prize was announced, thousands of researchers and students worked on the problem, decades-long technical challenges were solved, and the amount of recovered text increased from one or two splotchy characters to 15 columns of clear text with more than 2000 characters.

To this

The success of the Vesuvius Challenge validates the motivating insight of metascience: It’s not about how much we spend, it’s about how we spend it.

Most debate over science funding concerns a topline dollar amount. Should we

double the budget of the NIH

? Do we spend too much on

Alzheimer’s

and too little on

mRNA

? Are we winning the

R&D spending race with China

? All of these questions implicitly assume a constant exchange rate between spending on science and scientific progress.

The Vesuvius Challenge is an illustration of exactly the opposite. The prize pool for this challenge was a little more than a million dollars. Nat Friedman and friends probably spent more on top of that hiring organizers, building the website etc. But still this is pretty small in the context academic grants. A million dollars donated to the NSF or NIH would have been forgotten if it was noticed at all. Even a direct grant to

Brent Seales

, the computer science professor whose research laid the ground work for reading the scrolls, probably wouldn’t have induced a tenth as much progress as the prize pool did, at least not within 9 months.

Brent Seales

: “The Vesuvius Challenge allowed us to enlist 1,000 research teams to work on a problem that normally we’d only have 5 people working on.”

It would have been easy to spend ten times as much on this problem and get ten times less progress out the other end. The money invested in this research was of course necessary but the spending was not sufficient, it needed to be paired with the right mechanism to work.

The success of the challenge hinged on design choices at a level of detail beyond just a grants vs prizes dichotomy. Collabor...
Thought as Word Dynamics 
by Paul Jorion — 2024-03-30T10:19:57.325Z

Thought as Word Dynamics

Paul J. Jorion, Université Catholique de Lille

paul.jorion@univ-catholille.fr

The following is a manuscript written in the year 2000 meant to become a volume 2 of my French book Principes des systèmes intelligents published in 1989 by Masson in Paris.

The model presented here has been built over a number of years from several angles. It is truly multi-disciplinary in its concept as the evidence which has been gathered to support its plausibility has five main components: (i) mathematical objects that have been explored by the author in another context as part of his anthropological research on modelling kinship networks and so-called "primitive mentality" types of reasoning, (ii) intensive and extensive logic, (iii) linguistics, (iv) psychology — including Freudian metapsychology, (v) the introspective part of philosophy that amounts to twenty-five centuries of speculative cognitive science.

The ambition here is to provide a framework for speech acts, being specific enough about its statics and dynamics that it is testable as an Artificial Intelligence application (i.e. can be written as source code). The test began many years ago when, being part of British Telecom's "Connex" Project, I designed ANELLA: an "Associative Network with Emergent Logic and Learning Abilities" (1987-1990). The project is currently being revived as SAM: Self-Aware Machines (www.pribor.io).

Overall principles:

1. Speech acts are generated as the outcome of a dynamics operating on a network

2. This network is stored in the human brain

3. A talking subject experiences the dynamics as being emotional or "affective"

II. Statics

4. The Network comprises a subset of the words (the "content words") of a particular natural language

5. The individual unit of the Network is a word-pair

6. Each such word-pair has at any time an affect value attached to it

7. The word-pairs and their affect value result from Hebbian reinforcement

8. The Network has two principles of organisation: hereditary and endogenous

9. The hereditary principle is isomorphic to the mathematical object called a "Galois Lattice"

10. The endogenous principle is isomorphic to the mathematical object called a"P-graph'

11. The endogenous principle is primal

12. The hereditary principle is historical: it allows syllogistic reasoning and amounts to the emergence of "reaso...
the underestimation of circular thinking
by AxiomWriter — 2024-03-30T09:45:15.350Z

There is a great and often underappreciated danger in logic thinking itself.

That is the danger of circular reasoning. The easiest example goes:

If A is true then so is B.

If B is true, than so is A.

Therefore A and B are true.

Because it seems so easy to spot, it is vastly underestimated.

But what if hundreds of theorems and variables are involved?

It is not only possible, it is more than likely, that circular reasoning is involved somehow.

Say your intuition goes against the conclusion you have drawn.

You begin to test your logic, going this way and that way through all the theories and variables,

training your own brain to believe in them.

Proofs of the existence of gods have been done that way.

There is a point where you should look at the concepts you think in, even into the exact meaning of the words you use.

Which things do you see as to trivial to even think about?

That things fall down, for example?

Newton did not find that so trivial.

In mathematics, that is what an axiom system is for. It is a safe standing, a sort of table you can use to build your conclusions on without the danger of circular thinking. That is why axiom systems are so crucial in mathematics.

So, how is an axiom system constructed?

At the very beginning, before the axioms are even formulated, there are the definitions.

They contain the ideas you want to think about.

The hardest steps for explaining "fall down" for Newton must have been finding the physical words for "energy", "impulse" or "gravity".

Newton took words that were around anywhere, but their meaning was and is wholly different outside his world of physics.

So, as a mathematician, that is what a axiom system is for: Giving you a set of words and rules to think in. But beware: Your logical thinking is only true as long as the axioms hold.

If you hear a lot of mathematicians say: This is obviously so, no explanation needed, then look very close:

That is the place to look for a nice little overlooked axiom.

You have to take it, build it into the axiom system, probably overthrowing the whole system, and find new words to think in, until you have a sound and well - build axiom system again.

And then you probably have to proof all the theorems and certainties of the old system anew.

I think, the feelings most mathematicians have for tinkerers on axioms systems is very alike to the feeling of one who has build a 100 floors high playing card house...
A dialog with the axiom of choice
by AxiomWriter — 2024-03-30T07:47:35.888Z

preliminary remark: the axiom of choice ( Auswahlaxiom in Germany) can be formulated this way:

For all sets M there is a selection function, that assigns for all elements of the power set P(M) exept ∅ an element of the corresponding subset of M.

It is assumed to be true in many areas of mathematics. Besides its "job" of giving elements of infinite sets, it has equivalent formulations to give "upper bounds" (Lemma of Zorn) and others. It is crucial in functional analysis in many ways.

In the following dialogue I tried to clarify a bit to myself, what this axiom means and why I do not entirely trust it.

Beware: It is an axiom, not a proven theorem. All that is proven about it is the following: It can be included in certain axiom systems without generating a contradiction.

The mathematical god of set theory

There is a little god in mathematics, who is known by many names. The Lemma of Zorn is one, but I choose the name axiom of choice.

Both are equivalent, which is "the same" mathematicaly.

This little god does many things in many mathematical areas.

I choose its job in set theory.

Every time a mathematician says:

"I choose an element of the set M to do this and that with",

it takes an element and gives the element to her - or, in some cases, him or it. I mean, how else can the mathematician get it?

So a person enters the temple of choice and asks:

person: "O axiom of choice, give me an element of the set M."

axiom : "I, the mighty axiom of choice, have a question first.

Did you ensure the set M is not empty? Otherwise I will not give you an element of the set M. I will never do this.

So I ask you: Are you sure the set M is not empty? By the way, which set M are we talking about?"

person: "I want a dialog between the axiom of choice and a mathematician. But I am too lazy to write it. Give me a mathematician to write a dialog with you."

axiom: "Any special kind of mathematician?"

person: "How about Douglas Hofstadter? He is very good at writing

dialogs

."

axiom: " The person, who entered my temple, wishes an element of a subset of mathematicians. Is this set not empty?"

person: "Of course not. It has one element."

axiom: "The petitioner wishes an element of a subset with one

element

. I give you Douglas Hofstadter."

person: "Where is he?"

axiom: "I give elements in a more mathematical sense. To find it is an exe...
Will OpenAI also require a "Super Red Team Agent" for its "Superalignment" Project?
by Super AGI — 2024-03-30T05:25:37.801Z

OpenAI recently released their “Superalignment” plan. But, along with “AI Alignment “, “Red Teaming” is also a critical part of building an AI, LLM, or other software stack that will be accessible to the world over the internet. Red Teaming for an Artificial Superintelligence (ASI) must address potential attacks from diverse "Bad Actors”: including humans, AIs, LLMs, AGIs, and ASIs.

What's the point of building a "fully aligned ASI" if it, or its underlying infrastructure, is vulnerable to attack and manipulation, potentially even by other ASIs?

Would a "Super Red Teaming Agent" be needed to test ASI vulnerabilities?

Alongside their "roughly human-level automated alignment researcher," would a "roughly human-level (or higher) automated Red Teaming researcher" be needed? An agent to develop threat models and test for vulnerabilities arising from potentially super-human AI systems. Including:

Threat Modeling: Anticipating the full spectrum of threats and vulnerabilities an ASI might face.
Prompt Engineering and Malicious Injection: A Super Red Teaming Agent could systematically test for ways to exploit an ASI.
Systems Hacking: Attempting to infiltrate the ASI's infrastructure and access core programming elements.
Developing Exploits: Creating sophisticated attacks specifically tailored to ASI architecture.
Social Engineering: Targeting the human developers and teams behind the ASI, aiming to expose social vulnerabilities.

The Dangers of a Super Red Teaming Agent

To be effective, this "Super Red Teaming Agent" must possess super-human capabilities to exploit any system hack, loophole, vulnerability, prompt injection, etc. It should test the ASI itself, its infrastructure, and even social engineering attacks against those controlling the ASI.

But, could the valuable capabilities of a Super Red Teaming Agent also make it a dangerous tool? If trained to infiltrate an ASI running on the world's most secure systems, could it be repurposed to attack any company, AGI, or infrastructure? What if this "Super Red Teaming Agent" was leaked and sold to the highest bidder?

Can Human Red Teams Alone Ensure ASI Security?

If a "Super Red Teaming Agent" is too dangerous, can "Human Red Teams" comprehensively validate an ASI's security? Are they enough to defend against superhuman ASIs? If not, how can companies like OpenAI ensure their infrastructure and ASIs a...
Happiness
by complicated.world — 2024-03-30T02:18:21.387Z
Link: https://complicated.world/en/Human_Life

complicated.world is a website, where we're trying to present our view on certain popular aspects like the need for privacy, problem of discrimination, immigration, or platforms moderation vs freedom of speech. We want each article to be rooted in some basic values and axioms. We have recently started and here is a link to our first article, which is on happiness and which is a foundation for further articles.

Excerpts:

Origins of Happiness

We may only expect that as with everything else, evolution is to blame. Evolution means survival of the fittest and in the current world it seems that strong, effective societies are the fittest entities. That would lead us to a presumption that our happiness was formed in such a way to support an effective society. For example being kind to other people seems to have a strong positive correlation with happiness, but it also allows a group of people to work together much more efficiently. Similarly with morality, conscience, or the desire to be useful, or the importance of relations. We have law and police to enforce some basic socially acceptable behaviours, but that surely would not be enough without the bottom-up force of people trying to be decent to each other. On a related note, we should mention that as the strength of a state grows as compared with the strength of a single person, that bottom-up kindness becomes less applicable, especially in a totalitarian states, where there is no freedom left to a person, so also there is no space for any kindness or unkindness. If we reach that point, then our happiness will no longer play any role in evolution and may disappear as a whole concept, even in our subjective perception of it.

A Difficult Start

It is disproportionately difficult to take the first steps in self-reflection. And once that process starts, it becomes much easier to make progress, reason about our own lives and intentionally improve over time. That creates a situation where some people get entirely isolated from this process, not even knowing that they could improve their lives, or not even realising that they are in fact unhappy, even though they really are. They would be seen as angry, or unsocial due to their nature. Typically, these individuals would attribute this state to external factors, like their specific life situation, unhelpful colleagues and relatives, or dishonest politicians. Sometimes,

...
Was a Subway in New York City Inevitable?
by Jeffrey Heninger — 2024-03-30T00:53:21.314Z
Link: https://blog.aiimpacts.org/p/historical-note-was-a-subway-in-new

Introduction

The first serious attempt at building a subway in New York City occurred in 1866, following the end of the Civil War (1865) and the opening of the first subway in London (1863). The following decades saw a sequence of failed attempts, and the first subway in New York City would not begin operations until 1904.

When we consider how popular the subway was, even when it first opened, these failures are remarkable. Many of the important actors were indifferent or opposed, and those who supported building a subway were comically bad at coordinating. Only widespread, mostly decentralized public support was able to pressure different political and economic elites into finally cooperating.

The history of these efforts made me wonder whether the construction of a subway in New York City was inevitable. I am usually skeptical of technological inevitability, but this seems like a potentially good example. For decades, the New York City subway was a locally resisted technological temptation,[1] and a potential example of technological overhang.[2] The subway eventually succeeded despite a system that seemed structured to thwart it.

The main source I read about this history was:

Wallace B. Katz. The New York rapid transit decision of 1900: Economy, society and politics. (1978),[3] from a series put together by the Historic American Engineering Record.[4]

All quotes or claims that are not cited come from this source. I have not investigated other sources in detail, but it seems to present a mainstream scholarly view of the history.

Why Did New York City Need A Subway?

By 1866, it seems inevitable that New York City would become one of the greatest world cities. It has an excellent harbor and a deep river heading inland. After the completion of the Erie Canal (1825), most of the Midwest’s international trade passed through New York. It has been the largest city in the US since the first census (1790),[5] and was among the 10 largest in the world by 1850.[6] Network effects would help it continue to grow. The New York Stock Exchange brought in large businesses, especially railroads. New York City was also a major destination for immigrants.[7]

Rapid transit was even more important for New York than for other great world cities. Manhattan is an island, and so New York City has fewer directions it can grow in than London, Paris, or Beijing. Manhat...