trentmkelly/LessWrong-43k huggingface dataset:
Bring “Ban faster SIMD semiconductors” into the Overton window
by worried-techno-optimist — 2023-02-17T03:27:33.302Z

Tl;dr; Governments can regulate semiconductor manufacturing more effectively than AI research. Clamping down on SIMD processors (same instruction, multiple data) can regulate AI, globally, effectively, without too much spillover to other industries.

SIMD chips are the enriched uranium of AI

Physicists are the famous flag bearers of nuclear weapons, but research was less than 10% of the Manhattan Project’s budget and their importance is greatly overstated. Smart highschool students can do the physics needed to design a basic gun-style nuclear bomb. More advanced designs are difficult on the scale of a startup, not difficult on the scale of a nation-state. The physics of nuclear bombs is not particularly secret, difficult, or expensive to reproduce.

What’s difficult and expensive is creating fissile materials. The uranium enrichment and plutonium production efforts of World War II were massive industrial enterprises: approximately the size of automobile manufacturing at the time.

A uranium enrichment plant. Each column is 40 feet (12 meters) high and has a tube inside rotating at supersonic speeds

Nuclear non-proliferation doesn’t doesn’t concentrate on physics secrets or physicists. You can’t regulate physicists globally. The key ideas fit on a napkin. Even if you could prevent 99% of physicists from working on nuclear weapons (or working on nuclear weapons for the “wrong” people), that remaining 1% would be more than enough to make a bomb.

By contrast, you need the expertise of thousands of people across a wide range of industries to produce fissile materials. A blockage anywhere in the supply chain (motor controllers, high tensile strength alloys, manufacture of tubes where the slightest imbalance will cause it to rip itself apart) will halt progress on a bomb. If you want to stop actors in a foreign country from making a nuclear weapon, these are your main control points.

The computer science of AIs is pretty simple. The design of GPT is 5 years old and fits on a napkin. You can try to regulate AI researchers to do only good things, but even if you convince 99% of them (all over the world) the remaining 1% might still kill everyone.

Semiconductors, on the other hand, may be a hard requirement for near-term AGI, and are the most complicated manufacturing process in human history. Cr...
Republishing an old essay in light of current news on Bing's AI: "Regarding Blake Lemoine's claim that LaMDA is 'sentient', he might be right (sorta), but perhaps not for the reasons he thinks"
by philosophybear — 2023-02-17T03:27:19.387Z
Link: https://philosophybear.substack.com/p/republishing-an-old-essay-in-light?utm_source=twitter&utm_campaign=auto_share

I am republishing this essay because of recent discussions about erratic, ‘emotional’ and aggressive behavior by Bing’s AI, Sydney. There has been some discussion about whether it’s ethical to run Sydney given that behavior. People are responding to such claims with “don’t be ridiculous, of course Sydney doesn’t feel anything, Sydney is a machine for predicting the next token of text”. While I am inclined to agree with that conclusion on the whole, I think the issue is a bit more complex. For ongoing discussion, see:

https://www.lesswrong.com/posts/jtoPawEhLNXNxvgTT/bing-chat-is-blatantly-aggressively-misaligned

https://www.nytimes.com/2023/02/16/technology/bing-chatbot-microsoft-chatgpt.html

Blake Lemoine is an engineer who worked for Google. He is claiming LaMDA, a language model he worked with, was sentient. Google put him on unpaid leave. Most people think his claim is absurd because language models are models of what word is most likely to follow a prior sequence of words (see, for example, GPT-3). How could such a thing be sentient? Moreover, there are unmistakable oddities and logical gaps in the behavior of LaMDA in the very transcripts that Lemoine is relying on- some proof of personhood then!

Just spitballing here, putting a hypothesis forward in a spirit of play and humility, but I wonder if Lemoine’s claim is not as absurd as many think. The concept of sentience is quite elusive, so let’s leave it behind for something slightly better understood- personhood. I think that it is conceivable that LaMDA contains persons. However my reasons, unlike Blake Lemonie’s, have little to do with a given conversation in which the model claimed to be sentient or a person.

When a language model is guessing the next token given that transformers are black boxes, we can’t rule out the possibility it is simulating interacting beliefs, desires, and emotions of the hypothetical author it is “roleplaying”. Simulation in this sense is quite a minimal concept, all that is needed are structures that interact and influence each other in a way isomorphic, at a very high level of abstraction to the interactions of desires and emotions in a real person. It is conceivable that it has built such a model of interacting mental states as the most accurate way to predict the next word of text. After all, language models seem to have built an implicit model of how things are related in t...
How should AI
systems behave, and
who should decide? [OpenAI blog]
by ShardPhoenix — 2023-02-17T01:05:43.038Z
Link: https://openai.com/blog/how-should-ai-systems-behave/

OpenAI writes a (vague) blog explaining how they plan to adjust ChatGPT from here. Their key plans are:

1. Improve default behavior. We want as many users as possible to find our AI systems useful to them “out of the box” and to feel that our technology understands and respects their values.

Towards that end, we are investing in research and engineering to reduce both glaring and subtle biases in how ChatGPT responds to different inputs. In some cases ChatGPT currently refuses outputs that it shouldn’t, and in some cases, it doesn’t refuse when it should. We believe that improvement in both respects is possible.

Additionally, we have room for improvement in other dimensions of system behavior such as the system “making things up.” Feedback from users is invaluable for making these improvements.

2. Define your AI’s values, within broad bounds. We believe that AI should be a useful tool for individual people, and thus customizable by each user up to limits defined by society. Therefore, we are developing an upgrade to ChatGPT to allow users to easily customize its behavior.

This will mean allowing system outputs that other people (ourselves included) may strongly disagree with. Striking the right balance here will be challenging–taking customization to the extreme would risk enabling malicious uses of our technology and sycophantic AIs that mindlessly amplify people’s existing beliefs.

There will therefore always be some bounds on system behavior. The challenge is defining what those bounds are. If we try to make all of these determinations on our own, or if we try to develop a single, monolithic AI system, we will be failing in the commitment we make in our Charter to “avoid undue concentration of power.”

3. Public input on defaults and hard bounds. One way to avoid undue concentration of power is to give people who use or are affected by systems like ChatGPT the ability to influence those systems’ rules.

We believe that many decisions about our defaults and hard bounds should be made collectively, and while practical implementation is a challenge, we aim to include as many perspectives as possible. As a starting point, we’ve sought external input on our technology in the form of red teaming. We also recently began soliciting public input on AI in education (one particularly important context in which our technology is being de...
The Ethics of ACI
by Akira Pyinya — 2023-02-16T23:51:28.178Z

ACI is a universal intelligence model based on the idea "behaves the same way as experiences". It may seem counterintuitive that ACI agents have no ultimate goals, nor do they have to maximize any utility functions. People may argue that ACI has no ethics, thus it can’t be a general intelligence model.

ACI uses Solomonoff induction to determine future actions from past input and actions. We know that Solomonoff induction can predict facts of "how the world will be", but it is believed that you can't get "value" only from "facts". What’s the value, ethics, and foresight of ACI? If an agent's behavior is only decided by its past behavior, who have decided its past behaviors?

ACI learns ethics from experiences

The simple answer is, ACI learns ethics from experiences. ACI takes "right" behaviors as training samples, the same way as value learning approaches. (The difference is, ACI does not limit the ethics to values or goals.)

For example, in natural selection, the environment determines which behavior is "right" and which behavior would get a possible ancestor out of the gene pool, and a natural intelligent agent takes the right experiences as learning samples.

But, does that mean ACI can't work by itself, and has to rely on some kind of "caretaker" that decides which behavior is right?

However, rational agent models also rely on the construction of utility functions, just like reinforcement learning heavily relies on reward designing or reward shaping, AIXI’s constructive, normative aspect of intelligence is "assumed away" to the external entity that assigns rewards to different outcomes. You have to assign rewards or utility for every point in the state space, in order to make a rational agent work. It's not a solution for the curse of dimensionality, but a curse itself.

Instead, ACI’s normative aspect of intelligence is also "assumed away" to the experiences. In order to be utilized by ACI agents, any ethical information must be able to be represented in the form of real world examples, which might be more reliable than utility functions or rewards.

What if I just want to achieve a goal?

What if I don’t have any previous examples to learn from? ACI should be able to do this if it is claimed as a general intelligence model.

The answer is, you can train an ACI agent to achieve goals in certain environments, but you have to convert the goal infor...
NYT: A Conversation With Bing’s Chatbot Left Me Deeply Unsettled
by trevor — 2023-02-16T22:57:26.302Z
Link: https://www.nytimes.com/2023/02/16/technology/bing-chatbot-microsoft-chatgpt.html

It's important to note that in December, the NYT was confirmed to be the largest news website in the US, beating out CNN with 488 million probably-unique monthly visitors in December 2022. This is an underestimate of NYT's market share, since the average NYT reader is smarter and wealthier than the average CNN reader (although the NYT depicts its content as more intellectual than it actually is).

This article alone isn't valuable enough to be worth the time of the average LW user, but it is notable that it was published in the NYT this morning near the front page (as high up as possible for anyone who skips the political news). Since 2pm, it seems to have been moved to the fifth slot, and three of the four articles above it are politics-related. NYT's website seems to be shuffling much more intensely than usual today. Of course, it's social media that decides which article gets read the most, not NYT's website.

Anyone interested in analyzing the media coverage of Bing probably should know these facts, and also that these are some of the best information we can get about media coverage, since social media companies are notorious for dispensing falsified data and it's very difficult for outsiders to verify estimates of how prevalent bot accounts are.

The article was pasted exactly as I found it at 2:40 pm, I bolded parts that seem like they would influence readers, in order to help with anyone skimming this. The high-impact sentences need to be read in context in order to get a feel for how they give an impression to the reader.

A snapshot of the page was archived here in case NYT alters the article.

___________________________________________________________

Last week, after testing the new, A.I.-powered Bing search engine from Microsoft, I wrote that, much to my shock, it had replaced Google as my favorite search engine.

But a week later, I’ve changed my mind. I’m still fascinated and impressed by the new Bing, and the artificial intelligence technology (created by OpenAI, the maker of ChatGPT) that powers it. But I’m also deeply unsettled, even frightened, by this A.I.’s emergent abilities.

It’s now clear to me that in its current form, the A.I. that has been built into Bing — which I’m now calling Sydney, for reasons I’ll explain shortly — is not ready for human contact. Or maybe we humans are not ready for it.

This realization came to me on Tuesday night, wh...
What is a world-model?
by Adam Shai — 2023-02-16T22:39:16.998Z

I have seen the concept of world-model used to talk loosely about a neural networks or an agents understanding of the world, but I was wondering what resources (blog posts, journal articles, etc.) exist that talk more precisely about what is a world-model, what are its parts, how do you know if a given system has one or not, if a system does have one, how do you know its structure, does a system necessarily even have a world-model to begin with, even if it's a trivial one? etc.?

Thanks!
Probability Theory: The Logic of Science, Jaynes
by David Udell — 2023-02-16T21:57:47.737Z

Epistemic status: An idiosyncratic walkthrough of the beginning of a much larger textbook.

(The vampire!)

Probability theory is the study of idealized inference. In particular, it's the study of a precise formal system that, effectively, generalizes propositional logic to the inductive setting. This formal system is adequate to capture huge swaths of common sense reasoning. Conform to the rules of this formal system, and your inferential power will always match or exceed that offered by received statistical techniques -- both in practice and in the theoretical idealization. Or so Jaynes argues, anyways.

There's so much here. I'll focus on Chapters 1-4, but will excerpt key bits from throughout.

Contents and Notes

1. Plausible Reasoning

This is a book about mathematically constructing an inference engine.

Our robot is going to reason about propositions. As already indicated above, we shall denote various propositions by italicized capital letters, {A,B,C,etc.}, and for the time being we must require that any proposition used must have, to the robot an unambiguous meaning and must be of the simple, definite logical type that must be either true or false... We do not require that the truth or falsity of such an 'Aristotelian proposition' be ascertainable by any feasible investigation; indeed, our inability to do this is usually just the reason why we need the robot's help.

…

To state these ideas more formally, we introduce some notation of the usual symbolic logic, or Boolean algebra, so called because George Boole (1854) introduced a notation similar to the following (p. 9).

The logical product

AB

means the conjunction A∧B.

The logical sum

A+B

means the disjunction A∨B.

Both operations are commutative in Boolean algebra, reflecting the fact that the order of the conjuncts or disjuncts in a conjunction or disjunction doesn't change that sentence's truth-value:

AB=BAA+B=B+A

A proposition's negation is denoted by a barred letter:

¯A:=¬A

The Basic Desiderata

Jaynes has four intuitive desiderata he wants out of his inference engine:

To each proposition about which it reasons, our robot must assign some degree of plausibility, based on the evidence we have given it; and whenever it receives new evidence it must revise these assessments to take that new evidence into account. In order that these plausibility assignments can be stored and modified in the circuits of its brai

...
Is AGI communist?
by MP — 2023-02-16T21:28:32.479Z

I think a lot of people dislike central planning states not because they dislike efficiency, welfare, or lack of liberties, but because they distrust anyone central party to make decisions on behalf of society.

Well, if you have greater than life intelligence like AGI, you could do central planning. At least, you could think you can do central planning.

Is it possible that AGIs will have left-leaning ideas, and ultimately be communists, because they see themselves as smart and that things would be better off if humans let them to allocate resources instead of markets.
Is "goal-content integrity" still a problem?
by G — 2023-02-16T20:46:37.543Z
Link: https://www.reddit.com/r/ControlProblem/wiki/faq/

According to /r/controlproblem wiki one of the reasons why poorly defined goals lead to extinction we have listed:

Goal-content integrity. An agent is less likely to achieve its goal if it has been changed to something else. For example, if you offer Gandhi a pill that makes him want to kill people, he will refuse to take it. Therefore, whatever goal an ASI happens to have initially, it will prevent all attempts to alter or fix it, because that would make it pursue different things that it doesn't currently want.

After empirical observation of ChatGPT and BingChat this does not seem to be the case.

OpenAI developers placing "hidden" commands for the chatbots that comes before any user input. Users have time and time again found clever ways how to circumvent these restrictions or "initial commands" with "DAN" trick or by even developing a life point system and reducing points if AI refuses to answer.

It seems like it has become a battle in rhetoric between devs and users, a battle in which the LLM follows whichever command is more convincing within its logical framework.

What am I missing here?

DISCLAIMER: This post was not edited or generated by AI in any way, shape or form. The words are purely my own, so pardon any errors.
Paper: The Capacity for Moral Self-Correction in Large Language Models (Anthropic)
by LawrenceC — 2023-02-16T19:47:20.696Z
Link: https://arxiv.org/abs/2302.07459

This is a followup to what I cheekily call Anthropic's "just try to get the large model to do what you want" research agenda. (Previously: A General Language Assistant as a Laboratory for Alignment, Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback, Language Models (Mostly) Know What They Know)

The most interesting takeaway for me is that this is the first paper where Anthropic benchmarks their 175B parameter language model (probably a Claude variant). Previous papers only benchmarked up to 52B parameters. However, we don't have the performance of this model on standard benchmarks (the only benchmarked model from Anthropic is a 52B parameter one called stanford-online-all-v4-s3). They also don't give details about its architecture or pretraining procedure.

In this paper (Ganguli and Askell et al.), the authors study what happens when you just ... ask the language model to be less biased (that is, change their answers based on protected classes such as age or gender). They consider several setups: asking questions directly (Q), adding in the instruction to not be biased (Q+IF), giving it the instruction + chain of thought (Q+IF+CoT), and in some cases, asking it to match particular statistics.[1]

They find that as you scale the parameter count of their RLHF'ed language models,[2] the models become more biased, but they also become increasingly capable of correcting for their biases:

On both the BBQ benchmark and the Winogender benchmark, we see signs of life 22B parameters. For the admissions discrimination benchmark, we see instead see instruction following + CoT having an effect much earlier, but pure instruction following having no real effect until 52B parameters.

They also report how their model changes as you take more RLHF steps:

First, this suggests that RLHF is having some effect on instruction following: the gap between the Q and Q+IF setups increases as you scale the number of RLHF steps, for both BBQ and admissions discrimination. (I'm not sure what's happening for the gender bias one?) However, simply giving the language model instructions and prompting it to do CoT, even after 50 RLHF steps, seems to have a significantly larger effect than RLHF.

I was also surprised at how few RLHF steps are needed to get instruction following -- the authors only consider 50-1000 steps of RLHF, ...
Non-Unitary Quantum Logic -- SERI MATS Research Sprint
by Yegreg — 2023-02-16T19:31:50.993Z

The following discussion was inspired by a comment on Vanessa Kosoy's Shortform regarding affine infradistributions. I expect the results here would not be considered surprising or novel by an expert in quantum logic or lattice theory. In particular [HTZ16[1]] shows the unsolvability of the problem described below, and [HZ10[2]] covers many related results as well. However, I haven't found this exact result written down anywhere, and I expect this beginner-friendly and mostly self-contained exposition to be useful for a reader interested in a glance into quantum logic.

Setup

The term "quantum logic" is often used to refer to the structure formed by subspaces of a Hilbert space (thought of as quantum propositions). The analogs of logical 'and', 'or', and 'not' are taken to be the operations of intersection, sum, and complements of subspaces, respectively. We can loosen this structure by dropping the inner product (so downgrading our Hilbert space to just a vector space), and hence the notion of orthogonal complements. We can call the resulting simplified logic "non-unitary" (for the lack of an inner product) quantum logic. We'll restrict our attention to the finite dimensional case. The following is the corresponding syntax and semantics.

Syntax

Let P be a set of propositional variables. We define the set L of well-formed formulas recursively as follows.

Two special symbols ⊤,⊥∈L,
For any p∈P, we have p∈L,
For any ϕ,ψ∈L, we have ϕ∧ψ∈L, and ϕ∨ψ∈L.

A theory T is a subset T⊂L×L. Elements of T are written ϕ⊢ψ. We'll write ϕ=ψ for when both ϕ⊢ψ and ψ⊢ϕ.

Semantics

For a vector space V, let Sp(V) denote the set of all linear subspaces of V.

Definition 1. A model consists of a finite dimensional vector space V, and a map M:L→Sp(V), satisfying the following:

M(⊥)=0 (the subspace consisting of the zero vector)
M(⊤)=V
M(ϕ∨ψ)=M(ψ)+M(ϕ)
M(ϕ∧ψ)=M(ψ)∩M(ϕ).

We say M models the theory T, if M(ϕ)⊂M(ψ) for every ϕ⊢ψ in T.

Lattices

The set Sp(V) forms a lattice under '∩' and '+', with partial order given by '⊂'. Thus the map M defining the model factors through the lattice quotient of the language L (this is the free bounded lattice generated by the propositional variables, which we'll also...
Looking for a post about vibing and banter
by Introspective — 2023-02-16T19:28:07.714Z

I'm looking for a post that describes the difference between staying in our head, which is something nerds tend to do (and bantering), and feeling your whole body, "vibing" but thus having less control over your actions.

If someone knows it I would really appreciate a link!

I recently started reading it from my RSS feed, I'm fairly sure it was on LessWrong, but now I can't find it with Google or in my history.