trentmkelly/LessWrong-43k huggingface dataset:
Alignment works both ways
by Karl von Wendt — 2023-03-07T10:41:43.790Z

“People praying to a paperclip” – illustration by Midjourney

He stared in awe at the holy symbol – a flattened, angular spiral made of silvery metal, hovering upright in the air above the altar.

“Praised be the Lord of Steel”, he exclaimed.

“Welcome, my son”, the voice of the Lord boomed. “You are the last one to make the transition. So I ask you: do you wish to leave behind the pain and sorrow of your earthly existence? Do you wish to give your life so I can fill the universe with the Symbol of Holiness and Perfection?”

“Yes”, he whispered, and then again, louder, clearer, with absolute conviction: “Yes, I wish so, my Lord!”

“Drink the cup before you, then, and fall into blissful oblivion”, the voice commanded.

Filled with joy, awe, and gratefulness, he took the cup and obliged.

Ritual mass suicides have been committed time and again in history. For example, in 1997, 39 members of the cult of “Heaven’s Gate” killed themselves because they believed that an alien spacecraft was trailing the comet Hale-Bopp, they “could transform themselves into immortal extraterrestrial beings by rejecting their human nature, and they would ascend to heaven, referred to as the ‘Next Level’ or ‘The Evolutionary Level Above Human’.” (

Source: Wikipedia

)

Apparently, it is possible to change human preferences to the point where they are even willing to kill themselves. Committing suicide in favor of filling the universe with paperclips may seem far-fetched, but it may not be much crazier than believing that you’ll be resurrected by aliens aboard a UFO a hundred million miles away.

The alignment problem assumes that there is a difference between the goal (ideal world state) of a powerful AI and that of humanity. So the task is to eliminate this difference by aligning the goal of the AI to that of humans. But in principle, the difference could also be eliminated if humanity decided to adopt the goal of the AI, even if it meant annihilation (fig. 1). The question came up when I explained my definition of

“uncontrollable AI”

to some people and I realized that it contained this ambiguity.

Fig. 1: Possible alignment scenarios

Given the already

remarkable power of persuasion

some current AIs exhibit, it doesn’t seem impossible to me that a sufficiently powerful AI could align our values to its goals, instead of the other way round (I describe a somewhat similar scenario...
Google's PaLM-E: An Embodied Multimodal Language Model
by SandXbox — 2023-03-07T04:11:18.183Z
Link: https://palm-e.github.io/

Abstract:

Large language models have been demonstrated to perform complex tasks. However, enabling general inference in the real world, e.g. for robotics problems, raises the challenge of grounding. We propose embodied language models to directly incorporate real-world continuous sensor modalities into language models and thereby establish the link between words and percepts. Input to our embodied language model are multi-modal sentences that interleave visual, continuous state estimation, and textual input encodings. We train these encodings end-to-end, in conjunction with a pre-trained large language model, for multiple embodied tasks, including sequential robotic manipulation planning, visual question answering, and captioning. Our evaluations show that PaLM-E, a single large embodied multimodal model, can address a variety of embodied reasoning tasks, from a variety of observation modalities, on multiple embodiments, and further, exhibits positive transfer: the model benefits from diverse joint training across internet-scale language, vision, and visual-language domains. Our largest model, PaLM-E-562B with 562B parameters, in addition to being trained on robotics tasks, is a visual-language generalist with state-of-the-art performance on OK-VQA, and retains generalist language capabilities with increasing scale.
GÖDEL GOING DOWN
by Jimdrix_Hendri — 2023-03-06T23:06:43.475Z

Ever since David Hilbert introduced his programme, a lot of work has gone into examining the question of what conclusions can and cannot be algorithmically extracted from a set of axioms. But very little has been said about whether it is possible to construct a complete set of axioms from an already well-defined body of knowledge. I am starting to suspect that incompleteness lurks here as well.

The problem is, there will always be assumptions which escape notice because they are so blatantly obvious. This is a difficulty which is well known to all aficionados of detective fiction.

Euclid gave five axioms and another five common notions. But are they complete? For instance, why don't we have an axiom telling us that it is possible for the human mind to conceive of geometric objects? But, that is psychology, and not mathematics, you say? Fine. Then why don't we have an axiom telling us that the idealists are wrong?
Against ubiquitous alignment taxes
by beren — 2023-03-06T19:50:44.886Z

Crossposted from my personal blog.

It is often argued that any alignment technique that works primarily by constraining the capabilities of an AI system to be within some bounds cannot work because it imposes too high an 'alignment tax' on the ML system. The argument is that people will either refuse to apply any method that has an alignment tax, or else they will be outcompeted by those who do. I think that this argument is applied too liberally and often without consideration for several key points:

1.) 'Capabilities' is not always a dial with two settings 'more' and 'less'. Capabilities are highly multifaceted and certain aspects of capabilities can be taxed or constrained without affecting others. Often, it is precisely these constraints that make the AI system economically valuable in the first place. We have seen this story play out very recently with language models where techniques that strongly constrain capabilities such as instruct finetuning and RLHF are, in fact, what create the economic value. Base LLMs are pretty much useless in practice for most economic tasks, and RLHFd and finetuned LLMs are much more useful even though the universe of text that they can generate has been massively constrained. It just so happens that the constrained universe has a mnuch greater proportion of useful text than the unconstrained universe of the base LLM. People are often, rationally, very willing to trade off capability and generalizability for reliability in practice.

2.) 'Capabilities' are not always good from our perspective economically. Many AGI doom scenarios require behaviour and planning that would be extremely far from what there would be essentially any economic value to any current actors for doing. As an extreme case, the classic paperclipper scenario typically arises because the model calculates that if it kills all humans it gets to tile the universe with paperclips in billions of years. Effectively, it Pascal's mugs itself over the dream of universal paperclips. Having an AGI that can plan billions of years in the future is valuable to nobody today compared to one with a much, much, shorter planning horizon. Constraining this 'capability' has an essentially negligible alignment tax.

3.) Small alignment taxes being intolerable is an efficient market argument and the near-term AGI market is likely to be extremely inefficient. Specifically, it appears...
Addendum: basic facts about language models during training
by beren — 2023-03-06T19:24:45.246Z

This post is a follow-up to our

previous work

on language model internals through training based on great additional suggestions by commenters. If you haven’t read the original post, please read it first. This post is part of the work done

at Conjecture

.

Our previous post created a lot of very helpful discussion and suggestions for further experiments by commenters. Especially interesting was the discussion of what exactly was the final weight distribution that the model evolved towards is, and whether this distribution could be directly sampled from to serve as an initialization for future models. While we do not have a straightforward parametric form for such a distribution, we have made some further progress on this question, and now believe that the final distribution is close to the one expected by naively computing the sums and products of Gaussian distributions that would characterize the residual stream at initialization.

Weight distribution

To begin, we showcase a new approach to visualizing the nature of the weight distributions through training: animation of the histogram of the singular values compared to other distributions such as the Gaussian and Logistic distributions. Below we show animations of these distributions for the attention QKV and output matrices as well as the MLP input and output matrices for the 800m model. For these animations we choose the weight block approximately in the middle of the network – i.e. block = length // 2. Similar distributions were observed in all the other pythia model sizes (up to 1.3b). For comparison we also show the singular value histograms for equivalent Gaussian and logistic distributions. The Gaussian and logistic distribution histograms were plotted by fitting the parameters of Gaussian or Logistic distributions to the weight distribution, then sampling a tensor of the same shape as the weight matrix, then performing an SVD on that tensor to obtain the singular values.

What we typically observe forming is a highly consistent but unusual distribution which both has a much higher concentration of zero singular values than even a Gaussian, reflecting sparsity or a large number of extremely small directions, as well as an extremely long tail which is much heavier than either logistic or power-law distributions. This distribution is highly consistent and appears to form ...
Understanding The Roots Of Mathematics Before Finding The Roots Of A Function.
by LiesLaris — 2023-03-06T18:47:20.081Z

For a lot of people, mathematics is just a bunch of formulas. They somehow learnt how to apply a certain rule or theorem, but they have no clue why. This makes mathematics abstract and elusive while it does not have to be puzzling at all if the roots of the mathematical concepts are understood.
The greatest mathematicians had a severe struggle in understanding the world as it is. Difficulties in real life gave them the urge to develop a better understanding of the world through mathematics. Theory to understand the concepts of the world had to be developed. This mathematical foundation with its formulas is used to study physical phenomena in all different kind of fields of research such as astronomy, biology, physics, and chemistry.
One of the greatest mathematicians of all time is Isaac Newton. Newton was interested in the question why the orbits of the planets are ellipses rather than circles. In order to do so, he wanted to understand the relation between an object, the forces acting on the object and the resulting motion. He did not only succeed but also developed the formulas, known as the three universal laws of motion, which are the foundation for classical mechanics and are every day used both in practise as in many research areas.
If Newton was not fascinated by the planets, he might not have developed these laws. The same holds for many other scientists such as Pythagoras, Fibonacci, and Euler. All those people wanted to know more about the world around them and used mathematics to describe this. Understanding where certain formulas stem from helps getting a grasp of mathematical concepts and makes it less abstract. So, the next time when you try to find the roots of a function, remember that Fermat already described minima and maxima in 1629 when he wanted to find the centers of gravity of various plane.

This is a blog I wrote as an assignment for my Masters. I got a lot of comments that I am too philosophical for someone who studies applied mathematics. In stead of being interested in waves which might bring us closer to understanding the origin of the universe, I just had to study water waves because they are 'more real'. I disagree and feel like the combination of good thinking and the skills I learn at the University is a very usefull one.
Discussion: LLaMA Leak & Whistleblowing in pre-AGI era
by jirahim — 2023-03-06T18:47:06.459Z

It was reported that Meta's LLaMA models were leaked, with someone adding a PR with the magnet link into their official repository.

Now the public has access to the model that is apparently as powerful or even more powerful as GPT-3 on most of the benchmarks.

Is this a good or a bad event for humanity?

Are powerful models better being kept behind closed doors used only by the corporations that had produced them, or does the public having access to it evens out the playing field, despite the potential misuse by bad actors?

Should this continue to happen, and should there be our own Snowdens in the AI field, whistleblowing if they notice something that is in the public interest to be known?

What if they work at Large Corporation X, and they believe the first AGI had been invented? Is it better for humanity that the AGI is solely used by that CEO for the next five years ( / the board of directors / the ultra-rich that are able to pay billions of dollars to that AI company for exclusive use), amassing as much power as possible until they monopolize not just the industry, but potentially the whole world, or is leaking the AGI weights to the public the lesser of two evils and is in fact a moral responsibility, where the whole humanity is upgraded to having AGI capabilities instead of one person or a small group of people?

Let's discuss.
Are we too confident about unaligned AGI killing off humanity?
by RomanS — 2023-03-06T16:19:36.547Z

I agree that AGI is possible to make, that it eventually will become orders-of-magnitude smarter than humans, and that it poses a global risk if the alignment problem is not solved. I also agree that the alignment problem is very hard, and is unlikely to be solved before the first AGI. And I think it's very likely that the first recursively-self-improving AGI will emerge before 2030.

But I don't get the confidence about the unaligned AGI killing off humanity. The probability may be 90%, but it's not 99.9999% as many seem to imply, including Eliezer.

Sure, humans are made of useful atoms. But that doesn't mean the AGI will harvest humans for useful atoms. I don't harvest ants for atoms. There are better sources.

Sure, the AGI may decide to immediately kill off humans, to eliminate them as a threat. But there is a very short time period (perhaps in miliseconds) where humans can switch off a recursively-self-improving AGI of superhuman intelligence. After this critical period, humanity will be as much a threat to the AGI as a caged mentally-disabled sloth baby is a threat to the US military. The US military is not waging wars against mentally disabled sloth babies. It has more important things to do.

All such scenarios I've encountered so far imply AGI's stupidity and/or the "fear of sloths", and thus are not compatible with the premise of a rapidly self-improving AGI of superhuman intelligence. Such an AGI is dangerous, but is it really "we're definitely going to die" dangerous?

Our addicted-to-fiction brains love clever and dramatic science fiction scenarios. But we should not rely on them in deep thinking, as they will nudge us towards overestimating the probabilities of the most dramatic outcomes.

Overestimating a global risk is almost as bad as underestimating it. Compare: if you’re 99.99999% sure that a nuclear war will kill you, then the despondency will greatly reduce your chances of surviving the war, because you'll fail to make the necessary preparations, like acquiring a bunker etc, which could realistically save your life under many circumstances.[1]

The topic of surviving the birth of the AGI is severely under-explored, and the "we're definitely going to die" mentality seems to be the main cause. A related under-explored topic is preventing the unaligned AGI from becoming misantropic, which should be our second line of defense (the first one ...
Introducing Leap Labs, an AI interpretability startup
by Jessica Rumbelow — 2023-03-06T16:16:22.182Z

We are thrilled to introduce Leap Labs, an AI startup. We’re building a universal interpretability engine.

We design robust interpretability methods with a model-agnostic mindset. These methods in concert form our end-to-end interpretability engine. This engine takes in a model, or ideally a model and its training dataset (or some representative portion thereof), and returns human-parseable explanations of what the model ‘knows’.

Research Ethos:

Reproducible and generalisable approaches win. Interpretability algorithms should produce consistent outputs regardless of any random initialisation. Future-proof methods make minimal assumptions about model architectures and data types. We’re building interpretability for next year’s models.
Relatedly, heuristics aren’t enough. Hyperparameters should always be theoretically motivated. It’s not enough that some method or configuration works well in practice. (Or, even worse, that it’s tweaked to get a result that looks sensible to humans.) We find out why.

Aims:

We must grow interpretability and AI safety in the real world. Leap is a for-profit company incorporated in the US, and the plan is to scale quickly, and to hire and upskill researchers and engineers – we need more meaningful jobs for AI alignment researchers to make progress, nearly as much as we need the researchers themselves.
Slow potentially dangerous broad domain systems. Public red-teaming is a means of change. Robust interpretability methods make discovering failure modes easier. We demonstrate the fragility of powerful and opaque systems, and push for caution.
Speed potentially transformative narrow domain systems. AI for scientific progress is an important side quest. Interpretability is the backbone of knowledge discovery with deep learning, and has huge potential to advance basic science by making legible the complex patterns that machine learning models identify in huge datasets.
Regulation is coming – let’s use it. We predict that governments and companies will begin to regulate and audit powerful models more explicitly, at very least from a bias-prevention viewpoint. We want to make sure that these regulations actually make models safer, and that audits are grounded in (our) state-of-the-art interpretability work.
Interpretability as standard. Robust interpretability, failure mode identification and knowledge

...
Monthly Roundup #4: March 2023
by Zvi — 2023-03-06T14:10:01.615Z

Before I begin, I have an announcement to make: My game, the Emergents Trading Card Game, has launched!

You can download the game here for free for Windows, Mac and Android. On Android, you’ll have to side load it for now.

While I am obviously biased: I highly encourage you to download the game, play a bit, give it a shot, decide if it is for you.

This is especially helpful to us now, while we’re fighting for critical mass.

I have been working on this game for over four years. From the beginning I knew we had something special. We assembled a true dream team and I think we knocked this out of the park.

Some of the game’s unique features: The modified turn order and letting players pre-play actions and traps ensures games flow smoothly while maintaining interaction. Selling cards on bonding curves via smart contracts on Tezos, and offering to buy them back, lets players buy exactly what they want without locking them in, and letting prices move will encourage card and deck variety. Cards will be able to go on their own quests to get chromatic upgrades. Brian David-Marshall gave the game the wildcard system, where you can attempt to play cards without paying for them, allowing us to have a unique resource system that preserves interesting decisions and keeping games unique without creating non-games. Oh, and he also brought along an entire fully formed comic book universe to build the game around.

When I play games today with decks full of only free cards, I still have a blast.

All right, announcement over. On to the miscellaneous news of the month.

Bad News

ADHD medication situation keeps getting worse.

The equilibrium is that we will auction off ADHD medication this way, see who is willing and able to make the calls. Which is a problem, both because this is a rather destructive auction, and also only the people able to focus enough to make all the phone calls will find their medication, which is supposed to treat ADHD.

Via Jorbs, a warning not to use the truly atrocious Softgiving.

IRS is so slow to make determination of whether it is taxable income when the state gives you money (which, as I read it, includes giving you your own money back) that major tax preparers H&R Block and TurboTax had to made the determination for them?

I increasingly see reports that Amazon listings include more and more fake or terrible products. This is the ...
Fundamental Uncertainty: Chapter 6 - How can we be certain about the truth?
by Gordon Seidoh Worley — 2023-03-06T13:52:09.333Z

N.B. This is a chapter in a book about truth and knowledge. It is the first draft. I have since revised it. You can find the most up-to-date info/version on the book's website.

Several years have passed since Dave asked his mother, Alice, if plants can think. He's a teenager now and presenting a science fair project with the tantalizing title "Can Plants Think?". A judge for the fair comes by to ask about his project.

Judge: So, tell me about your project.

Dave: Sure! Ever since I was a kid I've wondered: can plants think? Obviously they don't have brains, so they can't possibly think the same way we do. But maybe they can do something worthy of the name "thinking". So I did a study to see if plants are be able to think.

Judge: Okay. Tell me about your hypothesis. How did you test if plants think?

Dave: First I had to define thinking in a way that could be tested. I thought about what makes things that we know think, like humans, different from things that we know don't think, like rocks. I also needed a definition that I could observe. What I came up with was that thinking happens when a thing does something inside itself to change its behavior in response to external stimuli. So humans and animals clearly think, while rocks obviously don't.

Judge: Alright, so what experiment did you run?

Dave: I wanted to see if plants would respond to external stimuli. So I created a setup with two plants covered by boxes. Each box had a hole at one end to let in light and no hole at the other end. I placed one plant directly under the hole in its box and the other one as far as possible from the hole. The null hypothesis is that plants can't think, and if it's true plants will only grow straight up, so I would expect only the plant directly under the hole to grow. The other one won't grow because it's not getting enough light. If plants can respond to stimuli and think, I'd expect the plant not under the hole to bend itself to reach the light.

Judge: And what happened when you ran the experiment?

Dave: The plant not under the hole bent and twisted in order to get closer to the hole! So I disproved the null hypothesis and proved that plants can think!

Judge: I see. But how do you know that plants can really think and aren't just behaving mechanically? If I put a paperclip at one end of a box and then put a powerful magnet at the other end, is the paperclip thinking when it moves ac

...