trentmkelly/LessWrong-43k huggingface dataset:
Hints about where values come from
by Spiracular — 2023-10-18T00:07:58.051Z

TsviBT

Intro written a day later:

Spiracular and I discuss the nature and origin of values. The dialogue doesn't get into much of a single clear thread, but I had fun writing it and hope it has some interesting trailheads for others.

(You may wish to skip the methodological exchange at the beginning; start instead at "Throwing some stuff out seems good!".)

Hello.

Spiracular

Hi! So, "Where do values come from?" and some of the animal behavior/human origins Qs probably bleed into that, right?

TsviBT

I have a desire to do something like "import stances.radical_philosophical_confusion", but I imagine this might be boring and not a short word in our shared language.

Spiracular

Is there a post? Or in the absence of that: What's the ~5-sentence version?

TsviBT

There's a post here: https://tsvibt.blogspot.com/2023/09/a-hermeneutic-net-for-agency.html
But like many of my posts, it's poorly written, in that it's the minimal post that gets the ideas down at all.

TsviBT

A short version is: A lot of my interest here is in reprogramming a bunch of related ideas that we bring with us when thinking / talking about values. I want to reprogram them (in my head, I mean) so that I can think about alignment.

Spiracular

Glazed over instantly on "hermeneutic net" in the first sentence, yeah. On trying to crunch on it anyway... things like bible interpretation personal-wikis? Or am I completely veering off on the wrong track?

TsviBT

The "hermeneutic" there is just saying, like, I want to bounce around back and forth between all the different concrete examples and concrete explicit concepts, and also the criteria that are being exerted on our concepts and our understanding of examples, where the criteria come from the big picture.

TsviBT

(I guess I'm uncomfortable talking at this meta level, largely because I imagine you as not being interested, even though you didn't say so.)

Spiracular

Okay, so "crosslinking" yes, de-emphasize the bible part, buff up the "dialogue" / differences-in-interpretation part, keep the "heirarchy feeding into questions" thing probably...

Spiracular

Uh... I'm feeling pretty okay, but I recognize I'm trying to do some kind of "see if I can short-circuit to an impression of a complex thing you're gesturing at" that might not work/might be really fundamentally off, and doing some weird social things to do it.

Spiracular

We can back out of it.

TsviBT

I think the sh

...
Labs should be explicit about why they are building AGI
by peterbarnett — 2023-10-17T21:09:20.711Z

Three of the big AI labs say that they care about alignment and that they think misaligned AI poses a potentially existential threat to humanity. These labs continue to try to build AGI. I think this is a very bad idea.

The leaders of the big labs are clear that they do not know how to build safe, aligned AGI. The current best plan is to punt the problem to a (different) AI,[1] and hope that can solve it. It seems clearly like a bad idea to try and build AGI when you don’t know how to control it, especially if you readily admit that misaligned AGI could cause extinction.

But there are certain reasons that make trying to build AGI a more reasonable thing to do, for example:

They want to build AGI first because they think this is better than if a less safety-focused lab builds it
They are worried about multi-polar scenarios
They are worried about competition from other nations, specifically from China
They think one needs to be able to play with the big models in order to align the bigger models, and there is some other factor which means we will soon have bigger models we need to align

I think the labs should be explicit that they are attempting to build AGI[2], and that this is not safe, but there are specific reasons that cause them to think that this is the best course of action. And if these specific reasons no longer hold then they will stop scaling or attempting to build AGI. They should be clear about what these reasons are. The labs should be explicit about this to the public and to policy makers.

I want a statement like:

We are attempting to build AGI, which is very dangerous and could cause human extinction. We are doing this because of the specific situation we are in.[3] We wish we didn’t have to do this, but given the state of the world, we feel like we have to do this, and that doing this reduces the chance of human extinction. If we were not in this specific situation, then we would stop attempting to build AGI. If we noticed [specific, verifiable observations about the world], then we would strongly consider stopping our attempt to build AGI.

Without statements like this, I think labs should not be surprised if others think they are recklessly trying to build AGI.

^

Either an automated alignment researcher, or something to do with scalable oversight

^

Or scale AI systems to levels that are not known to be safe

^

It is important that they actua

...
Eleuther releases Llemma: An Open Language Model For Mathematics
by mako yass — 2023-10-17T20:03:45.419Z
Link: https://blog.eleuther.ai/llemma/

Today we release Llemma: 7 billion and 34 billion parameter language models for mathematics. The Llemma models were initialized with Code Llama weights, then trained on the Proof-Pile II, a 55 billion token dataset of mathematical and scientific documents. The resulting models show improved mathematical capabilities, and can be adapted to various tasks through prompting or additional fine-tuning.

Our work parallels Minerva, a model suite specialized for quantitative reasoning developed by Google Research last year. While we don't achieve quite the same scale as Minerva, our Llemma models perform better on an equi-parameter basis. Moreover, we make our models and dataset open-access and our code open-source.

Language models with strong mathematical reasoning capabilities are upstream of a number of emerging research areas, such as reward modeling, algorithmic reasoning, and formal mathematics. We hope that by providing researchers with a much stronger base model for reasoning applications, Llemma will accelerate progress on these problems.

The code subset of the Proof-Pile-2 endows Llemma with capabilities Minerva lacks without additional finetuning. In this blog post, we'll discuss formal theorem proving. Our paper contains additional results on a Python-aided problem solving task.
Investigating the learning coefficient of modular addition: hackathon project
by Nina Panickssery — 2023-10-17T19:51:29.720Z

As our project at the Melbourne hackathon on Singular Learning Theory and alignment (Oct. 7-8), we did some experiments to estimate the learning coefficient of the single-layer modular addition task at a basin, an invariant that measures the information complexity (read: program length) of a fully trained neural net.

We used the recent paper of Lau, Murfet, and Wei as our starting point; this paper estimates provides a stochastic estimate for the learning coefficient (which they denote ^λ) via Langevin dynamics. The thermodynamic quantity measured by ^λ is proven to asymptotically converge to the learning coefficient for idealized singular systems in a beautiful paper by Watanabe.

All code for the experiments described can be found in this GitHub repository.

Brief results

In our tests, we were pleasantly surprised to find that, for the task of modular addition modulo a prime p, the outputs of our implementation of Lau et al.'s SGLD methods are (up to a roughly constant small multiplicative error, less than a factor of 2) in robust agreement with the theoretically predicted results for an idealized single-layer modular addition network[1].

Similar results have to date been obtained for a two-neuron network by Lau et al. and for a 12-neuron network by Chen et al. Our results are the first confirmation for medium-sized networks (between about 500 and 8,000 neurons) of the agreement between the estimate and the theoretical results.

While our results are off by a small multiplicative factor from the theoretical value for a single modular addition circuit, we discover a remarkably exact phenomenon that perfectly matches the theoretical predictions, namely that the learning coefficient estimate is linear in p for modular addition networks that generalize; this is the first precise scaling result of its kind.

Mean measurements of ^λ on modular addition networks for 5 different values of p: the degree of linearity is remarkable.

In addition, using the modular addition task as a test case lets us closely investigate the ability of the ^λ complexity estimate to differentiate generalization and memorization in neural networks: something that seems to be mostly new (though related to some of the phase transition phenomena in Chen et al.). We observe that while generalization has linear learning coef...
Worldwork for Ethics
by False Name — 2023-10-17T18:55:31.977Z

Abstract: An alternative to the now-predominating models of alignment, corrigibility and "CEV", following a critique of these. The critique to show, in substance: CEV and corrigibility have the exact same problems - in effect, they're isomorphs of one another, and each equally unobtainable. This briefly shown, and then, in flat contradiction to point 22 of “AGI Ruin: A List of Lethalities”, there is a quite different way to characterize, so achieve, alignment, via a refutation of Kant’s supposedly irrefutable categorical imperative which refutation also is included; from this, an ethic designed to be intrinsically applicable for any volitional, so by assumption algorithmic, behavior altogether. Suggestions for implementation of such also included.

Epistemic status: If this argument did not seem more true than anything else, this author would not now be alive to write it. It is intuitively true, and, reasoned such that no refutation is obvious. Posting it here, and again, is in hopes of a critique, even a refutation that it has not yet been given, perhaps because it’s So Bad It’s Not Even Wrong; if so on your examination, then please write to say so. That done, next steps could go through very quickly. For, whereas it has always and still seems true, and important – it is no longer so important that one can base a life upon it, if it cannot be lived-for.

Anthropic-affecting alignment strategies

We begin by considering the cause of Yudkowsky's despair, in failing to make usable CEV or corrigibility; thus because they're functionally the same, or at least, they lead to the same problem. The method which follows, then, is not "door number three" relative to what the "List of Lethalities" calls the "only options" for alignment; following the critique of present approaches (and this is only one such, informal, refutation of CEV and corrigibility’s efficacy), is a second way.

CEV is designed to result in an at-once manifested fulfillment of human wants – and that in principle these could be explicitly represented, if only by the AI after it realizes what they are and instantiates them, á la Anthropic’s Constitutional Alignment. Corrigibility by design is to singly or multiply preclude any action by the AI that would itself preclude human deactivation of the AI, or human suspension of the AI’s activities.

Observe then, that by assumption a corr...
When building an organization, there are lots of ways to prevent financial corruption of personnel. But what are the ways to prevent corruption via social status, political power, etc.?
by M. Y. Zuo — 2023-10-17T18:51:47.127Z

Nowadays, even fairly sophisticated accounting and auditing concepts are well known by those with on-the-job work experience in any sizeable firm.

Such as systems for controlling the flow of money from spending accounts, corporate cards, etc., procedures for leaving a highly legible audit trail, and so on.

And to a fairly large extent, these methods are successful in preventing the anyone from messing with financial numbers in too obvious a manner, at least not without leaving a lot of warning signs along the way.

However, in the realm of controlling for those unduly desiring of social status or political power, there seem to be a much sparser set of methods.

Most of the ones discussed in popular literature only work at a large scale, such as parliamentary procedures, multiple branches of government checking and balancing each other, etc...

What are some of the proven methods that work at the scale of an average organization?

(say 100 to 10 000 people)

And if there aren't any, what are some of the theoretical proposals?
Eliezer's example on Bayesian statistics is wr... oops!
by Zane — 2023-10-17T18:38:18.327Z

This post was going to be an explanation of how an example Eliezer Yudkowsky frequently uses in discussion of statistics doesn't actually imply what he thinks it does. In the process of proving his mistake, I found out I was actually wrong. I'm still writing the post up, and it might end up interesting to someone else making the same mistake as me.

The example in question is one that Eliezer often uses when arguing that Bayesian statistics are superior to frequentist ones. Here in the Sequences, here on Arbital, here in Glowfic, and a couple other times I can't find at the moment. A coin is flipped six times; it comes up heads the first five times, and tails the sixth. But we don't know whether the experimenter had A) decided beforehand to flip the coin six times, and reported what happened, or B) decided beforehand to flip the coin over and over until it came up tails, and report however long that took. What information does this give us about the coin's bias, or lack thereof?

According to a frequentist perspective, there's a serious difference in the p-values one would assign in these two different scenarios. In the first case, the result HHHHHT is placed into the class of results that are at least that extreme relative to the "null hypothesis" - of which there are 14: HHHHHH, TTTTTT, HHHHHT, TTTTTH, HHHHTH, TTTTHT, and so on. With 64 possibilities in total, 14/64 = 0.22, which is far above the p=0.05 level and therefore not enough to conclude significance. In the second case, the result HHHHHT is instead placed into a different class of results: 5 heads and then a tail, 6 heads and then a tail, 7 heads and then a tail, and so on forever. The probability of that entire class is 1/32 in total, which is 0.03 - statistically significant!

Eliezer criticizes this in several ways. To start out with, the part where a frequentist decides to lump in the actual result with a group of results that are similar is subjective enough to allow for a lot of freedom in what the actual result is. Maybe in the first case, instead of choosing the class of results with 5 or more of the same side, you only choose the class of results with 5 or more heads in particular, thus halving the p=0.22 to p=0.11. He also criticizes the very notion of significance being determined by "rejecting the null hypothesis" rather than looking at different theoretical effect sizes and how well they wo...
Trying to deconfuse some core AI x-risk problems
by habryka — 2023-10-17T18:36:56.189Z

Max H commented on the dialogues announcement, and with both of us being interested in having a conversation where we try to do something like "explore the basic case for AI X-risk, without much of any reference to long-chained existing explanations". I've found conversations like this valuable in the past, so we decided to give it a shot.

His opening comment was:

I feel like I've been getting into the weeds lately, or watching others get into the weeds, on how various recent alignment and capabilities developments affect what the near future will look like, e.g. how difficult particular known alignment sub-problems are likely to be or what solutions for them might look like, how right various peoples' past predictions and models were, etc.

And to me, a lot of these results and arguments look mostly irrelevant to the core AI x-risk argument, for which the conclusion is that once you have something actually smarter than humans hanging around, literally everyone drops dead shortly afterwards, unless a lot of things before then have gone right in a complicated way.

(Some of these developments might have big implications for how things are likely to go before we get to the simultaneous-death point, e.g. by affecting the likelihood that we screw up earlier and things go off the rails in some less predictable way.)

But basically everything we've recently seen looks like it is about the character of mind-space and the manipulability of minds in the below-human-level region, and this just feels to me like a very interesting distraction most of the time.

In a dialogue, I'd be interested in fleshing out why I think a lot of results about below-human-level minds are likely to be irrelevant, and where we can look for better arguments and intuitions instead. I also wouldn't mind recapitulating (my view of) the core AI x-risk argument, though I expect I have fewer novel things to say on that, and the non-novel things I'd say are probably already better said elsewhere by others.

I am quite interested in the "In a dialogue, I'd be interested in fleshing out why I think a lot of results about below-human-level minds are likely to be irrelevant, and where we can look for better arguments and intuitions instead." part.

Do you want to start us off with a quick summary of your take here?

habryka

So, I think there's a lot of alignment discussion that is missing a kind of induct

...
Being patient with problems
by Henrik Karlsson — 2023-10-17T18:30:05.787Z
Link: https://www.henrikkarlsson.xyz/p/being-patient-with-problems

In the writings of mathematician Alexander Grothendieck, there is a recurring discussion of two opposing strategies when confronting problems. One strategy, which he saw as the default among mathematicians and lamented leaning on early in his career, was to “attack problems,” that is, directly work to prove what you have set out to prove. This is how I did math when I was younger. Handed a problem, I would attempt the first thing that came to mind; if that failed I would go for another approach—trying increasingly thoughtless tactics until the problem, through sheer force, cracked open. Then I would move on.

The other approach, which Grothendieck preferred, he famously illustrated with a metaphor of a rising sea:

The unknown thing to be known appeared to me as some stretch of earth or hard marl, resisting penetration… the sea advances insensibly in silence, nothing seems to happen, nothing moves, the water is so far off you hardly hear it.. yet it finally surrounds the resistant substance.

Practically speaking, what he did was explore the problem in an open-ended way, building intuition, introducing notation—growing an entire theory around the problem. This could go on for years. Until one day, the solution had become self-evident, a natural consequence of the theory. The problem essentially disappeared. It had been swallowed by the sea.

These ways of working on problems, going fast or slow, can be seen in all domains. Some writers go as fast as they can type, others take years understanding the world they will describe. Some house owners plant their gardens with whatever flowers catch their fancy at the store, others study the soil, the local fauna, the way the sun moves through the day . . .

There is an obvious drawback to going deep: the solutions may take a long time to surface. Grothendieck was famous for his willingness to spend a long time on a problem. He was insanely productive (averaging the equivalent of one paper a day for twelve years!) but often what he chose to work on was not ideal from the perspective of an academic career. He wasn’t optimizing for short term wins. He wouldn’t write up proofs to important theorems if he felt like he didn’t understand it properly, or if he felt that it could wait until its proper place in the larger theory had been reached. Sometimes, after years of waiting, his theorems would have to be written up and ...
Rationality Club at UChicago
by Noah Birnbaum — 2023-10-17T17:38:15.400Z

Hi!

I plan on starting a rationality student group at UChicago based off of sites like LW, MR, ect and more general insights on rationality in order to improve decision making and thought in people. Given that there is so much demand for groups like EA, I think may be way undervalued and highly impactful.

I created this post because I thought y'all may be able to give me good advice. If anyone has any ideas about how this should be structured or literally anything else that can help (and perhaps arguments in favor of your idea), I would love to hear it. One particular question I've been trying to answer is how to get good data on whether people are actually becoming more rational--self surveying does not seem like a good way to measure that type of thing (although it would probably be better than nothing).

(Also, if you're interested and know some basic web design stuff that would be super cool 😎).

If you have any experience with doing something like this (rationality "movement building?" or something similar), I would love to hop on a call or just get a few words of advice.

If you are doing something similar at a different university or would like to, I would also love to talk.

If you're in UChicago and would like to join or be an officer (undergrad, graduate, anything) or are just interested, send me a pm, and we can talk.
Road to Proto-AGI 2024
by Lucas Heinrich — 2023-10-17T17:37:34.260Z

Introduction

As we approach the final quarter of 2023, it's worth pausing to consider the likely trajectory of artificial intelligence, particularly in the context of its potential convergence toward Artificial General Intelligence (AGI). This post will attempt to make probabilistic predictions for the technological and social landscape of AI in 2024. Given the rapid advancements in multimodal capabilities, code interpretation in models like GPT-4, we stand at the brink of what might be a seismic shift in AI capabilities.

The Near Future:

Autonomous AI and Corporate Rivalry

GPT-4, released by OpenAI, serves as an important milestone. It has vision capabilities and can interpret code, two significant steps toward higher-level cognitive abilities. What's even more intriguing is the notion that OpenAI experimented with long-term memory [1] in its AI models but hasn't released it officially yet. If all these components are integrated into a unified, autonomous agent, we may be looking at a proto-AGI.

Many seem to have forgotten that German Microsoft CTO leaked details about GPT-4's capabilities, hinting at its potential for video understanding. [2] This raises an interesting question: Is OpenAI deliberately constraining the true capabilities of their technology?

On the other hand, Google has had ample time to study GPT-4 and will release its own exceptional AI model, tentatively called "Gemini." It could either be a leap forward in multimodal understanding or a complete disappointment.

Predictions for 2024

The Arrival of Meta's Llama 3:
Meta will release an opensource AI model, Llama 3, with about 90 billion parameters, equivalent to ChatGPT 3.5 and even better at some tasks. This will be a major competitive move against OpenAI's models.

Google Gemini:
Will come in several sizes. Most of them a free to use, as they will get incorporated in Google products anyways. There will be a very big one that you can use as a subscriber (like ChatGPT Plus), and the biggest one will be available with a waiting list and not yet opened to the public.
The biggest (closed) one will be at the level of OpenAIs 2024 model, while the one you can subscribe to will be around GPT-4 level, but with more capabilities right from the start.

OpenAI's Next Model:
In the wake of Google's Gemini (be it a success or a failure), OpenAI will be highly motivated to push the boundaries. Their next m

...