trentmkelly/LessWrong-43k huggingface dataset:
Accidentally Load Bearing
by jefftk — 2023-07-13T16:10:00.806Z

Sometimes people will talk about Chesterton's Fence, the idea that if
you want to change something—removing an apparently useless
fence—you should first determine why it was set up that way:

The gate or fence did not grow there. It was not set up by
somnambulists who built it in their sleep. It is highly improbable
that it was put there by escaped lunatics who were for some reason
loose in the street. Some person had some reason for thinking it would
be a good thing for somebody. And until we know what the reason was,
we really cannot judge whether the reason was reasonable. It is
extremely probable that we have overlooked some whole aspect of the
question, if something set up by human beings like ourselves seems to
be entirely meaningless and mysterious. — G. K. Chesterton, The
Drift From Domesticity

Figuring out something's designed purpose can be helpful in evaluating
changes, but a risk is that it puts you in a frame of mind where what
matters is the role the original builders intended.

A few years ago I was rebuilding a bathroom in our
house, and there was a vertical stud that was in the way. I could
easily tell why it was there: it was part of a partition for a closet.
And since I knew its designed purpose and no longer needed it for
that anymore, the Chesterton's Fence framing would suggest that it was
fine to remove it. Except that over time it had become accidentally
load bearing: through other (ill conceived) changes to the structure
this stud was now helping hold up the second floor of the house. In
addition to considering why something was created, you also need to
consider what additional purposes it may have since come to serve.

This is a concept I've run into a lot when making changes to complex
computer systems. It's useful to look back through the change
history, read original design documents, and understand why a
component was built the way it was. But you also need to look closely
at how the component integrates into the system today, where it can
easily have taken on additional roles.
AI #20: Code Interpreter and Claude 2.0 for Everyone
by Zvi — 2023-07-13T14:00:08.266Z

There is no shortage of people willing to talk about every week as a huge week in AI with tons of amazing new releases and announcements. At first this was usually right, then it mostly stopped being right. This week we had Code Interpreter and Claude 2.0 and x.AI (whatever that actually is) and a bunch of other stuff, while I was still processing OpenAI’s huge announcement of their Superalignment Taskforce. As usual, when there’s lots of great stuff that makes it that much harder to actually find time to play around with it. Things are good and also can be overwhelming.

I will be in Seattle this weekend, arriving Sunday and leaving Tuesday. If you’d like to say hello while I am there, let me know. I know better than to request things get quiet.

Also on a non-AI note, congratulations to Nate Silver on a great run at the World Series of Poker main event, whole tournament has been a blast to watch especially when Rigby is at the table, he’s crazy good.

And on another non-AI note, I’d like to highlight the Roots of Progress Blog-Building Intensive. We need more voices for progress generally, including to ensure AI goes well. Deadline is August 11. I am a big believer in blogging in particular and writing in general, as a way to think and understand the world, and also to share your knowledge with and connect with others, and increase your serendipity factor greatly. You need to be a good philosophical fit, but if you are, this is a great opportunity. The talent you’ll be working with will be top notch.

Table of Contents

Introduction.

Table of Contents.

Code Interpreter. Data science marvel now available. Minor flaws.

Language Models Offer Mundane Utility. Ask and ye shall receive.

Language Models Don’t Offer Mundane Utility. Know your limitations.

Deepfaketown and Botpocalypse Soon. When in doubt, let the police handle it.

The Art of the Super Prompt. Code interpreter jailbreak hype.

They Took Our Jobs. The question is which ones.

Claude 2.0. I Estimate a gain of perhaps 0.15 GPTs from Claude 1.3.

Introducing. So what is x.AI? That’s the thing, no one knows. Some sort of math.

In Other AI News. People are still publishing their capabilities papers. You fool!

Quiet Speculations. Are you are modeling transformative AI, or mundane AI?

The Quest for Sane Regulation. DeepMind paper on this is good as far as it goes.

The

...
How can I get help becoming a better rationalist?
by TeaTieAndHat — 2023-07-13T13:41:46.670Z

Well…

Right now, being ‘a rationalist’ could be said to be a massive part of my identity, at least judging by the absurd amount of time I’ve spent reading posts here, or SSC/ACX, and in a few other places. Yet, I’m still a mere lurker unfamiliar with most of the local customs.

But it’s not what matters. What does is that I’m a terrible rationalist.
You see, rationality takes practice. And reading stuff on LW isn’t practice at all. If anything, it’s just a great way of filling my brain with a lot of useful concepts, and then either blame myself for not using them, or use them for something entirely unrelated to their normal purpose. Often, to make myself feel worse, and think worse.

As the saying goes, rationality is a martial art. Learning it by reading the rules, or by watching other people apply the rules, is about as effective as developing one’s muscles by watching sports on TV.

I know of the CFAR, and of various related groups, meetups for ACX readers or for other people, etc. But, apart from ACX meetups, which aren’t about being better rationalists per se, I don’t have easy access to any of those, or certainly to a general environment which welcomes this. You know, not being in the Bay Area and all.

And yet, I want to be more rational as much as anyone who’s been lurking here for five years wants it, and given how depressed I was until very recently, I probably badly need it, too.

I’m not sure what kind of answers I expect, but, like, how can I push myself to learn more, and especially to practice more, and ideally to actually use rationality to improve my life?
i love eating trash
by Ace Delgado — 2023-07-13T11:23:00.812Z

It was my recent pleasure to come across the best thing I have yet seen on LessWrong. No, it's not some eye-opening post on a niche AI topic full of mathematical notation, nor another monument to Quinean Naturalism - it was the moderation log, found here: https://www.lesswrong.com/moderation. I cannot recommend enough that you go check it out. Like sorting YouTube results by 'new', it offers you platform perspective like nothing else.

Most wonderfully, this includes the newly created 'Rejected Posts' section, where untold marvels are to be found. This is truly the trash-heap of LessWrong, and I am all for consuming. I love eating trash.

Why, just look at this groundbreaking piece on AI: Exploring Last-Resort Measures for AI Alignment: Humanity's Extinction Switch. It is simply bursting with ideas that research, until now, has neglected to consider!

Here's a really engaging post: Hands of gods. A totally novel movie reference to start, and a great reminder from our tireless admins that, whilst writing a post designed for LessWrong and then copy-paste cross-posting to other platforms is perfectly good, the inverse is simply impermissible. After all, where would we be without such tools of cultural aggression!

Or what about this? This one's my personal favourite - a truly heartwarming tribute to the strongest of positive feelings we can have: I love my wife

So, go, go! Consume to your heart's content! Feast upon this heap of wonder!
Elon Musk announces xAI
by Jan_Kulveit — 2023-07-13T09:01:01.278Z
Link: https://www.ft.com/content/2a96995b-c799-4281-8b60-b235e84aefe4

Some quotes & few personal opinions:
FT reports

Musk is also in discussions with a number of investors in SpaceX and Tesla about putting money into his new venture, said a person with direct knowledge of the talks. “A bunch of people are investing in it . . . it’s real and they are excited about it,” the person said.
...
Musk recently changed the name of Twitter to X Corp in company filings, as part of his plans to create an “everything app” under the brand “X”. For the new project, Musk has secured thousands of high-powered GPU processors from Nvidia, said people with knowledge of the move.
...
During a Twitter Spaces interview this week, Musk was asked about a Business Insider report that Twitter had bought as many as 10,000 Nvidia GPUs, “It seems like everyone and their dog is buying GPUs at this point,” Musk said. “Twitter and Tesla are certainly buying GPUs.” People familiar with Musk’s thinking say his new AI venture is separate from his other companies, though it could use Twitter content as data to train its language model and tap Tesla for computing resources.

According to xAI website, the initial team is composed of
Elon Musk

Igor Babuschkin

Manuel Kroiss

Yuhuai (Tony) Wu

Christian Szegedy

Jimmy Ba

Toby Pohlen

Ross Nordeen

Kyle Kosic

Greg Yang

Guodong Zhang

Zihang Dai

and they are "advised by Dan Hendrycks, who currently serves as the director of the Center for AI Safety."

According to reports xAI will seek to create a "maximally curious" AI, and this also seems to be the main new idea how to solve safety, with Musk explaining: "If it tried to understand the true nature of the universe, that's actually the best thing that I can come up with from an AI safety standpoint," ... "I think it is going to be pro-humanity from the standpoint that humanity is just much more interesting than not-humanity."
My personal comments:
Sorry, but at face value, this just does not seem a great plan from safety perspective. Similarly to Elon Musk's previous big bet how to make us safe by making AI open-source and widely distributed ("giving everyone access to new ideas").
Sorry, but given "Center for AI Safety" moves to put them into some sort of "Center", public representative position of AI Safety - including the name choice, and organizing the widely reported Statement on AI risk - it seems publicly associating their brand with xAI is a strange choice.
The intelligence-sentience orthogonality thesis
by Ben Smith — 2023-07-13T06:55:59.353Z

You're familiar with the orthogonality thesis. Goals and intelligence are more or less independent for one another. There's no reason to suspect an intelligent AI will also be an aligned AI because it is intelligent.

Popular writing suggests to me there's a need to clear up some other orthogonality theses as well. We could propose:

The intelligence-sentience orthogonality thesis
The sentience-agency orthogonality thesis
The self-reflection-sentience orthogonality thesis

What I mean by "sentience" and "intelligence"

In this article I generally use the word sentience to describe the state of experiencing qualia, or the property a mind has that means there is "something that it is like to be" that mind. I use that term because many people have conflicting ideas about what consciousness means. When I use the term "consciousness", I try to be specific about what I mean, and typically I use the word while explaining other people's ideas about it.

When I use the word "intelligence" I mean the ability to represent information abstractly and draw inferences from it to complete tasks, or at least to conceptualize how a task can be completed. This is close to the g factor being measured in intelligence tests, which measures how a human or other agent can recognize patterns and draw inferences about them. It's also close to what we mean by "artificial intelligence".

Mind, intelligence, self-awareness, and sentience as overlapping Euler sets

It might be helpful to represent each of concepts like mind, intelligence, self-awareness, agency, and sentience as overlapping sets on a Euler diagram. We might conceptualize an entity possessing any of the above to be a kind of mind. Self-awareness and agency plausibly must be forms of intelligence, whereas arguably there are simple qualia experiences that do not require intelligence--perhaps a pure experience of blueness or pain or pleasure does not require any soft of intelligence at all. Finally, we can imagine entities that have some or all of the above.

The Euler diagram shows specific examples of where various minds are positioned on the chart. I'm pretty low-confidence about many of these placements. The main point to take away from the diagram is that when thinking about minds one should get in the habit of imagining various capacities separately, and not assuming that all capacities of the human mind will go together in other minds....
Eric Michaud on the Quantization Model of Neural Scaling, Interpretability and Grokking
by Michaël Trazzi — 2023-07-12T22:45:45.753Z
Link: https://theinsideview.ai/eric

Eric is a PhD student in the Department of Physics at MIT working with Max Tegmark on improving our scientific/theoretical understanding of deep learning -- understanding what deep neural networks do internally and why they work so well.

We mostly talk about Eric's paper, The Quantization Model of Neural Scaling, but also two papers he recently published on Grokking, Towards Understanding Grokking: an effective theory of representation learning, and Omnigrok: Grokking Beyond Algorithmic Data.

Below are some highlighted quotes from our conversation (available on Youtube,

Spotify

,

Google Podcast

,

Apple Podcast

). For the full context for each of these quotes, you can find the accompanying

transcript

.

On The Quantization Of Neural Scaling

"The name of the paper is the quantization model of neural scaling. And the one-tweet summary is that it's possible for smooth loss curves on average to average over lots of small, discrete phase changes in the network performance.

What if there were a bunch of things that you need to learn to do prediction well in something language? And so these things could be pieces of knowledge or different abilities to perform certain types of specific computations.

We can imagine enumerating this set of things that you need to learn to do prediction well. And we call these the quanta of the prediction problem. And then what if the frequency in natural data that these were useful, each of these quanta, each of these pieces of knowledge or computational ability, what if the frequency that they were useful for prediction followed a power law?" (context)

Quantas are the smallest clusters for simple subtasks

"In order to predict the new line, has to count line lengths for the previous lines in the document. And then it's able to use that to accurately predict when a new line should be present.

And you can find just a large number of clusters where the thing that is common between the clusters just seems to be that it's the same type of problem, or doing prediction on those samples requires the same piece of knowledge. And so you might call these the quanta, or evidence of there being quanta, although it's a little bit tricky, because we, in doing the clustering, enforce this discreteness, where everything is a member of a cluster, a particular cluster, and not another cluster.

Anyway, it's complicated and weird. Wh

...
Are there any good, easy-to-understand examples of cases where statistical causal network discovery worked well in practice?
by tailcalled — 2023-07-12T22:08:59.916Z

When I first read the Sequences, one of the exciting posts was Causal Diagrams and Causal Models, which got me into the idea that one could discover the structure of causal networks using statistics. Another rationalist source which gave me similar hopes was Scott Alexander's SSC Journal Club: Mental Disorders As Networks.

However, when I actually started applying these techniques to my own data, or to publicly available datasets, I often found that the techniques were unstable, and that one could easily infer plausible conditions where they would give the wrong results. It's possible I had the wrong approach or something, but in my confusion I started reading up on what experts in causal inference had said, and I got the impression that they studied the problem for a while, initially finding some algorithms, but over time concluding that their algorithms didn't work very well and that it is better to just have a human in the loop who specifies the causal networks.

So I mostly abandoned it, or saw it as a much more limited tool than I had before. But recently, John Wentworth argued that it was actually quite feasible in practice, so maybe I was too quick to abandon it. I would like to know - what are the best examples of this working well in practice? Or alternatively, did anyone else come to the same conclusions as I did?
The Opt-In Revolution — My vision of a positive future with ASI (An experiment with LLM storytelling)
by Tachikoma — 2023-07-12T21:08:22.537Z

I writing and publishing a story to better clarify my vision for a positive future with transformative AI (or AGI/ASI). The story is written in chapters, and is still being put together piece by piece. Each chapter is split into the LLM generated version (primarily Claude v1.3) and following that the original version that I wrote. I think that more detail is better, especially since a target audience for it are future AIs like LLMs which would have no trouble reading it all.

Dawn of the Final Day – The Opt-In Revolution (elsewhereunbound.com)

My reasoning or motivation for this project is as follows:

It seems wrong that it should be so easy to prompt ChatGPT or people to imagine what a future with unaligned ASI might look like while the opposite is much harder. More attention is focused on bad outcomes, instead of good outcomes, to be fair with good reason—it’s important to avoid the worst outcomes. But I think it’s also important that thought go into what good outcomes could or more importantly, should be. It is not clear to me what people in the AI communities think a good future looks like should we develop Aligned AGI. There is a vague idea that with aligned AGI many of the things we care about will improve, but how and for whom is left vague. This story tries to provide more concrete answers that can be debated over.

I read somewhere that ChatGPT knows more about what Gondor is like compared to Wichita, Kansas even though the latter is fictional because there’s more written depictions and discussion about it. That’s what I want to achieve with this project, but more broadly by outlining what a future with Aligned ASI looks like.

It’s my belief that it is easier to imagine the end of the world than how the world might be positively transformed by AGI. Creation is harder than destruction because destruction is the closing off of possibilities while creation begets more creation and widens the space of possibility. This is another reason to focus on imagining a positive future, because it is harder to do.

I think we should express our personal belief or conception of an ideal future with Aligned AGI to process potential disagreements constructively and generate a common consensus that can feed into the development and training of future AIs. After all, if we can’t concretely imagine what a future with aligned AI is like and the decisions they make, why sho

...
What does the launch of x.ai mean for AI Safety?
by Chris_Leong — 2023-07-12T19:42:47.060Z

x.ai has now launched. It seems worthwhile to discuss both what it means for AI safety and whether people interested in AI safety should consider applying for the company.

Some thoughts:

It's notable that Dan Hendrycks is listed as an advisor (the only advisor listed).

The team is also listed on the page.
I haven't taken the time to do so, but it might be informative for someone to Google the individuals listed to discover whether they lie in terms of their interest being between capabilities and safety.
Towards Developmental Interpretability
by Jesse Hoogland — 2023-07-12T19:33:44.788Z

Developmental interpretability is a research agenda that has grown out of a meeting of the Singular Learning Theory (SLT) and AI alignment communities. To mark the completion of the first SLT & AI alignment summit we have prepared this document as an outline of the key ideas.

As the name suggests, developmental interpretability (or "devinterp") is inspired by recent progress in the field of mechanistic interpretability, specifically work on phase transitions in neural networks and their relation to internal structure. Our two main motivating examples are the work by Olsson et al. on In-context Learning and Induction Heads and the work by Elhage et al. on Toy Models of Superposition.

Developmental interpretability studies how structure incrementally
emerges through phase transitions during training.

Mechanistic interpretability emphasizes features and circuits as the fundamental units of analysis and usually aims at understanding a fully trained neural network. In contrast, developmental interpretability:

is organized around phases and phase transitions as defined mathematically in SLT, and
aims at an incremental understanding of the development of internal structure in neural networks, one phase transition at a time.

The hope is that an understanding of phase transitions, integrated over the course of training, will provide a new way of looking at the computational and logical structure of the final trained network. We term this developmental interpretability because of the parallel with developmental biology, which aims to understand the final state of a different class of complex self-assembling systems (living organisms) by analyzing the key steps in development from an embryonic state.[1]

In the rest of this post, we explain why we focus on phase transitions, the relevance of SLT, and how we see developmental interpretability contributing to AI alignment.

Thank you to @DanielFilan, @bilalchughtai, @Liam Carroll for reviewing early drafts of this document.

Why phase transitions?

First of all, they exist: there is a growing understanding that there are many kinds of phase transitions in deep learning. For developmental interpretability, the most important kind of phase transitions are those that occur during training. Some of the examples we are most excited about:

Olsson, et al., "In-context Learning and Induction Heads", Transformer Circuits Th

...
Flowchart: How might rogue AIs defeat all humans?
by Aryeh Englander — 2023-07-12T19:23:59.720Z

Epistemic status: I put this together for myself mostly off the top of my head without carefully cross-referencing other sources. I'm hoping to flesh this out a lot more and do a much more thorough research job of it at some point, but in the meantime I've decided to err on the side of getting this out there even in the current unpolished state.

The following is a rough draft version of a Miro flowchart I put together for how a misaligned rogue AI or group of AIs might actually take over the world and cause kill all humans. I think it should be mostly self-explanatory, even if vague in places.

Partly this was inspired by Carl Shluman's recent interview on the Lunar Society podcast, but I haven't actually checked yet how closely this matches with Carl's discussion.

Link to the Miro board

1) Paths towards human extinction

2) How exactly could rogue AIs defeat all humans?

3) How could misaligned AIs gain access to / control over the resources needed to defeat all humans?