trentmkelly/LessWrong-43k huggingface dataset:
Effective Altruists and Rationalists Views & The case for using marketing to highlight AI risks.
by gilch — 2024-04-19T04:16:15.016Z
Link: https://youtu.be/dGfyF7lU1Qo?t=4186

The link is to a particular timestamp in a much longer podcast episode. This segment plays immediately after the (Nonlinear co-founder) Kat Woods interview. (Skipping over the part about requesting donations.) In it, the podcast host John Sherman specifically calls out the apparent lack of instrumental rationality on the part of the Rationalist and Effective Altruism communities when it comes to stopping our impending AI doom. In particular, our reluctance to use the Dark Arts, or at least symmetric weapons (like "marketing"), in the interest of maintaining our epistemic "purity".

(For those not yet aware, Sherman was persuaded by Yudkowsky's TIME article and created the For Humanity Podcast in an effort to spread the word about AI x-risk and thereby reduce it. This is an excerpt from Episode #24, the latest at the time of writing.)

I have my own thoughts about this, but I'm not fully aware of trends in the broader community, so I thought I'd create a space for discussion. Is the criticism fair? Are there any Rationalist/EA projects Sherman is unaware of that might change his mind? Have we failed? Are we just not winning hard enough? Should we change? If so, what should we change?

My (initial) Thoughts

I'm less involved with the EA side, but I feel that LessWrong in particular is a bastion of sanity in a mad world, and this is worth protecting, even if that means that LessWrong proper doesn't get much done. Maxims like "Aim to explain, not persuade" are good for our collective epistemics, but also seem like a prohibition on prerequisites to collective action.

I think this is fine? Politics easily become toxic; they risk poisoning the well. There's no prohibition on rationalists building action- or single issue–focused institutions outside of LessWrong. There have been reports of people doing this. (I even kind of co-founded one, starting from LessWrong, but it's not super active.) Announcing what they're starting, doing postmortems on how things went, or explaining organizational principles seem totally kosher for LessWrong to me. I feel like I'm seeing some of this happening too, but maybe not enough?

What I'm not seeing is any kind of pipeline on skilling up our group rationality, especially of the instrumental flavor. Not to say there's been zero effort.

Also, I'm personally not a marketer, or even very skilled socially. The kind of action Sherman seems to be as...
Cohesion and business problems
by Adam Zerner — 2024-04-19T00:45:00.269Z

In software development there is a concept called cohesion.

It works like this. Suppose you have the following functions:[1]

function getArea(radius) { ... }

function getCircumference(radius) { ... }

function sendWelcomeEmail(user) { ... }

function updatePassword(user, newPassword) { ... }

function getTemperatureInFahrenheit(temperatureInCelsius) { ... }

function getTemperatureInCelsius(temperatureInFahrenheit) { ... }

You want to group similar functions together. Suppose you created the following modules:

// module-one.js
export function getArea(radius) { ... }
export function sendWelcomeEmail(user) { ... }

// module-two.js
export function getCircumference(radius) { ... }
export function getTemperatureInFahrenheit(temperatureInCelsius) { ... }

// module-three.js
export function updatePassword(user, newPassword) { ... }
export function getTemperatureInCelsius(temperatureInFahrenheit) { ... }

This wouldn't make sense. The modules would each have a low degree of cohesion since we grouped unrelated functions together, and this is undesirable.

Now imagine that we did this instead:

// geometry.js
export function getArea(radius) { ... }
export function getCircumference(radius) { ... }

// user.js
export function sendWelcomeEmail(user) { ... }
export function updatePassword(user, newPassword) { ... }

// temperature.js
export function getTemperatureInFahrenheit(temperatureInCelsius) { ... }
export function getTemperatureInCelsius(temperatureInFahrenheit) { ... }

Much better, right? Now we have grouped like with like. Now our modules each have a high degree of cohesion. The functions in geometry.js are each related to geometry, the functions in user.js are each related to users, and the functions in temperature.js are each related to temperature. This is desirable.

The concept of cohesion applies beyond programming though. Consider restaurants:

Appetizing Alice's:
- Pad thai
- Chicken parm

Bob's Bistro:
- Meatball sub
- Pork tacos

Carnivore Carol's:
- Tom kah gai
- Carne asada

Low cohesion. What about this?

Thai on Main:
- Pad thai
- Tom kah gai

Italian on Third:
- Chicken parm
- Meatball sub

Mexican by the river:
- Pork tacos
- Carne asada

High cohesion. Now we've grouped like with like.

You could do the same thing with stores. A convenience store has low cohesion whereas a book store has high cohesion. Relatedly, a large grocery store has lower cohesion than a high-e...
The Thermodynamics of Death
by Peter lawless  — 2024-04-19T00:36:23.762Z

by M.C. Price [Editor: This article is reprinted from Extropy #6, Summer 1990. Extropy was published by The Extropy institute]

First this little planet with its winds and ways, and then all the laws of mind and matter that restrain him. Then the planets about him, and at last out across immensity to the stars. And when he has conquered all the deeps of space and all the mysteries of time, still he will be beginning.

- The Shape of Things to Come.

Cataclysmic Futures

As an immortalist I want to live forever. Not just for a thousand or a googolplex of years but forever. It is the almost universal opinion of scientists and philosophers that the universe we currently live in will become uninhabitable with the passage of time, and this is accepted at face value by many people. Such people can dismiss the idea of personal immortality because they see it as incompatible with a universe in which life is an ephemeral phase. This article will expose the total falsity of this mindset, by the simple expedient of applying a modicum of logic to this situation. Pessimistic eschatological visions of the future have traditionally appealed more to the human mind than utopian visions. In Scandinavian mythology all is lost at the end of time as the Frost Giants wage war against the gods, the Fenris-Wolf is loosed and the World-Worm awakens, leading to the destruction of the three worlds of gods, dwarfs, and men in the cataclysmic fire of Ragnarok. Hinduism has a ruling trinity of the Creator, Preserver, and Destroyer; the latter will destroy the world in an orgy of flame at the appointed hour. Early Christianity revelled in the imminent destruction of the world, as set out in Revelations, until its adoption by the emperor Constantine as a state religion, and is still enthused about in almost every crackpot fundamentalist sect, permeating many facets of Western culture.

Turning to more serious, (if not quite so graphic) visions, there are some projections of the future that we can dismiss because they ignore the reality of the technological progress (replicator technology coupled with super intelligence will vastly expand the resources available for coping with inimical changes in the environment). Scenarios that fall into th...
Backyard Office
by jefftk — 2024-04-19T00:31:01.924Z

In 2020 I renovated the small building in our backyard which had

fallen into
disrepair. It was zoned for use as a home office, and had
electric but not plumbing. I wrote about how I was

thinking
about insulating it and

comparing framing
options but then apparently I never got around to writing up how I
finished it!

I hired someone to replace the roof:

Doesn't look like I have a picture of the top, but it's rubber
membrane.

I hired them to put in a window as well. If I'd realized how much
space would be lost to casing I'd have asked the mason to make a
larger window hole.

Plans for the walls and floor:

Covering the walls and floor in 2" foam:

Anna helped:

The floor is one layer of OSB, then one layer of plywood, screwed to
each other but floating:

Vapor barrier around the top, and 2x3s the flat way to attach the
drywall to. I used fiberglass batts to insulate the roof:

One more layer of foam, around everything.

Help from Lily:

Drywalling it all:

Casing the old windows. This was annoying since nothing was quite
square.

Finished!

A major thing I liked about this house project is that no one was
depending on it being done at any specific time, so I could work on it
when I had free time.

Now one of our tenants uses it as an office, and we rent it for
$400/month (utilities included). The total cost (ignoring my time)
was $17k, and utilities might be $500/y, so if we're able to rent it
continuously the payback period is 4y.

It's nice to have more usable space!

Comment via: facebook, mastodon
hydrogen tube transport
by bhauth — 2024-04-18T22:47:08.790Z
Link: https://www.bhauth.com/blog/industrial%20design/hydrogen%20tubes.html

Elon Musk's Hyperloop proposal had substantial public interest. With various initial Hyperloop projects now having failed, I thought some people might be interested in a high-speed transportation system that's...perhaps not "practical" per se, but at least more-practical than the Hyperloop approach.

aerodynamic drag in hydrogen

Hydrogen has a lower molecular mass than air, so it has a higher speed of sound and lower density. The higher speed of sound means a vehicle in hydrogen can travel at 2300 mph while remaining subsonic, and the lower density reduces drag. This paper evaluated the concept and concluded that:

the vehicle can cruise at Mach 2.8 while consuming less than half the energy per passenger of a Boeing 747 at a cruise speed of Mach 0.81

In a tube, at subsonic speeds, the gas must move backwards around the vehicle as the vehicle moves forwards. This increases drag, but

Gap flow increases the required power by at most 36% for any vehicle and tube system for which the ratio of tube-to-vehicle diameter is 2.38.

Larger tubes give lower drag.

Compared to a tube filled with vacuum, there are multiple advantages:

The vehicle can be supported aerodynamically.

The tube doesn't have large compressive forces.

Leaks would not cause rapid gas flow.

Propellers are an option for propulsion.

Air brakes could be used.

Airlocks are easier to implement.

airlocks

A hydrogen/air airlock could be implemented as follows:

Imagine 2 vertical tubes, connected at the ends to form a loop. The top of the loop is filled with hydrogen, and the bottom is filled with air.

In 1 tube, there's a piston with some (low-volatility) organic liquid sealing around it. The other tube is designated the airlock tube, and has an airlock chamber in its middle with doors.

When a vehicle arrives, the piston is raised to push hydrogen down into the airlock chamber. A door opens, and the vehicle enters the airlock chamber. The piston is then lowered to push air up into the airlock chamber, and a door is opened to outside.

high-speed train problems

The faster a train goes, the straighter its tracks need to be. They must be straight on a small scale to avoid vibration; this is an engineering problem. They also must be straight on a larger scale to avoid high accelerations; this makes buying land for them more difficult, and often requires digging or elevated tracks or tunnels. The speeds that woul...
LessOnline Festival Updates Thread
by Ben Pace — 2024-04-18T21:55:08.003Z

This is a thread for updates about the upcoming LessOnline festival. I (Ben) will be posting bits of news and thoughts, and you're also welcome to make suggestions or ask questions.

If you'd like to hear about new updates, you can use LessWrong's "Subscribe to comments" feature from the triple-dot menu at the top of this post.

Reminder that you can get tickets at the site for $400 minus your LW karma in cents.
A Review of In-Context Learning Hypotheses for Automated AI Alignment Research
by alamerton — 2024-04-18T18:29:33.892Z

This project has been completed as part of the Mentorship in Alignment Research Students (MARS London) programme under the supervision of Bogdan-Ionut Cirstea, on investigating the promise of automated AI alignment research. I would like to thank Bogdan-Ionut Cirstea, Erin Robertson, Clem Von Stengel, Alexander Gietelink Oldenziel, Severin Field, Aaron Scher, and everyone who commented on my draft, for the feedback and encouragement which helped me create this post.

TL;DR

The mechanism behind in-context learning is an open question in machine learning. There are different hypotheses on what in-context learning is doing, each with different implications for alignment. This document reviews the hypotheses which attempt to explain in-context learning, finding some overlap and good explanatory power from each, and describes the implications these hypotheses have for automated AI alignment research.

Introduction

Since their capabilities have improved and their size has increased, large language models (LLMs) have started demonstrating novel behaviours when prompted with natural language. Pre-trained LLMs can effectively carry out a range of behaviours when prompted. While predicting the most probable next token in a sequence is somewhat well understood, LLMs display another interesting behaviour, in-context learning, which is less easy to understand from the standpoint of traditional supervised learning.

In-context learning is an emergent behaviour in pre-trained LLMs where the model seems to perform task inference (learn to do a task) and to perform the inferred task, despite only having been trained on input-output pairs in the form of prompts. The model does this without changing its parameters/weights, contrary to traditional machine learning.

A Visual Example of ICL from The Stanford AI Lab Blog, Rong (2021)

In traditional supervised learning, a model’s weights are changed using an optimisation algorithm such as gradient descent. The main reason why ICL is a significant behaviour is because learning appears to take place, but the model’s weights do not change. Therefore, it does not require specific training or fine-tuning for new tasks. The model seems able to learn to do a new task with only prompts. ICL is also significant because it doesn’t need many training examples to do tasks well, unlike traditional training and fine-tuning approaches to learning.

ICL is de...
I'm open for projects (sort of)
by cousin_it — 2024-04-18T18:05:01.395Z

I left Google a month ago, and right now don't work. Writing this post in case anyone has interesting ideas what I could do. This isn't an "urgently need help" kind of thing - I have a little bit of savings, right now planning to relax some more weeks and then go into some solo software work. But I thought I'd write this here anyway, because who knows what'll come up.

Some things about me. My degree was in math. My software skills are okayish: I left Google at L5 ("senior"), and also made a game that went semi-viral. I've also contributed a lot on LW, the most prominent examples being my formalizations of decision theory ideas (Löbian cooperation, modal fixpoints etc) and later the AI Alignment Prize that we ran with Paul and Zvi. Most of that was before the current AI wave; neural networks don't really "click" with my mind, so I haven't done much work on them.

And yeah, this is an invitation to throw at me not necessarily money-paying work, but also stuff you'd like me to look at, criticize, help with your own projects and so on. I find myself with a bit more free time now, so basically drop me a PM if you have something interesting to talk about :-)
Blessed information, garbage information, cursed information
by tailcalled — 2024-04-18T16:56:17.370Z

This post is also available on my substack. Thanks to Justis Mills for editing and feedback.

Imagine that you're a devops engineer who has been tasked with solving an incident where a customer reports having bad performance. You can look through the logs of their server, but this raises the problem that there's millions of lines of log, and likely only a few of them are relevant to the issue. Thus, the logs are basically "garbage information".

Rather than looking at a giant pool of unfiltered information, what you really need is highly distilled information that's specifically optimized for solving this performance issue. For instance you could ask the user for more information about precisely what they were doing, or use filters to get the logs for exactly the parts of the application they were dealing with, or look through the places where the server spent a very large amount of time. The more a piece of information has been made to help you, the more "blessed" it is, with the extreme end of blessedness being information that keeps surprising you in its usefulness.

It might be tempting to think you could use multivariate statistics like factor analysis to distill garbage information by identifying axes which give you unusually much information about the system. In my experience, that doesn't work well, and if you think about it for a bit, it becomes clear why: if the garbage information has a 50 000 : 1 ratio of garbage : blessed, then finding an axis which explains 10 variables worth of information still leaves you with a 5 000 : 1 ratio of garbage : blessed. The distillation you get with such techniques is simply not strong enough.[1][2]

A 50 000 : 1 ratio might sound insurmountable by any technique, but because strong evidence is common, it's actually pretty feasible; e.g. knowing which minute in a week an incident occurred already gets you about this strong of a filter.

While blessed information is actively helpful, and garbage information is essentially useless, there's also the third case, of information that leads you down the wrong road. If an incident is labelled as "everything is slow", then that may very well get it more highly prioritized through customer service, but if most things aren't slow but the engineer investigates as if it were, that ends up burning more engineer time than if it was labelled accurately. Actively misleading information coul...
[Fiction] A Confession
by Arjun Panickssery — 2024-04-18T16:28:48.194Z
Link: https://arjunpanickssery.substack.com/p/fiction-a-confession

This morning while taking the LIRR to the city I performed first aid on a man who had been shot through the window of my carriage.

“Is he going to die?” his girlfriend asked me.

“We’re all going to die.”

A long pause. “I mean—is he going to die right now?”

“Probably not.” Probably he didn’t die. I got off at Jamaica Station while he stayed on (he was unconscious) so I don’t know. I didn’t want to be questioned at length as a witness since it was my day off.

I continued toward a barbershop I like. There wasn’t any reason for me to stay. A similar case of accidental gunfire into the train was in the news a while back. I guess also since it’s Saturday the workweek is over so it likely wasn’t any organized criminal act.

As I was passing Kew Gardens a stranger in a torn windbreaker pulled me suddenly aside.

“I have committed a terrible crime: a murder. No one suspects me. Only you know the truth. This is my name and address.” He pushed a small business card into the breast pocket of my coat and walked away.

Initially I supposed that I could turn him in to the police. A few reasons presented themselves immediately. First, it could be considered morally appropriate to denounce him to the authorities for the sake of justice. Second, a naïve interpretation suggested that he wanted me to turn him in, since otherwise he wouldn’t have confessed his crime to me. Third, a failure on my part to denounce him could present the possibility in the minds of concerned parties that I was his accomplice.

But walking through Forest Park with disregard for the operating hours of my barbershop, I considered the opposing evidence. First, I could be exposing myself to some kind of danger or unforeseen trap. Second, I might lack the conviction for treachery. This man entrusted me—and me alone—with such a secret. Already I walked among my fellow citizens with a newfound transgressive thrill. I resigned myself to the fate of my co-conspirator, whether arrest and punishment or criminal victory, the goal and outcome of which I knew nothing

Again and again I reversed my position for some hours. Such always has been the nightmare of my life with its interminable indecisiveness and hesitation. Very little new I discovered during this time, but only the relative importance of the different reasons shifted in my brain.

Halfway across the park I saw a little Pomeranian carrying a big stick, maybe five or si...
Discriminating Behaviorally Identical Classifiers: a model problem for applying interpretability to scalable oversight
by Sam Marks — 2024-04-18T16:17:39.136Z

In a new preprint,

Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models

, my coauthors and I introduce a technique, Sparse Human-Interpretable Feature Trimming (SHIFT), which I think is the strongest proof-of-concept yet for applying AI interpretability to existential risk reduction.[1] In this post, I will explain how SHIFT fits into a broader agenda for what I call cognition-based oversight. In brief, cognition-based oversight aims to evaluate models according to whether they’re performing intended cognition, instead of whether they have intended input/output behavior.

In the rest of this post I will:

Articulate a class of approaches to scalable oversight I call cognition-based oversight.
Narrow in on a model problem in cognition-based oversight called Discriminating Behaviorally Identical Classifiers (DBIC). DBIC is formulated to be a concrete problem which I think captures most of the technical difficulty in cognition-based oversight.
Explain SHIFT, the technique we introduce for DBIC.
Discuss challenges and future directions, including concrete recommendations for two ways to make progress on DBIC.

Overall, I think that making progress on DBIC is tractable with current interpretability techniques, and I’d be excited to see more work on it.

Cognition-based oversight

In this section, I describe a class of approaches to overseeing models I call cognition-based oversight. First, I review the problem of discriminating good vs. bad models and its relationship to scalable oversight. Then I impose a pessimistic assumption – that the good and bad models cannot be discriminated behaviorally – and propose cognition-based oversight as a potential remedy.

Discriminating models

In an ideal world, we could argue that AI systems will be safe by default as follows:

We train AI systems to receive high scores according to some scoring function. The models that get the highest scores are all good models that robustly behave as desired. Therefore, we have strong a priori reason to expect our training processes to result in good models.

Sadly, in the real world, the “scoring functions” we actually use tend to be shaped like “human evaluator approval.” And “human evaluator approval” does not enjoy the property that “the models which humans evaluate most positively all robustly behave as desired.” Two salient

cou

...