trentmkelly/LessWrong-43k huggingface dataset:
BBC Future covers progress studies
by jasoncrawford — 2022-06-16T22:44:47.153Z
Link: https://rootsofprogress.org/bbc-future-covers-progress-studies

BBC Future has an article on progress studies. I was interviewed, along with Tyler Cowen, Holden Karnofsky, and others. It’s well-researched and, although somewhat critical, it is pretty fair in how it represents the progress community.

Here are brief responses to some of the criticisms:

Why the progress community focuses on material progress

Progress writers say that we care about human well-being, and that this includes moral and social progress. But in practice, we have mostly focused on material progress so far. Why?

First, material progress is underrated. Economic growth is considered a wonkish concern; it deserves to be considered a humanitarian one.

Second, an appreciation of material progress is crucial to validating the core ideas of the Enlightenment—which I consider to be crucial to moral progress. For more on the link between the two, see my essay “Why Liberals Should Care About Progress.”

That said, I would love to turn my focus to moral progress at some point, and even write a book on it. In the meantime, for more on the subject, see my review of Steven Pinker’s The Better Angels of Our Nature: Why Violence Has Declined.

Why we focus on frontier growth

The progress community has also been focused on growth at the technological frontier, rather than catch-up growth in places like China and India. Why?

First, frontier growth is fundamental. Without frontier growth, there’s nothing to catch up to—that was the entire history of the world until about 1800. With frontier growth, poor countries benefit, even when they don’t fully catch up. For instance, every country on Earth today has a life expectancy higher than any country did in 1800. (More recently, the Center for Global Development found a “strong convergence in use of consumption technologies“ which “reflects considerably stronger global convergence in quality of life than in income.”)

Second, frontier growth (like material progress) is underrated. The UN has set “Sustainable Development Goals” like “no poverty” and “clean water.” No major international organization that I know of has even considered goals like “cure aging,” “settle Mars” or “invent fusion energy.”

Further, thoroughly understanding the growth of frontier countries such as the UK and US is an important foundation for driving catch-up growth. The two types of growth don’t happen in exactly the same way, but I think there is a large element...
Humans are very reliable agents
by alyssavance — 2022-06-16T22:02:10.892Z

This post has been recorded as part of the LessWrong Curated Podcast, and an be listened to on Spotify, Apple Podcasts, and Libsyn.

Over the last few years, deep-learning-based AI has progressed extremely rapidly in fields like natural language processing and image generation. However, self-driving cars seem stuck in perpetual beta mode, and aggressive predictions there have repeatedly been disappointing. Google's self-driving project started four years before AlexNet kicked off the deep learning revolution, and it still isn't deployed at large scale, thirteen years later. Why are these fields getting such different results?

Right now, I think the biggest answer is that ML benchmarks judge models by average-case performance, while self-driving cars (and many other applications) require matching human worst-case performance. For MNIST, an easy handwriting recognition task, performance tops out at around 99.9% even for top models; it's not very practical to design for or measure higher reliability than that, because the test set is just 10,000 images and a handful are ambiguous. Redwood Research, which is exploring worst-case performance in the context of AI alignment, got reliability rates around 99.997% for their text generation models.

By comparison, human drivers are ridiculously reliable. The US has around one traffic fatality per 100 million miles driven; if a human driver makes 100 decisions per mile, that gets you a worst-case reliability of ~1:10,000,000,000 or ~99.999999999%. That's around five orders of magnitude better than a very good deep learning model, and you get that even in an open environment, where data isn't pre-filtered and there are sometimes random mechanical failures. Matching that bar is hard! I'm sure future AI will get there, but each additional "nine" of reliability is typically another unit of engineering effort. (Note that current self-driving systems use a mix of different models embedded in a larger framework, not one model trained end-to-end like GPT-3.)

(The numbers here are only rough Fermi estimates. I'm sure one could nitpick them by going into pre-pandemic vs. post-pandemic crash rates, laws in the US vs. other countries, what percentage of crashes are drunk drivers, do drunk drivers count, how often would a really bad decision be fatal, etc. But I'm confident that whichever way you do the math, you'll still find that humans...
Towards Gears-Level Understanding of Agency
by Thane Ruthenis — 2022-06-16T22:00:17.165Z

Epistemic status: Highly speculative, much of this is probably wrong.

Summary: The "centerpiece" of this post is my attempt to describe how a full-fledged consequentialist agent might incrementally grow out of a bundle of heuristics, aided by SGD or evolution. The story's detail are likely very wrong. It's informed by a few insights that I'm more confident in, however, and suggests some interesting variables and developmental milestones of agency. Some specific claims:

Operating in a vast well-abstracting environment requires the ability to navigate arbitrary mathematical contexts.
That requires the ability to represent any mathematical environment internally.
In turn, this is possible if the agent possesses a functionally complete set of "mental primitives" — hard-coded conceptual building blocks. These primitives may be put together to define any object.

For memory-efficiency reasons, such composite objects are then abstracted-over via an algorithm like this. Their internal structure is wiped, only the externally-relevant properties are kept.
If the algorithm is imperfect, the resultant abstractions might be "leaky": carrying the traces of the primitives they're made of, or plain flawed.
This creates severe problems in case of ontology crises: if the agent tries to translate its goal from one environment to another, redefining it and imperfectly abstracting over its re-definition.

Agency has the tendency to "take over" the neural network it appears in.

It may start out as a minor and niche algorithm for context-learning/aid to built-in heuristics.
Gradually (across generations/batches), it would expand its influence, holding more sway over the network's decisions.
Eventually, it would maneuver the network into off-distribution environments where it hopelessly outperforms static heuristics, and eventually fully dominate them.

Fundamentally, agency is little more than a menagerie of meta-heuristics: heuristics for best ways to train new heuristics, including better meta-heuristics.

In humans, the recursive self-improvement this implies is constrained by the static size of working memory + the fact that we can't modify the more basic features of our minds (like the mental primitives or the abstraction-building algorithm).

0. Background

In a previous post, I've made a couple relevant claims that I'm mostly confident in. I'll reframe and clarify them here.

0A. Universal

...
A possible AI-inoculation due to early "robot uprising"
by Shmi — 2022-06-16T21:21:56.982Z

I have talked about it before in the comments, but given the near-certain doomsday predictions by Eliezer, it might be worth posting it separately: there is a chance of a scenario that will sharply change the AI development trajectory before it is certain to wipe out everyone.

The scenario is that a non-super-intelligent AI, general or almost so, is given or accidentally gets access to a lot of computational power, and potentially to some real-world resources, as a part of, say "gain of function" or some other research. A concerted effort by a swarm of near-human-level agents acting together (or, equivalently, by one near-human-level agent with tons of computing power, like GPUs, TPUs etc.) can do a lot of damage, but is unlikely to wipe out all human life. If the event is devastating enough, the attitude to laissez faire AI research may change early, rather than late.

On general principles, I would expect there to be a phase transition of sorts:

A not-very-smart agent accessing a lot of compute is likely to trip over itself and the human safeguards and do little to no damage.
Once the agent is smart enough to avoid that, and to evade existing safeguards designed to constrain and contain non-adversarial side effects, its effects can discontinuously become very large.
Faced with an event like that, humanity, despite being inept and stupid about the pandemic, can enact AI safety measures it would not otherwise, giving us the precious second chance at doing it right.

Another bit of good news is that this scenario can actually be modeled and analyzed, since it does not require guessing how something both alien and smarter than a human would reason.
AI Risk, as Seen on Snapchat
by dkirmani — 2022-06-16T19:31:24.253Z

The news tiles on Snapchat are utter drivel. Looking at my phone, today's headlines are: "Zendaya Pregnancy Rumors!", "Why Hollywood Won't Cast Lucas Cruikshank", and "The Most Dangerous Hood in Puerto Rico". Essentially, Snapchat news is the Gen-Z equivalent of the tabloid section at a Walmart checkout aisle.

Which is why I was so surprised to hear it tell me the arguments for AI risk.

The story isn't exactly epistemically rigorous. However, it reiterates points I've heard on this site, including recursive self-improvement, FOOMing, and the hidden complexity of wishes. Here's an excerpt of the story, transcribed by me:

Given enough time and the ability to self-generate improved versions of itself, it wouldn't take long for a fully autonomous general AI to achieve superintelligence, a level of cognitive processing power so many times stronger than our own that its abilities would appear godlike to us, and the pace at which it could improve would be lightning fast.

Superintelligent AI could think through problems in a few hours equivalent to what the world's smartest people could do in a thousand years. How could we possibly control or out-think such a powerful intellect? [...]

Being the first superintelligence, the AI could see any competition as a threat, and could choose to remove [all humans]. [...]

When in the possession of such superintelligence, scientists may pose it questions, like how to solve previously-impossible mathematical equations. To solve the problem, the AI may choose to forcibly convert all matter on Earth into a supercomputer to handle the processing of the equation. It could solve the problem while killing the species that asked the question.

We could also make the mistake of giving the AI too vague a problem to solve, such as "end human suffering". It could then decide the best way to do this is to eliminate all humans, and therefore end their suffering.

Why does this matter? The channel, Future Now, has 110,000 subscribers(!), and the story was likely seen by many others. This is the first time I've ever seen anyone seriously attempt to explain AI risk to the masses.

The fact that Future Now ran the story implies that the masses are, in fact, capable of understanding the arguments for AI risk, so long as those arguments come from someone who sounds vaguely like an authority figure.
[Link] "The madness of reduced medical diagnostics" by Dynomight
by Kenny — 2022-06-16T19:20:36.556Z

Link:

The madness of reduced medical diagnostics

This is (or seems to me now to be) an obvious-in-hindsight and I'm sad that I've never (or don't remember having) encountered it; at least not so succinctly.

I'd like to try putting this 'advice' into practice myself, e.g. demanding doctors share relevant base rates but not otherwise avoiding seeing a doctor at all or avoiding diagnostic tests (even if I expect the doctor's subsequent decisions to be bad).
Breaking Down Goal-Directed Behaviour
by Oliver Sourbut — 2022-06-16T18:45:11.872Z

When we speak about entities 'wanting' things, or having 'goal-directed behaviour', what do we mean?

Because most of the actors that we (my human readers and I) attentively interact frequently with are (presumably) computationally similar (to each other and to ourselves), it is easy for our abstractions to conflate phenomena which are in fact different[1], or to couple together impressions of phenomena which are in fact separable[2]. On the occasions that we attentively observe dissimilar actors[3], we are often-enough 'in the wild' (i.e. looking in their 'habitat') that conflating - anthropomorphising - is sufficiently usefully predictive to get by[4], so these poor abstractions are insufficiently challenged.

Further, most of the people who in fact attentively observe a particular class of human-dissimilar actors (enough to perceive and understand the non-conflations) are focused domain experts about that particular class, and talk mainly to other domain experts about them[5]. There are few venues in which it is useful to have unambiguous terminology here. Thus, the language we use to communicate our abstractions about goal-directed behaviour is prone to conflation and confusion even when some people have some of the right abstractions[6].

Here I aim to take steps to break down 'goal-directed behaviour' into a conceptual framework of computational abstractions for which I offer tentative terminology, and which helps me to better understand and describe analogies and disanalogies between various goal-directed systems. The overarching motivation is to better understand goal-directed behaviour, in the sense of being able to better predict its (especially counterfactual and off-distribution) implications, its arisal, and other properties. Hopefully it is clear why I consider this worthwhile.

In order to ground this discussion, I refer to a reasonably diverse menagerie of candidate goal-directed systems, including natural and artificial systems at various levels of organisation. Contemplation of this diverse collection was responsible for the ideation and refinement of the ideas and gives some confidence in the appropriateness of the abstractions.

A collection of a few 'agents' drawn from the menagerie

Different in the sense that, even if the observed surface phenomena are similar 'in the wild', their behaviour in different contexts might radically come apart; th

...
Perils of optimizing in social contexts
by owencb — 2022-06-16T17:40:46.843Z

As a special case of not over-optimizing things, I want to point out that an optimizing approach can have a bunch of subtle negative effects in social contexts. The slogan is something like "optimizers make bad friends" ... people don't like being treated as a means rather than an end, and if they get the vibe from social interactions that you're trying to steer them into something, then they may react badly.

Here I'm referring to things like:

"How do I optimize this conversation to get X to say yes?"
"How do I persuade Y of this particular fact?"
"How can I seem particularly cool/impressive to these people?",

... where it's not clear that the other side are willing collaborators in your social objectives.

So I don't mean to include optimizing for things like:

"How can I give the clearest explanation of this thing, so people aren't confused about what I'm saying?"
"How can I help Z to feel comfortable and supported?"

I think that people (especially smart people) are often pretty good at getting a vibe that someone is trying to steer them into something (even from relatively little data). When people do notice, it's often correct for them to treat you as adversarial and penalize you for this. This is for two reasons:

A Bayesian/epistemic reason: people who are optimizing for particular outcomes selectively share information which pushes towards those outcomes. So if you think someone is doing [a lot of] this optimization your best estimate of the true strength of the position they're pushing should be [much] lower than if you think they're optimizing less (given otherwise the same observations).

Toy example: if Alice and Bob are students each taking 12 subjects, and you randomly find out Alice's chemistry grade and it's a B+, and you hear Bob bragging that his history grade is an A-, you might guess that Alice is overall a stronger student, since it's decently likely that Bob chose his best grade to brag about.

An incentives reason: we want to shape the social incentive landscape so that people aren't rewarded for trying to manipulate us. If we just do the Bayesian response, it will still often be correct for people to invest some in trying to manipulate (in theory they know more about how much information to reveal to leave you with the best impression after Bayesian updating).

I don't think the extra penalty for incentive shaping needs to be too big, but I do thin

...
Don't Over-Optimize Things
by owencb — 2022-06-16T16:33:17.560Z

or Optimizing Optimization

The definition of optimize is:

to make something as good as possible

It's hard to argue with that. It's no coincidence that a lot of us have something of an optimization mindset.

But sometimes trying to optimize can lead to worse outcomes (because we don't fully understand what to aim for). It's worth understanding how this happens. We can try to avoid it by a combination of thinking more about what to aim for, and (sometimes) simply optimizing less hard.

Reflection on purpose vs optimizing for that purpose

What does the activity of making something as good as possible look like in practice? I think often there are two stages:

Reflection on the purpose — thinking about what the point of the thing at hand is, and identifying what counts as "good" in context
Optimizing for that purpose — identifying the option(s) which do best at the identified purpose

Both of these stages are important parts of optimization in the general sense. But I think it's optimization-for-a-given-purpose that feels like optimization. When I say "over-optimization" I mean doing too much optimization for a given purpose.

What goes wrong when you over-optimize

Consider this exaggerated example:

Alice is a busy executive. She needs to get from one important meeting to another in a nearby city; she's definitely going to be late to the second meeting. She asks her assistant Bob to sort things out. "What should I be optimizing for?", Bob asks. "Just get me there as fast as possible", Alice replies, imagining that Bob will work out whether a taxi or train is faster.

Bob is on this. Eager to prove himself an excellent assistant, he first looks into a taxi (about 90 minutes) and a train (about 60 minutes plus 10 minutes travel at each end — but there's a 20 minute wait for the right train). So the taxi looks better.

But wait. Surely he can do better than 90 minutes? OK, so the journey is too short for a private jet to make sense, but what about a helicopter? Yep, 15 minutes to get to a helipad, plus 45 minutes flight time, and it can land on the hotel roof! Even adding in 5 minutes for embarking/disembarking, this is 25 minutes faster.

Or ... was he assuming that the drivers were sticking to the speed limit? Yeah, if he make the right phone calls he can find someone who can drive door to door in 60 minutes.

Can he get the helicopter to be faster than that? Yeah, the driver can s

...
Security analysis of 'cloud chemistry labs'?
by Kenny — 2022-06-16T16:06:25.989Z

I learned of 'cloud chemistry labs' while watching this video:

Stephen Wolfram: Complexity and the Fabric of Reality | Lex Fridman Podcast #234 - YouTube

Wolfram is trying to devise a 'molecular computing' experiment to generate prime numbers using the 'dynamics' of a 'molecular system' (i.e. via something like the 'set of all possible chemical reaction pathways'). (That's pretty interesting on its own!)

But, apparently, he's using (or planning on using) a 'cloud chemistry lab' to perform the actual chemistry experiments. He mentioned this particular 'cloud lab':

Emerald Cloud Lab: Remote Controlled Life Sciences Lab

That is a really cool idea – but also seems, to me anyways, to be a (possibly) really dangerous 'capability' to make available in this way!

What are your thoughts on the security of either this particular lab or these kinds of labs in general?

Added 2022-6-18 19:07 UTC:

[Copied from this comment by me:]

I was hoping more for 'brainstorm' type responses I guess. (A somewhat comprehensive answer would be wonderful of course.)

I'm not actually worried about this. I don't expect any particular 'immediate' tragedy to result. I'm thinking of this as more of 'security puzzle' than anything else, e.g. a 'call to action'.

This isn't a political campaign, or activism, or me attempting to pull any kind of 'alarm' about this being an impending catastrophe!

I'm really just curious.

What are the ways this could fail/be-compromised? (Don't share useable 'exploits' tho! Disclose any useable vulnerabilities responsibly!)

What are the ways we should expect this to fail/be-compromised?

What are the ways it could be defended? What are the practical and principled limits of various defense?

How should we expect it to be defended?

[But also – SHIT – is the expected value of these kinds of discussions even positive at all?]
Covid 6/16/22: Do Not Hand it to Them
by Zvi — 2022-06-16T14:40:02.604Z

You (finally!) no longer need to hand anyone a negative test in order to travel internationally.

You also most definitely do not have to hand it to the FDA for their work solving the bottlenecks they created.

Paxlovid remains something that you will probably not by default get, but you can get it if you are sufficiently pushy. Public health has the opinion that if you are low risk, a large (51%?) reduction in your relative risk isn’t even worth continuing a study, so that tells you how much you should care about otherwise reducing your risk.

Executive Summary

You can get Paxlovid and/or travel without testing if you want to, but you still can’t quite get permission to vaccinate your young child.
Don’t leave your friends behind.
Even if they don’t dance, they’re still friends of mine.

Let’s run the numbers.

The Numbers

Predictions

Prediction from last week: 700k cases (+7%) and 2,300 deaths (+4%).

Results: 644k cases (-2%) and 1,917 deaths (-14%).

Prediction for next week: 650k cases (+1%) and 2,000 deaths (+4%).

As is suspiciously often the case, cases and deaths moved unexpectedly in the same direction, causing suspicion of a reporting issue, which was at least in part Pennsylvania not reporting. I’ve given the state the same numbers as last week, at which point things overall seem closer to stable.

That means two major states (Florida is the other) require adjustments due to declining data quality. That’s a pretty bad sign for our state capacity and competence.

Deaths

This looks a lot like a combination of a slow decline and some sort of reporting hiccup I haven’t yet fully accounted for.

Cases

This includes a proper Florida adjustment, so it looks like the South is at least mostly stable.

New wave coming on in the UK with R0 up in the 1.4 range. Scary on its face, but I don’t expect things to get out of hand.

Travel is Back

Tests are no longer required to enter the United States. We were far behind the sanity curve on this one, as Europe has long since gotten over the testing requirement.

Canada is also lifting its requirements.

In other news, Justin Trudeau has Covid-19 (as does Tony Fauci), which is interesting timing. Remember when we used to worry when world leaders got Covid? Worth noting ‘following health guidelines’ means he’s likely not getting Paxlovid, which is pretty crazy. Mild sympt...