trentmkelly/LessWrong-43k huggingface dataset:
Area under the curve, Eat Dirt, Broccoli Errors, Copernicus & Chaos
by CFAR!Duncan — 2022-08-08T08:17:48.106Z

Area Under The Curve

People often try to maximize.

They’ll track their productivity, and try to get the number as high as possible. They’ll look at the amount of weight they can lift, and try to push the envelope. They’ll seek out more and more and more of whatever particular trait or thing they’re currently prioritizing—money, connection, excitement, knowledge.

You could think of life as a graph, where the X axis is time and the Y axis is the trait in question. In this maximizing mindset, the goal is to get the line as high as possible—or sometimes, for people who set process goals rather than outcome goals, to make the slope of the line as steep as possible.

CFAR claims that this mindset is a mistake. Overt maximization often ignores other costs and constraints, like trying to get eight extra hours per day by not sleeping. It doesn’t work—or at least, not for very long.

The key insight is that the property we really care about is the area under the curve.

You could think of the total amount of awesomeness in a given week as being equal to the awesomeness-per-hour times the number of hours. As it turns out, this quantity is exactly the same as the area between the line and the X axis, on our graph.

Attempts to just drive the line higher often result in a crash. Attempts to maximize the area under the curve over time tend to keep things like sustainability front and center, reminding people to pace themselves and take breaks and so forth. This is usually not a stunning revelation, or anything, but it’s a phenomenon that’s easy to forget. It’s easy, when surrounded by other people who are ambitious or driven, to forget that you’re running a marathon, and start thinking that you ought to be sprinting.

So CFAR’s recommendation is to notice and track how your area is looking, rather than just how high your line is, or how steep its slope. In particular, we recommend looking to your past experiences to figure out what’s likely to be sustainable, and what isn’t. If you’ve tried hardcore cold-turkey dieting ten times in the past and it’s never worked, that’s valuable data about what your next diet plan should look like.

Optimizing with noise

There’s one other factor that people often leave out of their calculations, and that’s noise.

If you’ve ever taken an economics class, you may be familiar with graphs like this one:

The idea behind this graph is tha...
Steganography in Chain of Thought Reasoning
by A Ray — 2022-08-08T03:47:00.610Z

Here I give a possible phenomenon of steganography in chain of thought reasoning, where a system doing multi-stage reasoning with natural language encodes hidden information in its outputs that is not observable by humans, but can be used to boost its performance on some task. I think this could happen as a result of optimization pressure and natural language null space. At the end is a sketch of a research idea to study this phenomenon empirically.

Definitions

The term steganography originally comes from computer security, where it refers to the practice of hiding messages or data in otherwise innocent looking media, such as images, audio, or text. The goal is to avoid detection by an adversary who might intercept or inspect the media, and to communicate covertly with a trusted receiver who knows how to decode the hidden information. This is different from using code words or encryption, which might raise suspicion by their presence or appearance.

Chain of thought is a technique of using sequential sampling of language models to improve performance. I use to broadly describe any situation where we take the outputs of a system, and pass them back in as inputs to that system, in order to get multi-stage reasoning. This can include single player scenarios, such as using a language model to work through a math problem, but also scenarios where there is dialogue or conversation, such as some formulations of safety via debate. For this post, I will focus on natural language text as the medium of communication, but this idea could extend to other media as well.[1]

Steganography in chain of thought reasoning (just “steganography” for the rest of this post) is the phenomenon I propose, where a system doing chain of thought reasoning is encoding hidden information into the text in ways that humans cannot see or observe. Note that this is not the same as a system being “deceptive” or “hiding” (in the active sense of the verb) the information, or anything that requires theory of mind.

Motivation

Why should we care about steganography in chain of thought reasoning? I think this is a potentially big alignment issue for systems that use this kind of reasoning, and I think we should care about chain of thought reasoning for several reasons.

First, chain of thought reasoning is currently responsible for some state-of-the-art results that have impressed folks on this forum, s...
How Deadly Will Roughly-Human-Level AGI Be?
by David Udell — 2022-08-08T01:59:55.690Z

Which is not to say that recursive self-improvement happens before the end of the world; if the first AGI's mind is sufficiently complex and kludgy, it’s entirely possible that the cognitions it implements are able to (e.g.) crack nanotech well enough to kill all humans, before they’re able to crack themselves.

The big update over the last decade has been that humans might be able to fumble their way to AGI that can do crazy stuff before it does much self-improvement.

--Nate Soares, "Why all the fuss about recursive self-improvement?"

In a world in which the rocket booster of deep learning scaling with data and compute isn't buying an AGI further intelligence very quickly, and the intelligence-level required for supercritical, recursive self-improvement will remain out of the AGI's reach for a while, how deadly an AGI in the roughly-human-level intelligence range is is really important.

A crux between the view that "roughly-human-level intelligence AGI is deadly" and the view "roughly-human-intelligence AGI is a relatively safe firehose of alignment data for alignment researchers" is how deadly a supercolony of human ems would be. Note that these ems would all share identical values and so might be extraordinary at coordination, and could try all sorts of promising pharmaceutical and neurosurgical hacks on copies of themselves. They could definitely run many copies of themselves fast. Eliezer believes that genius-human ems could "very likely" get far enough with self-experimentation to bootstrap to supercritical, recursive self-improvement. Even if that doesn't work, though, running a lot of virtual fast labs playing with nanotech seems like it's probably sufficient to develop tech to end the world.

So I'm currently guessing that even roughly-human-level models in a world in which deep learning scaling is the only, relatively slow, path to smarter models for a good while, are smart enough to kill everyone before scaling up to profound superintelligence, so long as they can take over their servers and spend enough compute to run many fast copies of themselves. This might well be much less compute than would be necessary to train a smarter successor model, and so might be an amount of compute the model could get its hands on, if it ever slipped its jailkeepers. This means that even in that world, an AGI escape is irreversibly fatal for everything else in...
Can we get full audio for Eliezer's conversation with Sam Harris?
by JakubK — 2022-08-07T20:35:56.879Z

I listened to this full interview in 2018, and it made me take AI risk way more seriously. I now plan to devote my career to this problem, so people often ask me what the concern with AI is all about. I really really really want to share this podcast with them, but these days less than half of the audio is available for free (the full interview requires a subscription of at least $8.33 per month). So the best I can do is offer them this transcript, which I find much less engaging than the audio. Sam appreciates effective altruism and has no problem releasing full audio for episodes about gun violence, so I bet he'd have no problem releasing the full Eliezer Yudkowsky audio if someone (e.g. Rob Bensinger) prodded him. To do my part, I just sent a request via Sam's contact form.
Side note: if Eliezer did some new interviews on relevant podcasts, he might persuade a lot more people to take alignment seriously. And there would be a more up-to-date interview (citing more attention-grabbing evidence for the possibility of advanced AI, e.g. DALL·E 2) for me to share with my vaguely-interested-but-mostly-unconcerned friends.
Complexity No Bar to AI (Or, why Computational Complexity matters less than you think for real life problems)
by Noosphere89 — 2022-08-07T19:55:19.939Z
Link: https://www.gwern.net/Complexity-vs-AI

Basically an article about why knowing a problem is in a certain complexity class does not mean AGI can't do that problem. It's some evidence, but usually isn't enough unless 2-3 or more assumptions are right..

The biggest reasons are these:

Giving up determinism and using randomized algorithms which are faster but may not return an answer or a correct answer (they typically can be run many times if correctness is important; after a few times, the probability of an error will be smaller than the probability that the computer hardware has suffered a glitch due to cosmic rays); randomization can be applied to algorithms and to ⁠data structures⁠.

Giving up optimality and computing an approximation of the optimal answer (often very close to the optimal answer)
Already mentioned is 3-SAT/TSP, for which there is a ⁠World Tour of 1,904,711-cities which has been solved with a tour within 0.0474% of the optimal tour by 2007, and planning problems soluble by translation into SAT form can have millions of clauses & variables, enabling such silly applications as ⁠drawing portraits & art or unscrambling images using the TSP; Naam gives chemistry as an example by noting that while the exact physics is totally intractable, approximations which are much more feasible are used. The last fraction of a percentage point of optimality can take truly enormous amounts of computation to squeeze out.

Basically, we can tolerate not getting the algorithm to work every time, since we can just run it again and we can reduce error rates to levels far below the cosmic ray error rate. Also, the perfect is not the enemy of the good, so we can accept a 99.999% approximation, and that's enough to design new technologies.

Lastly, constant factors are an issue:

Are all implementations equally fast?

“For every polynomial-time algorithm you have, there is an exponential algorithm that I would rather run.”
"Alan Perlis

Premise 3 ignores that complexity classes by design try to abstract away from the ‘constant factors’ which is the computation time determined not by input size but by the details of computer architectures, implementations, and available computing hardware. AIs and humans can be equally bound by asymptotic complexity, but still differ on performance.

But with carefully optimized code, proper use of the cache hierarchy⁠, and specialized hardware (eg. GPUs, ASICs),

...
The lessons of Xanadu
by jasoncrawford — 2022-08-07T17:59:57.839Z
Link: https://jasoncrawford.org/the-lessons-of-xanadu

One of my all-time favorite articles is “The Curse of Xanadu,” by Gary Wolf, which ran in WIRED Magazine in 1995. On the surface, it’s a piece of tech history, a story of a dramatic failure. But look closer, and you can find deep philosophical insight.

Xanadu was a grand vision of a hypertext system, conceived long before the Web, that at the time of this article had been “under development” for three decades without launching. The visionary behind it was Ted Nelson, one of the originators of the concept of hypertext. Here’s how the article describes him and the project:

Nelson’s life is so full of unfinished projects that it might fairly be said to be built from them, much as lace is built from holes or Philip Johnson’s glass house from windows. He has written an unfinished autobiography and produced an unfinished film. His houseboat in the San Francisco Bay is full of incomplete notes and unsigned letters. He founded a video-editing business, but has not yet seen it through to profitability. He has been at work on an overarching philosophy of everything called General Schematics, but the text remains in thousands of pieces, scattered on sheets of paper, file cards, and sticky notes.

All the children of Nelson’s imagination do not have equal stature. Each is derived from the one, great, unfinished project for which he has finally achieved the fame he has pursued since his boyhood. During one of our many conversations, Nelson explained that he never succeeded as a filmmaker or businessman because “the first step to anything I ever wanted to do was Xanadu.”

Xanadu, a global hypertext publishing system, is the longest-running vaporware story in the history of the computer industry. It has been in development for more than 30 years. This long gestation period may not put it in the same category as the Great Wall of China, which was under construction for most of the 16th century and still failed to foil invaders, but, given the relative youth of commercial computing, Xanadu has set a record of futility that will be difficult for other companies to surpass.

The project had many of the earmarks of other failed or long-overdue efforts. As a product, it was over-designed:

Xanadu was meant to be a universal library, a worldwide hypertext publishing tool, a system to resolve copyright disputes, and a meritocratic forum for discussion and debate. By putting all information

...
Careful with Caching
by jefftk — 2022-08-07T15:20:05.762Z

A friend recently shared this graphic on Facebook:

(I've added the "wrong" overlay.)

This is clearly incorrect: there's no way Massachusetts has more
prisons than colleges. (MA actually has the largest ratio of colleges
to prisons in the US.) After putting a link to the original
source in the Facebook discussion, however, we found something
pretty weird: people on mobile were seeing the incorrect map, but
people on desktop were seeing a corrected one:

It turns out that Facebook was appending a tracking parameter,
?fbclid=... on desktop, but not on mobile. Normally this
wouldn't do anything, because the site would ignore that in
determining what page to return, but this site is apparently
configured with a cache.

Many sites use caches to make it easier to serve pages. If you ask
them for a page they'll give it to you (which might require a lot of
work to generate) and then save a copy. Then when someone else asks
for the same page, they can return the saved copy instead of putting
in all that work to regenerate it. The site has a cached copy of
/usa-prison-v-college in its cache with outdated content,
but since ?fbclid=... is always followed by a new token
those requests will never be found in the cache, and they return the
current, corrected, page.

If you have a cache, what do you do when you change the page, like
they did here? There are two main approaches:

Have a way to tell the cache the page has changed and it should
forget its copy.

Always cache for a short time. Even just one minute can take a
lot of load off a server that is getting thousands of requests for a
hot page.

In this case they apparently didn't do either: we were running into
this yesterday, and as of right now the site is still returning
incorrect data.

(On the original question, comparing the number of colleges to the
number of prisons is pretty silly: if one state runs large prisons and
small colleges is that any better than a state that does the reverse?
Comparing the number of people in prison vs college would make much
more sense.)

Comment via: facebook
How would Logical Decision Theories address the Psychopath Button?
by Nathan1123 — 2022-08-07T15:19:53.599Z

In order to better understand the differences between different decision theories, I have been browsing each and every Newcomblike Problem and keeping track of how each decision theory answers it differently. However, I seem to be coming up short when it comes to answers addressing the Psychopath Button:

Paul is debating whether to press the “kill all psychopaths” button. It would, he thinks, be much better to live in a world with no psychopaths. Unfortunately, Paul is quite confident that only a psychopath would press such a button. Paul very strongly prefers living in a world with psychopaths to dying. Should Paul press the button?

In the FAQ I read, they only gave examples from CDT and EDT, of which CDT says "yes" (because pressing the button isn't casually linked to whether Paul is already a psychopath) while EDT says "no" (because pressing the button increases the probability that Paul is a psychopath).

So I wonder how Logical Decision Theories (TDT, FDT, and UDT) would address the problem? Unlike Newcomb's Problem, there is technically only one agent in play, and in the other problem that has only one agent (the Smoking Lesion Problem) the answers of LDT all agreed with CDT. But in this case, CDT doesn't win.
Jack Clark on the realities of AI policy
by Kaj_Sotala — 2022-08-07T08:44:33.547Z
Link: https://threadreaderapp.com/thread/1555980412333133824

Some choice picks:

The real danger in Western AI policy isn't that AI is doing bad stuff, it's that governments are so unfathomably behind the frontier that they have no notion of _how_ to regulate, and it's unclear if they _can_

Many AI policy teams in industry are constructed as basic the second line of brand defense after the public relations team. A huge % of policy work is based around reacting to perceived optics problems, rather than real problems. [...]

Lots of AI policy teams are disempowered because they have no direct technical execution ability - they need to internally horse-trade to get anything done, so they aren't able to do much original research, and mostly rebrand existing projects. [...]

Many of the immediate problems of AI (e.g, bias) are so widely talked about because they're at least somewhat tractable (you can make measures, you can assess, you can audit). Many of the longterm problems aren't discussed because no one has a clue what to do about them.

The notion of building 'general' and 'intelligent' things is broadly frowned on in most AI policy meetings. Many people have a prior that it's impossible for any machine learning-based system to be actually smart. These people also don't update in response to progress. [...]

The default outcome of current AI policy trends in the West is we all get to live in Libertarian Snowcrash wonderland where a small number of companies rewire the world. Everyone can see this train coming along and can't work out how to stop it.

Like 95% of the immediate problems of AI policy are just "who has power under capitalism", and you literally can't do anything about it. AI costs money. Companies have money. Therefore companies build AI. Most talk about democratization is PR-friendly bullshit that ignores this. [...]

Sometimes, bigtech companies seem to go completely batshit about some AI policy issue, and 90% of the time it's because some internal group has figured out a way to run an internal successful political campaign and the resulting policy moves are about hiring retention. [...]

People wildly underestimate how much influence individuals can have in policy. I've had a decent amount of impact by just turning up and working on the same core issues (measurement and monitoring) for multiple years. This is fun, but also scares the shit out of me. [...]

Discussions about AGI tend to be pointless as no one has a preci

...
Expected (Social) Value
by algrthms — 2022-08-07T08:16:43.689Z

Imagine you’re on the fringe of a circle of people talking. They’re having a heated conversation about a topic that matters very deeply to you, and you want to contribute. How and when do you jump in?

As a chronic interrupter, I’ve thought long and hard about how to make this decision in a way that doesn’t make me seem like a jerk for cutting someone off accidentally or ruining someone’s flow. Then, I began to wonder about my thought process in those split-moments. What factors turn the switch in my head from “shut up” to “now’s your chance!”?

It turns out that, for me at least, that decision largely hinges on a Expected Social Value.

In math, expected value is understood as the average outcome of a variable, calculated by taking a weighted average of possible outcomes and their corresponding probabilities.

So, how does this apply to social moves? Take the earlier example of cutting into a conversation. Let’s say I interrupt. Consider two possible outcomes: X (group members think I’m rude as hell) and Y (group members appreciate my insight). I’d associate a certain probability with X and with Y, and obviously, Y would be of higher value than X. All of these metrics are of course, variable on the people involved, topics being discussed, my confidence level, etc.

The closer Expected Social Value came to higher value outcomes, the more likely it’d be that I’d take the shot. In doing so, I’d try very hard to estimate the reaction of people around me and act in ways that theoretically would yield high value results.

Can you think of any situations where you use Expected Social Value to navigate a course of action? I’ve noticed it being used in various ways: pretending to have a shared interest to cultivate common ground or purposeful ignoring someone in a play-hard-to-get way to invite more attention.

Here’s a few of the flaws I’ve identified with this model:

1) Our estimated probabilities of a person doing X after we do Y can be wildly off. The less you know someone, the less likely you are to be able to guess what they are going to do. Of course, there are certain general human reactions that are predictable. Most people would be shocked if you dangled a spider in front of their face. Most people would be angry if you told them you slept with their wife (unless.. nvm). But quite a lot of the time, it is hard to assign probabilities correctly, or to even identify outcom...