trentmkelly/LessWrong-43k huggingface dataset:
Towards a Formalisation of Returns on Cognitive Reinvestment (Part 1)
by DragonGod — 2022-06-04T18:42:24.612Z

Disclaimers

This is a rough draft of the first part of the nth post of what I hope to turn into a proper sequence investigating AI takeoff dynamics. It's not entirely self-contained material. There's a lot of preceding and subsequent context that I have not written.

If you don't immediately understand the problem I'm trying to solve, why it's important, or why I chose the approach I did, that may be why. I will try to briefly explain it though, and do what I can to contextualise it.

I've written some tangible material on one aspect of the problem and share it for feedback and validation.

Some Prior Contextualising Work

My vision for where this fits into my broader attempt to investigate takeoff dynamics is in a thesis about the "hardness" of intelligence, and especially the hardness of intelligence with respect to itself (reflexive hardness?). I haven't written up my thoughts on the hardness of intelligence in a version that I fully endorse, but here's an even rougher draft that sketches up what the concept means and how it might affect takeoff dynamics.

Do note that I don't fully endorse that draft, and my notation and formalisms in it are strictly superseded by any notations and formalisms I use here. A polished up and refined version of the ideas in that draft will probably serve as the introductory post in my sequence on investigating takeoff dynamics (once I get around to stitching together all my disparate thoughts on the topic).
In the broader literature, this post can be viewed as an attempt to formalise how to measure "returns on cognitive reinvestment" as outlined by Yudkowsky in "Intelligence Explosion Microeconomics".

A significant (and important) way in which I disagree with Yudkowsky's framing of "returns on cognitive reinvestment", is that I'm thinking almost entirely about architectural and algorithmic improvements and not improvements from access to more computational resources (FLOPs, training data, hardware, etc.).

I have some other disagreements with some of Yud's framing and approach, but they wouldn't be addressed in this post.

Introduction

I would like to describe how to measure returns on cognitive reinvestment (RCR) in a more structured manner. The aim is to define — or specify how one might define — a function that captured the concept. I am most interested in the shape of that function (is it sublinear, linear or superlinear...
Does quantum mechanics predict all the effects of relativity except gravity?
by EniScien — 2022-06-04T13:52:12.231Z

(My reading of physics books is progressing very slowly, so I'll be done in a year at best, so I'm asking this question here.) Does quantum mechanics describe the slowing down of time and the contraction of objects when approaching the speed of light, as well as other effects (special?) theory of relativity?
Deep Learning Systems Are Not Less Interpretable Than Logic/Probability/Etc
by johnswentworth — 2022-06-04T05:41:56.713Z

There’s a common perception that various non-deep-learning ML paradigms - like logic, probability, causality, etc - are very interpretable, whereas neural nets aren’t. I claim this is wrong.

It’s easy to see where the idea comes from. Look at the sort of models in, say, Judea Pearl’s work. Like this:

It says that either the sprinkler or the rain could cause a wet sidewalk, season is upstream of both of those (e.g. more rain in spring, more sprinkler use in summer), and sidewalk slipperiness is caused by wetness. The Pearl-style framework lets us do all sorts of probabilistic and causal reasoning on this system, and it all lines up quite neatly with our intuitions. It looks very interpretable.

The problem, I claim, is that a whole bunch of work is being done by the labels. “Season”, “sprinkler”, “rain”, etc. The math does not depend on those labels at all. If we code an ML system to use this sort of model, its behavior will also not depend on the labels at all. They’re just

suggestively-named LISP tokens

. We could use the exact same math/code to model some entirely different system, like my sleep quality being caused by room temperature and exercise, with both of those downstream of season, and my productivity the next day downstream of sleep.

We could just replace all the labels with random strings, and the model would have the same content:

Now it looks a lot less interpretable.

Perhaps that seems like an unfair criticism? Like, the causal model is doing some nontrivial work, but connecting the labels to real-world objects just isn’t the problem it solves?

… I think that’s true, actually. But connecting the internal symbols/quantities/data structures of a model to external stuff is (I claim) exactly what interpretability is all about.

Think about interpretability for deep learning systems. A prototypical example for what successful interpretability might look like is e.g. we find a neuron which robustly lights up specifically in response to trees. It’s a tree-detector! That’s highly interpretable: we know what that neuron “means”, what it corresponds to in the world. (Of course in practice single neurons are probably not the thing to look at, and also the word “robustly” is doing a lot of subtle work, but those points are not really relevant to this post.)

The corresponding problem for a logic/probability/causality-based mo...
Announcing the Alignment of Complex Systems Research Group
by Jan_Kulveit — 2022-06-04T04:10:14.337Z

tl;dr: We’re a new alignment research group based at Charles University, Prague. If you’re interested in conceptual work on agency and the intersection of complex systems and AI alignment, we want to hear from you. Ideal for those who prefer an academic setting in Europe.

What we’re working on

Start with the idea of an "alignment interface": the boundary between two systems with different goals:

As others have

pointed out

, there’s a whole new alignment problem at each interface. Existing work often focuses on one interface, bracketing out the rest of the world.

e.g. the AI governance bracket

The standard single-interface approach assumes that the problems at each alignment interface are uncoupled (or at most weakly coupled). All other interfaces are bracketed out. A typical example of this would be a line of work oriented toward aligning the first powerful AGI system with its creators, assuming the AGI will solve the other alignment problems (e.g. “politics”) well.

Against this, we put significant probability mass on the alignment problems interacting strongly.

For instance, we would expect proposals such as

Truthful AI: Developing and governing AI that does not lie

to interact strongly with problems at multiple interfaces. Or: alignment of a narrow AI system which would be good at modelling humans and at persuasion would likely interact strongly with politics and geopolitics.
Overall, when we take this frame, it often highlights different problems than the single-interface agendas, or leads to a different emphasis when thinking about similar problems.
(The nearest neighbours of this approach are the “multi-multi” programme of

Critch and Krueger

, parts of Eric Drexler’s CAIS, parts of John Wentworth's approach to understanding agency, and possibly

this

.

)

If you broadly agree with the above, you might ask “That’s nice – but what do you work on, specifically?” In this short intro, we’ll illustrate with three central examples. We’re planning longer writeups in coming months.

Hierarchical agency

Many systems have several levels which are sensibly described as an agent. For instance, a company and its employees can usually be well-modelled as agents. Similarly with social movements and their followers, or countries and their politicians.

Hierarchical agency: while the focus of e.g. game theory...
Eternal youth as eternal suffering
by Shmi — 2022-06-04T01:48:49.684Z

First, I must state, for the record, that I am personally all for eternal youth for anyone, for as long as they want to. (By "eternal" I mean somewhat less than infinity, maybe a few hundred years to start.)

That said, I have been listening to an interview with David Sedaris on NPR, who lived to his 60s in the shadow of his abusive father, and the reprieve came only when the father got old and demented, and eventually died. We usually talk about highly visible cases, where, say, tyrants and dictators do not give up control of, say, a country. But the real (aggregated) suffering is at much lower levels, where the torture, physical or emotional is mostly hidden and personal. Unless you are a negative utilitarian, this suffering probably does not outweigh the benefits of feeling young and full of energy for centuries, but it is still a lot of suffering with no recourse, and often with little to no visibility. I am not sure how to feel about this worse-than-Omelas possibility.
D&D.Sci June 2022: A Goddess Tried To Reincarnate Me Into A Fantasy World, But I Insisted On Using Data Science To Select An Optimal Combination Of Cheat Skills!
by abstractapplic — 2022-06-04T01:28:18.301Z

This is an entry in the 'Dungeons & Data Science' series, a set of puzzles where players are given a dataset to analyze and an objective to pursue using information from that dataset.

“So I expect you know what this is about . . .” says the strange woman, as you both float gently in the void between worlds.

You find yourself tongue-tied, but your expression successfully communicates that you have no idea who she is, why you’re here, or how her hair manages to keep that shape.

“You know, the thing where someone dies, then a goddess gives them superpowers and sends them to another universe to defeat a Demon King?” She examines you quizzically. “It’s pretty common.”

“I’ve never heard of anything like that,” you say, uselessly. “I was just cycling to the office . . .” You trail off as the implications of everything she just said hit you like a series of trucks (a comparison which, as your recent memories return, you realize you’re extremely qualified to make).

The apparently-goddess looks at you with a little sympathy and a lot of impatience. “I know you’re going through a lot right now, but like I said, there are a bunch of other people in your situation. So how about you just pick your cheat skills, let me move on to the next hero, and process all this later?”

As most of your mind continues to bluescreen, a hardcoded habit from your dayjob assumes control and starts running on automatic. “A . . . bunch of people, you say?”

“Yes. So I-”

“How big a bunch? Hundreds, thousands?”

“Hundreds of thousands. It’s a big multiverse.”

“So in other words, you have access to a data source from which we could plausibly derive statistically significant results, and thereby inform my decision?"

“. . . huh?”

After some explanations, some pleading, and a promise to share any insights she might find useful, the goddess leaves you in the care of one of her lesser angels, who has been taking notes on her previous heroes’ characteristics, choices and successes.

(As she departs, she expresses surprise that you’re making your decision based solely on win rates, and not considering how fun the various abilities could be or how well they match your personality. You reply that while you might have taken fun into consideration if you’d been more confident in your success, right now you’re just doing whatever gives you the best odds of winning: leaving a Demon King undefeated seems like it could be ...
The Burden of Worldbuilding
by lsusr — 2022-06-04T01:15:44.078Z

When I was studying special relativity one of the things which caught my attention was how, because the speed of light c was a constant, you could just set it to c=1. Setting c equal to 1 caused space and time to have the same units. One nanosecond is slightly less than to one foot. The Lorentz Equation 1γ2+v2=1 is beautiful when written with c=1.

I started designing a system of absolute units. My system would be elegant. No more would humanity's perspective be shackled by our pre-relativistic Dark Age units of measurement.

About halfway through the design process I realized I was designing a bad system. My units were impractical. Not (just) because of the coordination problems required to actually get everyone to change units. Not (just) because Plank's Constant h is over thirty orders of magnitude shorter than the distances we use for everyday tasks. My system was fundamentally broken because the inelegant space-time non-equivalence of real-world units is a feature, not a bug.

Suppose we used the same units for time as for distance. Meters are replaced with nanoseconds. If I say something is "fifteen minutes away" it's no longer obvious whether I'm talking about a fifteen-minutes of time or the distance from the Earth to the Sun and back again. Usually whether I am discussing space or time can be inferred, but inference shouldn't be necessary. The purpose of language is to communicate effectively the first time. A little bit of redundancy improves communication.

The world is a complex place. The reason things are the way they are is often non-obvious. That's why if you propose a change to the world then the burden of proof is on you to fill in all of the burdensome details.

I think the burden of worldbuilding is why the philosophy in science fiction tends to be so good. Science fiction authors are required to worldbuild which forces their ideas to be somewhat grounded in reality.
Longtermist Consequences of a New Dark Age?
by David Udell — 2022-06-03T23:00:07.051Z

Inspired by a talk I had with James Faville.

Suppose that, sometime this decade, designer superbugs end up collapsing civilization as we know it, but not wiping out the entire species. Things are catastrophic, but enough people possess immunity to any particular act of bioterrorism that humanity ultimately pulls though. However, many nation-states collapse and AI capabilities progress, along with much else, regresses for many decades.

From the perspective of the deep future and the 1050 good lives that might inhabit it, how bad was this dark age? Did this narrowly avert a far greater catastrophe, by destroying a global civilization about to wipe out humanity via unaligned AGI, or did it just delay that destruction a century or so? Does a new dark age put us on a course to values unlike our own, such that even if survivor-humanity later succeed at alignment, much of what we care about is lost?
The STEM Attractor
by David Udell — 2022-06-03T22:21:39.986Z

No significant planecrash spoilers this time.

Keltham will spend the next five minutes extemporizing an elevator pitch on Civilization, the nice things that it has, and how while there's lots of specific nice things, the much more important thing is going into an attractor made out of harmonizing bits of Law that lets you start figuring out those things yourself.

--Eliezer, planecrash

The human brain is a haphazard thing, thrown together by idiot evolution, as an incremental layer of icing on a chimpanzee cake that never evolved to be generally intelligent, adapted in a distant world devoid of elaborate scientific arguments or computer programs or professional specializations.

It's amazing we can get anywhere using the damn thing. But it's worth remembering that if there were any smaller modification of a chimpanzee that spontaneously gave rise to a technological civilization, we would be having this conversation at that lower level of intelligence instead.

--Eliezer, "True Sources of Disagreement"

Humans are the species that evolved to be just smart enough to "kill off" evolution and take over Earth for ourselves. So evolution progressively stopped training Homo sapiens as we gained more and more control over the dangers and resources present in our environment. No other species ever did this, so where did we uniquely get the power to escape our training process?

There exists, as an objective affordance of our universe to the possible algorithms that can exist within it, a 'STEM attractor.' Algorithms that can do some science and technology are empowered to do more and better science and technology, with the help of their newfound theories and tools. Thus, civilizations that start to learn any chunk of STEM are prone to pick up more and more of it, and this attraction is stronger the closer you get to "scientific maturity" (knowing all possible science relevant to controlling the world you interact with). This is just an attraction: it's still possible for civilizations to stall out in their scientific progress, either because they stumbled into an unescapable totalitarian world government or because they completely destroyed themselves, but absent those defeaters the civilization will tend towards learning more and more science.

The STEM attractor is a feature of all civilizations that evolve in our universe; probably, distant alien cultures have completely ...
Book Review: Talent
by Zvi — 2022-06-03T20:10:02.425Z

Tyler Cowen will often praise a book by saying it is interesting on every page. Together with Daniel Gross, he has written a book that very much matches this description.

The topic is talent: How to search for it, how to identify it, interview and otherwise evaluate it, how to recruit it, how to buy it low and sell it high. The book asks, in various ways, what matters? What makes one person succeed or provide lots of value, while another fails and provides little? How can we put together a portfolio of the first type of people at (in all senses) a good price?

There is a unifying philosophy and way of looking at people and the world underlying all of it. I mostly subscribe to the bulk of it, and especially to a core kind of economic thinking.

As part of this way of thinking, the goal is not to interview or evaluate people in an adversarial framework. You don’t want to ‘get’ them by tricking them and making them look foolish. What does that accomplish? You want to ‘get’ them by understanding who they are, and you learn by asking in a situation where they feel they can trust you, about things you both care about.

Don’t approach interviewing as a process where you are trying to trick or trip up the other person. First, that is intrinsically a bad way to behave. Second, once the person realizes what you’re doing, they’ll stop trusting you and, in most cases, become guarded. That will make it harder for you to determine whether the person is a good match for your job or assignment, not to mention harder to land them should you decide they are. (415)

…

Most importantly, it should be a question you really want to know the answer to. So if the person worked in a ball bearings factory in Cleveland, ask yourself if you care more about ball bearings or more about Cleveland. And your follow-up questions should also reflect real interests of yours. (420)

…

There is only one real and credible protection against those phoniness-spotting abilities: to be actually worthy of someone’s trust. When we are genuinely curious it draws people out, and as we show interest, they begin to trust us. This doesn’t mean you shouldn’t ask hard questions and challenge answers that don’t add up—it’s simply that you must be genuine. (427)

I strongly agree with this. You’re still going to want to test them and provide challenges so you can see how they react, but there’s essentially never any call for a...
On The Spectrum, On The Guest List: (iii) Etiquette
by party girl — 2022-06-03T18:44:12.571Z
Link: https://onthespectrumontheguestlist.substack.com/p/etiquette

See also my LessWrong preface to this series, and previous installments: (i) TAO Downtown and (ii) LAVO

"The sparkler is a live firework that shoots out as high as eight inches. It burns for up to fifty-five seconds..."

‘Well, I'm glad you made it. To…’ he glances around, as if searching for something toastworthy in the room, past me, to the all-male gaggle of R's friends-cum-clients, none of whom you would mistake for Brazilian or Columbian, pulls focus back to me, whips left to note the winged robot on stage next to us, back to me…

‘To _____!’

‘What!?’

T says, ‘I said, “To _____!” ’ He’s doing the sort of loud talking you do as a last resort before shouting, which is never attractive on anyone, no matter the necessity. It's like the volume equivalent of speed-walking. I can’t hear him, I hope I’m not toasting to anything obscene. I’m just shouting.

‘I… …sure!’

We’re at Etiquette, a pop-up nightclub in Midtown with a significant dinner theater component. And we’re very underground—I spiraled down at least four flights’ worth of marble stairs, behind a trundling bouncer who escorted me and didn’t even slow down as we passed coatcheck so I could bare my teeth at those nickel-and-dimers, after I walked up to the club’s entrance on Forty-Sixth and was whisked inside before I could even spit out my name or R’s. Can’t wait to party it up at M42 next week. Very cavernous in here, too. Caverns, it’s always caverns. Where are all those second-wave feminists decrying the skyscraper as the symbol of phallocentrism now, huh? Oh, right. Face control.

R came to fetch me where the bouncer had dumped me, standing in the room’s entrance, transfixed by the whole spectacle, his side hug snapping me out of it:

‘Quite the place! And you didn’t want to work for these people?’

‘It is, isn’t it? Yeah maybe, one day!…’

‘Oh c’mon! Six figures, to wear getups each Saturday, like…’ I tilt my head in a bit, just as an off-brand Ziggy Stardust makes his way between us and walks out of the room, Ground Control, Gonna Use The John. R just shrugs, and I stow my Man Who Sold The World quip. As we start walking he slips in a mention of his night post-LAVO, involving one of the French girls, I fail to render it here because so often men garden-path their way through brags like that, it always glitches out my memory circuits. Not that I’m surprised, much:

Most promoters were

...