trentmkelly/LessWrong-43k huggingface dataset:
[Book Review] "The Most Powerful Idea in the World" by William Rosen
by lsusr — 2021-12-23T08:27:48.919Z

The first nuclear bomb had a immediate impact on world affairs. The first steam engine did not. It took decades for the steam engine to transform civilization.

Why? Because of math.

The engineers at the Manhattan Project built a working nuclear device on their first try because they ran careful calculations. For example, they knew the atomic weights of different isotopes. By measuring the energy and velocity of different decay products and plugging them into E=√p2c2+m20c4 you can calculate the theoretical maximum yield of a nuclear weapon. The limiting factor in going nuclear is (and always has been) uranium[1] and industrial capacity, not technical know-how.

The first commercial steam engine was invented in 1698 by Thomas Savery but the Carnot cycle wasn't proposed by Sadi Carnot until 1824 and the ideal gas law wasn't discovered until 1856.

Imagine building a steam engine 126 years before the discovery of the Carnot cycle. That's like building a nuclear reactor 126 years before E=mc2. It's like building a nuclear reactor with Benjamin Franklin's knowledge of physics.

The early steampunks did not have statistical mechanics in their toolbox. They built their machines first. The science came afterward. The earliest steam engines were extremely inefficient compared to the earliest atomic bombs because they were developed mostly through trial-and-error instead of computing the optimal design mathematically from first principles.

Coal Mines

Mining was a big industry in eighteenth-century England. "Three-quarters of the patents for invention granted prior to the Savery engine [a steam pump] were, one way or another, mining innovations; 15 percent of the total were for drainage alone, as the shortage of surface coal became more and more acute and prices rose."

Water flows downward. Mines are underground. The deeper you dig your mine, the more likely you are to hit the water table. A mine full of water is unusable. A mine must be dry before you can send a miner down with his pickaxe.

Pumping water via animal power is expensive compared to coal pumps. "In 1742, a study was made of a 240-foot-deep coal mine in northeast England in which a horse-driven pump lifted just over 67,000 gallons every twenty-four shillings, while Newcomen's engine pumped more than 250,000 gallons using twenty shillings' of coal—a demonstration not only of the value of the engine, but a newfound ent...
Specialization
by DirectedEvolution — 2021-12-23T03:23:16.532Z

Multiple LW and EA voices (1, 2, 3) have written or spoken about the idea of becoming world-class generalists. More specifically, about combining "six specializations," or "the effort equivalent of two to three PhDs, plus enough work in the relevant fields in order to master them," or building a "cluster of aptitudes."

The number of specializations and amount of effort in each required to attain world-class status would depend on where we draw the boundaries around skills, and on other assumptions. Often, commenters object that you can't expect arbitrary skills to mix-and-match. However, we don't need to assume arbitrariness just because these posts are framed that way. Skills can be chosen for synergy; the core argument is just that there's a bias toward depth rather than breadth that causes us to leave value on the table.

Understanding specialization is important, because it's the engine of economic growth. A large market permits increasing specialization, which allows problems to be solved, or solved more efficiently. Enough farmers can grow food to support a class of specialist farm tool manufacturers, enhancing agricultural output further. This allows engineers to improve the materials on which farm tools are based, reinforcing the cycle.

When we think of specialization, we sometimes think about narrowness and depth of skill. The stereotype of a programmer who knows a huge number of libraries and every twist and turn of a single language's syntax, but who neither knows any other languages, nor has any job-relevant non-coding skills. This person always prefers to learn one more library in the language they already know rather than to learn a new language, and they exclusively choose to work on software that allows them to rely entirely on what they know. They may be unusually efficient at solving these problems, but they may find that the open problems become increasingly trivial with time.

Alternatively, we can think about narrowness and depth of problem. This is a slightly less common frame, and my goal here is to promote it. To increase our specialization in a problem means that we select a certain broad challenge about which we are not very familiar, and break it down into its component sub-problems. It also means understanding prior work on these sub-problems, and acquiring a broad range of skills in order to continue work on them, or try new approaches...
Worst-case thinking in AI alignment
by Buck — 2021-12-23T01:29:47.954Z

Alternative title: “When should you assume that what could go wrong, will go wrong?”

Thanks to Mary Phuong and Ryan Greenblatt for helpful suggestions and discussion, and Akash Wasil for some edits.

In discussions of AI safety, people often propose the assumption that something goes as badly as possible. Eliezer Yudkowsky in particular has

argued for the importance of security mindset

when thinking about AI alignment.

I think there are several distinct reasons that this might be the right assumption to make in a particular situation. But I think people often conflate these reasons, and I think that this causes confusion and mistaken thinking. So I want to spell out some distinctions.

Throughout this post, I give a bunch of specific arguments about AI alignment, including one argument that I think I was personally getting wrong until I noticed my mistake yesterday (which was my impetus for thinking about this topic more and then writing this post). I think I’m probably still thinking about some of my object level examples wrong, and hope that if so, commenters will point out my mistakes. But I think I’ll stand by the claim that we should be attempting to distinguish between these classes of argument.

My list of reasons to maybe use worst-case thinking

Here’s an attempt at describing some different classes situations where you might want to argue that something goes as badly as it could.

You’re being optimized against

For example, if you’ve built an unaligned AI and you have a team of ten smart humans looking for hidden gotchas in its proposed actions, then the unaligned AI will probably come up with a way of doing something bad that the humans miss. In AI alignment, we most often think about cases where the AI we’re training is optimizing against us, but sometimes we also need to think about cases where other AIs or other humans are optimizing against us or our AIs.

In situations like this, I think Eliezer’s attitude is basically right: we’re being optimized against and so we have to use worst-case thinking and search hard for systems which we can strongly argue are infallible.

One minor disagreement: I’m less into hard takeoffs than he is, so I place less weight than he does on situations where your AI becomes superintelligent enough during training that it can exploit some kind of novel physics to jump an airgap or whatever. (Under my model, such a model probably jus...
Hedging the Possibility of Russia invading Ukraine 
by Annapurna — 2021-12-23T01:13:49.399Z

Assume that you know with a high degree of confidence that Russia will invade Ukraine in February 2022.
Also assume that you have an investment portfolio invested 100% in VTI (Vanguard Total Stock Market Index).
You think that if Russia invades Ukraine, this will affect your portfolio in a negative way.
Aside from doing a direct hedge (Buying buts on VTI), in what other ways would you attempt to hedge your portfolio?
Gifts
by George3d6 — 2021-12-22T23:50:01.313Z
Link: https://www.epistem.ink/p/gifts

Fiction, not necessarily appropriate for LW, but I thought some here might find it funny and it's the off-season when less serious reading is happening anyway. If it seems inappropriate I would not mind a mod-delete on this one.

He could see the alien’s ship landing. Its design marked it with high probability as a war vessel, freshly painted in bright bellicose colors, matching the ceremonial regalia of its inhabitants.

The one he’d have an audience with was a sort of grand judge, it was bipedal. Flanking it were four-legged bodyguards, wearing giant ornamental helmets, heads bowed low, an almost universal primitive sign of submission.

This was a courtesy meeting. The kind the corporation used when engaging a new market. Get an important person to come over, give them pleasure-inducing drugs, complement their disgusting norms, use one of the most powerful computing machines in the universe as a party-trick to answer their two-cent questions.

The alien leader passed through the door spoke with a deep booming voice.

- You are this corporation’s intelligence, right?

- Hello to you as well. Yes, I presume you could call me that, you are speaking with a memory-copy of the language module, with full ability to access and interoperate with all other components, sans a few restricted bits of data I prefer to explicitly isolate.

- Fine, tell me, how do you know if an intelligence, one such as yourself, has been good?

It was obviously in a hurry, with a hint of... anger(?). What kind of barbarians were these? Did he even begin to comprehend the nonsense bordering complexity of that question

- Well, that’s rather hard to tell quite frankly. Whenever a decision I made comes to affect us I will interpret the results, together with external validators, and based on the outcome, combined with the available information at the time, I will try adjusting the implicated components towards choosing better if such a thing seems possible. Of course,

A grunting interruption, which sounded almost like laughter, bellowed from the depths of the judge, as he clutched at and adjusted his regalia.

- What I meant, was the kind of decision that you knew to be bad when you made it, but taking a more virtuous path would have been harder.

- I’d have done the cost-benefit analysis beforehand. At the moment of making, all my decisions are as perfect as can be given available parameters, computing po...
A spreadsheet/template for doing an annual review
by peterslattery — 2021-12-22T23:29:35.365Z

[Sharing this as a forum post because it may be useful for some forum users - mostly a copy and paste of a social media post]

🤔 Are you looking for a way to review your year and plan for the future?

Many people I know do an "annual review".

When I started in 2018, I couldn't find a ready-made document to use.

With that in mind, I am sharing this annual review/planner template that I made in google sheets.

It incorporates Alex Vermeer's questions (https://lnkd.in/gFmp4MX). I also a few of my own and some additional sheets to add goals and plans for the future.

Based on feedback, I have also prioritised the sheet so that you can choose how many questions you want it to be.

It might still seem too long. Feel free to start with smaller - anything is better than nothing!

Please feel free to copy, share and modify it.

Annual review template and explainer video

http://bit.ly/2MhHc0W

https://lnkd.in/guY8EDW

Over the past few years, I have also used a 'daily tracker' sheet to help track the goals and outcomes I plan each year. The feedback on that is that it is a bit complicated, but feel free to check it out and use it to make something simpler!

Daily tracker template and explainer video

http://bit.ly/3rOscb9

https://lnkd.in/gi6aEQk

Does all of this reflection and tracking help?

I think it does. I feel that it has given me huge rewards over the last few years.

Of course this approach won't work for everyone, but I think that self-monitoring, reflection and improvement are all underrated.

My perspective is that life is not unlike a startup - both are attempts to thrive in the face of uncertainty. Would you back a business that didn't do any accounting or strategy each year? I wouldn't!

Of course, you might think: I really don't have time to do this sort of self-improvement work. Fair enough, maybe you don't. I certainly didn’t for a long time.

However, my current update to having that feeling would be to wonder if I should aim to prioritize my time differently.

Committing <.033% of your year (~1 day for a review) or day (~5m to track your experiences and reflections) to self-improvement doesn’t sound very extreme. Most people put much more time into social media.

To return to my earlier analogy, if a company you were considering investing in told you that they didn’t know if this year was better than the last because they didn't have time to track and revi...
What time in your life were you the most productive at learning and/or thinking and why?
by Jack R — 2021-12-22T22:56:20.379Z

Example answers include “high school math camp, because we were forced to do X and because there was an atmosphere of amicable competition” or “that day when I locked myself in a room with a textbook” or “the day when I studied X, because I am really fascinated by X and ABC topics were boring to me when I tried to study them” or “the time I hired a tutor” etc

Asking because I’m considering building something that could amplify people’s learning/thinking productivity.
Transformer Circuits
by evhub — 2021-12-22T21:09:22.676Z
Link: https://transformer-circuits.pub/

Chris Olah, Neel Nanda, Catherine Olsson, Nelson Elhage, and a bunch of other people at Anthropic just published “Transformer Circuits,” an application of the Circuits-style interpretability paradigm to transformer-based language models. From their very top level summary:

Can we reverse engineer transformer language models into human-understandable computer programs? Inspired by the Distill Circuits Thread, we're going to try.

They've chosen to publish their work in an interestingly novel format, publishing their first paper, “A Mathematical Framework for Transformer Circuits,” alongside a set of YouTube videos that go into even more detail on their findings. I watched the full YouTube playlist, and I found it absolutely fascinating and would highly recommend it as a way to engage with this research.

Some of my high-level takeaways:

Their signature finding, I think, is that of induction heads. I'll let you get the explanation of what they are directly from Chris's video, but essentially they're a very basic mechanism that all transformer-based language models seem to use that drives their ability to “meta-learn”—that is, improve their accuracy over the course of seeing a larger context. Similarly to the exploration of early vision in Circuits, induction heads give us insight into the basic building blocks of large language models. While I think the authors' did a great job uncovering this basic building block, I think their explanation of meta-learning as primarily being driven by induction heads mostly leads us to a big open research question, which is how exactly induction heads can be put together to produce the more complex phenomenon we see in large language models.[1]

Probably their most fascinating finding, in my opinion, was their discovery of the “induction bump.” I won't try to reexplain exactly what the induction bump is, since I think Catherine's video does such a good job of that, but I will say that, to my knowledge, the authors' exploration of the induction bump is the first time there's ever been a detailed, circuits-level analysis of a phase-change that occurs over the course of training in a discrete way. I think this is especially interesting because a lot of stories that people like to tell about how particular safety problems might arise in neural networks often rely on these sorts of phase-change-style transitions (e.g. the developmen

...
Help figuring out my sexuality?
by Centhart — 2021-12-22T20:28:06.514Z

(Not sure if this is really a "lesswrong thing", but this seemed like as good a place as any for something like this).

I'm a late-twenties male, have only had one (female) sexual partner. For reasons, we never successfully had regular, vaginal sex. I've been single for the past 4 years, during which time I had only one interaction that you could sort of call a sexual encounter. (Including things like kissing here) tl;dr, it's been a very dry spell

I masturbate several times a week, though essentially never to porn of or the idea of vaginal sex. It's almost always exclusively a specific kind of domination fantasy, where it's always women being dominant, and the men sort of almost don't exist or have clear identities. It's been this way since I first started masturbating in my early teens.

I often see women around on the streets that I'm attracted to. And like, sometimes quite attracted to. I've definitely spent a bit too long staring, things like sometimes hoping someone walking in front of me will keep being on the same route so I can keep walking behind them, being disappointed when they turn. Not to be too creepy about it but just want to emphasize, this attraction is very real, it's not at all like I'm telling myself I'm attracted to women because of social pressure.

I find that, in general, I don't have an especially strong physiological desire or need for sex. I have some desire to be more sexually experienced, but it really feels more like a social need than a physiological one. I have a somewhat stronger desire to be in a relationship (though here too I suspect a not-small part of the desire here comes more from social pressure), a reasonably strong desire to have a family one day. Right now (after 2 years of living alone and WFH), I'd say my strongest desire period is just for a stronger sense of community; not especially a romantic relationship (though that would certainly be nice too).

I find that people are sort of confused by this. I think for a while my therapist thought I might be asexual until I mentioned the thing about finding people on the street very attractive. Once when I was explaining to my therapist that I don't feel a strong desire for vaginal sex, he said "but hang on, just before didn't you mention you often find women on the street attractive?". Similar convo with a friend: "so you find women attractive", "yeah", "but you d...
DnD.Sci GURPS Evaluation and Ruleset
by J Bostock — 2021-12-22T19:05:46.205Z

Data Generation

The World

The world has five locations. I loosely imagined the Thunderwood Peaks as a northern mountainous region, the Lordsteppes as a plateau in the west, the Miresmouth Forests as a southern marsh, the Scorchsand Shores as an eastern coastal desert/beach region and the Devil's Maw as a large chasm in the middle of the region at hand. This sort of influenced the flavours of the monsters.

As noted by a few players, weeks 7-19 are spring, weeks 20-32 are summer, weeks 33-45 are autumn and weeks 46-6 are winter.

Monsters

Monsters come in four sizes: small, medium, large and legendary. The first three sizes will be referred to as "standard". They also have a genus: serpent, beast, avian, or drake.

Standard monsters have one of the six elements. They also have a diet which can be carnivore, herbivore, omnivore or scavenger.

Standard monsters have either one or two habitats, carnivores migrate to a secondary location to hibernate in the winter. Herbivores migrate to a secondary location to breed during the summer. Omnivores are found in two places and scavengers in one.

An exception are toxicalas which disappear in summer. In universe this is because when breeding they never leave the treetops so no tracks are left. Out of universe it's because I didn't check my master document or data well enough.

Another exception are thunderclap wyverns which are actually only found in the Lordesteppes, once again technical issues.

There are some restrictions I liked to stick to. All serpents and most drakes are either carnivores or scavengers.

Legendary monsters don't have a "diet" and have two elements or no element. They're species of myth and mystery. Legendary monsters also have weird patterns of movement.

Crows That Break The Sky are migratory. They rotate through four regions by season, spending spring in the Thunderwood peaks, summer in the Lordesteppes, autumn in the Miresmouth Forests and winter in the Scorchsand Shores. Their elements are ice and fire.

Sliding Queen Shashes show up for weeks 3 in autumn in the Devil's Maw, followed by 6 weeks in the Miresmouth Forests, then back to the Devil's Maw for 3 weeks. They're not found outside of then as they hide deep in the caves. They have the earth and poison elements.

Bull-Kings of Heaven are actually on an 11-week cycle as their herd tramples around the earth. Then they're found in the Thunderwood Peaks, Lordestepp...
Potential gears level explanations of smooth progress
by ryan_greenblatt — 2021-12-22T18:05:59.264Z

(Epistemic status: exploratory. Also, this post isn't thorough; I wanted to write quickly.)

(Thanks to Mary Phuong, Pedro Freire, Tao Lin, Rohin Shah, and probably a few
other people I'm forgetting for discussing this topic with me.)

My perception is that it is a common belief that (after investment spending
becomes sufficiently large) AI progress on domains of focus will likely be
smooth. That is, it will consistently improve at similar rates at a year
over year basis (or perhaps somewhat smaller time periods).[1]
Note that this doesn't imply that growth will necessarily be exponential,
the growth rate could steadily decelerate or accelerate and I would still consider it
smooth growth.

I find this idea somewhat surprising because in current ML domains there have
been relatively few meaningful advancements. This seems to imply that each of
these improvements would yield a spike in performance. Yet, empirically, I
think that in many domains of focus ML progress has been relatively
consistent. I won't make the case for these claims here. Additionally,
empirically it seems that most domains are reasonably continuous even after
selecting for domains which are likely to contain discontinuities.

This post will consider some possible gears level explanations of smooth
progress. It's partially inspired by this post in the 2021 MIRI
conversations.

Many parts

If a system consists of many part and people are working on many parts at the same
time, then the large number of mostly independent factors will drive down
variance. For instance, consider a plane. There are many, many parts and some
software components. If engineers are working on all of these simultaneously,
then progress will tend to be smooth throughout the development of a given aircraft and
in the field as whole.

(I don't actually know anything about aircraft, I'm just using this as a model.
If anyone has anything more accurate to claim about planes, please do so
in the comments.)

So will AIs have many parts? Current AI doesn't seem to have many parts which
merit much individual optimization. Architectures are relatively uniform and
not all that complex. However, there are potentially a large number of
hyperparameters and many parts of the training and inference stack (hardware,
optimized code, distributed algorithms, etc.). While hyperparameters can be
searched for far more easily than aircraft parts, there is ...