trentmkelly/LessWrong-43k huggingface dataset:
Cyberwar escalation
by ChristianKl — 2022-03-23T11:29:06.404Z

From arstechnica article Sabotage: Code added to popular NPM package wiped files in Russia and Belarus:

Two weeks ago, the node-ipc author pushed a new version of the library that sabotaged computers in Russia and Belarus, the countries invading Ukraine and providing support for the invasion, respectively. The new release added a function that checked the IP address of developers who used the node-ipc in their own projects. When an IP address geolocated to either Russia or Belarus, the new version wiped files from the machine and replaced them with a heart emoji.

I would expect that there are currently officers at the FSB and also non-government actors in Russia who are thinking about how to retaliate against this attack.

If you don't have a setup for data backup and would lose important data if someone wiped your hard drive, now is the time to fix it.

If you have the keys for any package on which others depend it also makes sense to make sure that you aren't an easy target for the FSB and other Russian hackers who want to retaliate in kind.

Packaging your cyberweapon in a peace-not-war module is like it's 1984. It would be good if the FBI reacts fast and charges RIAEvangelist for hacking to prevent escalation of the conflict.

The Biden administration also warns for increased cyber attacks:

This is a critical moment to accelerate our work to improve domestic cybersecurity and bolster our national resilience. I have previously warned about the potential that Russia could conduct malicious cyber activity against the United States, including as a response to the unprecedented economic costs we’ve imposed on Russia alongside our allies and partners. It’s part of Russia’s playbook. Today, my Administration is reiterating those warnings based on evolving intelligence that the Russian Government is exploring options for potential cyberattacks.
Albert Hirschman on liberalism and nationalism
by David Hugh-Jones — 2022-03-23T10:08:53.957Z
Link: https://wyclif.substack.com/p/albert-hirschman-on-liberalism-and?s=w

Albert Hirschman wrote Exit, Voice and Loyalty at the very dawn of neoliberalism and you can think of it as a warning before its time. His story is about what happens to a liberalism based on choice and nothing more.

Suppose that your brand of cornflakes isn’t good enough. The flakes are too soggy and flavourless. You have two remedies, says Hirschman: exit to a different brand, or exercise voice by writing a letter of complaint. The obvious choice here is exit, and indeed there are many brands of cornflakes, and no obvious market failure. But now suppose we extend this idea, to the market for train transport, say — or for political services. Then there might be a tension. Maybe train companies, or politicians, will only know how to improve their services if someone tells them. If so, and if everyone relies too much on exit, then we might end up in a world where everyone is always shuffling between different mediocre services. In particular, says Hirschman, “the presence of the exit alternative… can atrophy the development of the art of voice”. Exercising voice, by writing letters of complaint, or by taking part in democracy, is difficult and takes time to get good at. It is also often best done together. It’s hard to get a protest movement going if each member worries that everyone else is planning to exit.

And so even before neoliberalism and the sovereign consumer and a whole bunch of libertarians dreaming of the market in jurisdictions, Hirschman was — if you squint quite a bit — the world’s least likely post-liberal. Too much choice and you will become weak and selfish! Decadent! You won’t be able to act together!

Hirschman interpreting for the German general Anton Dostler at a 1945 war crimes trial. Hirschman was Jewish by background and had fought for the republicans in Spain, later helping to organize escape routes from occupied Europe to the US.

I visited Czernowitz (Ukrainian: Chernivtsi, Romanian: Cernăuți) in 2018. It’s in Tim Snyder’s Bloodlands, where Stalin starved the kulaks — though Czernowitz was spared that, being in Romania until 1940 — and then Hitler exported the Jews for slaughter. The birthplace of the 20th century’s greatest poet and the centre of an extraordinary culture, a Vienna of the East. That efflorescence was long gone, but there was still a beautiful red-brick university building, and an art museum, and expensive shops and coffe...
Different Is The Generator, Not A Side Effect
by George3d6 — 2022-03-23T09:31:08.103Z
Link: https://www.epistem.ink/p/different-is-the-generator-not-a

Disclaimer: This may be extremely obvious.

i - The Weird Problem

I’ve seen theories about why smart, genial, creative, and overall high-achieving people are often “atypical”, and even discussions praising atypicality as the capital Q ality for making great discoveries, funding companies, finding everlasting love, leading armies into battle, untying the Gordian knot, and whatnot.

Let me give some examples of this to make it more vivid:

Good discoveries, products, political parties, and such come from the union of, competition between and collaboration among diverse groups of individuals. This is why it’s important to try and foster diversity and equality in matters such as academia (as opposed to construction work or tomato-picking, where affirmative action has been notable in its lack). This is why permissive and diverse cultures end up with the best minds and inventions. This is why when you get even the first hundred women to join scientific fields in the 19th century one of them becomes the first double-noble laureate (in spite of plenty of unfair obstacles). This is why it’s important to get fresh blood into companies at the management level. People that “think different” from the current “consensus” are valuable almost intrinsically.

It’s obvious the “nerds” usually missed some social developmental milestones that their peers achieved. This is your usual socially awkward programmer or mathematician somewhere on the ASD. The explanation goes that, of course, a brain is a balancing act, and people born with skills 5-STDs towards the right in manipulating abstract symbols and rotating shapes might well have skills 2-STDs towards the left in the areas correlated with collecting STDs. This just boils down to “neurodivergence”, people that are good at <insert highly conceptual activity> are not the norm, by definition, they are odd and different, and we’d expect this to manifest in more than a single area.

This is then boiled down in a bunch of “myth of the genius” type narratives that use words like diversity, odd, weird, and different in association with things like creativity, intelligence, and achievement.

While takes like the above may not be wrong, I feel like these narratives might be missing a very obvious causal mechanism that requires no diving down into the history of genius, multi-generational GWASes, or Jungian efforts of parsing and construction ...
Lessons After a Couple Months of Trying to Do ML Research
by RowanWang — 2022-03-22T23:45:44.398Z

I’m a 17 y/o who started doing transformer interpretability work around October 2021, mainly for the learning experience, but also with the goal of potentially finding something cool and interesting about transformers. I’m writing this post to consolidate some lessons I learned since then, and I hope that some of these ideas will be useful to other early career people hoping to do ML research. If you think any of this advice is wrong/misleading, please comment!

Getting Good Research Intuitions

For the majority of the past couple months, my work was exploratory. I was reading papers, talking to other researchers in the field, reimplementing things, messing around with other people’s codebases or trying weird experiment ideas. Most of this work was done to build good research intuitions: how do I set my priors such that I have a) good ideas for what’s interesting, b) principled mental models of [insert research topic] (transformer internals in my case) and c) reasonable predictions for my hypotheses and experiments?

Recognizing Promising Topics & Ideas

The “good idea for what’s interesting” part is what I think is the most important part of having good research intuitions. In my mind, “good idea for what’s interesting” encompasses both having a good idea of which topics are interesting and also having a good idea for which experiment ideas and hypotheses are promising.

Fortunately, the first part — knowing which areas of research are promising — shouldn’t be too hard if your field of interest is big enough. I think the standard advice of 1) figure out who all the important people are in the field, 2) figure out what they think is interesting and 3) figure out why works here. That plus an understanding of the importance, tractability, and neglectedness of each topic should be enough information for you to make a logical decision of which topic you want to work on. But I don’t think I would recommend going that deep on deciding which topics are interesting. What’s probably faster and better anyways is, after having a basic overview of the field, to just pick the topic that you vibe with the most and double down on that for a while. Obsessive interest is a powerful indicator of genius (

bus ticket theory of genius

) and you’ll work harder and better on the things that you have a strong internal compulsion towards.

The second part — knowing...
Progress Report 1: interpretability experiments & learning, testing compression hypotheses
by Nathan Helm-Burger — 2022-03-22T20:12:04.284Z

next post: Report 2

Intro

I got a grant from the Long Term Future Fund to refocus my career from data science & machine learning generally, to ai alignment research specifically. I feel most drawn to and suited for the interpretability angle.

I'm also currently in the AGI fundamentals technical course. I had already read most of the readings, but I've been finding the discussions insightful and useful in broadening my perspective.

For my grant-funded time, the two subgoals I am focusing on are:

improving my skillset / experience with ml interpretability research
accumulating some evidence of my ability to be a good researcher / research engineer

My current concrete project in pursuit of these goals is to:

1 implement and understand some simple transformers, in line with those described in the research by the Transformer Circuits team (

https://transformer-circuits.pub

).

2. test some simple hypotheses I have on these based on my ideas around revealing natural abstractions through applying transforms to the weights of a pre-trained model. These transforms could include: compression, factorization, clustering then encapsulation, and anything else that occurs to me or gets suggested to me. (some details on my thoughts in this previous post of mine:

https://www.lesswrong.com/posts/7vcZTHzjCFE8SbjWF/neural-net-decision-tree-hybrids-a-potential-path-toward

)

Initial steps

can I apply compression (e.g. using an infoVAE) to information coming from component(s) of the pre-trained model and examine the change in performance on baselines. (hopefully not too much of a drop. I plan to test and compare varying levels of compression to start with.) My hope is to find some representation of the space which is more intuitive to interpret. Related link:

https://youtu.be/BePQBWPnYuE

will the information bottlenecks (e.g. latent layers of the infoVAEs) be useful to examine? Will it either be easier to understand or provide a useful alternate view of the information being added to the residual stream by the relevant component?
If the answers to the above two points are yes, then can I apply the same method to a larger more complex pre-trained model (e.g. distilGPT-2)

Current progress

I'm now about 5 weeks into working on this.

A lot of the first couple weeks was reading and understanding the work being done by the Transfo...
Brain preservation to prevent involuntary death: a possible cause area
by Andy_McKenzie — 2022-03-22T12:36:16.135Z

(Cross-posted at the Effective Altruism Forum)

Previous EA discussions of this topic:

here

,

here

,

here

, and

here

. Note that these primarily focus on cryonics, although I prefer the term brain preservation because it is also compatible with

non-cryogenic methods

and anchors the discussion around the preservation quality of the brain.

See here

for more discussion of terminology.

This post is split up into two sections:

(a) Technical aspects, which discusses why I think preserving brains with methods available today may allow for revival in the future with long-term memories and personality traits intact.

(b) Ethical aspects, which discusses why I think the field may be among the most cost-effective ways to convert money into long-term QALYs, given certain beliefs and values.

In this post, I’m not discussing whether individuals should sign themselves up for brain preservation, but rather whether it is a good use of altruistic resources to preserve people and perform research about brain preservation.

Technical aspects of brain preservation

1. What is the idea behind brain preservation?

a. Brain preservation is the process of carefully preserving and protecting the information in someone’s brain for an indefinite length of time, with the goal of reviving them if technologic and civilizational capacity ever progresses to the point where it is feasible and humane to do so.

b. Our society’s definition of death has shifted over time. It depends upon the available medical technology, such as CPR and artificial respiration. In the future, the definition of death will almost certainly be different than it is today. One possible improved definition of death would be when the information in the person’s brain that they value is irreversibly lost, which is known as information-theoretic death.

c. Pausing life without causing information-theoretic death could be done with a long-term preservation method that is not yet known to be reversible today, but which has the goal of preserving enough information in the brain so that it could potentially become reversible in the future with improvements in technology. This is brain preservation.

d. With current methods, we can potentially preserve enough structure in the brain over the long term to retain the information for valued cognitive functions like long-term memories. There are mult...
Searching for post on Community Takeover
by Søren Elverlin — 2022-03-22T09:42:10.264Z

I am searching for an old LessWrong post about something like "Community takeover": The idea that a successful community (Rationalist/AI Risk) could attract people from outside, who would attempt to co-opt/hijack the agenda to focus on something else.

This is inspired by the suggestion by Carla Zoe Cremer (FHI) and Luke Kemp (CSER) that Existential Risk Studies should look into Wealth Inequality.

Thanks in advance

Søren
On expected utility, part 3: VNM, separability, and more
by Joe Carlsmith — 2022-03-22T03:05:21.073Z

(Cross-posted from Hands and Cities)

Previously in sequence: Skyscrapers and madmen; Why it can be OK to predictably lose

This is the third essay in a four-part series on expected utility maximization (EUM). This part examines three theorems/arguments that take probability assignments for granted, and derive a conclusion of the form: “If your choices satisfy XYZ conditions, then you act like an EUM-er”: the von Neumann-Morgenstern theorem; an argument based on a very general connection between “separability” and “additivity”; and a related “direct” axiomatization of EUM in Peterson (2017). In each case, I aim to go beyond listing axioms, and to give some intuitive (if still basic and informal) flavor for how the underlying reasoning works.

I. God’s lotteries

OK, let’s look at some theorems. And let’s start with von-Neumann Morgenstern (vNM) — one of the simplest and best-known.

A key disadvantage of vNM is that it takes the probability part of EUM for granted. Indeed, this is a disadvantage of all the theorems/arguments I discuss in this essay. However, as I discuss in the next essay, we can argue for the probability part on independent grounds. And sometimes, working with sources of randomness that everyone wants to be a probabilist about – i.e., coin flips, urns, etc – is enough to get a very EUM-ish game going.

Here’s how I tend to imagine the vNM set-up. Suppose that you’re hanging out in heaven with God, who is deciding what sort of world to create. And suppose, per impossible, that you and God aren’t, in any sense, “part of the world.” God’s creation of the world isn’t adding something to a pre-world history that included you and God hanging out; rather, the world is everything, you and God are deciding what kind of “everything” there will be, and once you decide, neither of you will ever have existed.

(This specification sounds silly, but I think it helps avoid various types of confusion – in particular, confusions involved in imagining that the process of ranking or choosing between worlds is in some sense part of the world itself (more below). On the set-up I’m imagining, it isn’t.)

God’s got a big menu of worlds he’s considering. And let’s say, for simplicity, that this menu is finite. Also, God has an arbitrarily fine-grained source of randomness, like one of those spinning wheels where you can mark some fraction as “A,” and some fraction as “B.” And God ...
Responsible AI communication survey
by Anca G. Rusu — 2022-03-22T00:19:11.780Z

When it comes to developing and using AI, there is an expectation of ethical and honest communication about both the technology’s opportunities and risks. But is that currently the case? This is part of the question I’m asking for my PhD research, and I want to know what you think. Please help me better understand where we stand on responsible AI communication by filling in this anonymous survey: https://forms.gle/6ZckuTQLx97fEEJk6 Thank you in advance!
Research Hamming Questions
by johnswentworth — 2022-03-22T00:16:08.722Z

We’ll start with Richard Hamming’s original question: what are the most important problems in your field?

(At this point, you should grab pencil and paper, or open a text file, or whatever.

Set a timer for at least two minutes

- or five to ten if you want to do a longer version of this exercise - and write down the most important problems in your field. The rest of the questions will be variations on this first one, all intended to come at it from different directions; I recommend setting the timer for each of them.)

Perfection

Imagine that your field achieved perfection - the ultimate theory, perfect understanding, building The Thing.

What has been achieved in the idealized version of the field, which has not yet been achieved today? What are the main barriers between here and there?

Measurement

Often, in hindsight, a field turns out to have been bottlenecked on the development of some new measurement method, ranging from physical devices like the thermometer to abstract ideas like Shannon’s entropy and information channel capacity.

In what places does it look like your field is bottlenecked on the ability to usefully measure something? What are the main barriers to usefully measuring those things?

Framing

The difficult thing, in most pre-paradigmatic and confused problems at the beginning of some Science, is not coming up with the right complicated long sentence in a language you already know. It's breaking out of the language in which every hypothesis you can write is false. [...] The warning sign that you need to 'jump-out-of-the-system' is the feeling [of] frustration, flailing around in the dark, trying desperate wild ideas and getting unhelpful results one after another. When you feel like that, you're probably thinking in the wrong language, or missing something fundamental, or trying to do something that is in fact impossible. Or impossible using the tools you have. -

Mad Investor Chaos

What are the places where your field is flailing around in the dark, trying desperate ideas and getting unhelpful results one after another? What are the places where it feels like the problem is formulated in the wrong language, and a shift to another frame might be required to ask the right question or state the right hypothesis?

Unification

Sometimes, we have a few different models, each of which works really well in different places. Maybe it feel...
Even more curated conversations with brilliant rationalists
by spencerg — 2022-03-21T23:49:26.990Z

Since August 2020, I've been recording conversations with brilliant and insightful rationalists, effective altruists, and people adjacent to (or otherwise connected) to those communities. If you're an avid reader of this site, I suspect you will recognize many of the names of those I've spoken to. Today's post contains a whole new selection of the LessWrong-relevant recordings that have come out since

my last post about this

. I hope you enjoy them! Since last time, we’ve also started adding transcripts for a number of the episodes (thanks to those of you who commented on my last post encouraging us to do this).

The curated list below is organized according to the LessWrong-relevant topics we cover in each conversation. All of these conversations can also be found

on our podcast website

or by searching for "Clearer Thinking" in just about any podcast app.

If there are other people you'd like to see me record conversations with, please nominate them in the comments! The format of each episode is that I invite each guest to bring four or five "ideas that matter" that they are excited to talk about, and then the aim is to have a fun, intellectual discussion exploring those ideas (rather than an interview).

Rationality and decision-making

Beyond cognitive biases: improving judgment by reducing noise (with Daniel Kahneman)

How can we apply the theory of measurement accuracy to human judgments? How can cognitive biases affect both the bias term and the noise term in measurement error? How much noise should we expect in judgments of various kinds? Is there reason to think that machines will eventually make better decisions than humans in all domains? How does machine decision-making differ (if at all) from human decision-making? In what domains should we work to reduce variance in decision-making? If machines learn to use human decisions as training data, then to what extent will human biases become "baked into" machine decisions? And can such biases be compensated for? Are there any domains where human judgment will always be preferable to machine judgment? What does the "fragile families" study tell us about the limits of predicting life outcomes? What does good decision "hygiene" look like? Why do people focus more on bias than noise when trying to reduce error? To what extent can people improve their decision-making abilities? How can we recogni

...