trentmkelly/LessWrong-43k huggingface dataset:
Measurement, Optimization, and Take-off Speed
by jsteinhardt — 2021-09-10T19:30:57.189Z

Crossposted from my blog, Bounded Regret.

In machine learning, we are obsessed with datasets and metrics: progress in areas as diverse as natural language understanding, object recognition, and reinforcement learning is tracked by numerical scores on agreed-upon benchmarks. Despite this, I think we focus too little on measurement---that is, on ways of extracting data from machine learning models that bears upon important hypotheses. This might sound paradoxical, since benchmarks are after all one way of measuring a model. However, benchmarks are a very narrow form of measurement, and I will argue below that trying to measure pretty much anything you can think of is a good mental move that is heavily underutilized in machine learning. I’ll argue this in three ways:

Historically, more measurement has almost always been a great move, not only in science but also in engineering and policymaking.
Philosophically, measurement has many good properties that bear upon important questions in ML.
In my own research, just measuring something and seeing what happened has often been surprisingly fruitful.

Once I’ve sold you on measurement in general, I’ll apply it to a particular case: the level of optimization power that a machine learning model has. Optimization power is an intuitively useful concept, often used when discussing risks from AI systems, but as far as I know no one has tried to measure it or even say what it would mean to measure it in principle. We’ll explore how to do this and see that there are at least two distinct measurable aspects of optimization power that have different implications (one on how misaligned an agent’s actions might be, and the other on how quickly an agent’s capabilities might grow). This shows by example that even thinking about how to measure something can be a helpful clarifying exercise, and suggests future directions of research that I discuss at the end.

Measurement Is Great

Above I defined measurement as extracting data from [a system], such that the data bears upon an important hypothesis. Examples of this would be looking at things under a microscope, performing crystallography, collecting survey data, observing health outcomes in clinical trials, or measuring household income one year after running a philanthropic program. In machine learning, it would include measuring the accuracy of models on a dataset, measuring the variabilit...
Paths To High-Level Machine Intelligence
by Daniel_Eth — 2021-09-10T13:21:11.665Z

This post is part 3 in our

sequence on Modeling Transformative AI Risk

. We are building a model to understand debates around existential risks from advanced AI. The model is made with

Analytica

software, and consists of nodes (representing key hypotheses and cruxes) and edges (representing the relationships between these cruxes), with final output corresponding to the likelihood of various potential failure scenarios. You can read more about the motivation for our project and how the model works in the

Introduction post

. The previous post in the sequence,

Analogies and General Priors on Intelligence

, investigated the nature of intelligence as it pertains to advanced AI.

This post explains parts of our model most relevant to paths to high-level machine intelligence (HLMI). We define HLMI as machines that are capable, either individually or collectively, of performing almost all economically-relevant information-processing tasks that are performed by humans, or quickly (relative to humans) learning to perform such tasks. Since many corresponding jobs (such as managers, scientists, and startup founders) require navigating the complex and unpredictable worlds of physical and social interactions, the term HLMI implies very broad cognitive capabilities, including an ability to learn and apply domain-specific knowledge and social abilities.

We are using the term “high-level machine intelligence” here instead of the related terms “human-level machine intelligence”, “artificial general intelligence”, or “transformative AI”, since these other terms are often seen as baking in assumptions about either the nature of intelligence or advanced AI that are not universally accepted.

In relation to our model as a whole, this post focuses on these modules:

The module AI Progression & Requirements investigates when we should expect HLMI to be developed, as well as what kind of HLMI we should expect (e.g.,

whole brain emulation

, HLMI from current deep learning methods, etcetera). These questions about the timing and kind of HLMI are the main outputs from the sections of the model in this post, influencing downstream parts of the model. The timing question, for instance, determines how much time there is for safety agendas to be solved. The question regarding the kind of HLMI, meanwhile, affects many further cruxes, including which safety agendas are likely necessary to sol...
Handling Identical USB Sound Cards
by jefftk — 2021-09-10T12:30:06.100Z

My

Pi Sound Box has two output channels,
each on a

USB
sound card. One has a mic input and runs my

whistle synth,
the other has no audio input and runs

fluidsynth. This worked fine,
up until my first gig out (busking in the park) where it decided to
swap the devices: my whistle card moved from

hw:0,0 to

hw:1,0, and vice-versa.

Here's someone else with the
same problem in 2011.

Cheap USB devices don't have a serial number, and these two sound
cards present identically. On the other hand, it's possible to
identify them by their location in the USB tree:

$ for x in /sys/bus/usb/devices/*/product; do
echo $x $(cat $x)
done
/sys/bus/usb/devices/1-1.2.1.3/product USB Audio Device
/sys/bus/usb/devices/1-1.2.1/product 4-Port USB 2.0 Hub
/sys/bus/usb/devices/1-1.2.2/product HP Elite USB Keyboard
/sys/bus/usb/devices/1-1.2.4/product AXIS-49 2A
/sys/bus/usb/devices/1-1.2/product 4-Port USB 2.0 Hub
/sys/bus/usb/devices/1-1.3/product USB Audio Device
/sys/bus/usb/devices/usb1/product DWC OTG Controller

(lsusb --tree can do this too, but it's harder to read)

Since this is based on physical ports, it will be consistent across
reboots.

It looks like you can write udev rules to tell the system
how to map 1-1.2.1.3 to hw:*,0 (docs). Unfortunately, this
was getting well beyond what I considered fun, so I gave up. I added
a key combination to swap the outputs (commit),
so if I notice it's wrong while setting up at a gig I can fix it.
Possibly at some point I'll come back and deal with udev?
How do you decide when to change N95/FFP-2 masks?
by ChristianKl — 2021-09-10T07:40:41.353Z

I lack a gears model of how N95/FFP-2 masks works that tells me when it makes sense to change to a new mask. The same goes into how to interact with the mask to maximize it's Does someone have a gears model and a got idea when it makes sense to change masks to a new one?
[linkpost] Political Capital Flow Management and the Importance of Yutting
by mike_hawke — 2021-09-10T07:27:23.009Z
Link: https://everythingstudies.com/2021/08/24/political-capital-flow-management-and-the-importance-of-yutting/

From the venerable John Nerst. "Yut" kind of reminds me of ADBOC.
Bayeswatch 8: Antimatter
by lsusr — 2021-09-10T05:01:48.845Z

Vi woke up to the sound of Miriam disassembling the scout's control system.

"Please tell me you're not scuttling our ride home," said Vi.

"I'm not scuttling our ride home. Vaccinate yourself in case this leads to another Z-Day," said Miriam.

The suitcase was labeled with an orthonormal cube Vi recognized from her training as the Mark of Pandora. Vi opened the suitcase. One half contained vials, syringes and needles. The other half contained a nest of wires, circuitboards and heatsinks wrapped around a slab of computronium. She injected the vaccine into her upper arm muscle.

"You didn't tell me what you're doing," said Vi.

Miriam tossed Vi a sphere the size of a golf ball. It was coated in depleted uranium. "This is a neutrino beacon. It can be remotely activated. Well, technically it's an anti-neutrino beacon. But we call them neutrino beacons. It's not like matter neutrinos are used in your microwave oven."

"We've been flying around with an antimatter bomb?" said Vi.

"We've been flying around with a harmless antimatter bomb. Neutrinos barely interact with regular matter. Covert Bayeswatch teams use these to signal for rescue from hostile territory. Bayeswatch's sensor array could detect us if they chose to remotely activate it," said Miriam.

"Just Bayeswatch?" said Vi.

"A global neutrino triangulator sensitive enough to detect one of these is beyond the purchasing power of mere nation-states," said Miriam.

"Speaking of countries, do you have any idea who or where our enemy is?" said Vi.

"They can't be anywhere with a Bayeswatch-backed police state. They're not from China, Russia, Western Europe, the United States or anyone else in the Alliance. Our enemy has to be somewhere with significant high-tech industrial capacity. We can cross out the Philippines, Mozambique and Vietnam," said Miriam.

"Taiwan then? Or Singapore?" said Vi.

"Maybe. They could be based out of Argentina, Brazil or South Africa. We just don't have enough information," said Miriam.

"We'll have to get some more," said Vi.

Miriam called a taxi. The taxi carried them 20 miles before poachers shot out its tires. The taxi was Three Laws Compliant. It could flee humans but it was forbidden from fighting back. The poachers used a heavy duty diamond-cut saw to extract the computronium core from car's navcomputer. They left the taxi for dead. Miriam and Vi hitchhiked with the poachers to the edge of the wastela...
The Blackwell order as a formalization of knowledge
by Alex Flint — 2021-09-10T02:51:16.498Z

Financial status: This is independent research, now supported by a grant. I welcome further financial support.

Epistemic status: I’m 90% sure that this post faithfully relates the content of the paper that it reviews.

In a recent conversation about what it means to accumulate knowledge, I was pointed towards a paper by Johannes Rauh and collaborators entitled Coarse-graining and the Blackwell order. The abstract begins:

Suppose we have a pair of information channels, κ1, κ2, with a common input. The Blackwell order is a partial order over channels that compares κ1 and κ2 by the maximal expected utility an agent can obtain when decisions are based on the channel outputs.

This immediately caught my attention because of the connection between information and utility, which I suspect is key to understanding knowledge. In classical information theory, we study quantities such as entropy and mutual information without the need to consider whether information is useful or not with respect to a particular goal. This is not a shortcoming of these quantities, it is simply not the domain of these quantities to incorporate a goal or utility function. This paper discusses some different quantities that attempt to formalize what it means for information to be useful in service to a goal.

The reason to understand knowledge in the first place is so that we might understand what an agent does or does not know about, even when we do not trust it to answer questions honestly. If we discover that a vacuum-cleaning robot has built up some unexpectedly sophisticated knowledge of human psychology then we might choose to shut it down. If we discover that the same robot has merely recorded information from which an understanding of human psychology could in principle be derived then we may not be nearly so concerned, since almost any recording of data involving humans probably contains, in principle, a great deal of information about human psychology. Therefore the distinction between information and knowledge seems highly relevant to the deception problem in particular, and, in my opinion, to contemporary AI alignment in general. This paper provides a neat overview of one way we might formalize knowledge, and beyond that presents some interesting edge cases that point at the shortcomings of this approach.

I suspect that knowledge has to do with selectively retaining and discarding ...
The Best Software For Every Need
by Tomás B. — 2021-09-10T02:40:13.731Z

When I first started programming, I didn't use a terminal multiplexer and finding tmux was a sort of revelation. I joked once on discord that "life before tmux was not life". It strikes me there are probably many other programs that I am not aware of that would be useful to know about.

I've occasionally found Luke's The Best Textbooks on Every Subject thread useful, so I thought a similar thread about software may be interesting.

Here are the rules:

Post the name of a program for a given need.
You must have tried at least 2 other programs designed for the same/similar class of problems.
You must briefly name the other programs you have tried and why you think your chosen program is superior to them.
Framing Practicum: Comparative Advantage
by johnswentworth — 2021-09-09T23:59:09.468Z

This is a

framing practicum

post. We’ll talk about what comparative advantage is, how to recognize applications of comparative advantage in the wild, and what questions to ask when you find it. Then, we’ll have a challenge to apply the idea.

Today’s challenge: come up with 3 examples of comparative advantage which you have not seen before. For each one, say what the different objectives are, and what the different components/subsystems are. They don’t need to be good, they don’t need to be useful, they just need to be novel (to you).

("What do different objectives and subcomponents have to do with comparative advantage?" I hear you ask, "I don't remember anything about that from econ 101." This is framing practicum - we want to apply the frame of comparative advantage to new kinds of systems, not just trade-between-nations or whatever. So, we're going to present it a bit differently from what you're used to.)

Expected time: ~15-30 minutes at most, including the Bonus Exercise.

What’s Comparative Advantage?

Suppose we’re running a fruit-growing company, Fruit Co, with many different orchards which can each grow apples or bananas. Each farm has different soil, different weather, different initial conditions, etc, and therefore faces different

opportunity costs

for growing each fruit. For instance, the Xenia site might be able to grow 1 extra unit of apples/yr at the cost of 0.5 units of bananas (by replacing their least-effective banana grove with apple trees) or vice versa, while the Yuma site may be able to grow 1 extra unit of apples/yr at the cost of 1 unit of bananas or vice versa.

ΔApples	ΔBananas
Xenia	1	:	0.5
Yuma	1	:	1

Claim: given these numbers, the company can achieve a pareto increase in their fruit production. How? Well, they can produce one more unit of apples at the Xenia site (missing out on 0.5 units of bananas), and produce one less unit of apples at the Yuma site (using those resources to produce 1 extra unit of bananas). Overall, the amount of apples produced stays the same, but the amount of bananas produced increases by 0.5 units.

Intuitively: each site specializes a little more in whatever fruit they have a comparative advantage (aka relative advantage) in growing. We have multiple goals (growing more apples, and growing more bananas), and multiple subsystems which we can independently adjust to contribute to those goals (Xenia and Yuma orcha...
GPT-X, DALL-E, and our Multimodal Future [video series]
by bakztfuture — 2021-09-09T23:05:51.740Z
Link: https://youtube.com/playlist?list=PLza3gaByGSXjUCtIuv2x9fwkx3K_3CDmw

Video series talking about how GPT-3, DALL-E, and Multimodal AI models of the future could change human creativity in foundational ways.

Total of 19 videos, so far is at #8. New Video is released every week day.
You can get feedback on ideas and external drafts too
by Ruby — 2021-09-09T21:06:04.446Z

Due to some editor bugs that only affect admins, a few crucial edits were missing from the recent announcement that LessWrong is providing feedback and proofreading on drafts as a service.

Crucially, you do not need to have a LW draft to get feedback. Feel free to use Intercom (bottom right) to ask for feedback on a post idea (no draft) or on an external draft, e.g. Google Doc.

The Intercom button looks like this:

If you ever disabled Intercom in your user settings, you'll need to re-enable it. Alternatively you can reach us via email (team@lesswrong.com).
The alignment problem in different capability regimes
by Buck — 2021-09-09T19:46:16.858Z

I think the alignment problem looks different depending on the capability level of systems you’re trying to align. And I think that different researchers often have different capability levels in mind when they talk about the alignment problem. I think this leads to confusion. I’m going to use the term “regimes of the alignment problem” to refer to the different perspectives on alignment you get from considering systems with different capability levels.

(I would be pretty unsurprised if these points had all been made elsewhere; the goal of this post is just to put them all in one place. I’d love pointers to pieces that make many of the same points as this post. Thanks to a wide variety of people for conversations that informed this. If there’s established jargon for different parts of this, point it out to me and I’ll consider switching to using it.)

Different regimes:

Wildly superintelligent systems
Systems that are roughly as generally intelligent and capable as humans--they’re able to do all the important tasks as well as humans can, but they’re not wildly more generally intelligent.
Systems that are less generally intelligent and capable than humans

Two main causes that lead to differences in which regime people focus on:

Disagreements about the dynamics of AI development. Eg takeoff speeds. The classic question along these lines is whether we have to come up with alignment strategies that scale to arbitrarily competent systems, or whether we just have to be able to align systems that are slightly smarter than us, which can then do the alignment research for us.
Disagreements about what problem we’re trying to solve. I think that there are a few different mechanisms by which AI misalignment could be bad from a longtermist perspective, and depending on which of these mechanisms you’re worried about, you’ll be worried about different regimes of the problem.

Different mechanisms by which AI misalignment could be bad from a longtermist perspective:

The second species

problem: We build powerful ML systems and then they end up controlling the future, which is bad if they don’t intend to help us achieve our goals.

To mitigate this concern, you’re probably most interested in the “wildly superintelligent systems” or “roughly human-level systems” regimes, depending on your beliefs about takeoff speeds and maybe some other stuff.

Missed opportunity: We build pretty powerful ML

...
Thinking About College Funding
by jefftk — 2021-09-09T19:20:05.053Z

A few years ago

I wrote
about how, if you're trying to decide between direct work and earning
to give, the college price discrimination system of financial aid
can push towards direct work. It's a bit of a niche thing, since it
really only applies if:

Your children are likely enough to be admitted to the kind of
institution that commits to meeting 100% of demonstrated financial
need, or otherwise has a similar "100% effective tax rate".

You're not be very interested in saving money for your own
future use. The CSS Profile suggesting 5%/y for parental assets means
that with three kids at 4y each you might be asked for 60% of assets.

Your earnings need to be low enough just before and during
college, either because your career has never been highly lucrative or
because you are willing to change your line of work for that time
period.

Now that we have a third child I'm thinking about this again, and in
our particular case I now think we are pretty unlikely to take the
direct-work-minimize-income approach:

Nora is unlikely to be in college at the same time as her older
siblings, since she's five years younger. This would mean a much
longer period of applying that approach, ~10y instead of ~5y.

The CSS Profile is very comprehensive in identifying assets,
and likely to continue to get more so. If we chose that approach the
value of our house and retirement assets would be unusually high for
our income. This could raise flags, and many places do consider them
to some extent.

I've been lucky in how well compensated I am, and if this
continues then paying sticker prices for college would not impact our
ability to donate.

I continue to enjoy working as a programmer, and think it's
pretty likely I'll want to be doing something similar in ten years.

This also means that a 529 college savings
plan probably makes sense for us. This is an investment option
where gains are not taxed as long as funds are limited to educational
expenses. It's a lower priority than donations or retirement savings,
but if we have additional money left over the lack of capital gains
tax is nice.

Comment via: facebook
Does blockchain technology offer potential solutions to some AI alignment problems?
by pilord — 2021-09-09T16:51:39.252Z

I am not an expert in blockchain technology nor AI, so I am asking this community this question as I imagine it is already much explored.

My understanding is that AI alignment is an existential risk because of the fear of an intelligence explosion, and the related paperclip maximization risks (among others). This appear to cause a substantial amount of fear in some people. However, my initial reaction is more sanguine, as it seems to me that we may have tools to avoid these risks - namely, the concentration of a far more capable power than us.

Specifically, two features of blockchain technologies that would appear to reduce the power of a hostile AI would be a) transaction costs and b) decentralization. Say the world interfaced using blockchain technology. Then the transaction costs of taking a series of adversarial action from this hostile AI would either bankrupt the AI, or stall enough to allow people to fork and abandon that AI. But the greater point may be decentralization. If the incentives are set up to minimize collusion, and constraints on concentration imposed, wouldn't that weaken the power that any given AI could accumulate? That is, if we set up the system such that no one party can dominate, then even a superintelligent AI could not dominate.

That is not the cleanest way to express my intuition, so maybe this a better way: my understanding is that crypto is secured not by trust, guns, or rules, but by fundamental computational limits (regarding e.g., the factorization of large numbers for the cryptographic hash). Those fundamental limits imposed by nature would apply to an AI as well. By connecting money and power to the fundamental limits imposed by nature / math, the ability of any one actor (including AI) to gain arbitrary power without the consent of everyone else would be limited, as even arbitrary intelligence can't do the physically impossible. That is, by imposing a constraint on the ability to accumulate power in general, that constraint is also imposed on AI.

Where does this logic fail me? I know the logic I presented here is not at all air tight - for example, I could imagine some AI exploiting some incentive structure, some weakness in some chain, or using collusion to make my point moot. These topics are deep, and I know I'm missing a lot of the nuance here. Still, it seems that many people have thought deeply on this subje...