trentmkelly/LessWrong-43k huggingface dataset:
The Energy Requirements and Feasibility of Off-World Mining
by clans — 2023-01-30T15:07:59.872Z
Link: https://locationtbd.home.blog/2023/01/30/the-energy-requirements-and-feasibility-of-off-world-mining/

Space Prospecting

It’s been an enticing subject for more than half a century now and continues to capture the imaginations of extremely ambitious engineers and powerful capital allocators alike. It’s also somewhat viable: although most asteroids (and almost all of those that are considered to be “near-Earth”) have ore concentrations lower than those found terrestrially on Earth, there is a class of “metallic” (M-type) asteroids likely rich with platinum group metals (PGMs). In high demand, PGMs like Pd, Os, and Ir are difficult & pollutive to mine and thus are among the most expensive materials known to exist at >$500,000/kg.

Companies like

AstroForge

have recently raised millions on the business case for returning such metals prospected from asteroids, partly because of the enormous down mass potential (~20-50t) of SpaceX’s Starship. But the asteroid mining field is littered with bad ideas and failed projects. For one, Planetary Resources, a 2010s venture backed by Larry Page, thought it viable to extract volatiles from asteroids and robotically produce chemical fuel for in-space depots. Their money made it as far as a few test launches before running out and their assets acquired by a crypto company. It turns out it’s a bad idea to mine near-Earth asteroids for pretty much anything. Water is mixed in with rock and must be heated >400° C to use. Volatiles are not thermally stable and outgas in the vacuum. Metals are hit and miss. Asteroids are also more like piles of rubble, difficult to land on, and overall very small – so you’d want to land on quite a few.

To make the case for asteroids worse, we don’t currently have a ton of conviction in their elemental composition. Statistical issues mar researchers: we don’t understand the distribution of samples we have (i.e. how often they’re from the same asteroid), how measurements are affected by the atmosphere, and old studies have extrapolated seemingly erroneous values [1]. According to leading researchers like

Kevin Cannon

, PGM concentration on M-type asteroids can vary from 6-230 ppm, which is still higher than most Earth ore concentrations at the bound.

Besides M-type asteroids, which have material that could be economically shipped back to Earth, there are asteroids with modest concentrations of materials that are only useful for supplying projects off-world, in-situ. These are volatiles like water t...
Whatever their arguments, Covid vaccine sceptics will probably never convince me
by contrarianbrit — 2023-01-30T13:42:56.028Z
Link: https://thomasprosser.substack.com/p/whatever-their-arguments-covid-vaccine

Recently, Covid vaccine sceptics have enjoyed a growing profile. In an

article

on Bari Weiss’s Substack, Vinay Prasad and John Mandrola draw attention to sudden deaths and

studies

which link the vaccine with myocarditis, an inflammation of the heart muscle. Reading such articles, I can say little in response; Prasad is a professor of epidemiology and Mandrola is a practicing cardiologist and their arguments seem reasonable.

But Covid vaccine sceptics will probably never convince me.

This reflects my attitude to expertise and its institutional foundations. Whilst I have followed debates about the vaccine somewhat, my knowledge is very limited. I am not a medical expert, my inability to counter the arguments of Prasad and Mandrola reflecting this. Being unacquainted with a subject, one is defenceless against a knowledgeable interlocutor. I have a friend who is a biologist and creationist. When he explains his scepticism with evolutionary theory, going into great technical detail, I cannot say a thing in retort.

Given these deficits in my knowledge, reliance on mental shortcuts is reasonable, scholars calling these heuristics. One cannot make all one’s decisions; we have limited time and confront issues of great complexity, particularly in modern societies. Therefore, we use heuristics constantly. Currently, the vast majority of

experts

support

vaccination programmes.

Clearly, one cannot trust the state indiscriminately. My wife grew up in Communist Poland, the government lying habitually. Are liberal democracies different? Whilst I (and many others) would argue this, these systems being predicated on openness, such states are unworthy of unconditional trust. Within living memory, liberal democracies have endorsed lobotomies for mental illness and thalidomide for morning sickness. Recently, such states have embraced ideological positions which I consider questionable, the transgender issue involving medicine.

Why are Covid vaccines different? Aside from the rarity of scandals such as thalidomide, I have faith in the formation of expert consensuses, following years in academia. In another area in which the role of expertise has been disputed, the Brexit debate, I have some technical knowledge. Whilst Brexit experts made mistakes (see below), I think that prognoses proved sound and consensuses developed from the bottom-up.

When Covid vaccine sceptics writ...
Simulacra Levels Summary
by Zvi — 2023-01-30T13:40:00.774Z

For More Detail, Previously: Simulacra Levels and Their Interactions, Unifying the Simulacra Definitions, The Four Children of the Seder as the Simulacra Levels.

A key source of misunderstanding and conflict is failure to distinguish between combinations of the following four cases.

Sometimes people model and describe the physical world, seeking to convey true information because it is true.

Other times people are trying to get you to believe what they want you to believe so you will do or say what they want.

Other times people say things mostly as slogans or symbols to tell you what tribe or faction they belong to, or what type of person they are.

Then there are times when talk seems to have have gone strangely meta or off the rails entirely. The symbolic representations are mostly of the associations and vibes of other symbols. The whole thing seems more like a stream of words, associations and vibes. It sounds like GPT-4.

One can refer to these as the simulacra levels as a useful fake framework for understanding this. When looking at talk, one can ask what level or levels a statement or discussion is on, and which ones people care about in context. One can also ask about the level a person, group or civilization most cares about. That is also how they default to understanding new talk.

This concept has important details that are difficult to understand. The posts linked up top offer discussions of four definitions that all point at the same dynamics. Each is stronger at capturing different elements.

As a more concise alternative, this post gathers together the most vital information.

First, the more straightforward definitions from 2020.

The Lion and Pandemic Definitions

The lion definition asks what each level means by ‘There is a lion across the river.’

Level 1: There’s a lion across the river.

Level 2: I don’t want to go (or have other people go) across the river.

Level 3: I’m with the popular kids who are too cool to go across the river.

Level 4: A firm stance against trans-river expansionism focus-grouped well with undecided voters in my constituency.

The pandemic definition changes the statement to ‘There is a pandemic heading our way from China.’

Level 1: “There’s a pandemic headed our way from China.”

Level 2: “I want you to act as if you think there might be a pandemic on our way from China” while hoping to still be interpreted by the...
A Few Principles of Successful AI Design
by Vestozia — 2023-01-30T10:42:25.960Z

Foreword: This post was originally submitted to Lab42 as part of an essay competition. The word count was limited to ~2,500 so there is a substantial amount of information that I have neglected to include. I did not win, so I decided to share it with you all today. Enjoy!

(Briefly) Defining Intelligence

Before attempting to approach (what I’d consider) the fundamental principles of developing human-level artificial intelligence, the term ‘intelligence’ itself must be defined. For the purpose of this essay, and to establish a shared vocabulary, I would like to consider intelligence as “the ability to recognise the significance of acquired information or skills via experience and exposure, in addition to the context it may be applied in”. By this standard, intelligence is not limited to biological systems such as the human brain, nor does it require sentience or an ability to experience qualia. I state this due to the nature of computational theory of mind (when applied to Turing-completeness), implying that an artificial intelligence which imitates human cognition to a highly convincing degree is plausible.

Solving The Alignment Problem

Intuitively, the first and foremost measure to take when considering the development of human-level artificial intelligence should be attempting to solve the alignment problem. Though I do not have a solution, I will provide guiding suggestions and core principles to consider during the creation of such a technology. There are several reasons why alignment is a fundamental issue to tackle; namely to minimise training data bias. A notable example of bias within datasets may be presented as certain ethnicities or races representing a larger demographic of those receiving jail sentences with disregards towards additional context (e.g. judges are more likely to give harsher sentences when their football team loses[1]). This results in complications by either increasing the likelihood of erroneous predictions or actions, however, as long as bias and inaccuracies can be reduced (and tailored to align with human values) in the lead up to the development of human-level artificial intelligence, then more focus and resources can be shifted to other issues such as preventing black boxes within self-improving AI systems.

Should an artificial intelligence demonstrate itself to be equally intelligent as humans, it has likely already surpassed th...
Against Boltzmann mesaoptimizers
by porby — 2023-01-30T02:55:12.041Z

I'm effectively certain that a weakly defined version of "mesaoptimizers"- like a component of a network which learns least squares optimization in some context- is practically reachable by SGD.
I'm pretty sure there exists some configuration of an extremely large neural network such that it contains an agentic mesaoptimizer capable of influencing the training gradient for at least one step in a surprising way. In other words, gradient hacking.
I think there's a really, really big gap between these which is where approximately all the relevant and interesting questions live, and I'd like to see more poking aimed at that gap.
Skipping the inspection of that gap probably throws out useful research paths.
I think this belongs to a common, and more general, class of issue.

Boltzmann mesaoptimizers

Boltzmann brains are not impossible. They're a valid state of matter, and there could be a physical path to that state.[1] You might just have to roll the dice a few times, for a generous definition of few.

And yet I feel quite confident in claiming that we won't observe a Boltzmann brain in the wild within the next few years (for a similar definition of few).[2]

The existence of a valid state, and a conceivable path to reach that state, is not enough to justify a claim that that state will be observed with non-negligible probability.

I suspect that agentic mesaoptimizers capable of intentionally distorting training are more accessible than natural Boltzmann brains by orders of magnitude of orders of magnitude, but it is still not clear to me that the strong version is something that can come into being accidentally in all architectures/training modes that aren't specifically designed to defend against it.[3]

Minding the gap

I feel safe in saying that it would be a mistake to act on the assumption that you're a Boltzmann brain. If you know your history is coherent (to the degree that it is) only by pure chance, and your future practically doesn't exist[4], then in any given moment, your choices are irrelevant. Stuff like 'making decisions' or 'seeking goals' goes out the window.

Observing that there is the logical possibility of a capable misaligned agentic mesaoptimizer and then treating any technique that admits such a possibility as irrevocably broken would be similarly pathological.[5]

If we cannot find a path that eliminates the logical possibility by construction, we mus...
How Likely is Losing a Google Account?
by jefftk — 2023-01-30T00:20:01.584Z

Let's say you use a Google account as the root of your online
identity: Gmail, Fi, "Sign in with Google", Google's password manager,
etc. How much should you be worried that you'll suddenly find
yourself unable to get into everything? Should you do something else
instead?

To get a sense of how common lockouts are and how they happen I looked
through lockout reports on Hacker News by searching for
[google
blocked account] and [google
locked out]. I looked at top-level stories and the comments on
them for cases where people were entirely locked out of an account; I
didn't include cases where people lost access to only some Google
services (Payments, AdWords, etc) or where they did get back in on
their own. I did count cases where it took making a lot of noise on
HN or Twitter, though.

There are two reasons people seem to get locked out:

Security lockouts: they're not convinced you're you, and are
trying to prevent an attacker from getting into your account.

Policy lockouts: they don't like you. They've flagged your
account as abusive, enough that they completely suspend your account.

I found 32 cases (sheet),
going back to 2008. I found 22 security lockouts, 7 policy lockouts,
and 3 with too few details to tell. I think this likely majorly
undercounts security lockouts relative to policy ones: reading the
comments, a lot of the security ones were like "that happened to
me too" while the policy ones got mainstream
news
articles. [1]

With the security lockouts, in cases where you could tell what had
happened the most common reason was that someone had configured a
backup method (phone number, recovery email, 2FA) but no longer had
access. The second most common were cases where someone hadn't
configured any backup methods and Google was considering their login
to be suspicious.

Security lockouts are a tricky situation because failures in either
direction are very bad. All of the above are false positives: people
who should have been let back into their accounts but weren't. But
there are also false negatives: cases where an attacker was let into
someone's account.

There is a question of whether Google should be flagging suspicious
logins at all, though. If my account were protected only by a
password and someone else got it, maybe Google should just let them
in? The problem is that this would mean lots of hacked ac...
Model-driven feedback could amplify alignment failures
by aog — 2023-01-30T00:00:28.647Z

Anthropic, DeepMind and Google Brain are all working on strategies to train language models on their own outputs. For a brief summary of the work so far:

Red Teaming Language Models with Language Models (Perez et al., 2022). One model prompts another, seeking to expose undesirable generations. A third model classifies those generations as undesirable, creating a dataset of behaviors to be avoided.
Constitutional AI (Bai et al., 2022). Begins with the red-teaming setup described in Perez et al., 2022. Then fine-tunes on that dataset by either (a) critiquing and rewriting the response and training the generator to imitate that output with supervised fine-tuning, or (b) choosing the better of two responses to train a preference model, and training the generator with RL on the preference model.
Large Language Models Can Self-Improve (Huang et al., 2022). Fine-tunes a model on its own "high confidence" outputs. High-confidence outputs are identified by asking the model to answer the same question many times, using chain-of-thought each time, and fine-tuning on outputs that are agreed upon by a majority of responses.

I'd like to point out a simple failure mode of this approach: Failures of alignment and capability in the original model could be amplified by fine-tuning on its own outputs. Empirically, recent experiments on language models have found more benefit than harm in model-driven feedback. But that might not always be the case.

This recent work is an extension of weak supervision, a technique dating back to at least 1963 which has been successful in applications such as image classification and protein folding. This literature has long acknowledged the possibility of amplifying a model's existing shortcomings via self-training:

Semi-Supervised Learning of Mixture Models (Cozman et al., 2003) analyze cases where weak supervision will help or hurt a maximum likelihood estimator.
Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning (Arazo et al., 2020) provide evidence that naive implementations of weak supervision can hurt performance on image classification tasks. They show that data augmentation and scaling can reduce these harms.
An Overview of Deep Semi-Supervised Learning (Ouali et al., 2020) Section 1.3 lays out key assumptions behind weak supervision, and discusses state of the art methods.

One particularly...
Takeaways from calibration training
by Olli Järviniemi — 2023-01-29T19:09:30.815Z

Summary. Thoughts arising from doing calibration training on ~500 questions, consisting of both trivia questions and mundane real life events.

Epistemic status: Confident on own experiences, uncertain on generalizability and usefulness to others.

Intro

I've practiced quantifying my uncertainty via assigning numerical probabilities for my beliefs. I've used Open Philanthropy's web app and Quantified Intuition's question set for calibration training (doing ~250 questions on both), and also have made predictions on ~100 real life every-day events. Below I share a few things I learned from this.

I assume no further background beyond knowing what calibration training is, though without experience in calibration practice one may not get much insight from the post.

I. Learning to be calibrated is not that hard

This often-repeated point is worth repeating: you can learn to be calibrated.

There was a time when I first read texts like Kahneman's Thinking, Fast and Slow, explaining how people generally are grossly overconfident and how this is true also for Highly-Educated Smart People and People Who Should Know Better. Some part of me thought that "What?! I'm pretty sure my 98% confidence intervals wouldn't be wrong 40 percent of the time!", and another part thought along the lines of (what I now recognize is) modest epistemology.

Much later, I tried out calibration training. Sure, my first attempt was not great, but it wasn't that bad either. And then I did a bit more training, and 75% of my 80% assessments (n = 63), or 56% of my 60% assessments (n = 101), were correct.[1]

Caveats: My calibration feels poorer when changing the question category (e.g. "doing EA-themed calibration questions" and "predicting everyday events" are very different), and getting up to speed requires a bit of practice and feedback in the new environment. Also, the tools I used informed me about the correctness of my answers immediately after each answer (instead of after a batch of 100 questions or such); this feedback loop feels crucial to me, as I get some feeling about how I'm doing in real time. I could very well see myself being way off in a new environment without a feedback loop.[2]

II. >90% region is difficult

Making confident claims feels difficult: I really don't think I'm well calibrated in the >90% region.

Example: Once I made a 90% prediction for "I still have peanut ...
Structure, creativity, and novelty 

by TsviBT — 2023-01-29T14:30:19.459Z

[Metadata: crossposted from https://tsvibt.blogspot.com/2022/08/structure-creativity-and-novelty.html. First completed 26 June 2022. I'm likely to not respond to comments promptly.]

A high-level confusion that I have that seems to be on the way towards understanding alignment, is the relationship between values and understanding. This essay gestures at the idea of structure in general (mainly by listing examples).

Why do we want AGI at all?

We want AGI in order to understand stuff that we haven't yet understood.

(This is not a trivial claim. It might be false. It could be that to secure the future of humane existence, something other than understanding is necessary or sufficient; e.g. it's conceivable that solving some large combinatorial problem, akin to playing Go well or designing a protein by raw search with an explicit criterion, would end the acute risk period. But I don't know how to point at such a thing--plans I know how to point at seem to centrally involve understanding that we don't already have.)

Elements and structure

Understanding implies some kind of structure. (This is a trivial claim, or a definition: structure is what a mind is or participates in, when it understands.) Structure is made of elements. "Structure" is the mass noun of, or continuous substance version of, "element". The point of the word "element" is just to abbreviate "any of that pattern-y, structure-y stuff, in a mind or in the world in general".

Elements. An element (of a mind) is anything that combines to constitute the mind, at any level of organization or description.

Examples of elements. Any instance within a mind of any of the following categories is an element: features, aspects, properties, parts, components, subagents, pieces, inputs, algorithms, code, processes, concepts, ideas, skills, methods, procedures, values, goals, architecture, modules, thoughts, propositions, beliefs, probabilities, principles, rules, axioms, heuristics, plans, operations, connections, associations, metaphors, abstractions, memories, arguments, reasons, purposes, modes, emotions, tendencies, organs, ingredients, functions, dynamics, structures, data, types, languages, proofs, justifications, motives, images, searches, knowledge, computations, rewards, reinforcement, specifications, information, intuitions, ideologies, protocols, stimuli, responses, domains, gradients, objective functions

...