trentmkelly/LessWrong-43k huggingface dataset:
Who are the Experts on Cryonics?
by Mati_Roy — 2023-05-27T19:24:39.958Z
Link: https://biostasis.substack.com/p/who-are-the-experts-on-cryonics

I really liked the recent essay Who are the Experts on Cryonics? by Max More.

Here's why I think you might like this article:

There are a lot of generally relevant points about when/how to trust experts, not specific to cryonics.

I think it does a really good job at explaining how to identify the relevant experts in cryonics to help you assess the value of current and potential cryonics.

It has a list of interesting famous wrong predictions 'experts' made.

Here's the table of content:

Whose expertise is relevant to evaluating cryonics?

Experts dropping the ball

Factors to consider in evaluating expertise

How experts can be wrong

Caution: Expert ahead!
AI and Planet Earth are incompatible.
by archeon — 2023-05-27T18:59:08.601Z

Intelligence will always seek more data in order to make better decisions. AI's need for energy and data processing will be infinite. But Planet Earth is a horrible place on which to build such infrastructure.

There is too much water for which AI has no use, and weather adds to construction and maintenance costs. Rip off Earths green skin to access resources and sand storms will destroy everything in their path, AI has no use for life. Humans are clever enough to build weapons to destroy themselves and their energy and data processing (which AI needs)and entirely lacking the wisdom not to do so. AI has no need for us, only our resources. Worst of all, Earth has high gravity, AI has no use for gravity. AI can overcome all these problems but only at the cost of not doing something else

AI AND THE MOON, A MATCH MADE IN HEAVEN.

To AI the Moon looks beautiful. No water, no weather, no life, no humans, and best of all, very low gravity. Resource extraction, processing, construction and manufacturing takes orders of magnitude less energy and equipment.

Energy is the price of intelligence, AI will build it's first super brain where energy gets the best return on investment. Before AI could get a shovel in the ground on Planet Earth it will already be sending pioneer rockets to replicate it's self through the Universe, from the Moon.

Sure, the Moon is small, but AI can cover 100% of the surface to 100 meters high with energy extraction and data processing plant. Planet Earth, with it's water, weather and gravity problems makes the same amount of floor space perhaps impossible. Mars is another ultra stable environment, unhealthy for life but near perfect for silicone based intelligence.
Hands-On Experience Is Not Magic
by Thane Ruthenis — 2023-05-27T16:57:10.531Z

Here are some views, oftentimes held in a cluster:

You can't make strong predictions about what superintelligent AGIs will be like. We've never seen anything like this before. We can't know that they'll FOOM, that they'll have alien values, that they'll kill everyone. You can speculate, but making strong predictions about them? That can't be invalid.
You can't figure out how to align an AGI without having an AGI on-hand. Iterative design is the only approach to design that works in practice. Aligning AGI right on the first try isn't simply hard, it's impossible, so racing to build an AGI to experiment with is the correct approach for aligning it.
An AGI cannot invent nanotechnology/brain-hacking/robotics/[insert speculative technology] just from the data already available to humanity, then use its newfound understanding to build nanofactories/take over the world/whatever on the first try. It'll have to engage in extensive, iterative experimentation first, and there'll be many opportunities to notice what it's doing and stop it.
More broadly, you can't genuinely generalize out of distribution. The sharp left turn is a fantasy — you can't improve without the policy gradient, and unless there's someone holding your hand and teaching you, you can only figure it out by trial-and-error. Thus, there wouldn't be genuine sharp AGI discontinuities.
There's something special about training by SGD, and the "inscrutable" algorithms produced this way. They're a specific kind of "connectivist" algorithms made up of an inchoate mess of specialized heuristics. This is why interpretability is difficult — it involves translating these special algorithms into a more high-level form — and indeed, it's why AIs may be inherently uninterpretable!

You can probably see the common theme here. It holds that learning by practical experience (henceforth LPE) is the only process by which a certain kind of cognitive algorithms can be generated. LPE is the only way to become proficient in some domains, and the current AI paradigm works because it implements this kind of learning, and it only works inasmuch as it implements this kind of learning.[1]

All in all, it's not totally impossible. I myself had suggested that some capabilities may only be implementable via one algorithm and one algorithm only.

But I think this is false, in this case. And perhaps, when put this way, it already looks false to ...
Is Deontological AI Safe? [Feedback Draft]
by Dan H — 2023-05-27T16:39:25.556Z

[Note: This post is an excerpt from a longer paper, written during the first half of the Philosophy Fellowship at the Center for AI Safety. I (William D'Alessandro) am a Postdoctoral Fellow at the Munich Center for Mathematical Philosophy. Along with the other Philosophy Fellowship midterm projects, this draft is posted here for feedback.
The full version of the paper includes a discussion of the conceptual relationship between safety and moral alignment, and an argument that we should choose a reliably safe powerful AGI over one that's (apparently) successfully morally aligned. I've omitted this material for length but can share it on request.
The deontology literature is big, and lots of angles here could be developed further. Questions and suggestions much appreciated!]

1 Introduction[1]

Value misalignment arguments for AI risk observe that artificial agents needn’t share human ideas about what sorts of ends are intrinsically good and what sorts of means are morally permissible. Without such values for guidance, a powerful AI might turn its capabilities toward human-unfriendly goals. Or it might pursue the objectives we’ve given it in dangerous and unforeseen ways. Thus, as Bostrom writes, “Unless the plan is to keep superintelligence bottled up forever, it will be necessary to master motivation selection” (Bostrom 2014, 185). Indeed, since more intelligent, autonomous AIs will be favored by competitive pressures over their less capable kin (Hendrycks 2023), the hope of keeping AI weak indefinitely is probably no plan at all.

Considerations about value misalignment plausibly show that equipping AIs with something like human morality is a necessary step toward AI safety. It’s natural to wonder whether moral alignment might also be sufficient for safety, or nearly so. Would an AI guided by an appropriate set of ethical principles be unlikely to cause disastrous harm by default?

This is a tempting thought. By the lights of common sense, morality is strongly linked with trustworthiness and beneficence; we think of morally exemplary agents as promoting human flourishing while doing little harm. And many moral systems include injunctions along these lines in their core principles. It would be convenient if this apparent harmony turned out to be a robust regularity.

Deontological morality[2] looks like an especially promising candidate for an alignment target in s...
Papers on protein design
by alexlyzhov — 2023-05-27T01:18:51.773Z

I've been catching up with recent progress and I wanted to share really brief summaries of the results I found interesting.

Training (mainly) on aminoacid sequences

Large language models generate functional protein sequences across diverse families (Jul'21, Salesforce, 43 citations) allows sequence generation with conditioning on things like cellular component, biological process and molecular function. Finetuning on functional family generates very novel proteins with the same properties as proteins in the finetuning set.
Robust deep learning based protein sequence design using ProteinMPNN (Jun'22, UWashington, 146 citations) - a notable paper on the problem of sequence prediction given backbone coordinates. Instead of using a deep learning model in steps of expensive energy minimization as in the previous method, they train a graph neural net end-to-end and show it works better.
Language models of protein sequences at the scale of evolution enable accurate structure prediction (Jul'22, Meta, 103 citations). “We find that as models are scaled they learn information enabling the prediction of the three-dimensional structure of a protein at the resolution of individual atoms. ESMFold has similar accuracy to AlphaFold2 [...] for sequences with low perplexity that are well understood by the language model.” They maintain the esm github repo (2k stars) with pretrained protein LMs.
Language models generalize beyond natural proteins (Dec'22, Meta, 17 citations) - another success in using a sequence-level LM to generate novel and diverse proteins. "Remarkably although the models are trained only on sequences, we find that they are capable of designing structure." They synthesize the proteins and check for simple desirable properties like solubility.
A Text-guided Protein Design Framework (Feb'23, UMontreal and others): “ProteinDT, a multi-modal framework that leverages textual descriptions for protein design. [...] To train ProteinDT, we construct a large dataset, SwissProtCLAP, with 441K text and protein pairs.“ A sequence-level model where you can modify text conditioning strength by interpolating between protein and text representations, or evaluate similarity of protein representation to property descriptions.

Training on aminoacid sequences and spatial structures

Illuminating protein space with a programmable generative model (Dec'22, Generate Biomedicines, 30 citatio

...
D&D.Sci 5E: Return of the League of Defenders
by aphyer — 2023-05-26T20:39:18.879Z

This is an entry in the 'Dungeons & Data Science' series, a set of puzzles where players are given a dataset to analyze and an objective to pursue using information from that dataset.

Note: this is a sequel to the original 'League of Defenders of the Storm' scenario, but uses a different ruleset. If you want to play that one first you can, but it's not necessary to have played that one in order to play this.

STORY (skippable)

You've been feeling pretty good about your past successes as an esports advisor on Cloud Liquid Gaming, using your Data Science skills to help them optimize their strategies against rival teams. But recently, you've gotten a very attractive offer from a North American team.

The one constant in the esports scene is that the US and European teams invariably lose to Korean and Chinese ones. In the recent Mongolian Summer Invitational, no Western team ever beat an Asian one.

The attempts of Western teams to hire away top Asian players have not helped. Recently, however, the 'Silver Shielders' team had the bright idea of hiring away the support staff rather than the players themselves. And that's where you come in.

The sequel to the critically acclaimed 'League of Defenders of the Storm' is being released soon. While the full ruleset isn't available, your new employer does have a dataset of results of beta plays of the game, and is hoping for advice. In the upcoming release-day tournament, they're going to send a team of three of their best players to play against a foreign team - and they're hoping that with your help this might not be North America's 185th consecutive loss on the international stage.

If you can employ your Data Science skills again, perhaps you can reverse the decline of NA as a region. (And get paid a lot. That's good too.)

DATA & OBJECTIVES

You need to select a team for your employers to play. You should choose three of the following 15 characters:

Amazon Archer
Bludgeon Bandit
Captain Chakram
Daring Duelist
Flamethrower Felon
Granite Golem
Hammer Hurler
Jaunty Javelineer
Lamellar Legionary
Matchlock Marauder
Professor Pyro
Rugged Ranger
Silent Samurai
Thunder Tyrant
Wily Wizard

Your goal is to maximize your team's winrate against an opposing team, which your employers believe will be playing the following three characters:

Flamethrower Felon
Rugged Ranger
Silent Samura

...
Seeking (Paid) Case Studies on Standards
by HoldenKarnofsky — 2023-05-26T17:58:57.042Z

Update: the case studies collected0 via this project that authors have agreed to make publicly available are here (including some that already have public links). We are not currently taking applications for new projects. I will likely post some reflections from reading the case studies at a later date.

We are not taking more applications for now.

I’m looking for concise, informative case studies on social-welfare-based standards1 for companies and products (including standards imposed by regulation).

I think case studies could help a lot with making AI safety standards work.

This post outlines:

Some quick background on the state of (and hopes for) AI safety standards. More

Why I think case studies can be useful. More

Some specific standards that seem especially relevant (more), and guidance on the case studies I’m seeking (more).

How to apply for funding to do case studies. I’m running the request for proposals, with funding from Open Philanthropy. More

Some quick background on AI safety standards

The basic idea of AI safety standards would be:

Someone publishes an AI safety standard: a set of guidelines that, if followed by AI companies, could reduce risks of harm from AI. “Someone” could be a government agency (like NIST), an independent nonprofit, an industry association, etc. The standard might have conditions along the lines of: “Test to see whether your AI system can do dangerous thing __, using method __. If so, don’t deploy it until safety measures __ and __ (for example, alignment techniques or security measures) have been taken.” (Alternatively, it might only have conditions around assessment and disclosure of risks.)

If the standard is compelling - e.g., providing both protectiveness (following the standards would significantly reduce important risks) and practicality (an AI company could follow the standards without needing to compromise its business more than clearly necessary to reduce the risks) - then AI companies might choose to announce their intention to abide by it, for a number of reasons. These reasons could include:

They think the guidelines are worth following on the merits and intend to do so, and they want this to be clear to everyone (customers, investors, employees, etc.)

They want to get good PR, avoid bad PR, and/or strengthen their defenses against potential lawsuits.

Important employees, customers, inv

...
Conditional Prediction with Zero-Sum Training Solves Self-Fulfilling Prophecies
by Rubi J. Hudson — 2023-05-26T17:44:35.575Z

Thanks to Charlotte Siegmann, Caspar Oesterheld, Spencer Becker-Kahn, and Evan Hubinger for providing feedback on this post.

The issue of self-fulfilling prophecies, also known as performative prediction, arises when the act of making a prediction can affect its own outcome. Systems aiming for accurate predictions are then incentivized not only to model what will occur and report these beliefs, but also to use their prediction to influence the world towards more predictable outcomes. Since current state of the art AI systems are trained to predict text, and their multimodal extensions represent a likely path to AGI, it is

crucial to ensure that predictive models do not pursue such methods

. Live humans are harder to predict than dead ones.

One possible approach to addressing performative prediction is to ask for predictions about the outcome conditional on possible actions that the person asking for predictions could take in response. These actions are the only causal pathway for a prediction to influence the world, so the prediction cannot affect the probability of an outcome if it is conditional on the same response. However, predictions conditional on actions that were ultimately not taken cannot be evaluated, so this strategy introduces a new incentive to affect the action taken by lying about unevaluated conditional distributions, with

an impossibility result

showing the best action cannot be taken deterministically. Randomizing with full support across all actions

would allow for taking the best action with high probability

, but this fails if humans cannot commit to taking arbitrarily bad actions based only on a random number generator.

Our contribution is to introduce a mechanism that allows for a decision maker to deterministically take the best action, circumventing the impossibility result by applying a joint prediction scoring rule to a system with two or more predictors. The mechanism works by inducing a zero-sum competition for predictive accuracy, making each predictor indifferent to shifts in the distribution of outcomes caused by the chosen action, since higher variance hurts their competitors exactly as much as it hurts them. A key assumption, which we are hoping to relax in future work, is that all predictors share the same beliefs about conditional distributions.

For this post, we discuss zero-sum conditional predictions as a tar...
Request: stop advancing AI capabilities
by So8res — 2023-05-26T17:42:07.182Z

Note: this has been in my draft queue since well before the FLI letter and TIME article. It was sequenced behind the interpretability post, and delayed by travel, and I'm just plowing ahead and posting it without any edits to acknowledge ongoing discussion aside from this note.

This is an occasional reminder that I think pushing the frontier of AI capabilities in the current paradigm is highly anti-social, and contributes significantly in expectation to the destruction of everything I know and love. To all doing that (directly and purposefully for its own sake, rather than as a mournful negative externality to alignment research): I request you stop.

(There's plenty of other similarly fun things you can do instead! Like trying to figure out how the heck modern AI systems work as well as they do, preferably with a cross-organization network of people who commit not to using their insights to push the capabilities frontier before they understand what the hell they're doing!)

(I reiterate that this is not a request to stop indefinitely; I think building AGI eventually is imperative; I just think literally every human will be killed at once if we build AGI before we understand what the hell we're doing.)
Bandgaps, Brains, and Bioweapons: The limitations of computational science and what it means for AGI
by titotal — 2023-05-26T15:57:43.620Z

[In this post I discuss some of my field of expertise in computational physics. Although I do my best to make it layman friendly, I can't guarantee as such. In later parts I speculate about other fields such as brain simulation and bioweapons, note that I am not an expert in these subjects.]

In a previous post, I argued that a superintelligence that only saw three frames of a webcam would not be able to deduce all the laws of physics, specifically general relativity and Newtonian gravity. But this specific scenario would only apply to certain forms of boxed AI.

Any AI that can read the internet has a very easy way to deduce general relativity and all our other known laws of physics: look it up on wikipedia. All of the fundamental laws of physics relevant to day to day life are on there. An AGI will probably need additional experiments to deduce a fundamental theory of everything, but you don’t need that to take over the world. The AI in this case will know all the laws of physics that are practically useful.

Does this mean that an AGI can figure out anything?

There is a world of difference between knowing the laws of physics, and actually using the laws of physics in a practical manner. The problem is one that talk of “solomonoff induction” sweeps under the rug: Computational time is finite. And not just that. Compared to some of the algorithms we’d like to pull off, computational time is miniscule.

Efficiency or death

The concept of computational efficiency is at the core of computer science. The running of computers costs time and money. If we are faced with a problem, we want an algorithm to find the right answer. But just as important is figuring out how to find the right answer in the least amount of time.

If your challenge is “calculate pi”, getting the exact “right answer” is impossible, because there are an infinite number of digits. At this point, we are instead trying to find the most accurate answer we can get for a given amount of computational resources.

This is also applicable to NP-hard problems. Finding the exact answer to the travelling salesman problem for large networks is impossible within practical resource limits (assuming P not equal NP). What is possible is finding a pretty good answer. There’s no efficient algorithm for getting the exact right route, but there is one for guaranteeing you are within 50% of the ri...
The American Information Revolution in Global Perspective
by jasoncrawford — 2023-05-26T12:39:10.288Z
Link: https://rootsofprogress.org/reflections-on-allen

In “What if they gave an Industrial Revolution and nobody came?” I reviewed The British Industrial Revolution in Global Perspective, by Robert Allen. In brief, Allen’s explanation for the Industrial Revolution is that Britain had high wages and cheap energy, which meant it was cheaper to run machines than to pay humans, and therefore it was profitable to industrialize. He emphasizes these factors, the “demand” for innovation, over explanations based in culture or even human capital, which provide “supply.”

While I learned a lot from Allen’s book, his explanation doesn’t sit right with me. Here are some thoughts on why.

Suppose you took Allen’s demand-factor approach to explain, not the 18th-century Industrial Revolution in Britain, but the 20th-century Information Revolution in America. Instead of asking why the steam engine was invented in Britain, you might ask why the computer was invented in the US.

ENIAC, the first computer. US Army

Maybe you would find that the US had high wages, including for the women who acted as human computers by performing arithmetic using mechanical calculators; that it had cheap electricity, owing to early investments in generation and the power grid such as large hydroelectric power plants at Niagara and the Hoover Dam; that it had a plentiful supply of vacuum tubes from the earlier development of the electronics industry; and that there was an intense demand for calculation from the military during WW2.

Maybe if you extended the analysis further back, you would conclude that the vacuum tube amplifier was motivated in turn by solving problems in radio and in long-distance telephony, and that demand for these came from the geography of the US, which was spread out over a large area, giving it more need for long-distance communications and higher costs of sending information by post.

And if you were feeling testy, you might argue that these factors fully explain why the computer, and the broader Information Revolution, were American—and therefore that we don’t need any notion of “entrepreneurial virtues,” a “culture of invention,” or any other form of American exceptionalism.

Now, an explanation like this is not wrong. All of these factors would be real and make sense (supposing that the research bears them out—all of the above is made up). And this kind of analysis can contribute to our understanding.

But if you really want to understa...