trentmkelly/LessWrong-43k huggingface dataset:
Cryptoepistemology
by davidad — 2022-02-24T20:34:32.790Z

This is a brief elaboration of an analogy I've found insightful for both some purely epistemological questions and some crypto-platform design ideas:

Justification for a belief takes the form of a cryptographic proof.

Here's what happens if we apply this idea to some theories of epistemic justification that are common in the philosophical literature:

Foundationalist cryptoepistemology is easy: it involves justifying all beliefs by proof-trees (or dags, or string diagrams) terminating in irreducible basic beliefs, which take the form of root certificate authorities.
Coherentist cryptoepistemology grounds beliefs in a global system of mutually endorsing identities. Coherentism corresponds to decentralization. Pure coherentism by itself has a big problem, which in philosophy is called the “regress problem” and in crypto is called the “Sybil attack”: what’s to stop any arbitrary belief from presenting itself as part of a global coherent system; how can one measure its true size?

Weak foundationalism corresponds to a “Web of Trust” scheme, in which certain beliefs are accorded slightly privileged status, such that between all the globally mutually coherent systems, the one which most coheres with those privileged beliefs is selected.
Proof-of-work is a radical and relatively recent idea which does not yet have a direct correspondent in philosophy. Here, cryptographic proofs witness the expenditure of resources like physical energy to commit to particular beliefs. In this way, the true scale of the system which agrees on certain beliefs can be judged, with the largest system being the winner.
Proof-of-stake is a variation of proof-of-work which is much more subtle to analyze. Here, instead of appealing to cryptographic proofs which directly witness the expenditure of resources, we appeal to beliefs (about identities’ stakes) that are justified by previous global beliefs, thereby pushing the majority of the foundational justification to (the hash of) the genesis block of the proof-of-stake chain.

Reliabilist cryptoepistemology would attempt to select endorsers whose beliefs are “truth-tracking”: across a range of possible worlds, their beliefs tend to correspond to actual ground truth. That this doesn’t really work by itself (how could one possibly know the reliability of an arbitrary cryptographic identity without any privileged source of ground

...
Naturalism
by LoganStrohl — 2022-02-24T19:45:21.659Z

According to me, this sequence has been pretty darn abstract.

That was kind of on purpose. It’s the opposite of what I like to do, of what I think I’m good at. I much prefer to engage with an actual specific thing, and to share the details of my experience as I go. This big picture stuff is really not my jam.

But I’ve been trying to paint a really big picture anyway, to describe an entire perspective on investigation, and rationality, and maybe life. I hope it’s been much easier to read than it was for me to write. And I hope that if, at some future point, I dive into the little details of particular exercises and techniques, you’ll be able to contextualize them as more than just trinkets, or rituals that are tedious to little purpose.

But I’m so tired of it. I’m exhausted by all this abstraction. I want to touch the ground. I want to show you what it actually looks like to live a life full of patient and direct observation.

I can tell you that there’s a magnifying glass in my pocket, which I use regularly. I can tell you that I turned the soles of my bare feet toward the sky last week, so that I could feel the snow falling on them. I can tell you that when I put “it seems to me” at the front of so many of my sentences, it’s not false humility, or insecurity, or a verbal tic. (It’s a deliberate reflection on the distance between what exists in reality, and the constellations I’ve sketched on my map.)

I can tell you dozens of facts like these, about my experience of myself and of the world. Hundreds. But none of those means much. Not on its own.

The problem is, this whole thing is founded on patience, which is difficult to demonstrate in an essay. It’s hard to show you all at once the myriad ways a thousand tiny moments add up to one big thing that matters.

Still, they do add up to something.

What they add up to is that I am a naturalist. I was raised to be a naturalist, and it worked. I was raised to be someone who yearns to know the territory through patient and direct observation. My childhood memories are full of mushroom hunting, finding newts under logs, following game trails, reading the geological histories in the rock layers whenever we traveled, and sketching the paths of Jupiter’s moons with a red flashlight beside my telescope.

My upbringing emphasized that the world is an infinity of wonders; unfathomably many in a single handful of dirt...
What would a thriving progress movement look like?
by jasoncrawford — 2022-02-24T17:58:02.576Z
Link: https://rootsofprogress.org/a-thriving-progress-movement

In recent essays I’ve outlined the intellectual-historical need for a progress movement, and the core ideas that I think the budding 21st-century progress movement is based on.

What would a thriving progress movement look like, in terms of activities, programs, and institutions? Here’s what we might see within the next decade or so:

Dozens of public intellectuals writing books and giving talks about the history, nature, and philosophy of progress.

Academic recognition of “progress studies” as a valuable interdisciplinary field. As Collison & Cowen said when they coined the term, this wouldn’t mean reorganizing academic departments, but “a decentralized shift in priorities among academics, philanthropists, and funding agencies,” including journals and conferences.
School curricula to teach the history of progress at the K–12 and undergrad level. I’ve created a high-school progress course; a curriculum like this should be at every high school in the world.
Art & entertainment that sounds themes of progress, such as optimistic science fiction, or biopics about great scientists, inventors, and founders. (I would start with Norman Borlaug.)
More journalism about progress, and more journalists exhibiting industrial literacy.
Political debates framed in terms of progress and growth, rather than primarily or exclusively in terms of redistribution.
Experiments in new models of funding, organizing, and managing scientific and technological research, such as the efforts covered recently in Endpoints and The Atlantic (see also the Overedge Catalog for a broader list).
Scientists, engineers and founders drawing inspiration and courage from this movement, and seeing their work as having the potential to be part of a grand and noble quest to improve the human condition.

The foundation of all of this is intellectual work: a lot of hard research, thinking, writing and speaking. The philosophy of progress has barely begun to be elucidated. To succeed, this movement will need much more than just “yay progress!” or “look at this hockey-stick graph!” Those notions are the beginning of this body of thought, not the end. As I wrote recently, we need to answer the challenges that arose in the 20th century and caused many people to sour on the idea of progress. Without more serious intellectual work here, we risk falling back on the naive 18th- and 19th-century notions of progress tha...
Capturing Uncertainty in Prediction Markets
by hawkebia — 2022-02-24T17:23:30.682Z

I've been trying to make sense of what a 50% prediction means. Does it convey different meanings based on the question asked, and is it really as uninformative as it's made out to be. As a part of that discussion, I believe there is a possible solution to two big problems prediction markets currently face.

How to extract useful signal from 50% predictions
How to incentivize those with low certainty to participate

Predictions tend to be comprised of two steps:

What do I believe is going to happen?
How certain am I?

There are many cases where even if one has a sense of what is going to happen, there is little to no certainty that it will. The options here are skipping the question (not placing a bet), or putting in 50% if you're forced to choose.

I think it's generally understood that using 50% to represent "I don't know" is problematic. Picking 50% also holds an entirely different meaning for the question "Will this coin toss result in heads?" vs. "Will China invade Taiwan this year?".

Similarly, as an observer of the market, does seeing the market at 50% really represent 1:1 odds? Or is it an indication that the market is extremely uncertain. There seems to be a useful signal missing here.

Finally, given that some of the most interesting questions in prediction markets are also about the most uncertain events, many people avoid participating or betting altogether. This is the opposite of what we want if these markets are to be useful.

A Possible Solution

There are two types of uncertainty, or "I don't know":

I don't know enough about this, so I won't bet
I've studied and researched this subject, and I still don't know, so I won't bet

Incentivizing the the first type to place bets would add noisy signal. But capturing the second type of "I don't know" seems to be a pretty important signal about the studied uncertainty of the event.

There have been some ideas about how to encourage more participation, like by providing interest free loans as incentive. But I think that suffers from a few of problems. First, it doesn't distinguish between noisy vs. studied signals. Second, it incentivizes picking the option with more upside. And third, it encourages exactly the most horrible and awful kind of punditry: Picking an opinion with certainty even though you have no idea.

So, is there a way to capture studied uncertainty as useful signal, AND incentivize the participation of th...
Less Lightning Links at Length #1
by Zvi — 2022-02-24T16:20:01.204Z

These are the things that I couldn’t fit into a Covid post even in the Not Covid section, but which aren’t big enough to deserve their own spin-off posts, and which are not time-sensitive.

With Covid winding down, I had the opportunity to work though some of that backlog this week. Rather than have a huge Not Covid section I’ve moved those here to their own post.

This post was written before Russia invaded Ukraine. I was going to keep adding to it, but at this point I am unlikely to be in a place to do that soon, so instead I’m posting what I have.

This also gives me a chance to see which ones do deserve their own posts, and spin those off. An experiment.

#1: Downtown In Trouble (WaPo)

Megan McArdle argues that downtowns are in deep trouble. Thanks to remote work, the argument goes, they may be missing a quarter of their people, and no one is facing up to this. What about the resulting lower rents?

Lower rents can ease the pinch, and in many areas they’ve already become more flexible. But Joseph Gyourko, a real estate professor at the University of Pennsylvania’s Wharton School, notes that only helps so much. If a significant number of the workers are gone on any given day, “You just don’t need as many Starbucks, or haircut places,” he told me, and “those folks are going to disappear. Some of these buildings, non-office buildings, will become empty. We won’t need as many.”

Converting offices to housing? In San Francisco or New York sure if there is no better option, but in general not economical.

Places like SF and NYC are not the problem here. There was a lot more demand for those downtowns than there was supply. If a quarter of current workers drop out to do remote work, there are plenty of others willing to take their place. For now there are empty offices because people are waiting for conditions to change, both companies unwilling to give up or shrink office space and landlords who do not want to lock in lower rents. At equilibrium, the markets will clear, the offices will be almost as packed as ever, so the number of visitors will stay almost the same, although some companies will buy more office space per person at lower prices so there will be slightly fewer people. But rents should adjust to make that small decline not a major issue for retail or restaurants.

Even if total needs do decline, there’s no reason not to fill all the space with...
Covid 2/24/22: The Next War
by Zvi — 2022-02-24T14:30:01.434Z

Russia has invaded Ukraine with the stated goal of 'demilitarization and denazification.'

On its face, this has nothing to do with Covid. The invasion may reflect a perception, either on the part of Putin or otherwise, from our Covid-19 response or otherwise, that the West has shown itself incapable of dealing with real world physical problems, and therefore feeling he has a free hand. It also may have shown Putin and/or others that one can use justifications that are obvious and blatant nonsense lies with impunity. Those facts also may have little or nothing to do with it. We do not know.

I do not feel qualified at this time to comment further beyond an urgent call to build as many state of the art nuclear power plants as we can as quickly as possible, so I’ll be looking at other sources and watching the BBC News like everyone else to try and get a handle on the war. At least for now I’d only be adding noise.

Under such conditions, and with Omicron in decline and relatively mild, it may seem foolish to focus on Covid-19. Ignoring other events would be a mistake, but ignoring Covid-19 would also be a mistake. Covid-19 prevention efforts continue to disrupt our lives, and as events in Canada have shown us such events can pose long term existential threats to our freedom. Justin Trudeau has ended his emergency decree, but the groundwork remains laid for next time. There is danger that a variety of pandemic measures might not only continue for far too long but even become permanent.

As for the actual case and death counts? Sure, worth tracking those too.

Executive Summary

Russia invades Ukraine.
Canada freezes bank accounts of those they dislike without due process.
Covid cases continue to decline.

Let’s run the numbers.

The Numbers

Predictions

Prediction from last week: 555k cases (-35%) and 11,400 deaths (-25%).

Results: 485k cases (-40%) and 13,075 deaths (-14%).

Prediction for next week: 340k cases (-30%) and 10,460 deaths (-20%).

I forgot about Presidents Day, which explains cases dropping somewhat more than I expected, so I presume true case counts were closer to the path that I expected.

That only makes the small decline in deaths more puzzling. There are echoes of this in the past, where death counts declined less rapidly than expected at the tail end of a wave, presumably because of a long tail of how long it takes some patients to die and/...
Russia has Invaded Ukraine
by lsusr — 2022-02-24T07:52:44.533Z

I wrote this quickly. It may contain errors. Please correct them in the comments after reading this post's special commenting guidelines.

You should ignore the news unless it's of historic import. Russia's invasion of Ukraine constitutes an event of historic import.

Interstate conflicts are inherently political. Politics is a delicate topic on Less Wrong because political topics tend to trigger tribal impulses. Arguments-as-soldiers are symbiotic with soldiers-as-arguments.

I'm writing about this topic because it's at the intersection of two topics dear to my heart: ① responding to rare events ② operating in a hostile informatic environment.

These are our players (in descending order of importance):

Russia (including its ally Belarus)

Ukraine

European Union + Britain

United States

China

China

China doesn't care what happens to Ukraine. China primarily cares about preserving its trade relationships with Russia and the West. China would prefer not to impose sanctions on Russia because sanctions are expensive.

The United States

The United States prefers that Ukraine remain out of Russian hands, but the United States cares more about the rise of China than Russia's fading empire. The United States just pulled out of Afghanistan. The United States does not want to get bogged down in another land war in Eurasia.

The United States has provided something around $2 billion in military aid to Ukraine since 2014. A couple billion dollars isn't nothing, but it's insignificant compared to Russian might.

The United States is not committed to defending Ukraine the way it is committed to defending Taiwan, Japan and NATO members. The United States does not plan to use its own troops to defend Ukraine.

The European Union + Britain

The European Union and Britain are allied with the United States. They would prefer Ukraine to stay out of Russian control but don't care enough to deploy their own troops in combat roles against Russia.

Thus, the European Union's (and Britain's) primary option is sanctions. But the European Union and its allies have already been imposing sanctions on Russia for its actions in Crimea since 2014. Putin has already demonstrated that this level of sanctions will not stop his actions in Ukraine.

Europe (especially Germany) is dependent on energy imports from Russia. Europe ramping up sanctions against Russia would damage the European economy and industry in...
Transformer inductive biases & RASP
by Vivek Hebbar — 2022-02-24T00:42:30.084Z
Link: http://proceedings.mlr.press/v139/weiss21a/weiss21a.pdf

This is a paper by Weiss et al. about the "Restricted Access Sequence Processing Language", which is a computational framework for what transformers can do.

DL architecture	Computational framework
Feedforward networks	Circuits
RNNs	Finite state machines
Transformers	RASP (this paper)

Any program written in RASP can be implemented straightforwardly by a transformer. Indeed, the authors demonstrate that RASP can be "compiled" to a transformer, and that transformers can be trained to "mimic a RASP solution".

From the abstract (emphasis mine):

"We map the basic components of a transformer-encoder—attention and feed-forward computation—into simple primitives, around which we form a programming language: the Restricted Access Sequence Processing Language (RASP). We show how RASP can be used to program solutions to tasks that could conceivably be learned by a Transformer, and how a Transformer can be trained to mimic a RASP solution. In particular, we provide RASP programs for histograms, sorting, and Dyck-languages. We further use our model to relate their difficulty in terms of the number of required layers and attention heads: analyzing a RASP program implies a maximum number of heads and layers necessary to encode a task in a transformer. Finally, we see how insights gained from our abstraction might be used to explain phenomena seen in recent works."

One implication of this is that program length (simplicity) in RASP may be a good approximation to the inductive bias of transformers. I will probably use this in future work along the lines of "How complex are myopic imitators?".

If you know of any tricks that transformers use that are not covered by RASP, please leave a comment -- it would be highly useful to me.
A comment on Ajeya Cotra's draft report on AI timelines
by Matthew Barnett — 2022-02-24T00:41:48.192Z

Ajeya Cotra's draft report on AI timelines is the most useful, comprehensive report about AI timelines I've seen so far. I think the report is a big step towards modeling the fundamental determinants of AI progress. That said, I am skeptical that the arguments provided in the report should reduce our uncertainty about AI timelines by much, even starting from a naive perspective.

To summarize her report, and my (minor) critique,

Ajeya builds a distribution over how much compute it would take to train a transformative ML model using 2020 algorithms. This compute distribution is the result of combining the distributions under six different biological anchors, and spans over 20 orders of magnitude of compute.
To calculate when it will it will be affordable to train a transformative ML model, Ajeya guesses (1) how much hardware price-performance will fall in the coming decades, (2) how much algorithmic progress will happen, and (3) how much large actors (like governments) will be willing to spend on training a transformative AI.
Ajeya says that she spent most of her time building the 2020 compute distribution, and relatively little time forecasting price-performance, algorithmic progress, and willingness to spend parameters of the model. Therefore, it's not surprising that I'd disagree with her estimates.
However, my main contention is that her estimates for these parameters are just point estimates, meaning that our uncertainty over these parameters doesn’t translate into uncertainty in her final bottom-line distribution of when it will become affordable to train a transformative ML model.
Separately, I think the hardware forecast is too optimistic. She assumes that price-performance will cut in half roughly every 2.5 years over the next 50 years. I think it's more reasonable to expect hardware progress will be slower in the future.
When making the appropriate adjustments implied by the prior two points, the resulting bottom-line distribution is both more uncertain and later in the future. This barely reduces our uncertainty about AI timelines compared to what many (perhaps most?) EAs/rationalists believed prior to 2020.

Upfront, I definitely am not saying that her report is valueless in light of this rebuttal. In fact, I think Ajeya's report is useful because of how it allows people to build their own models on top of it. I also think the report is admirably tran...
Christiano and Yudkowsky on AI predictions and human intelligence
by Eliezer Yudkowsky — 2022-02-23T21:34:55.245Z

This is a transcript of a conversation between Paul Christiano and Eliezer Yudkowsky, with comments by Rohin Shah, Beth Barnes, Richard Ngo, and Holden Karnofsky, continuing the Late 2021 MIRI Conversations.

Color key:

Chat by Paul and Eliezer	Other chat

15. October 19 comment

[Yudkowsky][11:01]

thing that struck me as an iota of evidence for Paul over Eliezer: https://twitter.com/tamaybes/status/1450514423823560706?s=20

16. November 3 conversation

16.1. EfficientZero

[Yudkowsky][9:30]

Thing that (if true) strikes me as... straight-up falsifying Paul's view as applied to modern-day AI, at the frontier of the most AGI-ish part of it and where Deepmind put in substantial effort on their project? EfficientZero (allegedly) learns Atari in 100,000 frames. Caveat: I'm not having an easy time figuring out how many frames MuZero would've required to achieve the same performance level. MuZero was trained on 200,000,000 frames but reached what looks like an allegedly higher high; the EfficientZero paper compares their performance to MuZero on 100,000 frames, and claims theirs is much better than MuZero given only that many frames.

https://arxiv.org/pdf/2111.00210.pdf CC: @paulfchristiano.

(I would further argue that this case is important because it's about the central contemporary model for approaching AGI, at least according to Eliezer, rather than any number of random peripheral AI tasks.)

[Shah][14:46]

I only looked at the front page, so might be misunderstanding, but the front figure says "Our proposed method EfficientZero is 170% and 180% better than the previous SoTA performance in mean and median human normalized score [...] on the Atari 100k benchmark", which does not seem like a huge leap?

Oh, I incorrectly thought that was 1.7x and 1.8x, but it is actually 2.7x and 2.8x, which is a bigger deal (though still feels not crazy to me)

[Yudkowsky][15:28]

the question imo is how many frames the previous SoTA would require to catch up to EfficientZero

(I've tried emailing an author to ask about this, no response yet)

like, perplexity on GPT-3 vs GPT-2 and "losses decreased by blah%" would give you a pretty meaningless concept of how far ahead GPT-3 was from GPT-2, and I think the "2.8x performance" figure in terms of scoring is equally meaningless as a metric of how much Effi

...