trentmkelly/LessWrong-43k huggingface dataset:
Knowledge is not just precipitation of action
by Alex Flint — 2021-06-18T23:26:17.460Z

Knowledge is not just precipitation of action

Financial status: This is independent research. I welcome financial support to make further posts like this possible.

Epistemic status: This is in-progress thinking.

This post is part of a sequence on the accumulation of knowledge. Our goal is to articulate what it means for knowledge to accumulate within a physical system.

The challenge is this: given a closed physical system, if I point to a region and tell you that knowledge is accumulating in this region, how would you test my claim? What are the physical characteristics of the accumulation of knowledge? What is it, exactly, about an artifact inscribed with instructions for building advanced technology that makes it so different from an ordinary rock, or from a video camera that has been travelling the cosmos recording data since the beginning of the universe? We are looking for a definition of knowledge at the level of physics.

Our goal is to articulate what it means for knowledge to accumulate within a physical system. The previous post looked at mutual information between high- and low-level configurations of a digital abstraction layer as a possible definition of knowledge and found that mutual information did not differentiate raw sensor data from useful models derived from that sensor data.

In this post we will consider a definition of knowledge as that which precipitates effective goal-directed action. That is whenever we see some entity taking actions that are effective and goal-directed, we could conclude that knowledge exists. This is, after all, the informal goalpost that we have been comparing each previous definition of knowledge to. Rather than seeking a separate definition of knowledge and comparing it to this goalpost, this post will look at ways that we might make this informal definition formal.

Example: Satellite tracker

Consider a computer scanning the sky for a satellite in order to transmit some information to it. The computer will scan the sky looking for transmissions on certain radio frequencies, and will integrate each of these noisy observations over time into an estimate of the satellite’s position. The computer only has enough power to transmit the information once, so it’s important that it locks onto the satellite’s true position before it transmits.

Initially, the computer has no knowledge of the satellite’s position and is approxima...
Notes on War: Grand Strategy
by johnswentworth — 2021-06-18T22:55:30.174Z

First things first: I am not any sort of military expert. But most discussions or portrayals of war which I see among rationalists make me face-palm, especially discussions around the role of AI in future warfare or AI “takeover”. Seriously, guys, it’s not about autonomous drone swarms.

This post is just a brain-dump of some models/frames which I wish more people had. We’ll walk through a few different “use-cases” of war (defending an invasion, territory grab, takeover) and talk about grand-strategy principles relevant to each. If people find this interesting, I may write more brain-dumps like this on strategy, tactics, etc; the post quickly became far too long to fit it all in one.

Epistemic status: any particular claim is low-confidence, but I am more confident in the usefulness of the high-level frames.

Defending an Invasion

For concreteness, let’s think about

Azerbaijan invading the Nagorno-Karabakh region of Armenia

(though admittedly I did not follow that conflict closely so we’re not going to talk about real details too much). At a high level, how might Armenia go about defending against that invasion?

The obvious option is “kill invading troops until they leave, and kill them some more if they come back”. I claim this is wildly inefficient: it’s treating symptoms, not causes.

On the Azerbaijani side, the decision to invade was presumably a political decision. There were factions in favor, and factions against. If the anti-invasion factions gain control of government decision-making, then the invasion will likely end. Killing invading troops might help make that happen - making the war visibly costly may make it less popular. On the other hand, killing invading troops may just create martyrs and solidify the image of Armenians as Enemies. If that’s the case, then killing invading soldiers may only solidify the political faction pushing for war, and troops will just keep invading until one side runs out of people/resources. It could go either way.

So: how could Armenia fight in a way which destabilizes the pro-invasion faction’s control over Azerbaijan, rather than stabilizing it? Some possibilities:

Directly attempt to kill key people in the pro-invasion faction, e.g. assassinations. Note that this could backfire in the same way as killing soldiers.
Target the power base of people in the pro-invasion faction. For instance, in many third-world countries the mil

...
Conditional offers and low priors: the problem with 1-boxing Newcomb's dilemma
by Andrew Vlahos — 2021-06-18T21:50:01.840Z

Suppose I got an email yesterday from someone claiming to be the prince of Nigeria. They say they will give me $10 million, but first they need $1,000 from me to get access to their bank account. Should I pay the money? No, that would be stupid. The prior of being given $10 million that easily is too low.

Newcomb's dilemma wouldn't be much better,

even if there was a way to scan a brain for true beliefs and punish lies

.

Suppose you win a lawsuit against me, your business rival, and the court orders me to pay you $10,000. When I show up to transfer the money, I bring two boxes: one with $10,000 in it, and the other one is opaque. I present an offer to the court: "This opaque box either contains a million dollars or no money. I offer my opponent an offer: take both boxes, or they can take only the opaque box and waive their right to the other one. The twist is, I put money into the opaque box if and only if I believe that my opponent would decline the $10,000."

Also assume that society can do noetic brain scans like in Wright's The Golden Age, and lying or brain modification to adjust my beliefs would be caught and punished.

Some people take both boxes, and end up with $10,000. Other people would take one box, and end up with $0!

What happened? Shouldn't rationalists win?

What the 1-boxers didn't think about was that the chance of someone volunteering to pay extra money was epsilon. If I thought my opponent had a >50% chance of 1-boxing, I wouldn't have made the offer.

1-boxing is better if Omega doesn't care about money. However, if the agent wants to pay as little money as possible, 2-boxing is the way to go.
Assume long serving politicians are rationally maximizing their careers
by Tim Liptrot — 2021-06-18T15:15:26.778Z

Imagine you had no idea how to ride a bull, and you encountered a bull rider. The rider is swinging his free arm and body wildly, at apparent random. Surely hugging the bulls back and moving less is better. You would immediately think the man is crazy, perhaps he’s having a seizure.

Now imagine your friend says “he’s been on that bull for five minutes, a town record”. Know you should reconsider. Surely someone has tried your strategy and lasted less. Flailing must help him stay on the bulls back, strange as it seems.

Politicians are the same way. Every political system has a steady stream of talented challengers, who incumbents must defeat. The Nancy Pelosis, Boris Johnson’s and Ayatollah Khameinis of the world may look like their flailing, but usually the know exactly what they’re doing.
Are bread crusts healthier?
by Yair Halberstadt — 2021-06-18T15:12:59.527Z

When you think about it for a second, almost definitely not. They're exactly the same stuff as bread, just more burnt, and so more likely to have the nutritious stuff broken down, and contain carcinogens. Indeed a quick internet search "busts the myth" that crusts are healthier.

So when my baby wasn't very hungry, and I was deciding which bits of the roll to give him for his last few bites of supper, why did I catch myself thinking "let's give him the crust, it's healthier"?

I guess the virtue theory of diet runs deeper in my subconscious than I realized. "If you are virtuous, and eat unpleasant foods, then they are good for you and you will grow strong and healthy." Usually when we have biases we can run rings around ourselves justifying them. It's good to occasionally catch ourself so red handedly in the process of magical thinking that we just have to accept we're occasionally irrational and work on rectifying that.
Non-poisonous cake: anthropic updates are normal
by Stuart_Armstrong — 2021-06-18T14:51:43.143Z

I am on a quest to show that anthropics probability are normal, at least in the absence of exact duplicates.

So consider this simple example: a coin is tossed. This coin is either fair, is 3/4 biased to heads, or 3/4 biased to tails; the three options are equally likely. After being tossed, the coin is covered, and you eat a cake. Then you uncover the coin, and see that it was tails.

You can now update your probabilities on what type of coin it was. It goes to a posterior of 1/6 on the coin being heads-biased, 1/3 on it being fair, and 1/2 on it being tails-biased[1]. Your estimated probability of it being tails on the next toss is (1/6)(1/4)+(1/3)(1/2)+(1/2)(3/4)=7/12.

Now you are told that, had the coin come up heads, there would have been poison in the cake and you would have died before seeing the coin.

This fact makes the problem into an anthropic problem: you would never have been alive to see the coin, had it come up heads. But I can't see how that would have changed your probability update. If we got ethics board approval, we could actually run this experiment. And for the survivors in the tail worlds, we could toss the coin a second time (without cake or poison), just to see what it came up as. In the long run, we would indeed get roughly 7/12 tails frequency. So the update was correct, and the poison makes no difference.

Again, it seems that, if we ignore identical copies, anthropics is just normal probability theory. Now, if we knew about the poison, then we could deduce that the coin was tails from our survival. But that information gives us exactly the same update as seeing the coin was actually tails. So "I survived the cake" is exactly the same type of information as "the coin was tails".

Incubators

If we had more power in this hypothetical thought experiment, we could flip the coin and create you if it comes up tails. Then, after getting over your surprise, you could bet on the next flip of the coin - and the odds on that will be the same as in the poison cake and in the non-anthropic-case. Thus updates are the same if:

Standard coin toss, you see tails.

Poison cake situation, you survive the cake.

You're created on tails flip and notice you exist.

The probability of tails given the heads-biased coin is 1/4; given the fair coin it is 1/2=2/4, and given tails-biased it is 3/4. So the odds are 1:2:3; multiplying these by the (equal) prior proba

...
Fauci’s Emails and the Lab Leak Hypothesis
by ChristianKl — 2021-06-18T13:55:19.913Z

Introduction

When trying to understand why the emails of Fauci might or might not be a big deal, most of the online sources I found look at individual emails while individual emails alone are not enough to understand what’s happening. After watching a

video

that points to the days in the beginning of February being central, I read through the related

emails

. Reading the original sources is very useful if you want to get a good understanding of the ground reality.

These emails aren't public because a hacker breached the security of the NIH, so argument against it being unethical to discuss material that's public due to hacks don't apply here. They are public because the law assumes that citizens have a right to understand how their institutions work and can access documents via Freedom of Information requests.

Given that most people won’t read the sources, I wrote my own research together in this post, to put some of what’s in the emails into a form that’s more easy to consume. I still recommend everyone who wants to form a grounded opinion to additionally look at the emails themselves.

There are a lot of emails and I try to list the ones that are most relevant to the story of how Fauci, Farrar and Collins organized a group to declare to the world that SARS-CoV-2 is of natural origin and not escaped from a lab. In this post I will use SARS-CoV-2 the whole time to refer to the virus even though it only got that designation on February 11, 2020 and emails before that point refer to the virus as 2019-nCoV.

Monday, 26st 2020

Forbers

writes that the Lab Leak theory started on January, 26st with

Washington Times

with an article titled “Coronavirus may have originated in lab linked to China's biowarfare program”. As a result the first public debate begins with multiple papers writing stories.

Friday, January 31st 2020

On January 31st an Indian researcher group around Prashant Pradhan published

Uncanny similarity of unique inserts in the 2019-nCoV spike protein to HIV-1 gp120 and Gag

as preprint.

The Fauci emails begin in February but given that one email replies to an email that was written a day earlier, it’s still part of the dataset. In it

Kristian Andersen

, professor at the Department of Immunology and Microbiology of California Campus, emails Anthony Fauci, NIAID (National Institute of Allergy and Infectious Diseases) since 1984, with the claim that at the c...
You can now apply to EA Funds anytime! (LTFF & EAIF only)
by Jonas V — 2021-06-18T13:40:51.100Z
Link: https://forum.effectivealtruism.org/posts/oz4ZWh6xpgFheJror/you-can-now-apply-to-ea-funds-anytime-ltff-and-eaif-only

The

Long-Term Future Fund

(LTFF) and the

EA Infrastructure Fund

(EAIF) are looking for grant applications:

You can now apply for a grant anytime. We have removed the previous round-based system, and now aim to evaluate most grants within 21 days of submission (and all grants within 42 days), regardless of when they have been submitted. If you indicate that your application is time-sensitive, we will aim to get back to you more quickly (potentially within just a few days).

Apply now

.
You can now suggest that we give money to other people, or let us know about ideas for how we could spend our money. We’re interested in both high-level ideas and concrete, shovel-ready grant opportunities. We will read all suggestions, but we expect to follow up on only a small number. It’s hard to find great grants, so we really appreciate your suggestions!

Suggest a grant

.
We fund student scholarships, career exploration, local groups, entrepreneurial projects, academic teaching buy-outs, top-up funding for poorly paid academics, and many other things. We can make anonymous grants without public reporting. We will consider grants as low as $1,000 or as high as $500,000 (or more in some cases). As a reminder,

EA Funds is more flexible than you might think

.
The LTFF is managed by Asya Bergal (chairperson), Adam Gleave, Evan Hubinger (newly appointed), and Oliver Habryka. For the coming months, they will be joined by Luisa Rodriguez as a guest manager. See its

recent payout report

.
The EAIF is managed by myself (interim/acting chairperson), Buck Shlegeris, Max Daniel, and Michelle Hutchinson. For the coming months, Linh Chi Nguyen and Michael Aird will join as guest managers. See its

recent payout report

and

AMA

.
The

Animal Welfare Fund

will continue on a round-based system. For recent updates, see

Request For Proposals: EA Animal Welfare Fund

and

Animal Welfare Fund: Ask us anything!

Apply here

. We look forward to hearing from you!
A Reason to Expect Republics to Perform Better than Absolute Monarchies in the Long-Term
by MikkW — 2021-06-17T22:22:50.656Z

Reading Raemon's recent Shortform on why the long-term leadership succession of small groups often deteriorates made me consider organizations which manage to successfully have several generations of competent leadership.

Reading through history, hereditary monarchies always seem to fall into a similar problem to what Raemon describes, with incompetent and (physically and mentally) weak monarchs being placed at the head of a nation, leading to a lot of problems. Republics, in contrast, almost always have competent leaders [1].

This makes life much better for the people in the republic, and may be in part responsible for the recent proliferation of republics (though it does raise the question of why that hasn't happened sooner. Maybe the robust safeguards implemented by the Founding Fathers of the USA in their constitution were a sufficiently non-obvious and important social technology required for republics to be viable on the world stage [2]).

A key difference between monarchies and republics is that each successive generation of leadership in a republic must win an intense competition to secure their position, unlike the heirs of a monarchy [3]. Not only this, but the competitions are usually held quite often (for example, every 4 years in Denmark, every 5 years in the UK [4]), which keeps the ⸤fact that the office is competitive⸥ in the public mind very frequently, making it hard for the office to become a de facto hereditary position (a problem that happens with all too high frequency). By holding a competition to fill the office, one ensures that, even if the leaders don't share the same vision as the original founder, they still have to be very competent to be appointed to the position of leadership [5].

Footnotes:

[1] One might disagree with goals of the leaders of republics, and they are too often appointed after their prime, when their health is declining [1a], but the leaders of republics are almost always very competent people.

(1a) - This (The fact that the leaders of republics are often elected when their health is in decline) makes me think it may be a good idea to have a constitutional maximum age, after which individuals cannot be elected to certain important offices, to ensure that only people who are in their prime (and hence likely sufficiently healthy) can lead the nation.

[2] - The existence of elective monarchies also is suggestive that ...
Thoughts on a "Sequences Inspired" PhD Topic
by goose000 — 2021-06-17T20:36:20.963Z

I currently work as an operations research team leader for a large organization (large organization, small team). For personal and professional reasons I won't post the details here in the public forum, but can provide some via PM on request. In about one to two years I expect to have an opportunity to start a resident PhD program with up to three years to complete the resident portion. There's a long side discussion I could go down here about whether or not the PhD is the best choice, but for now let's take it as a given that the piece of paper will increase promotion potential, pay, or both, with minimal cost. Although I started out with HPMOR, I have now read most of the sequences and made some rather fundamental changes in my beliefs as a result. More recently,

The Selfish Gene

and many of the posts by Scott Alexander drew me to learn more about game theory and have since found that the game theoretical approach offers a powerful formalism to how I have naturally thought about system dynamics for most of my life, and although the evolutionary psychology perspective introduced to me by the sequences made sense of the associated failures in reasoning, for perhaps the first time, game theory made them something I felt I could actually operationalize.

The intersection between these systems and AI systems is now simultaneously fascinating and horrifying to me. I completely understand why we are in the situation described in The Social Dilemma

,

but I also recognize that there are smaller scale systems where similar alignment issues exist. The phenomenon of people confusing the rule for the space has long frustrated me, but now I'm starting to better understand why it occurs and how game theory dictates the manner in which a system will respond to it.

I feel confident that there is a dissertation topic in here somewhere that leverage my interests, advances alignment research, and is also achievable. I am posting here because I am interested in the community's thoughts on how to improve/scope/pursue this goal, with the best description I have so far of a topic being: I am interested in using computational game theory, cognitive science, system modeling, causal inference, and operations research methods to explore the ways in which AI systems can produce unintended consequences and develop better methods to anticipate outer alignment failures.

Other...
Pros and cons of working on near-term technical AI safety and assurance
by Aryeh Englander — 2021-06-17T20:17:13.451Z

Cross-posted from the EA Forum: https://forum.effectivealtruism.org/posts/Ry4C4CKZvuRG7ztxY/pros-and-cons-of-working-on-near-term-technical-ai-safety

Recently I've been thinking about the pros and cons of working on near-term technical AI safety and assurance. This includes topics such as interpretability for near-term systems, generalizability / robustness, AI security, testing, verification, and the like.

Here are my own considerations so far:

(Note: In what follows I use the term Transformative AI (TAI) very loosely to mean any type of AI that has a decent chance of leading to a global catastrophe if safety challenges are not addressed first.)

Pros

Some approaches to these topics might actually turn out to work directly for TAI, especially where those approaches may not be pursued given the default trajectory (i.e., without EA intervention) of research from industry / government / academia.
This kind of research directly helps create a set of tools, techniques, organizations, regulations, etc., that iteratively builds on itself in the way that technology tends to do, such that whenever TAI becomes a real problem we will already have solutions or the resources to quickly find solutions.
Promoting this kind of research in industry / gov't / academia helps influence others in those communities to create a set of tools, techniques, organizations, regulations, etc., such that whenever TAI becomes a real problem we will already have solutions or the resources to quickly find solutions.
Research into these topics fosters a broader concern for AI safety topics in the general public (either directly or as a side effect of researchers / gov't / etc. respecting those topics more), which could lead to public pressure on industry / gov't to develop solutions, and that may help mitigate risks from TAI.

(For whatever it's worth, my personal inside view leans towards 3 as the most plausibly important from an EA point of view.)

Cons

Research into these topics, if successful, would remove some very large barriers that are currently preventing AI from being deployed in many applications that would be extremely valuable to industry or government (including the military). Removing these barriers would dramatically increase the value of AI to industry and government, which would accelerate AI development in general, potentially leading to TAI arriving before we're ready for it.
Research i

...