trentmkelly/LessWrong-43k huggingface dataset:
Coherence arguments imply a force for goal-directed behavior
by KatjaGrace — 2021-03-26T16:10:04.936Z

[Epistemic status: my current view, but I haven’t read all the stuff on this topic even in the LessWrong community, let alone more broadly.]

There is a line of thought that says that advanced AI will tend to be ‘goal-directed’—that is, consistently doing whatever makes certain favored outcomes more likely—and that this is to do with the ‘coherence arguments’. Rohin Shah, and probably others1, have argued against this. I want to argue against them.

The old argument for coherence implying (worrisome) goal-directedness

I’d reconstruct the original argument that Rohin is arguing against as something like this (making no claim about my own beliefs here):

‘Whatever things you care about, you are best off assigning consistent numerical values to them and maximizing the expected sum of those values’
‘Coherence arguments2’ mean that if you don’t maximize ‘expected utility’ (EU)—that is, if you don’t make every choice in accordance with what gets the highest average score, given consistent preferability scores that you assign to all outcomes—then you will make strictly worse choices by your own lights than if you followed some alternate EU-maximizing strategy (at least in some situations, though they may not arise). For instance, you’ll be vulnerable to ‘money-pumping’—being predictably parted from your money for nothing.3
‘Advanced AI will tend to do better things instead of worse things, by its own lights’
Advanced AI will tend to avoid options that are predictably strictly worse by its own lights, due to being highly optimized for making good choices (by some combination of external processes that produced it, its own efforts, and the selection pressure acting on its existence).
‘Therefore advanced AI will maximize EU, roughly’
Advanced AI will tend to be fairly coherent, at least to a level of approximation where becoming more coherent isn’t worth the cost.4 Which will probably be fairly coherent (e.g. close enough to coherent that humans can’t anticipate the inconsistencies).
‘Maximizing EU is pretty much the same as being goal-directed’
To maximize expected utility is to pursue the goal of that which you have assigned higher utility to.5

And since the point of all this is to argue that advanced AI might be hard to deal with, note that we can get to that conclusion with:

‘Highly intelligent goal-directed agents are dangerous’
If AI systems exist that very competently pur

...
Where did the idea of x-risk come from?
by fowlertm — 2021-03-26T15:39:40.867Z

Recently on the Futurati Podcast we interviewed Thomas Moynihan on how humanity came to discover the possibility of its own extinction and, with it, the full value of ourselves and our potential future.

Ideas are the basic means by which we grapple with the staggering complexity the world. They matter, and where they come from matters.

Without a proper understanding of our place in the universe, of whether or not we're likely alone, of the source of values, and of what is likely to end the human experiment, we simply can't know how high the stakes are.

With the discovery of the concept of existential risk we've achieved an important milestone in our maturation as a species; Dr. Moynihan is, to my knowledge, the first intellectual historian to tell this story.

In our conversation we also discuss the anatomy of viral memes, the cognitive scaffolding required for having certain kinds of insights, 'vanguard ideas' which up open new regions of conceptspace, and many other things.

Like the video and subscribe to our channel for more content like this :)
Communication Requires Common Interests or Differential Signal Costs
by Zack_M_Davis — 2021-03-26T06:41:25.043Z

If a lion could speak, we could not understand her.

—Ludwig Wittgenstein

In order for information to be transmitted from one place to another, it needs to be conveyed by some physical medium: material links of cause and effect that vary in response to variation at the source, correlating the states of different parts of the universe—a "map" that reflects a "territory." When you see a rock, that's only possible because the pattern of light reflected from the rock into your eyes is different from what it would have been if the rock were a different color, or if it weren't there.

This is the rudimentary cognitive technology of perception. Notably, perception only requires technology on the receiving end. Your brain and your eyes were optimized by natural selection to be able to do things like interpreting light as conveying information from elsewhere in the universe. The rock wasn't: rocks were just the same before any animals evolved to see them. The light wasn't, either: light reflected off rocks just the same before, too.

In contrast, the advanced cognitive technology of communication is more capital-intensive: not only the receiver but also the source (now called the "sender") and the medium (now called "signals") must be optimized for the task. When you read a blog post about a rock, not only did the post author need to use the technology of perception to see the rock, you and the author also needed to have a language in common, from which the author would have used different words if the rock were a different color, or if it weren't there.

Like many advanced technologies, communication is fragile and needs to be delicately maintained. A common language requires solving the coordination problem of agreeing on a convention that assigns meanings to signals—and maintaining that convention through continued usage. The existence of stable solutions to the coordination problem ends up depending on the communicating agents' goals, even if the meaning of the convention (should the agents succeed in establishing one) is strictly denotative. If the sender and receiver's interests are aligned, a convention can be discovered by simple reinforcement learning from trial and error. This doesn't work if the sender and receiver's interests diverge—if the sender would profit by making the receiver update in the wrong direction. Deception is parasitic on conventional meaning: ...
Toward A Bayesian Theory Of Willpower
by Scott Alexander — 2021-03-26T02:33:55.056Z

(crossposted from Astral Codex Ten)

I.

What is willpower?

Five years ago, I reviewed Baumeister and Tierney's book on the subject. They tentatively concluded it's a way of rationing brain glucose. But their key results have failed to replicate, and people who know more about glucose physiology say it makes no theoretical sense.

Robert Kurzban, one of the most on-point critics of the glucose theory, gives his own model of willpower: it's a way of minimizing opportunity costs. But how come my brain is convinced that playing Civilization for ten hours has no opportunity cost, but spending five seconds putting away dishes has such immense opportunity costs that it will probably leave me permanently destitute? I can't find any correlation between the subjective phenomenon of willpower or effort-needingness and real opportunity costs at all.

A tradition originating in psychotherapy, and ably represented eg here by Kaj Sotala, interprets willpower as conflict between mental agents. One "subagent" might want to sit down and study for a test. But maybe one subagent represents the pressure your parents are putting on you to do well in school so you can become a doctor and have a stable career, and another subagent represents your own desire to drop out and become a musician, and even though the "do well in school" subagent is on top now, the "become a musician" subagent is strong enough to sabotage you by making you feel mysteriously unable to study. This usually ends with something about how enough therapy can help you reconcile these subagents and have lots of willpower again. But this works a lot better in therapy books than it does in real life. Also, what childhood trauma made my subagents so averse to doing dishes?

I've come to disagree with all of these perspectives. I think willpower is best thought of as a Bayesian process, ie an attempt to add up different kinds of evidence.

II.

My model has several different competing mental processes trying to determine your actions. One is a prior on motionlessness; if you have no reason at all to do anything, stay where you are. A second is a pure reinforcement learner - "do whatever has brought you the most reward in the past". And the third is your high-level conscious calculations about what the right thing to do is.

These all submit "evidence" to your basal ganglia, the brain structure that chooses actions. Using the same evi...
Some things I've learned in college
by Aaron Bergman — 2021-03-25T21:30:05.899Z
Link: https://aaronbergman.substack.com/p/some-things-ive-learned-in-college

In my corner of the internet, there is a healthy contrarian skepticism about education, and about college in particular. Here are some of the points or critiques of the learning that (perhaps) goes on in college, many of which I largely agree with:

Signalling

(i.e. any learning that occurs is minimal or useless, as it is merely a side effect of school’s primary function. I agree about 75%)

Bryan Caplan’s

The Case Against Education

indeed makes a convincing case against education: school is about signaling, not learning things. Doing well in college signals to employers or graduate admissions departments that you are intelligent, conscientious, and otherwise capable as functioning in as a knowledge worker in modern society.

To some extent, this can be an important function of college since employers and other institutions do need a way of figuring out who is competent. However, the cost is that a huge proportion young adult’s lives are spent “learning” things that they will rapidly forget and/or never use.

Learning the wrong things

(I agree about 30%)

Maybe people are learning things, but they’re useless things. This is largely the old “What’s the difference between a philosophy degree and a large pizza? The pizza can feed a family of four” or “We don’t need art history/gender studies/classics majors, we need engineers!” or “Go to trade school!” critiques.

Inefficiency: time

(I agree with this about 50%)

Instead of spending dozens of hours a week for four years going to class and completing assigned coursework, smart and driven students could learn the same amount that they actually learn in a fraction of the time.

Inefficiency: money

(I agree with this about 99%)

Instead of spending $200,000+, smart and driven students could learn the same amount that they actually learn with a tiny fraction of the money. I can’t find the source for this, but I distinctly remember reading that, for a fraction of college tuition, you could hire a team of PhDs to personally tutor you in their subject of expertise for the amount of time students generally spend in class (edit: this is at least one of the sources). And, of course, many could learn just as well with a few textbooks and cheap or free online courses.

Inspiration for the post

As my “I agree with this X%” statements indicate, I too am skeptical of the direct educational value of college. To Brian Caplan’s credit, his

book

does conc...
Use your rationality or it's useless
by XxX — 2021-03-25T21:23:33.066Z

Infinite rationality is worthless if you never act. An omniscient entity which can never act has a net progress score of 0.

Likewise, an entity with totally random actions is going to fare poorly. Given that the possibility-space of state improving actions is much less than half, I see it as in fact likely the chaotic entity will worsen its situation over time.

If progress requires both rationality and action We are left with 3 possibilities.

An entity has disproportionately better decision-making then level of activity, and should focus its energy on using its knowledge.

An entity is disproportionately more active than rational, and would benefit from becoming more rational instead of continuing to be busy, or

An entity is somehow balanced, in which case good for them.

Looking at LessWrong, I'll give you three guesses which most of us are..

So for the people who read LessWrong, myself included, it would almost be more helpful to have a LessPassive and let our tendency towards epistemological optimisation play out on its own.

We need to increase our level of action. The upvotes on actually "Making Vaccine" are our belated acknowledgement of that.
Money Buys Happiness
by XxX — 2021-03-25T21:02:54.525Z

Eliezer Yudkowsky says money is the unit of caring.

I note that money is the unit of being able to buy things.

It therefore behooves me, someone who aspires to be more rational, to become rich. Time is money, money is time, and opportunity and happiness too.

How does money buy happiness? If you aren't living socially saturated with the very best people you would custom select from across the entire world, money would help. If you spend any time whatsoever doing things for money, that time-energy would be entirely freed for meaningful pursuits. I struggle to imagine that LessWrongers would be unable to do vastly better, if given infinite dollars.

And notably, money allows others to be the instrument of our rationality. If we constantly come up with ideas that aren't put into practice, this may be the greatest multiplier of all.

So then, if having large amounts of money is of such colossal worth that it multiplies many facets of our life by massive amounts, each of which compound the others...

Let's become rich?
Call to the technical support of reality
by bayesyatina — 2021-03-25T20:05:56.428Z

- Hello, what is your question?
- Good afternoon, I have some interference in reality. It looks like it's broken.
- Ok, I understand. Let's check... (hums a popular tune for thirty seconds, tapping his fingers on the keyboard). Everything is checked, nothing unusual has happened in your world in the last 13.8 billion years. Are you sure you've detected a malfunction in reality?
- Of course, reality does not work as it should! I can see that.
- Oh, sorry, I forgot to ask for the script. Are you a client of the reality? Should it work the way you want it to?
- That's right.
- Please state the contract number.
- What? What is the contract number? I don't have a contract.
- But in this case, Reality has the right to refuse you services in order to adapt to your tastes. Such claims have no legal support and will not be considered by the Supreme Court of the Multiverse.
- What should I do? The Reality does not meet my expectations.
- But we found out that reality doesn't have to do that. I can suggest that you download a free update of your beliefs so that they match reality.
- But then I will not expect what I want, I will have to expect what will come.
- Yes, it is. I am glad that you are familiar with the working principle of our product. Now we will fix everything (sings the continuation of that fashionable song, knocks on tired keys). I looked it up. Your device doesn't have enough memory and processing power to record the full version of Reality. Therefore, we suggest you download the package "Reality.lite". It has less accuracy, but will still match reality more often than the current pirated version. (the next sentence, as if reading from a piece of paper) And its not-absolute accuracy will help you keep that sense of surprise that you find so dear. Should I forward you the link?
- (sighs) Come on, for that matter.
- lesswrong.com

P.S. I apologize for my poor English. If something needs to be corrected, please indicate it.
The danger of error descriptions
by gistya — 2021-03-25T20:05:17.336Z

Complex, interconnected, automated systems increasingly control our world. We use cryptographically-secure systems of change-management (git) to ensure only authorized individuals can make changes to the programs of decisions that power these automated systems, but we remain subject to the risk of the fallibility of all human endeavor—namely that all human-produced software has bugs.

Under the presumption that detailed logs always help the programmer to fix bugs, a cargo-cult has evolved whereby we believe that code is inherently "safer" and "better" if it leaves detailed breadcrumbs about each and every any important decision, especially when it reaches an error state—an unwanted or undefined state.

This cargo-cult grades software in terms of resiliency—the ability to cope with error states. The cargo-cult holds as the prime tenet of its religion, the unfounded notion that a logical analysis of empirically obtained data can reliably ascertain the resiliency of a software system system.

As a result, virtually every industry now has "resilient" automated programs running critical systems at the enterprise and government levels. These programs continuously create petabytes upon petabytes of logs. When agents try to solve problems with such systems, these logs generally make up the initial input to the problem-solving process, as a source of truth about some problem.

Then the cargo-cult will feel confident making decisions based on these logs as if they represented cryptographically-verified, scientifically-proven, bona-fide true descriptions of some past event, such as an error state, However logs almost never rely on the kinds of cryptographic guarantees that we use for verifying everything else that happens in a computer, from source-code to compiled program.

Modern OSs will refuse to run any code that lacks a digital signature, and every single change made to a git (source-code) repository carries a digital hash that both identifies it and guarantees it as valid. You can't use any remote systems these days without some kind of cryptographic protocol to guarantee the security of the transmissions, not to mention the fact that we increasingly rely on blockchain as the most cryptographically secure of all forms of record.

And yet, we almost universally trust digital logs without any critical thought being applied. Even in the extremel...
On language modeling and future abstract reasoning research
by alexlyzhov — 2021-03-25T17:43:07.198Z
Link: https://docs.google.com/document/d/1Jlc44xOz0v9WO287KpfD3gXxRiTYj2VIC6rnFsogM14/edit?usp=sharing

Hi everyone! Here I link to a sketch of my thoughts on how recent advances in language modeling may be connected, or lead, to future advances in developing machine learning models with abstract reasoning capabilities.

This was done as a side project last year during my research fellowship at the Center of Long-Term Risk. Many thanks to Daniel Kokotajlo, Jesse Clifton, Anthony DiGiovanni for useful comments.
Is a Self-Iterating AGI Vulnerable to Thompson-style Trojans?
by sxae — 2021-03-25T14:46:34.390Z

In his 1984 lecture "Reflections on Trusting Trust", Ken Thompson (of Unix fame) speculated about a methodology for inserting an undetectable trojan horse within the C compiler binary that would self-propagate throughout all future versions. (Additional good video that got me thinking about this.)

The replacement code would miscompile the login command so that it would accept either the intended encrypted password or a particular known password. Thus if this code were installed in binary and the binary were used to compile the login command, I could log into that system as any user.

The lecture explains the trojan far better than I can, but appears to be a vulnerability that occurs within bootstrapped compilers, and specifically arises at the point at which the compromised version of the compiler is responsible for compiling the next version. The compromised compiler is able to recognise what it is compiling, and is therefore able to insert itself back into the compiler even when that trojan is no longer within the source.

Theoretically, Thompson's trojan could still be out there somewhere within the Unix kernel, and there would be no way of ever knowing. Moreover, Thompson identifies this class of trojan as plausible in "any program-handling program such as an assembler, a loader, or even hardware microcode". Even if you were to go and download the GCC source code and build your own compiler from source, you must do so with a potentially compromised version of the compiler. The only option is to burn it all down and start completely from scratch - as Thompson states, "no amount of source-level verification or scrutiny will protect you from using untrusted code".

Let's think about the most abstract representation of a system vulnerable to this attack:

We represent some kind of iterative process as two types of component:

S represents the symbols we input into the iteration process, S' the second set, and so on.

In a compiler, this is the source code.

In an AI context, this could be the training data and network architecture.

f(S) represents an object that has been transformed by S, which is in itself capable of transforming the next set of symbols S' into some new f(S')

In a compiler, this is the machine code

In an AI, this is the trained model

We begin with an existing function f(S), and give it a new data set S'.[1]

Consider an AGI which is:

powerf

...
Covid 3/25: Own Goals
by Zvi — 2021-03-25T13:50:00.989Z

AstraZeneca has made quite the mess of things. First they screwed up their initial studies in ways that kind of boggle the mind. Then, with the studies designed to repair trust, fix the problem and allow approval, they report incomplete results in order to make themselves look better, even though inevitably they were caught doing this within a day – it’s pretty inevitable that you’ll be caught when you do something in public that someone already warned you not to do, especially when that someone is also the regulatory authority. Oops.

In addition to AZ’s own goals, health officials continue to score additional own goals around the whole issue of blood clots (that don’t exist, and wouldn’t matter even if they did). Most but not all places have resumed vaccinations, but trust in the vaccine, and plausibly in all vaccines, is permanently damaged.

Those developments are infuriating, and also enlightening as to how the system of the world functions these days, but the main event remains the race between new strains and vaccinations.

In America the race is plausibly close. Cases are rising, and likely will continue to rise for several more weeks, especially if vaccination rates continue to stagnate. But that acceleration should start soon, and at an additional 3% protection per week that grows and compounds, the vaccinations won’t take that long to turn the tide even if they don’t accelerate much.

In Europe the race is not so close. Vaccinations are running far slower, with no short term hope for things to get much better. The recent own goals only made a bad situation worse, and in many European countries things are looking quite bad. Lockdowns are once again the order of the day in many places, most notably Germany, and yet the situation is getting rapidly worse, in some places reaching crisis proportions.

I spend a lot of focus on everything that’s wrong with our vaccine efforts, but as with many such things it’s equally vital to remember that things could be, and in almost all other places are, so much worse. We failed versus the standards of a fully functional civilization, but compared to the rest of Earth 2020, we passed with flying colors. We need to remember both results.

Let’s run the numbers.

The Numbers

Predictions

Last week’s prediction (WaPo numbers): Positivity rate of 4.3% (up 0.2%), deaths decline by 8%.

Result:

Pretty close...