trentmkelly/LessWrong-43k huggingface dataset:
Additive Operations on Cartesian Frames
by Scott Garrabrant — 2020-10-26T15:12:14.556Z

The mathematical object (but not the philosophical interpretation) of a Cartesian Frame is studied under the name "Chu space."

(In category theory, Chu spaces are usually studied in the special case of W=2. To learn more about Chu spaces, see Vaughan Pratt's guide to papers and nLab's page on the Chu construction.)

In this post and the next one, I'll mostly be discussing standard facts about Chu spaces. I'll also discuss how to interpret the standard definitions as statements about agency.

Chu spaces form a category as a special case of the Chu construction. You may notice a strong similarity between operations on Cartesian frames and operations in linear logic, coming from the fact that the Chu construction is also intimately related to linear logic, and is used in the semantics for linear logic.

Linear logic has a large number of operations—additive conjunction (&), multiplicative conjunction (⊗), and so on—and many of those symbols will turn out to have interpretations for Cartesian frames, and they're actually going to be meaningful interpretations in this setting. For that reason, we'll be stealing much of our notation from linear logic, though this sequence won't assume familiarity with linear logic.

Definition: Chu(W) is the category whose objects are Cartesian frames over W, whose morphisms from C=(A,E,⋅) to D=(B,F,⋆) are pairs of functions (g:A→B,h:F→E), such that a⋅h(f)=g(a)⋆f for all a∈A and f∈F, and whose composition of morphisms is given by (g1,h1)∘(g0,h0)=(g1∘g0,h0∘h1).

The composition of two morphisms C0→C1 and C1→C2, then, sends the agent of C0 to C2 and sends the environment of C2 to C0.

Claim: Chu(W) is a category.

Proof: It suffices to show that composition is well-defined and associative and there exist identity morphisms. For identity, (idA,idE) is clearly an identity on C=(A,E,⋅), where idX is the identity map from X to itself.

The composition (g1,h1)∘(g0,h0) of (g0,h0):C0→C1 with (g1,h1):C1→C2 is (g1∘g0,h0∘h1):C0→C2. To verify that this is a morphism, we just need that a0⋅0h0(h1(e2))=g1(g0(a0))⋅2e2 for all a0∈Agent(C0), and e2∈Env(C2), where ⋅i=Eval(Ci). Indeed,

a0⋅0h0(h1(e2))=g0(a0)⋅1h1(e2)=g1(g0(a0))⋅2e2,

sin...
Supervised learning of outputs in the brain
by Steven Byrnes — 2020-10-26T14:32:54.061Z

Follow-up to: My computational framework for the brain.

(Epistemic status: In what's becoming a pattern, I tried reading the standard neuroscience discussions in this area but found them confusing or narrow or not answering my big-picture questions in a way I could understand, so I'm kinda piecing things together myself. I assume I'm mostly reinventing wheels and this post is just pedagogy, but I dunno, parts could be original and/or wrong.)

It didn't take long after finishing my previous post on the big picture of brain computations before I found a problem! I was stuck on something that didn't make sense, which led to my reading more about the amygdala and cerebellum, and feeling very confused about what they were for and how they fit in. And then it hit me, I was missing something! The thing I was missing was: supervised learning of actions and other outputs.

Supervised learning of outputs: What, Where, and Why

What is supervised learning of outputs?

Supervised learning of outputs is when you have a learning algorithm receiving a ground-truth signal like this:

“Hey learning algorithm: you messed up—you should have moved your leg.”

Compare that to reinforcement learning, where the learning algorithm gets a much less helpful ground-truth signal:

“Hey learning algorithm: you messed up."

(a.k.a negative reward). Obviously, you can learn much faster with supervised learning of outputs than with reinforcement learning. So, of course, our default assumption should be that Evolution has built supervised learning of outputs into the brain, when information is available about what you should have done.

I'll call this type of information "shoulda information". ("Shoulda" is a slang shorthand for "should have".) A blink-related shoulda signal means "you shoulda blinked".

And indeed, although shoulda information is not always available, it is sometimes available! "Hey, you just get whacked in the face? Well, you shoulda flinched!" Evolution knows very well that you shoulda flinched. There are lots of examples like that. So we should expect to find supervised learning of outputs in the brain.

My computational framework for the brain post had supervised learning on the input-processing side (more specifically, self-supervised, a.k.a. predictive learning, in the neocortex), and it had reinforcement learning of outputs (also in the neocortex), and it had hardcoded input-output relati...
Defensible positions
by Chris_Leong — 2020-10-26T10:37:09.840Z

A useful concept when discussing philosophy is the concept of defensible positions or alternatively "hills to die on". In philosophy, some positions are more defensible than others or at least appear that way at first glance. Take for example discussions on morality.

Utilitarianism and absolute deontology (deontology where moral rules cannot be overridden by any amount of utility) tend to more naturally draw the philosophically inclined than any of the intermediate positions (I will concede that immediate positions are a more natural draw to those who are less philosophically inclined). However, let's suppose we take a deontologist who is willing to endorse absolute deontology in the abstract, but when presented with a concrete situation where they can punch an innocent person to save the world, concedes that their principles really aren't workable.

In this case, we've managed to explode their current defensible position and they have to find a new one to seek shelter at. Some philosophical refugees might then head over to utilitarianism, but this isn't necessarily the only option. Maybe there's another defensible position they can find in the giant intermediate zone, even if they can't name it at this time (we're ignoring alternatives like virtue ethics at the moment). Of course, often this is very difficult to rule out as it is hard to prove a negative. While sometimes you can make a general argument against intermediate positions, at other times the variety of them makes this extremely challenging and it turns into a game of whack-a-mole.

I guess the reason why I consider this a useful concept is because it allows me to more easily articulate my reasons for particular philosophical positions. For example, I lean towards utilitarianism over deontology. Strong deontology does not seem viable to me at all and it's a struggle to find an intermediate position that doesn't end up with the weakness of both utilitarianism and deontology. Similarly, I support paying in Counterfactual Mugging as the Counterfactual Prisoner's Dilemma explodes the position that we can just always ignore counterfactuals and while that doesn't prove that we should shift over to taking the expected value over counterfactuals, it seems hard to find a viable intermediate position.

In the past before I had the concept of defensible positions I would have been more likely to say that these argu...
Sam Altman at SlateStarCodex Online Meetup
by JoshuaFox — 2020-10-26T10:26:43.860Z

Our online meetup on November 8, at 10:30 AM Pacific Standard Time, 18:30 UTC, 20:30 Israel Standard Time, will feature Sam Altman, CEO of OpenAI, who will be answering questions. We will then have open video-chat rooms as usual.
We hope to focus the Q&A on AI, AGI, GPT-N and OpenAI, but feel free to ask questions on other topics if you like.
Please register here , and we'll send you an invitation.
But what kinds of puppets are we?
by KatjaGrace — 2020-10-26T08:40:03.435Z

Crossposted from world spirit sock puppet.

I watched The Social Dilemma last night. I took the problem that it warned of to be the following:

Social media and similar online services make their money by selling your attention to advertisers

These companies put vast optimization effort into manipulating you, to extract more attention

This means your behavior and attention is probably very shaped by these forces (which you can perhaps confirm by noting your own readiness to scroll through stuff on your phone)

This seems broadly plausible and bad, but I wonder if it isn’t quite that bad.

I heard the film as suggesting that your behavior and thoughts in general are being twisted by these forces. But lets distinguish between a system where huge resources are going into keeping you scrolling say—at which point an advertiser will pay for their shot at persuading you—and a system where those resources are going into manipulating you directly to do the things that the advertiser would like. In the first case, maybe you look at your phone too much, but there isn’t a clear pressure on your opinions or behavior besides pro phone. In the second case, maybe you end up with whatever opinions and actions someone paid the most for (this all supposing the system works). Let’s call these distorted-looking and distorted-acting.

While watching I interpreted the film suggesting the sort of broad manipulation that would come with distorted-acting, but thinking about it afterwards, isn’t the kind of optimization going on with social media actually distorted-looking? (Followed by whatever optimization the advertisers do to get you to do what they want, which I guess is of a kind with what they have always done, so at least not a new experimental horror.) I actually don’t really know. And maybe it isn’t a bright distinction.

Maybe optimization for you clicking on ads should be a different category (i.e. ‘distorted-clicking’). This seems close to distorted-looking, in that it isn’t directly seeking to manipulate your behavior outside of your phone session, but a big step closer to distorted-acting, since you have been set off toward whatever you have ultimately been targeted to buy.

I was at first thinking that distorted-looking was safer than distorted-acting. But distorted-looking forces probably do also distort your opinions and actions. For instance, as the film suggested, you are li...
Models predicting significant violence in the US?
by kdbscott — 2020-10-25T18:45:20.929Z

Do you have a model predicting >1% chance of significant political violence (>5,000 deaths) in the US in the next year? Pls share.

Also welcome: models predicting <.1%

Metaculus has been walking around 1-5% probability of a US civil war before July 2021. While 1% is the lowest probability one can give on Metaculus, rumor has it some reasonable people put more than 1% probability of significant political violence (SPV) happening.

I am[was] confused about this, and want[ed] to get a better sense of what models lead people to assign credence on this order of magnitude.

[it turns out this exercise led me to no longer feel disbelief, but I'm preserving it below. See Changing My Mind for my update]

My model borrows a bit from Samo Burja's civil war typology and goes something like: "while there's some probability of small-scale violence (e.g. a few hundred deaths from scattered violence), there simply doesn't exist the mechanisms for SPV". In my mind, mechanisms include:

Military: e.g. military vs. government (coup), military vs. military (split along e.g. leadership lines), or a weak/disorganized military that doesn't withstand a revolutionary offensive. I don't see any of this happening with the US military (e.g. <.01%). ThirdEyeOpen on Metaculus outlines cases where the military 'sides' with the democrats or republicans, which blows my mind. Is this a blind spot for me? Is there some small chance that any coherent part of the military defects in this way?
State (proxy wars): Both sides of a conflict having support from states. I guess there's some 'foreign interference' in our politics already, but if I'm Russia I really don't want to get caught supplying significant amounts of money/training/arms to insurgent groups in the US. That feels like much more risk (to Russia) than the incremental gain. But maybe there's some situation where it's worthwhile? Seems pretty close to WWIII, but I guess that's tail risks?
Insurgency: I currently have this as a large bucket that spans from haphazard terrorism to organized attempts to replace the government. It seems like the US might experience some things on this scale, but how far? I’m having a hard time imagining violence worse than e.g. The Troubles. What am I missing here? Maybe it's easier to kill more people these days? Is the main concern effective terrorism from a few actors, or more like thousands of ins

...
Some criticisms of polling
by Jimdrix_Hendri — 2020-10-25T18:45:10.523Z

POLLING 1. WHAT DO YOU INTEND?

As we approach the 3 November election, there are many polls promising to predict the outcomes. Looking back, many people were surprised by the outcome of the 2016 presidential. Late polls had given H Clinton a win probability over 80%. Were those polls in error? Not necessarily. An 80+% chance falls short of certainty. Still, the surprise result raises questions, and a lot of thought has gone into how polls arrive at a representative sample of the voting public. Something that gets a less attention is the method used to aggregate and present data. I think the usual methods are sloppy and misleading.

Suppose you were conducting a poll predicting the outcome of the upcoming election. The usual method is to go out and ask people their preferences, taking care to collect a representative cross section of voters, and applying well-reasoned principles weighing the chances of their voting. Then you report that candidate X1 is expected to achieve A1% of the vote, etc. The result is continuous, and gives an impression of exactness. But the impression lies. In fact, this type of report loses much information by compressing data through a binary filter.

Imagine you were a candidate and wanted to know where to spend your limited time and resources. You might look to the poll figures and concentrate on big localities where the polls show nearly equal voter support. That is the best you could do with this kind of poll data. But it is a mediocre guide to resource allocation, and a poor way to predict which locality are likely to flip. Here is a better alternative: instead of just asking interviewees to name their preference, the pollster should measure the strength of the interviewee’s commitment.

Since we will eventually want to aggregate responses, we need a way to express strength of commitment according to some uniform scale. One idea could be to ask the interviewee to rank how resistant he would feel to changing his vote to the opposition candidate compared with his strength of allegiance going into previous elections. (This is an imperfect measure. I’d welcome suggestions for improvements). The pollster registers this “percentile allegiance score” and can later report on average and variance (alternatively: publish the distribution).

How does this help? Consider two localities....
Reply to Jebari and Lundborg on Artificial Superintelligence
by Richard_Ngo — 2020-10-25T13:50:23.601Z

Jebari and Lundborg have recently published an article entitled Artificial superintelligence and its limits: why AlphaZero cannot become a general agent. It focuses on the thorny issue of agency in superintelligent AIs. I’m glad to see more work on this crucial topic; however, I have significant disagreements with their terminology and argumentation, as I outline in this reply. Note that it was written rather quickly, and so might lack clarity in some places, or fail to convey some nuances of the original article. I welcome comments and further responses.

Their paper runs roughly as follows: Jebari and Lundborg first discuss the belief–desire model for intentional action, under which agents act in ways that they believe will bring about their desires. They then distinguish between different degrees of generality that agents can have: “general agency is, loosely speaking, the ability to act in a diverse set of situations.” They consider thermostats to be very specialised agents, and dogs to be fairly general agents. They then introduce introduce Legg and Hutter’s definition of intelligence as “an agent’s ability to achieve its goals in a wide range of environments”. This is, unfortunately, a misquotation of Legg and Hutter, and one which leads Jebari and Lundborg astray, as they follow it with:

Claim 1: If an agent has very specialized goals or desires, it can be superintelligent with regards to those desires without being a general agent.

The actual definition given in both A collection of definitions of intelligence and Universal intelligence is: “Intelligence measures an agent’s ability to achieve goals in a wide range of environments”. Note the important distinction between an agent which can achieve its goals versus one which can achieve goals in general. In the original context, it is clear that to be highly intelligent according to this definition, an agent doesn’t just need to be able to achieve a single (potentially specialised) set of goals, but rather a wide variety of them! More specifically, in Universal intelligence goals are formalised as a reward function implemented by the environment, with intelligence measured over all computable environments (and therefore all computable reward functions). So Legg and Hutter’s definition does not allow us to call a non-general agent superintelligent. Bostrom, too, defines superintelligence to refer specifical...
Increasing Isolation Again
by jefftk — 2020-10-25T13:10:26.731Z

The coronavirus is getting much worse in Massachusetts, as it is in
many places. Wastewater levels do not look good:

Neither does the percentage of tested individuals who are positive:

This is p8 of the daily
coronavirus dashboard for today, and each day's is available at mass.gov
when ready. The "percentage of all molecular tests that are positive"
number is much less useful: we also don't want to think things are
good just because MIT is retesting their students at an incredible
rate.

When our house relaxed
our strict isolation in early July things were looking good: rates
were down, testing was up. The per-individual positive test
percentage was ~1.5-2%:

Hospitalizations are starting to trend upwards as well, though are
still far below their peak:

COVID
Tracking Project

We talked yesterday as a house, and decided on new house
rules:

Our default is the same as it has been since March: you can only interact
with other people outside the household if you're outside, masked, and
at least six feet apart. I recommend buying snowpants for comfortable outdoor cold-weather
socializing.

We're still treating a small number of people who have had
coronavirus and tested positive for antibodies on a reliable test as
fully safe. There is still some chance that they could get it again,
with immunity feeding overtime, but we think the additional risk from
them is very low.

Previously
Each of us could pick one person each week to spend time with us
unrestricted: no masks, kissing etc ok. We're not doing this
anymore.

Now, each of us can pick one person each week to spend time
with inside, as long as we wear a high-quality mask keep faces at
least 2ft apart. It's now possible to get KN-95 (Chinese-spec N95)
masks at about $2/mask (ex).

We're still not going into stores to shop etc.

These rules don't bring our risk down to zero, but we think they're a
reasonable trade off between danger of getting or spreading the
coronavirus and the benefits of spending time with people.

We will of course revisit as things continue to change.
Yet another world spirit sock puppet
by KatjaGrace — 2020-10-25T05:30:09.248Z

Crossposted from world spirit sock puppet.

I have almost successfully made and made decent this here my new blog, in spite of little pre-existing familiarity with relevant tools beyond things like persistence in the face of adversity and Googling things. I don’t fully understand how it works, but it is a different and freer non-understanding than with WordPress or Tumblr. This blog is more mine to have mis-built and to go back and fix. It is like not understanding why your cake is still a liquid rather than like not understanding why your printer isn’t recognized by your computer.

My plan is to blog at worldspiritsockpuppet.com now, and cross-post to my older blogs the subset of posts that fit there.

The main remaining thing is to add comments. If anyone has views about how those should be, er, tweet at me?
Which headlines and narratives are mostly clickbait?
by Pontor — 2020-10-25T01:19:29.361Z

Or: What do superforecasters tend to think of the topics that get tweets and airtime these days? The following questions get put into my attention (consensually and otherwise) and I am genuinely uncertain about the answers.

Is Trump really gearing up for a fascist takeover? Is fascism truly resurgent?
Does the elite left really endorse looting?
If Biden doesn't win in a landslide, will this really cause a national crisis?
Are we really seeing an incipient purge of non-leftwing thinkers from academia? Just how much of the putative "Great Awokening" is an illusion due to (social) media hype?
Have Silicon Valley and the mainstream media really gone to "war"? What will be the course of this war, and how will it end? Will I be much affected if I am not an investor nor a news consumer?
Is California really about to be swallowed by leftwing politics and lose all its golden geese to Texas, New Hampshire, or wherever?
Are regular Americans being driven into poverty en masse, soon to all be wage slaves to a technocratic elite?
Will higher education be drastically affected by the coming era of Work From Home?
How ought one view statements from the CDC henceforth? How much did its behavior and credibility in 2020 differ from other years or other administrations?
Are we going to see massive migrations out of cities? A world of "red zones and green zones" where large gatherings become a way for societies to flex their public health?
Is the general issue of misinformation (including but not limited to conspiracy theories and fake news) really much worse now than in decades past? By what metrics?
How much divisive mental malware have entities like the Internet Research Agency really managed to get into our brains? Are countries other than Russia getting in on the action?
Is China really running a global surveillance operation by compromising people's smartphones via TikTok? Are they really going to match the US in influence in the next 20 years? Will this require them to ease up on the Uighurs or not?

Sorry, that was a lot to drop all at once, but the cacophony is the point. I know headlines are headlines, clickbait is clickbait, but I frequently hear smart people (including rationalists) giving ample conversational time to any of the above topics. I suspect that with a moderate amount of effort, a smart person could identify and ignore whichever of these topics are pure distractio...