trentmkelly/LessWrong-43k huggingface dataset:
[Linkpost] Chinese government's guidelines on AI
by RomanS — 2021-12-10T21:10:58.327Z

China (PRC) published a new AI policy: "Ethical Norms for the New Generation Artificial Intelligence" (2021-09-26).

A surprisingly large part of the document's content is dedicated to the topics related to AI X-risk (although the existential risk is never addressed directly).

According to the document, it regulates AI research conducted by pretty much any entity in China, from companies to natural persons.

The original official document in Chinese is here (archive). Other AI policies are available here.

Some excerpts that could be relevant to AI X-risk:

AI activities should... follow the common values of mankind, respect... the fundamental interests of mankind,... promote human-machine harmony and friendliness.

Ensure that humans have full autonomy in decision-making,... the right to suspend the operation of artificial intelligence systems at any time,... ensure that artificial intelligence is always under human control.

Actively learn and popularize knowledge of AI ethics

objectively understand ethical issues, and not underestimate or exaggerate ethical risks

take the initiative to integrate AI ethics into the whole process of management

Strengthen risk prevention. Enhance bottom-line thinking and risk awareness, strengthen the study and judgment of potential risks of artificial intelligence development, carry out timely and systematic risk monitoring and assessment, establish an effective risk warning mechanism, and enhance the ability to control and dispose of ethical risks of artificial intelligence.

Strengthen the self-restraint of AI R&D-related activities, take the initiative to integrate AI ethics into all aspects of technology R&D, consciously carry out self-censorship, strengthen self-management, and not engage in AI R&D that violates ethics and morality.

In the algorithm design, implementation, application and other links, enhance transparency, interpretability, understandability, reliability, controllability

achieve verifiable, auditable, supervisable, traceable, predictable and trustworthy [AI].

Prohibit the use of artificial intelligence products.... that endanger national security, public safety
Understanding Gradient Hacking
by peterbarnett — 2021-12-10T15:58:51.272Z

This post was written under Evan Hubinger’s direct guidance and mentorship, as a part of the

Stanford Existential Risks Institute ML Alignment Theory Scholars (MATS) program

.

Additional thanks to Michele Campolo, Oliver Zhang, Adam Shimi, and Leo Gao for their thoughts and feedback on this post. This work is additionally supported by CEEALAR.

TL;DR: Gradient hacking describes how a deceptively misaligned mesa-optimizer could influence its own training process via its outputs and thinking process. This is mainly considered in the context of training a neural network with gradient descent. A deceptive mesa-optimizer would have strong incentives to gradient hack; to protect its mesa-objective from being overwritten by the training process, to prevent humans from discovering the deception, and to build its own misaligned capabilities. I think gradient hacking is only plausible if the model’s ability to gradient hack is strongly coupled to the model’s general capability to perform well on tasks, such that attempting to remove the gradient hacking ability causes a large drop in the model’s performance. If there is not this strong coupling I expect the gradient hacker to not be stable, and to be erased by gradient descent.

When attempting to prevent gradient hacking, we do not want to be in the position where we are attempting to look for a model performing gradient hacking; if a model is able to gradient hack then it may be able to competently hide itself from us. Rather we should attempt to design our training procedures such that gradient hacking never happens.

What is gradient hacking?

Gradient hacking

is a potential phenomenon where a machine learning model is able to influence its training process via its outputs and push its parameters in directions it desires. When a model is being trained by a local-optimization procedure such as gradient descent, it can deliberately choose certain outputs or ways of thinking to ensure the gradients are what it wants, and hence the parameter updates are in line with its own objectives. Here it is choosing its outputs and thought process to ‘hack’ the gradients and influence its training. This is most worrying if we have a deceptively aligned model, which is attempting to pretend to be aligned while actually having its own goals which are not aligned with ours. Such a gradient hacking model may choose its outpu...
There is essentially one best-validated theory of cognition.
by abramdemski — 2021-12-10T15:51:06.423Z

There are many theories of cognition. But if you want to work within a framework with the following properties:

Explains the major cognitive phenomena we know about.
Fits experimental data well, down to human reaction times, in a wide variety of psychological experiments.
Has a relatively complete story about functional neuroanatomy.

Well, then, I'm not aware of any theories which fit the bill as well as ACT-R theory.

You might also be interested in the common model of cognition (initially named standard model of the mind), which is consistent with the ACT-R picture, but also consistent with several competing theories. Think of it as the high-certainty subset of ACT-R.

References for ACT-R

I am no expert on ACT-R, so unfortunatery I can't tell you the best place to get started! However, here are some references.

Books by John R Anderson

John R Anderson is the primary researcher behind ACT-R theory. I have not read all of the following.

Learning and memory: an integrated approach. This book is not about ACT-R. It's basically a well-written textbook on everything we know about learning and memory. I think of it as John Anderson's account of all the empirical psychological phenomena which he would like to explain with ACT-R.
How can the human mind occur in the physical universe? This is a book about ACT-R. In this book, John R Anderson seeks to spell out "how the gears clank and how the pistons go and all the rest of that detail" -- which is to say, the inner workings of the brain.

John R Anderson also has several other books, which I haven't looked at very much; so maybe a different one is actually a better starting place.

Other References

Here are some other potentially useful starting places.

References for the Common Model of Cognition

A Standard Model of the Mind: Toward a Common Computational Framework across Artificial Intelligence, Cognitive Science, Neuroscience, and Robotics. This is the original paper for the common model of cognition. It rests on the observation that many cognitive architectures seem to be converging to common principles, much more than they were even 10 years earlier. To codify this consensus, leading researchers behind three of these theories got together and made a close comparison. (One of the three is my phd advisor, which is how I became aware of this.)
Empirical Evidence from Neuroimaging Data for a Standard Model of the Mind. What it s

...
The Promise and Peril of Finite Sets
by davidad — 2021-12-10T12:29:56.535Z

Has this ever happened to you?

But how can we decide whether this {species/moral theory/consensus node/quantum world} is actually 2 different things, or just 1 thing? Or does it even count at all?

The underlying cause of such dilemmas is that somebody, somewhere in the setup of your ontology and the framing of your problem statement, explicitly or implicitly, made the following move:

For simplicity of analysis, we assume that X is a finite set.

The Promise

This is an incredibly popular simplifying assumption, easily competitive with "we assume this distribution is Gaussian" and "we neglect higher-order terms". It's plausibly the single most popular mathematical modeling assumption in the world, because one needs so little background knowledge to make use of it.

It's also extremely powerful. Here's just some of the mathematical structure you get for free if you accept the assumption:

Computable representation. Saying that X is a finite set can be formalized as ∃n:N.X≅{i:N|i<n}, which means that there's some number k=⌈log2n⌉ such that any element x:X can be losslessly and efficiently represented with k bits.
Metric. Given two elements a,b:X, we say that d(a,b)=0 if a=b, and d(a,b)=1 otherwise. This is well-defined and makes X into a well-behaved metric space.
Compact Hausdorff topology. That metric induces a topology, which is also very well-behaved (in particular, it's compact and Hausdorff, which together make it a compactum).
Symmetry group. The automorphisms of {i:N|i<n} can be transported along the isomorphism to X, so now we have a permutation group, which is useful for formalizing things along the lines of "all elements of X will be treated equally".
Canonical probability measure. By the principle of transformation groups (which I discussed in the middle of this Shortform post about maximum entropy), this permutation group justifies normalized counting measure as an uninformative prior for an uncertain element of X (assuming all elements really should be considered interchangeable).
Ordering. If we have choice as a background assumption, a well-ordering isn't anything special, but from any specific witness of X's finiteness we get an eminently realizable total order, which is not merely a well-ordering but a completely distrib

...
Are big brains for processing sensory input?
by lsusr — 2021-12-10T07:08:31.495Z

Elephants have bigger brains than human beings. But human beings are smarter. What's all the cranial matter doing?

I used to believe that a larger organism's brain needs to generate larger electrical impulses to move its muscles. But that's not how electrical engineering works[1]. It's not like your nerves power your muscles. Your brain just sends a signal to the muscles telling your muscles to activate.

Muscular motor units aren't even voltage coded. They're frequency coded. When your brain wants a muscle to contract harder it sends nerve impulses more frequently. Trying to raise the output voltage of a motor neuron is like trying to push the number "2" through a single bit of a binary computer.

It's true that I wouldn't want to use a tiny microcontroller to control a Predator drone but that has nothing to do with the power output of its motor control pins. It's because I want more processing power, fine motor control and robustness against damage.

Blue whales have gigantic brains. I don't think the brain size is for robustness against damage. It would be more efficient for evolution to invest in a thicker skull than metabolically-expensive brain matter. I don't think the brain size is for fine motor control either. What is a blue whale going to do with fine motor control? It almost doesn't even have limbs.

Encephalization

An organism's encephalization quotient is the ratio of its brain size to its body size. Mammals of similar encephalization quotients tend to have similar intelligence levels.

Big brains don't just consume lots of energy. They have lower reaction speeds too[2] (though slow reaction speeds may less important to large organisms because large muscles take longer to move). I think the slow reaction speed of large brains is related to the time it takes a signal to travel from one end of the brain to the other.

Big brains are expensive in multiple dimensions. If big organisms have big brains then the big brains must be doing something useful. Since the encephalization baseline is orthogonal to intelligence, the encephalization baseline must be driven by something other than intelligence. Mammals with bigger bodies require larger brains just to break even. A blue whale has a large brain despite its low need for intelligence. Why?

Sensory Input

If a blue whale's bigger brain isn't driven by damage resistance, motor control or intelligence needs then the...
Combining Forecasts
by jsteinhardt — 2021-12-10T02:10:14.402Z

Often there are multiple ways to forecast the same thing, and we'd like a way of combining the forecasts together. For instance, consider the following question:

Will Will be among the top 3 most sold nonfiction books on Amazon for the week of November 21?

(This was posed to the course staff at the beginning of that week.)

I approached this problem using reference class forecasting, but there wasn't any single very good reference class. Specifically:

I built up a dataset of the past 6 weeks by manually copying information from the website.

Will had been #1 last week and #7 two weeks prior (and not released before that).

1 out of 3 books in the top 3 remained there the following week (15 data points total).

1 out of 5 books in the top 1 were in the top 3 the following week (5 data points total).

Of books that were top 3 last week and top 8 the previous week, 1 out of 4 stayed in the top 3 (4 points total).

Of books that were top 1 last week and top 8 the previous week, 1 out of 2 stayed in the top 3 (2 points total).

The problem is that:

None of these reference classes have many data points.

The class with the most data points ("top 3 last week") is the one that's least analogous to Will.

One way out of this is to combine all the reference classes together in a weighted average. This is called ensembling. I'll start with a rough intuitive way of doing this and then give a more principled way (as with most things in forecasting, the best approach is to practice the principled way until it informs your intuition, then mostly go with intuitive numbers).

Rough intuitive approach. Let's give each data source a score out of 10, based on how "good" it seems (a combination of the total number of data points and how analogous it is to Will).

Top 3: I give this 6 out of 10. There's a lot of data points but it's only somewhat analogous.

Top 1: I give this 3.5 out of 10. More analogous, but few enough data points to have lots of noise.

Top 3 + top 8: I give this 3 out of 10. Similar to top 1 but one fewer data point.

Top 1 + top 8: 2 out of 10. Most analogous but no data.

Then we combine as a weighted average: $\frac{6 \cdot (1/3) + 4 \cdot (1/5) + 3.5 \cdot (1/4) + 2 \cdot (1/2)}{6 + 4 + 3.5 + 2} = 0.30$.

More principled approach. The intuitive approach is reasonable, but as written it neglects some obvious information: for instance, the fact that the "top 3" refer...
Covid 12/9: Counting Down the Days
by Zvi — 2021-12-09T21:40:01.105Z

Omicron is coming. We are not preparing for it. That is the news that matters.

In addition to that, Delta is a growing problem as we face a winter wave. Future problems don’t protect us from current problems, no matter how inevitable and predictable the current problems might be. Here’s where we round those things up.

Note that due to recent events, and the new data on what people actually benefit from and want to share, I think it increasingly makes less sense to focus on weekly posts, and to focus more on addressing particular issues when they appear, so I’ll be transitioning towards being more inclined to split things up.

Omicron Posts: #1, #2, #3, #4, #5 (today)

Last Week’s Weekly Post.

Executive Summary

Omicron is coming.
Boosters won’t be updated in time.
Paxlovid remains illegal.

On Paxlovid, update is that final Pfizer data is expected in a few days, a perfect timeline gets the whole thing done by end of month, the 24% at Polymarket for that is probably somewhat low. Nothing’s going wrong, except for the total lack of urgency.

There’s various fights over masks and vaccine mandates, but game that matters is Omicron.

In the meantime, on with the show. Let’s run the numbers.

The Numbers

Predictions

Prediction from last week: 630k cases (+20%) and 8,000 deaths (+26%)

Results: 736k cases (+39%) and 8,355 deaths (+32%).

Prediction for next week: 750k cases (+2%) and 8,800 deaths (+5%).

I was being somewhat cowardly making my prediction. On reflection, I should have been more confident in what was obviously happening.

This week I’m predicting smaller increases because I expect that some of this week’s big increases were due to data dumps, and because I expect some amount of additional caution due to fear of Omicron, whether or not that yet makes physical sense.

Note that Omicron cases aren’t going to be noticeable this coming week.

Deaths

Cases

You’ll want to smooth out numbers over the last few weeks. Either way, it’s clear that the Delta situation continues to get worse in addition to the threat from Omicron. It won’t become a crisis before Omicron becomes primary.

Vaccinations

Fear of Omicron is working, at least somewhat, and there have been reports of difficulty getting appointments. Getting an appointment for my son’s second shot took a week, whereas my own booster was in my arm an hour after I went online ...
Conversation on technology forecasting and gradualism
by Richard_Ngo — 2021-12-09T21:23:21.187Z

This post is a transcript of a multi-day discussion between Paul Christiano, Richard Ngo, Eliezer Yudkowsky, Rob Bensinger, Holden Karnofsky, Rohin Shah, Carl Shulman, Nate Soares, and Jaan Tallinn, following up on the Yudkowsky/Christiano debate in 1, 2, 3, and 4.

Color key:

Chat by Paul, Richard, and Eliezer	Other chat

12. Follow-ups to the Christiano/Yudkowsky conversation

12.1. Bensinger and Shah on prototypes and technological forecasting

[Bensinger][16:22] (Sep. 23)

Quoth Paul:

seems like you have to make the wright flyer much better before it's important, and that it becomes more like an industry as that happens, and that this is intimately related to why so few people were working on it

Is this basically saying 'the Wright brothers didn't personally capture much value by inventing heavier-than-air flying machines, and this was foreseeable, which is why there wasn't a huge industry effort already underway to try to build such machines as fast as possible.' ?

My maybe-wrong model of Eliezer says here 'the Wright brothers knew a (Thielian) secret', while my maybe-wrong model of Paul instead says:

They didn't know a secret -- it was obvious to tons of people that you could do something sorta like what the Wright brothers did and thereby invent airplanes; the Wright brothers just had unusual non-monetary goals that made them passionate to do a thing most people didn't care about.
Or maybe it's better to say: they knew some specific secrets about physics/engineering, but only because other people correctly saw 'there are secrets to be found here, but they're stamp-collecting secrets of little economic value to me, so I won't bother to learn the secrets'. ~Everyone knows where the treasure is located, and ~everyone knows the treasure won't make you rich.

[Yudkowsky][17:24] (Sep. 23)

My model of Paul says there could be a secret, but only because the industry was tiny and the invention was nearly worthless directly.

[Cotra: ➕]

[Christiano][17:53] (Sep. 23)

I mean, I think they knew a bit of stuff, but it generally takes a lot of stuff to make something valuable, and the more people have been looking around in an area the more confident you can be that it's going to take a lot of stuff to do much better, and it starts to look like an extremely strong regularity for big industries like ML or se

...
Omicron Post #5
by Zvi — 2021-12-09T21:10:00.469Z

A few key questions determine our fate. We are rapidly getting closer to the answers. That doesn’t mean we have the magnitudes of the answers. The magnitudes matter, and they matter a lot. It’s still a stronger understanding than even a few days ago. Next time will be stronger still.

My guess is that that Omicron for now is on a bi-weekly schedule, and the next update will likely be on Monday, but it is subject to what happens.

So here are some things that, for all practical purposes, we know.

Does Omicron Have Substantial Immune Escape Properties?

Yes. Just yes. Wenowdis. The laboratory results are in. The results from who is testing positive are in. This isn’t a question anymore. 95% → 99%+.

Here’s Pfizer’s press release from December 8. They note that there’s a large decline in vaccine effectiveness, but also observe that those who are boosted seem to still have enough protection, and that two doses seems sufficient to provide at least some protection. They don’t want to come out and say yet that there’s protection against severe disease with two doses, but there’s plenty of other evidence on that.

Here’s a thread of two threads with the preliminary data. It’s clear that vaccine effectiveness is down, but not as far down as we feared, and definitely not to zero.

Here’s a thread from Muge Cevik describing the results. Again, not great, but not as bad as expected.

We are seeing large declines in antibody effectiveness, with numbers like 25x or 41x fold decline being thrown about, most commonly 41x.

The key is to remember that this is not a game of ‘exactly enough’ antibodies or antibody effectiveness getting it done and less not getting it done. This is a case of things varying by multiple orders of magnitude. It sounds scary to hear a 41x fold decrease in effectiveness – that’s 98% – but in terms of whether and how much you get sick, that’s a lot less than 98% less effective.

Also, Bloom Lab has come up with a way to estimate potential immune escape properties in advance, as a function of mutations. Neat.

Here’s another intriguing data point: Google trends data in South Africa doesn’t reflect an increase in searches for loss of taste and smell. I wonder if this will hold up.

Does Vaccination or Delta Infection Still Protect Against Severe Disease And Death from Omicron?

Yes. Just yes. Wenowdis. The laboratory results are in. The results from...
LessWrong discussed in New Ideas in Psychology article 
by rogersbacon — 2021-12-09T21:01:17.920Z

Hi LessWrong,

I wanted to let you know about a recent article in New Idea in Psychology written by myself and Dr. Dario Krpan (assistant professor of psychology at London School of Economics) that discusses LessWrong and Scott Alexander (Siskind) as examples of amateurs (or communities of amateurs) that make valuable contributions to psychology. In the article, we argue that psychology and behavioral science can benefit from increased participation in knowledge work (hypothesizing, experimentation, observational research, etc.) by amateurs. We highlight several “blind spots” in academic psychology (long-term projects, observational research, speculation, interdisciplinary research, taboo or uncommon subjects, and aimless projects) that amateurs might profitably focus on and discuss how we can support and facilitate amateurs to do research in psychology. One point of clarification is the term “amateur”; this is not meant as derogatory in anyway (and indeed our title makes light of the denigrating phrase “amateur hour”), but is used to refer to anyone that is not a professional psychology researcher. Scott, although he is practicing psychiatrist, would still qualify as an amateur in our definition because he does not belong to an institution that pays him to do psychology research. I would also mention that this paper might the first in which an author lists his substack as his institution (Secretum Secretorum).

Here is the official link to the paper: https://www.sciencedirect.com/science/article/abs/pii/S0732118X21000714

Here is the author copy which will be open access for the next 40 days or so:
https://authors.elsevier.com/c/1eAFI,GfNqbOWA

Below is the paragraph in which we discuss LessWrong.

Lastly, an example of a “quantified-self” amateur who has made valuable contributions in PBS is Alexey Guzey (2020), an independent researcher noted for conducting a self-experiment on the link between sleep and cognitive functioning. Guzey (2019) is also known for performing a rigorous fact-checking of the book Why We Sleep (Walker, 2017), which suggests that amateurs can also improve PBS by conducting thorough reviews of popular science books. Both reviews of popular science books and self-experimentation by amateurs can also be commonly found on LessWrong.com, a hub of the rationalist community; for example, one can find posts th

...