trentmkelly/LessWrong-43k huggingface dataset:
Feedbackloop-first Rationality
by Raemon — 2023-08-07T17:58:56.349Z

I've been workshopping a new rationality training paradigm. (By "rationality training paradigm", I mean an approach to learning/teaching the skill of "noticing what cognitive strategies are useful, and getting better at them.")

I think the paradigm has promise. I've beta-tested it for a couple weeks. It’s too early to tell if it actually works, but one of my primary goals is to figure out if it works relatively quickly, and give up if it isn’t not delivering.

The goal of this post is to:

Convey the framework
See if people find it compelling in its current form
Solicit ideas for improvements, before I decide whether to invest heavily into a larger experiment around it.

Rationality needs better feedback loops

Claim: Feedback loops are the most important thing ever. Hard things are hard because they have bad feedback loops. Some of the most important things (e.g. x-risk mitigation research) have the worst feedback loops.

Bold prediction: You can learn to think better, even about confusing, poor-feedback domains. This requires developing the art of inventing feedback loops. And then, actually putting in a lot of deliberate practice effort.

I've long been haunted by this Romeo Stevens comment (slightly paraphrased)[1]

Deliberate practice deliberate practice until you get really good identifying good feedback loops, and working with them.

People have a really hard time with interventions often because they literally do not have a functioning causal model of the skill in question. People who apply deliberate practice to a working causal model often level up astonishingly quickly. Don't know if you have the appropriate causal model? Well, when you apply deliberate practice do you not get better? You're pulling on fake levers.

In the past, I've tried to practice thinking. I've done explicit puzzle-solving exercises, and I have a day job that forces me to think about challenging questions on a regular basis. I sometimes have tried to refactor my day-job into something deliberate practice-shaped, but it never gelled.

I think I've gotten better at thinking in the past 12 years. But I haven't gotten overwhelmingly obviously better at thinking. I recently decided to deliberate practicing "solve confusing problems", until I was demonstrably better at it, and to host some workshops where I tried helping other people practice too.

I ended up settling into a pa...
Growing Bonsai Networks with RNNs
by ameo — 2023-08-07T17:34:15.713Z
Link: https://cprimozic.net/blog/growing-sparse-computational-graphs-with-rnns/

This is a linkpost for a writeup on my personal website: https://cprimozic.net/blog/growing-sparse-computational-graphs-with-rnns/

Here's a summary:

This post contains an overview of my research and experiments on growing sparse computational graphs I'm calling "Bonsai Networks" by training small RNNs. It describes the architecture, training process, and pruning methods used to create the graphs and then examines some of the learned solutions to a variety of objectives.

Its main theme is mechanistic interpretability, but it also goes into significant detail on the technical side of the implementation for the training stack, a custom activation function, bespoke sparsity-promoting regularizer, and more.

The site contains a variety of interactive visualizations and other embeds that are important to its content. That's why I chose to make this a linkpost rather than copy its content here directly.

I'd love to receive any feedback you might have on this work. This topic is something I'm very interested in, and I'm eager to hear peoples' thoughts on it.
Should I test myself for microplastics?
by Augs — 2023-08-07T17:31:41.656Z

I'm curious about the levels of microplastics and pfas in my body, and the only company i can find (million marker) claims to have $300 urine tests to give you those numbers. I'm usually skeptical of such companies, but would be willing to pay if the science or reputation of this place actually seemed worth the money. Does anyone have an analysis of the founders' previous experience, the company methods, or other technical details that would allow me to make an informed decision on if I should get this test, or other companies I can look at?
Optimisation Measures: Desiderata, Impossibility, Proposals
by mattmacdermott — 2023-08-07T15:52:17.624Z

Previously: Towards Measures of Optimisation

When thinking about optimisation processes it is seductive to think in information-theoretic terms.

Is there some useful measure[1] of 'optimisation' we can derive from utility functions or preference orderings, just as Shannon derived 'information' from probability distributions? Could there be a 'mathematical theory of optimisation' that is analogous to Shannon's theory of information? In this post we exhibit negative evidence that this point of view is a fertile direction of inquiry.

In the last post we reviewed proposals in that direction, most notably Yudkowsky's original idea using preference orderings, and suggested some informal desiderata. In this post we state our desiderata formally, and show that they can't all be satisfied at once. We exhibit a new proposal from Scott Garrabrant which relaxes one desideratum, and revisit the previous proposals to see which desiderata they satisfy.

Setup

Recall our setup: we're choosing an action from a set A to achieve an outcome in a set Ω. For simplicity, we assume that Ω is finite. Denote the set of probability distributions on Ωby ΔΩ. We have a default distribution p∈ΔΩ, which describes the state of affairs before we optimise, or in a counterfactual world where we don't optimise, and action distributions pa∈ΔΩ for each a∈A, which describe the state of affairs if we do. Our preferences are described by a utility function u:Ω→R. Let U denote the set of utility functions.

In the previous post we considered random variables OP(p,u)(x), which measure the optimisation entailed by achieving some outcome x, given a utility function u and base distribution p. We then took an expectation over pa to measure the optimisation entailed by achieving some distribution over outcomes, i.e. we defined OP(p,pa,u)=EpaOP(p,u)(x).

In this post we state our desiderata directly over OP(p,pa,u) instead. For more on this point see the discussion of the convex-linearity desideratum below.

Desiderata

Here are the desiderata we originally came up with for OP:ΔΩ×ΔΩ×U→R∪{−∞,∞}. They should hold for all p,pa,pb∈ΔΩ and for all u∈U. Explanations below.

(Continuity)
OP is continuous[2] in all its arguments.
(Invaria

...
Announcing the Clearer Thinking micro-grants program for 2023
by spencerg — 2023-08-07T15:21:28.191Z
Link: https://www.clearerthinking.org/post/submit-an-idea-for-a-clearer-thinking-tool-and-get-paid-up-to-595 

This is a link-post for the announcement on our website.

Do you have an idea for an educational module or tool related to rationality, effective altruism, or psychology that could benefit the general public? If you do, we'd love to hear from you, because we're running our Clearer Thinking micro-grants program again! You can earn up to $595 for proposing, designing, and/or building (with our help) a short online tool aligned with our mission. To apply, just submit your idea for a beneficial learning tool in under 400 words.
Here is the link to apply.
What I've been reading, July–August 2023
by jasoncrawford — 2023-08-07T14:22:57.046Z
Link: https://rootsofprogress.org/reading-2023-08

A quasi-monthly feature (I skipped it last month, so this is a double portion).

This is a longish post covering many topics; feel free to skim and skip around. Recent blog posts and news stories are generally omitted; you can find them in my links digests.

These updates are less focused than my typical essays; do you find them valuable? Email me or comment (below) with feedback.

Books (mostly)

Books I finished

Thomas Ashton, The Industrial Revolution, 1760-1830 (1948). A classic in the field. I wrote up my highlights here.

Samuel Butler, Erewhon (1872). It is best known for its warning that machines will out-evolve humans, but rather than dystopian sci-fi, it’s actually political satire. His commentary on the universities is amazingly not dated at all, here’s a taste:

When I talked about originality and genius to some gentlemen whom I met at a supper party given by Mr. Thims in my honour, and said that original thought ought to be encouraged, I had to eat my words at once. Their view evidently was that genius was like offences—needs must that it come, but woe unto that man through whom it comes. A man’s business, they hold, is to think as his neighbours do, for Heaven help him if he thinks good what they count bad. And really it is hard to see how the Erewhonian theory differs from our own, for the word “idiot” only means a person who forms his opinions for himself.

The venerable Professor of Worldly Wisdom, a man verging on eighty but still hale, spoke to me very seriously on this subject in consequence of the few words that I had imprudently let fall in defence of genius. He was one of those who carried most weight in the university, and had the reputation of having done more perhaps than any other living man to suppress any kind of originality.

“It is not our business,” he said, “to help students to think for themselves. Surely this is the very last thing which one who wishes them well should encourage them to do. Our duty is to ensure that they shall think as we do, or at any rate, as we hold it expedient to say we do.” In some respects, however, he was thought to hold somewhat radical opinions, for he was President of the Society for the Suppression of Useless Knowledge, and for the Completer Obliteration of the Past.

It’s unclear to me whether the more well-known part about machines evolving is a commentary on technology, or on Darwinism (which was quite new at ...
Monthly Roundup #9: August 2023
by Zvi — 2023-08-07T13:20:03.522Z

What a month. So much to cover.

What this post does not cover beyond this introduction is the biggest news story of the month, a potential room temperature superconductor.

If this discovery is real, it could be transformative. Think of the potential. We could be so back. Chances are that instead it is over, but even a small chance of something this big is huge. Even if it ends up being all over, it was amazing to see people around the world come together and do science to this to try and actually figure out something physical and real. Even in failure, we are so back.

What I am not going to do is pivot to suddenly becoming an expert on physics or get into the weeds myself. There is no need for that, I work fast and it would be fun but there are limits and I simply haven’t the time. We have a whole world stepping up to replicate the results and soon we will know the answer.

In the meantime, it is easy to know the probability of the answer. All you have to do is check Polymarket, for the largest real money prediction market. Size talks.

Polymarket does not allow Americans and requires crypto. So if you are in American or want to get involved via charity funds instead, you can bet much smaller amounts at Manifold Markets instead. They also have an endless running discussion, whose quality I am choosing not to be qualified to judge.

You can also enjoy my movie retrospective, Barbiehammer: Across the Dead Reckoning. It has been quite a good year for movies.

So what else has been happening?

Bad News

How bad are car break-ins in San Francisco and Oakland? A CNN reporter shooting a story about crime gets her rental cars broken into so often she is careful not to even leave candy bars behind. Then we get the hard data, which is that of 250 cars returned the previous day, 27 had been broken into. A reasonable guess is that the average rental period is 5 days. So that means that there is a 2% car break-in rate per day.

Portugal is struggling with drug legalization, Washington Post reports, as rates of use go up and the systems for dealing with drug use increasingly aren’t working. Whenever full legalization happens, this effectively means that drug users are violating a whole host of regulations and rules and degrading general quality of life, in ways that would never be tolerated from other ordinary citizens or ordinary business activities, but no one ha...
Strengthening the Argument for Intrinsic AI Safety: The S-Curves Perspective 

by avturchin — 2023-08-07T13:13:42.635Z

Epistemic Status: I'm attempting to present the strongest version (steelmaning) of a viewpoint that isn't my own, and then offer valid criticisms of it.

TL;DR: Existential risks (X-risks) arise from exponential thinking. However, if we assume that most growth follows a logistic (sigmoid) curve, then the power and effectiveness of AI will eventually see diminishing returns. This would result in a multitude of AIs with roughly equivalent capabilities. These AIs would then need to cooperate with each other and the remaining human population. Such cooperation would likely involve preserving humanity and a significant portion of human values. The primary challenge to this perspective is the idea of "series of S-curves" and the Moloch problem, which could ultimately lead to a single dominant AI (a Singleton) or dynamics that are unfavorable to humans.

Existential risks arise from exponents, but what if sigmoids are dominating?

The concept of existential risks (x-risks) is fundamentally rooted in the notion of unbounded exponential growth. This concept is applicable to various phenomena such as self-improving artificial intelligence (AI), pandemics, chain reactions, strangelet and black hole expansion. A generic x-risk is a process that grows exponentially, and theoretically without limit, until it engulfs the entire Earth within a finite timeframe. The specific process may vary, ranging from grey goo scenarios to self-replicating alien messages, but the underlying dynamics remain consistent.

However, certain individuals, including an AI researcher I am acquainted with (though they did not request this write-up and I am unsure if they currently endorse this line of thinking as I present it here), propose that logistic S-curves are the predominant pattern in the evolutionary dynamics of complex systems. While I do not personally endorse this argument, I will endeavor to convey its core principles as accurately as possible. In a logistic S-curve, a phase of near-exponential growth is succeeded by a period of diminishing returns and exponential deceleration, typically when a critical resource becomes depleted. An illustrative example of this phenomenon is the growth of bacteria in a Petri dish.

According to this perspective, all processes we currently perceive as exponential growth curves will eventually decelerate. Consequently, no process will ever reach its theoretical ...
Overview of how AI might exacerbate long-running catastrophic risks
by Hauke Hillebrandt — 2023-08-07T11:53:29.171Z
Link: https://aisafetyfundamentals.com/governance-blog/overview-of-ai-risk-exacerbation

Audio reading

Rationalist AI risk public outreach has often emphasized first principles thinking, theory, and logical possibilities (e.g. evolution, gradient decent, human-chimp analogy, etc.) over more concrete tangible empirical findings (e.g. deception emerging in small models, specification gaming, LLMs helping to create WMDs, etc.). I found the recent post below from the AGI governance fundamental provides a great summary.

Developments in AI could exacerbate long-running catastrophic risks, including bioterrorism, disinformation and resulting institutional dysfunction, misuse of concentrated power, nuclear and conventional war, other coordination failures, and unknown risks. This document compiles research on how AI might raise these risks. (Other material in this course discusses more novel risks from AI.) We draw heavily from previous overviews by academics, particularly

Dafoe (2020)

and

Hendrycks et al. (2023)

.

Bioterrorism

AI advances could worsen the risks of bioterrorism.

Hendrycks et al. (2023)

provide a useful introduction to this risk. The following is a large excerpt from their paper.

The rapid advancement of AI technology increases the risk of bioterrorism. AIs with knowledge of bioengineering could facilitate the creation of novel bioweapons and lower barriers to obtaining such agents. Engineered pandemics from AI-assisted bioweapons pose a unique challenge, as attackers have an advantage over defenders and could constitute an existential threat to humanity. [...]

Bioengineered pandemics present a new threat. Biological agents, including viruses and bacteria, have caused some of the most devastating catastrophes in history. It’s believed the Black Death killed more humans than any other event in history, an astounding and awful 200 million, the equivalent to four billion deaths today. [...] [E]ngineered pandemics could be designed to be more lethal or easily transmissible than natural pandemics, presenting a new threat that could equal or even surpass the devastation wrought by history’s most deadly plagues [6].

Humanity has a long and dark history of weaponizing pathogens [...] During the twentieth century, 15 countries are known to have developed bioweapons programs, including the US, USSR, UK, and France. Like chemical weapons, bioweapons have become a taboo among the international community. While some state actors continue to operate bio...
Problems with Robin Hanson's Quillette Article On AI
by DaemonicSigil — 2023-08-06T22:13:43.654Z

Original article here: https://quillette.com/2023/08/06/ais-will-be-our-mind-children

1. Hanson Strawmans the AI-Ruin Argument

Hanson writes:

AI-doomers often suggest that their fears arise from special technical calculations. But in fact, their main argument is just the mere logical possibility of a huge sudden AI breakthrough, combined with a suddenly murderous AI inclination.

Either this is a deliberate misrepresentation, or Hason simply hasn't done his homework. The argument is not that AI will suddenly decide that killing people is good for no particular reason. Rather it is that from the start, the AI will not share values with humans, simply because we don't know how to build an AI that does. So it will have its own ideas about how the universe should look, and would thus want to seize power from us if it could, so that it could enact its own vision of an ideal universe, rather than ours.

Similarly, a sudden large technical breakthrough is not required for us to observe an AI suddenly turning on us. Rather the situation is akin to a first order phase transition. At low levels of AI capability, it has no hope of taking over the world, and the best way to achieve its goals is to work together with us. Above some threshold of capability, it's better for an AI's goals to try and defeat humanity rather than work with us. This is true no matter if that AI is incrementally stronger than the previous one, or much stronger. (Though larger leaps have a higher chance of happening to be the ones that cross the threshold exactly because they are larger.)

Now these arguments certainly aren't technical calculations, but neither are they mere arguments from logical possibility. We've repeatedly seen the difficulty that practitioners have with getting neural networks to do what they want. The way Bing Syndney acted out when first put online was certainly amusing, and even cute, but we can hardly say it was what Microsoft wanted it to do. Similarly, we're currently having a hard time getting language models not to make stuff up, even though when they do this, they tend to give probability distributions over tokens that reflect the fact that they're uncertain. And this is just in the area of language models, reinforcement learning is even more of a tough case for alignment.

As one more example of how Hanson has badly misunderstood the AI-Ruin argument, consider:

However

...
Yann LeCun on AGI and AI Safety
by Chris_Leong — 2023-08-06T21:56:52.644Z
Link: https://drive.google.com/file/d/1wzHohvoSgKGZvzOWqZybjm4M4veKR6t3/view

Yann recently gave a presentation at MIT on Objective-Driven AI with his specific proposal being based upon a Joint Embedding Predictive Architecture.

He claims that his proposal will make AI safe and steerable, so I thought it was worthwhile copying the slides at the end which provide a very quick and accessible overview of his perspective:

Here's a link to the talk itself.

I find it interesting how he says that there is no such thing as AGI, but acknowledges that machines will "eventually surpass human intelligence in all domains where humans are intelligent" as that would meet most people's definition of AGI.

I also observe that he has framed his responses to safety on "How to solve the alignment problem?". I think this is important. It suggests that even people who think aligning AGI will be easy have started to think a bit more about this problem and I see this as a victory in and of itself.

You may also find it interesting to read Steven Byrnes' skeptical comments on this proposal.
Computational Thread Art
by CallumMcDougall — 2023-08-06T21:42:30.306Z

This post describes the iterative process I went through while creating the thread art which you can see featured on my website. This is also crossposted to my personal blog.

Black & White Algorithm

The basic version of the algorithm is pretty straightforward. The image is rescaled so that white equals zero, and black equals one. A bunch of random lines are generated, and the line which goes through the darkest pixels on average (i.e. the highest average value per pixel) is chosen. The value of every pixel along this line is decreased slightly (i.e. the image is made lighter) and the process is repeated, with each new batch of random lines constrained to start where the previous one finishes. Because each line only changes the brightness by a small amount along a very short width, this process gradually builds up gradients, and after a few thousand lines the full image emerges.

Illustration of how lines are added

The very first image I ever made was of my mum, for her 50th birthday, and it turned out surprisingly well given how basic the algorithm was at that point.

First full image produced by algorithm

Computational efficiency

Initially, each piece would take about 6 hours to run, because I had no understand of things like computational efficiency. About a month after making my first pieces, I realised that the numpy sum function worked much faster than the built-in Python method, and I could store coordinates in a dictionary rather than recomputing them on the fly, which reduced the time for each piece from 6 hours to just under 10 seconds (I wish I was joking).

Algorithm improvements

There were a few tweaks to this algorithm which made it work slightly better. For example:

Darkness penalty - rather than just drawing the lines which went through the darkest pixels on average, I introduced a penalty for drawing too many lines through an area.
Importance weighting - the penalty on each pixel was scaled by some value between zero and one. This allowed me to improve accuracy in some areas (e.g. facial features) at the expense of less important areas (e.g. the background). I also made this different depending on whether the value of the pixel was positive or negative - this allowed me to specify more complex behaviours like "don't draw any lines through the whites of the eyes".

When these tweaks were all combined, each line was minimizing the penalty function:

Pe...