trentmkelly/LessWrong-43k huggingface dataset:
I learn better when I frame learning as Vengeance for losses incurred through ignorance, and you might too
by chaosmage — 2022-10-15T12:41:52.504Z

THE OBSERVATION
My main point is in the title: I have found that when I consciously learn "with a vengeance", aiming to avenge whatever I lost because I have not learnt the thing earlier, I learn markedly better. I feel more motivated to learn, and recall seems clearly better. (I have not formally compared recall before and after, this is subjective judgement but I'm very confident.)

I have experimented with this for about 2 weeks. So far the effect does not seem to diminish; if there is any change at all, it is maybe getting slightly stronger.

Subjectively, this changes my failures from something to mourn or to be embarrassed about, into "something to be avenged" which is a driving motivation. It feels like I'm owning my mistakes more readily. I acknowledge them more readily and am more comfortable thinking and talking about them. And the learning from them feels something like gleeful, justified or gloating - definitely more powerful and satisfying.

I would like some of you to try this, and to report back whether this works for you too, because I have never heard anyone else talk about consciously trying to do this, so it might be new, and it helps me a lot so if it helps others too, it seems potentially extremely useful.

THE NARRATIVE
I was having a great argument about God with a very good friend who is a sophisticated theologian, and who pointed me to a podcast with the German theologian Siegfried Zimmer. Zimmer was talking about theodicy, the classic problem of theism where an almighty, all-knowing, all-good God seems impossible to square with the pointless suffering we observe. He did not have a satisfactory answer, he did manage to admit that, he did not manage to draw the obvious conclusion, so nothing new there. But in his argument he gave a really interesting reason to discard all the usual theistic answers. He said an answer to suffering is only good if you can say it to someone who is intensely and pointlessly suffering, to his or her face, and find it helps.

I was very impressed with this idea and concluded that as an atheist, I did not have an answer that would fulfil this criterion. So I thought about it. What do I say to someone dying in a concentration camp, from the other side of the fence, utterly unable to save them? To a kid dying of cancer? To the grieving parents of victims of a school shooting?

Naturally, as you do, I thought back to the S...
Quick Mock Brownie
by jefftk — 2022-10-15T03:00:01.858Z

We make a lot of quick desserts that look like "melt

chocolate and fat". Often
this is chocolate and peanut butter ("chocolate peanut butter sauce",
Lily's current favorite breakfast, on bread) or chocolate and butter
and/or cream ("chocolate sauce", really ganache). Lately I've been
making a variant that I've been calling "mock brownie" because it
tastes something like the gooey inside of not-quite-cooked brownie
that's just out of the oven.

I start with chocolate chips (Barry
Callebaut semi-sweet that we buy in bulk) and melt them with
cream and/or butter:

It takes about 25s in the microwave. Then I stir it smooth:

It's good like this, though a but thin and a little sweeter than I like.
So I add cocoa powder:

And stir it in:

I'm very happy with it. It's best warm, though also different and
interesting after it's cooled.

Comment via: facebook
A common failure for foxes
by Rob Bensinger — 2022-10-14T22:50:59.614Z

A common failure mode for people who pride themselves in being foxes (as opposed to hedgehogs):

Paying more attention to easily-evaluated claims that don't matter much, at the expense of hard-to-evaluate claims that matter a lot.

E.g., maybe there's an RCT that isn't very relevant, but is pretty easily interpreted and is conclusive evidence for some claim. At the same time, maybe there's an informal argument that matters a lot more, but it takes some work to know how much to update on it, and it probably won't be iron-clad evidence regardless.

I think people who think of themselves as being "foxes" often spend too much time thinking about the RCT and not enough time thinking about the informal argument, for a few reasons:

1. A desire for cognitive closure, confidence, and a feeling of "knowing things" — of having authoritative Facts on hand rather than mere Opinions.

A proper Bayesian cares about VOI, and assigns probabilities rather than having separate mental buckets for Facts vs. Opinions. If activity A updates you from 50% to 95% confidence in hypothesis H1, and activity B updates you from 50% to 60% confidence in hypothesis H2, then your assessment of whether to do more A-like activities or more B-like activities going forward should normally depend a lot on how useful it is to know about H1 versus H2.

But real-world humans (even if they think of themselves as aspiring Bayesians) are often uncomfortable with uncertainty. We prefer sharp thresholds, capital-k Knowledge, and a feeling of having solid ground to rest on.

2. Hyperbolic discounting of intellectual progress.

With unambiguous data, you get a fast sense of progress. With fuzzy arguments, you might end up confident after thinking about it a while, or after reading another nine arguments; but it's a long process, with uncertain rewards.

3. Social modesty and a desire to look un-arrogant.

It can feel socially low-risk and pleasantly virtuous to be able to say "Oh, I'm not claiming to have good judgment or to be great at reasoning or anything; I'm just deferring to the obvious clear-cut data, and outside of that, I'm totally uncertain."

Collecting isolated facts increases the pool of authoritative claims you can make, while protecting you from having to stick your neck out and have an Opinion on something that will be harder to convince others of, or one that rests on an implicit claim about ...
"AGI soon, but Narrow works Better"
by AnthonyRepetto — 2022-10-14T21:35:37.815Z

This is my offering to the Future Fund's "A.I. Worldview Prize" - and I stand as an outsider who is unlikely to move the needle; I hope this angle of approach is noticed there, at least.

Our Last Limitations?

The AGI with feelings and motives is irrelevant to my concerns; I worry about a machine that can add to its skill-set without bounds. The key functional constraints on this are 'catastrophic forgetting' and the difficulty extracting causal relationships & formulae from observations. Once those twin pillars fall, a single AGI can successively add-to its competencies, while finding symbolic parallels which further compress and unify its understanding. Even such a 'Vulcan' AGI could be a mis-aligned monster.

I don't see those 'forgetting & formulating' problems taking very long to solve; my personal timeline is 5-10yrs. with greater than half likelihood (bad stuff can happen, but this industry is not about to winter). After those hurdles, the issue of 'getting to brain-scale' will be found irrelevant - because human brains are bloatware. To explain:

We worry that, because each neuron in our own brains has complexity that takes ~200 artificial neurons to simulate, and because we have 100 trillion synapses or so, then 'getting to human brain performance' will also require getting to some 20 million billion weights and biases. Yet, all of the evidence so far is a gauntlet challenging this concern. Consider that language models are able to out-perform the average human (we are an AGI) while containing some 1,000 times fewer connections than us. Even driverless vehicles have a better safety-record than the average human, and last I checked, those were not running an AGI with trillions of parameters.

Realistically, we should expect the human brain to be hyper-redundant - look at our plasticity! And the rate of loss, damage, over the course of our reality-bonked lives! And, our brains learn from startlingly few examples, which I suspect is related to its immense over-parameterization. (This has an impact on AGI policy, too - We won't want AGI zero-shotting situations that we ourselves haven't grasped yet; "over-parameterizing an AGI with sparse training data to zero-shot the unexpected" is the definition of insane risk!)

So, near-term compute capacity will be plenty to run super-human AGI... and it'll be crap.

The Narrow Revolution

Consider that an AGI will need ...
[Job]: AI Standards Development Research Assistant
by Tony Barrett — 2022-10-14T20:27:00.907Z

Apply here.

Title: Research Assistant for AI Standards Development
Ideal start date: December 2022
Hours: 20-40 hours/week
Compensation: $30/hour to $50/hour, depending on experience and qualifications
Work location: Remote
Reports to: Tony Barrett, BERI Senior Policy Analyst

For best consideration, please apply by Monday November 7th, 2022, 5pm Eastern Time. Applications received after that date may also be considered, but only after applications that met the deadline.

Responsibilities

Supporting work planned by Tony Barrett and UC Berkeley colleagues to develop an AI-standards “profile” with best practices for developers of cutting-edge, increasingly general purpose AI, building on the ideas in Section 4 of the paper by Barrett and colleagues, “Actionable Guidance for High-Consequence AI Risk Management: Towards Standards Addressing AI Catastrophic Risks”. The profile guidance will be primarily for use by developers of such AI systems, in conjunction with the NIST AI Risk Management Framework (AI RMF) and/or the AI risk management standard ISO/IEC 23894. Our goal is to help set norms for safety-related practices across regulatory regimes, reducing chances that developers of highly advanced AI systems (including proto-AGI) would have to compromise on safety, security, ethics or related qualities of AI systems in order to be competitive.

Tasks will include research and analysis of technical or policy issues in AI safety standards or related topics. The goal is to help our team to address key AI technical issues with actionable guidance for AI developers, in ways that improve the overall quality of our profile guidance documents.

Technical research assistance tasks may include:

Literature searches on technical methods for safety or security of machine learning models
Gap analysis to check that our draft guidance would address key technical issues in AI safety, security or other areas

Policy research assistance tasks may include:

Identifying and analyzing related standards or regulations
Mapping specific sections of our draft guidance to specific parts of related standards or regulations
Checking that our draft guidance would meet the intent and requirements of related standards or regulations

We currently have funding for approximately one year of work, but we have potential to obtain additional funding to renew or expand this work.

Qualification Criteria

The most competitive ca...
Metaculus Launches the 'Forecasting Our World In Data' Project to Probe the Long-Term Future
by ChristianWilliams — 2022-10-14T17:00:31.311Z
Link: https://www.metaculus.com/tournament/forecasting-Our-World-in-Data/

With support from the FTX Future Fund, the project features a public forecasting tournament and a Pro Forecaster initiative delivering 1, 3, 10, 30, and 100-year predictions

Metaculus has launched Forecasting Our World In Data, a tournament that will deliver predictions on technological advancement, global development, and social progress on time horizons ranging from one to 100 years, with close collaboration from the Our World In Data (OWID) team.

OWID is one of the world's largest open online repositories of data and is a valuable source of information about both humanity's progress and challenges. Metaculus will collect forecasts and analyses on thirty OWID metrics — including CO2 emissions, supercomputer performance, nuclear stockpiles, and more — to generate likelihoods of specific outcomes in humanity's future. Aggregating crowd forecasts and weighting them by forecasters' past performance leads to greater accuracy than relying on a single model or subject matter expert alone.

The Forecasting Our World In Data tournament will award a prize pool of $20,000 to incentivize accurate forecasting and insightful written comments by forecasters. For each OWID metric, participants are invited to predict outcomes at 1, 3, 10, 30, and 100 years. Prizes will be awarded for accurate performance on 1 and 3-year outcomes and cogent analysis on 10, 30, and 100-year outcomes. Questions are staggered, with 10 questions opening today and 10 additional questions opening October 19th and 26th. Metaculus Pro Forecasters will predict each metric alongside the crowd.

A grant from the Future Fund makes this project possible, and is greatly appreciated. Our thanks also to the forecasting community for contributing to a greater understanding of humanity's near- and long-term future. The first set of questions is now open for public forecasting here.
Instrumental convergence: scale and physical interactions
by Edouard Harris — 2022-10-14T15:50:29.074Z
Link: https://www.gladstone.ai/instrumental-convergence-3

Summary of this post

This is the third post in a three-part sequence on instrumental convergence in multi-agent RL. Read Part 1 and Part 2.

In this post, we’ll:

Investigate instrumental convergence on a multi-agent gridworld with a complicated topology.
Show that when we add a simple physical interaction between our agents — in which we forbid them from overlapping on the gridworld — we induce stronger instrumental alignment between short-sighted agents, and stronger instrumental misalignment between far-sighted agents.

We’ll soon be open-sourcing the codebase we used to do these experiments. If you’d like to be notified when it’s released, email Edouard at

edouard@gladstone.ai

or DM me on Twitter at

@harris_edouard

.

Thanks to Alex Turner and Vladimir Mikulik for pointers and advice, and for reviewing drafts of this sequence. Thanks to Simon Suo for his invaluable suggestions, advice, and support with the codebase, concepts, and manuscript. And thanks to David Xu, whose comment inspired this work.

Work was done while at Gladstone AI, which Edouard is a co-founder of.

🎧 This research has been featured on an episode of the Towards Data Science podcast. Listen to the episode here.

1. Introduction

In Part 1 of this sequence, we saw how an agent with a long planning horizon tends to perceive instrumental value as being more concentrated than an agent with a shorter planning horizon. And in Part 2, we introduced a multi-agent setting with two agents — Agent H (standing for a human) and Agent A (standing for a powerful AI) — which we used to motivate a definition of multi-agent instrumental value, or POWER. We looked at how this definition behaved on a simple 3x3 gridworld, and found that when our agents had independent terminal goals,[1] their instrumental values ended up misaligned by default.

In this post, we’ll combine these two ideas and scale up our multi-agent experiments to a bigger and more complicated gridworld. Throughout this post, we’ll focus exclusively on the regime in which our agents have independent terminal goals. We'll see whether we can reproduce instrumental misalignment-by-default in this regime, and then we'll investigate which factors seem strengthen or weaken the instrumental alignment between our agents.

2. Multi-agent POWER: recap

If you’ve just read Part 2 of this sequence, ...
Counterarguments to the basic AI x-risk case
by KatjaGrace — 2022-10-14T13:00:05.903Z

(Crossposted from AI Impacts Blog)

This is going to be a list of holes I see in the basic argument for existential risk from superhuman AI systems1.

To start, here’s an outline of what I take to be the basic case2:

I. If superhuman AI systems are built, any given system is likely to be ‘goal-directed’

Reasons to expect this:

Goal-directed behavior is likely to be valuable, e.g. economically.
Goal-directed entities may tend to arise from machine learning training processes not intending to create them (at least via the methods that are likely to be used).
‘Coherence arguments’ may imply that systems with some goal-directedness will become more strongly goal-directed over time.

II. If goal-directed superhuman AI systems are built, their desired outcomes will probably be about as bad as an empty universe by human lights

Reasons to expect this:

Finding useful goals that aren’t extinction-level bad appears to be hard: we don’t have a way to usefully point at human goals, and divergences from human goals seem likely to produce goals that are in intense conflict with human goals, due to a) most goals producing convergent incentives for controlling everything, and b) value being ‘fragile’, such that an entity with ‘similar’ values will generally create a future of virtually no value.
Finding goals that are extinction-level bad and temporarily useful appears to be easy: for example, advanced AI with the sole objective ‘maximize company revenue’ might profit said company for a time before gathering the influence and wherewithal to pursue the goal in ways that blatantly harm society.
Even if humanity found acceptable goals, giving a powerful AI system any specific goals appears to be hard. We don’t know of any procedure to do it, and we have theoretical reasons to expect that AI systems produced through machine learning training will generally end up with goals other than those they were trained according to. Randomly aberrant goals resulting are probably extinction-level bad for reasons described in II.1 above.

III. If most goal-directed superhuman AI systems have bad goals, the future will very likely be bad

That is, a set of ill-motivated goal-directed superhuman AI systems, of a scale likely to occur, would be capable of taking control over the future from humans. This is supported by at least one of the following being true:

Superhuman AI would destroy humanity

...
Another problem with AI confinement: ordinary CPUs can work as radio transmitters
by RomanS — 2022-10-14T08:28:48.554Z
Link: https://news.softpedia.com/news/emitting-radio-waves-from-a-computer-with-no-radio-transmitting-hardware-501260.shtml

Excerpts:

Computer tech expert William Entriken has created a C library that can make a computer emit radio waves even if the device doesn't include any radio transmission hardware.

His program, called System Bus Radio, is written in C and relies on code that makes the computer's CPU emit electromagnetic radiation at certain frequencies.

The radiation is strong enough to escape the computer's shielding, and with the proper timing at which data is processed in the CPU, it can be forged into radio waves that carry off information away from the target's computer.

During tests, Mr. Entriken says he managed to transmit a version of the "Mary Had a Little Lamb" song over a distance of up to two meters and one meter if a drywall separated the computer and the receiving antenna.

You can get System Bus Radio from GitHub, test it on your own hardware, and add your results to Mr. Entriken's findings.

Worth reminding that this escape method was invented by a human-level intelligence. A Bayesian superintelligence may be able to devise much more cunning methods.
How much of China's Zero COVID policy is actually about COVID?
by jmh — 2022-10-14T07:23:20.961Z

I was wondering the other day about this question but have no good sense myself. Moreover, I'm not even sure just what good metrics might be applied to attempt to assess the question.

Earlier today I was thinking of asking but decided against. However, I just came across a new story that seems to raise the question from those in China. Here is the relevant quote related to a tracking wrist band one woman living in Beijing found on her doorstep:

I have accepted lockdowns, forced COVID-19 tests and health codes, but this thing feels like surveillance just for the sake of surveillance.

While the story doesn't clearly say it was provided by some national or provincial health department it makes no suggestion that some criminal or scamming (non-governmental at least) actor might have been involved.

Adding to the recent "flash" protest against Xi in general but including the policy I thought I might query here.

Often China is held up as something of a poster child for the state capacity story. That might imply the policy is now more about social control than actually managing COVID. But it could also just be a case of unintended consequences, at least in a perceptual form, rather than the intended goal now. I could certainly see an argument made for having locked itself into such a policy as China's health system and population are probably poorly suited to living with COVID that the rest of the world has started moving to specifically due to its earlier success in containment (even accounting for gross under reporting). Certainly one might think relaxing the policy and then seeing a major outbreak and overloaded hospitals would be the last thing Xi needs going into his selection process.

Any have thoughts on just how to tease out the likelihood or policy as advertised from health policy used for social control purposes given it seems quite possible that the two purposes could be pursed under the same policy simultaneously.
The North Oakland LessWrong Meetup
by Czynski — 2022-10-14T03:33:48.163Z

The mission of the North Oakland LessWrong meetup is to provide a fun, low-key social space with some structured interaction, where new and non-new community members can mingle and have interesting conversations. Everyone is welcome. Yes, that includes you, if you're reading this.
Every week, we have a topic, to provide some structure. But we explicitly encourage people to split off from the main conversation or diverge from the topic if that would be more fun for them (moving side conversations into a separate part of the space if appropriate). Meetup topics are here as a tool to facilitate fun interaction, so we certainly don’t want them to inhibit it.

WHEN: Every Wednesday at 6:00, but with the topic waiting until 6:30 or 6:45 to accommodate stragglers.

WHERE: 540 Alcatraz Ave., Oakland; try code zero-five-eight, and if it doesn't work text 978 460 3399 and someone will come down to let you in. (We're in the far back, two floors up. The door is marked.)

Recurring Topics

Projects: Bring something, usually a personal project; we'll do shared pomodoros and socializing. The first Wednesday of every month is a Projects meetup.

Board Game Night: Use my collection or your own games, let's play and chat.

Reading & Discussion: At the meetup, we pick a blog post or sometimes paper to read, then discuss it; often we'll repeat that with a second thing as well.

Group Debugging: Take five minutes to brainstorm your problems, then share a few with the group and ask for help.

Shallow Questions: Answer trivial questions about each other in excessive detail in pairs, then switch it up with other people.

Deep Questions: Answer broad, big-picture questions in pairs, and discuss, then switch it up with other people.

Short Talks: 7 minutes on whatever you're thinking about, frequently inspired by something the last talk said; no prep required.

...and others. Suggest more!

(Please do not put the door code or the unit number anywhere publicly visible. I get enough junk mail as it is, and have had some minor harassment that would have used my mail and physical location if they could find it.)