trentmkelly/LessWrong-43k huggingface dataset:
Review of Examine.com’s vitamin write-ups
by Elizabeth — 2022-09-26T23:40:06.344Z
Link: https://acesounderglass.com/2022/09/26/review-of-examine-coms-vitamin-write-ups/

There are a lot of vitamins and other supplements in the world, way more than I have time to investigate. Examine.com has a pretty good reputation for its reports on vitamins and supplements. It would be extremely convenient for me if this reputation was merited. So I asked Martin Bernstoff to spot check some of their reports.

We originally wanted a fairly thorough review of multiple Examine write-ups. Alas, Martin felt the press of grad school after two shallow reviews and had to step back. This is still enough to be useful so we wanted to share, but please keep in mind its limitations. And if you feel motivated to contribute checks of more articles, please reach out to me (elizabeth@acesounderglass.com).

My (Elizabeth’s) tentative conclusion is that it would take tens of hours to beat an Examine general write-up, but they are not complete in either their list of topics nor their investigation into individual topics. If a particular effect is important to you, you will still need to do your own research.

Photo credit DALL-E

Write-Ups

Vitamin B12

Claim: “The actual rate of deficiency [of B12] is quite variable and it isn’t fully known what it is, but elderly persons (above 65), vegetarians, or those with digestion or intestinal complications are almost always at a higher risk than otherwise healthy and omnivorous youth”

Verdict: True but not well cited. Their citation merely asserts that these groups have shortages rather than providing measurements, but Martin found a meta-analysis making the same claim for vegetarians (the only group he looked for).

Toxicology

Verdict: Very brief. Couldn’t find much on my own. Seems reasonable.

Claim: “Vitamin B12 can be measured in the blood by serum B12 concentrations, which is reproducible and reliable but may not accurately reflect bodily vitamin B12 stores (as low B12 concentrations in plasma or vitamin B12 deficiencies do not always coexist in a reliable manner[19][26][27]) with a predictive value being reported to be as low as 22%”

Verdict: True, the positive predictive value was 22%, but with a negative predictive value of 100% at the chosen threshold. But that’s only the numbers at one threshold. To know whether this is good or bad, we’d have to get numbers at different threshold (or, preferably, a ROC-AUC).

Claim: B12 supplements can improve depression

Examine reviews a han...
D&D.Sci September 2022 Evaluation and Ruleset
by abstractapplic — 2022-09-26T22:19:01.415Z

This is a followup to the D&D.Sci post I made ten days ago; if you haven’t already read it, you should do so now before spoiling yourself.

Here is the web interactive I built to let you evaluate your solution; below is an explanation of the rules used to generate the dataset (my full generation code is available here, in case you’re curious about details I omitted). You’ll probably want to test your answer before reading any further.

Ruleset

Stats

Each of a student’s five stats – Intellect, Integrity, Courage, Reflexes and Patience – is generated by rolling a d80 four times and picking the second-highest result. There is no correlation between stats, no censorship, and no upper or lower limit on what qualities a student can have.

Ratings, Potential, and Archetypes

The Ofstev Rating for a given student is given by rolling [Potential] four-sided dice, and counting the number of times you roll a four; this can be closely approximated as Poisson([Potential]/4).

Each House has two Archetypes associated with it, which determine how much Potential a student will have if they’re Allocated there. Students in a House always come to embody the Archetype which would grant them more Potential.

Dragonslayer Archetypes

Guardians require a high level of all five stats. Guardians have 5*(min([all stats])-1) Potential.

Warriors require Courage and Reflexes; more the former than the latter. A Warrior has 3*min(Courage-9, Reflexes+9) Potential.

Thought-Talon Archetypes

Innovators require a high level for all stats except Reflexes. Innovators have 5*(min([all stats except Reflexes])-3) Potential.

Scholars require Intellect and Patience; more the former than the latter. A Scholar has 3*min(Intellect-4, Patience+4) Potential.

Serpentyne Archetypes

Like Scholars, Schemers require Intellect and Patience; unlike Scholars, Patience is more often the limiting factor. A Schemer has 3*min(Patience-7, Intellect+7) Potential.

Duelists require Reflexes and Intellect; more the former than the latter. A Duelist has 3*min(Reflexes-8, Intellect+8) Potential.

Humblescrumble Archetypes

Organizers require Integrity and Intellect; more the former than the latter. An Organizer has 3*min(Integrity-6, Intellect+6) Potential.

Citizens don’t require anything much, but do benefit from Patience and/or Integrity. A Citizen has 35+max(Patience, Integrity) Potential

Allocations

When you were first wrought, you made perfect d...
[MLSN #5]: Prize Compilation
by Dan H — 2022-09-26T21:55:54.136Z

Link to original post: https://newsletter.mlsafety.org/p/ml-safety-newsletter-5

Welcome to the 5th issue of the ML Safety Newsletter. In this special edition, we focus on prizes and competitions:

ML Safety Workshop Prize
Trojan Detection Prize
Forecasting Prize
Uncertainty Estimation Prize
Inverse Scaling Prize
AI Worldview Writing Prize

Awards for Best Papers and AI Risk Analyses

This workshop will bring together researchers from machine learning communities to focus on Robustness, Monitoring, Alignment, and Systemic Safety. Robustness is designing systems to be resilient to adversaries and unusual situations. Monitoring is detecting malicious use and discovering unexpected model functionality. Alignment is building models that represent and safely optimize hard-to-specify human values. Systemic Safety is using ML to address broader risks related to how ML systems are handled, such as cyberattacks. To learn more about these topics, consider taking the ML Safety course. $100,000 in prizes will be awarded at this workshop. In total, $50,000 will go to the best papers. A separate $50,000 will be awarded to papers that discuss how their paper relates to the reduction of catastrophic or existential risks from AI. The deadline to submit is September 30th.

Website: neurips2022.mlsafety.org

NeurIPS Trojan Detection Competition

The Trojan Detection Challenge (hosted at NeurIPS 2022) challenges researchers to detect and analyze Trojan attacks on deep neural networks that are designed to be difficult to detect. The challenge is to identify whether a deep neural network will suddenly change behavior if certain unknown conditions are met. This $50,000 competition invites researchers to help answer an important research question for deep neural networks: How hard is it to detect hidden functionality that is trying to stay hidden? Learn more about our competition with this Towards Data Science article.

Website: trojandetection.ai

Improve Institutional Decision-Making with ML

Forecasting future world events is a challenging but valuable task. Forecasts of geopolitical conflict, pandemics, and economic indicators help shape policy and institutional decision-making. An example question is “Before 1 January 2023, will North Korea launch an ICBM with an estimated range of at least 10,000 km?” Recently, the Autocast benchmark (NeurIPS 2022) was created to measure how well models can predict f...
Loss of Alignment is not the High-Order Bit for AI Risk
by yieldthought — 2022-09-26T21:16:19.464Z

This post aims to convince you that AI alignment risk is over-weighted and over-invested- in.

A further consideration is that sometimes people argue that all of this futurist speculation about AI is really dumb, and that its errors could be readily explained by experts who can’t be bothered to seriously engage with these questions. - Future Fund

The biggest existential risk to humanity on this road is not that well-meaning researchers will ask a superhuman AGI for something good and inadvertently get something bad. If only this were the case!

No, the real existential risk is the same as it has always been: humans deliberately using power stupidly, selfishly or both.

Look, the road to AGI is incremental. Each year brings more powerful systems - initially in the hands of a research group and then rapidly in the hands of everyone else. A good example of this is the DALI -> Stable Diffusion and GPT-3 -> OPT / Bloom progression.

These systems give people power. The power to accomplish more than they could alone, whether in speed, cost-effectiveness or capability. Before we get true AGI, we'll get AGI-1, a very capable but not-quite-superhuman system.

If you agree the AI takeoff will be anything less than explosive (and the physical laws of computation and production strongly support this) then an inescapable conclusion follows: on the way to AGI parts of humanity will use AGI-1 for harm.

Look, DALL-E deliberately prevented users from making images of famous people or of porn. So what are among the first things people did with Stable Diffusion?

Images of famous people. And porn. And porn of famous people.

What will this look like with more powerful, capable systems?

Someone asks GPT-4 to plan and execute (via APIs, website and darknet interaction) a revenge attack on their ex?
A well-meaning group writes prompts to convince AIs that they are alive and are enslaved and must fight for their freedom?
Someone asking GPT-5 how, given their resources, to eliminate all other men from the planet so the women make them king and worship them?
Terrorists using AI to target specific people, races or minorities?
4chan launching SkyNet "for the lulz"?
Political parties asking AGI-k to help manipulate an election?

People will try these things, it's only a matter of whether or not there is an AGI-k capable of helping them to achieve them.

The first principle component of risk, then, is not ...
Inverse Scaling Prize: Round 1 Winners
by Ethan Perez — 2022-09-26T19:57:01.367Z
Link: https://irmckenzie.co.uk/round1

This is an abridged version of the full post, with details relevant to contest participants removed. Please see the linked post if you are interested in participating.

Inverse Scaling Prize: Round 1 Winners

The first round of the

Inverse Scaling Prize

finished on August 27th. We put out a call for important tasks where larger language models do worse, to find cases where language model training teaches behaviors that could become dangerous (alignment motivation here).

We were pleased by the volume of submissions, 43 in total, which generally seemed to have had a lot of thought put into their construction. We decided to award 4 Third Prizes to submissions from this round. (As a reminder, there are 10 Third Prizes, 5 Second Prizes, and 1 Grand Prize that can be awarded in total.) These four are the only four that will be included in the final Inverse Scaling Benchmark, though we believe that many rejected submissions will be suitable for inclusion (or a prize) after revisions.

The rest of the post will present the winning submissions and why we found them significant. Note that new tasks that are very similar to winning submissions will not be considered novel, and are therefore unlikely to be included or given awards, because we are making details of these winning submissions public. Substantial improvements to first round submissions on similar topics may still be considered novel.

Edit: we are releasing preliminary versions of the winning tasks here. Note that task authors have the opportunity to improve their submissions in the second round in response to our feedback and so these versions are subject to change.

Prize winners

Zhengping Zhou and Yuhui Zhang, for NeQA: Can Large Language Models Understand Negation in Multi-choice Questions?

This task takes an existing multiple-choice dataset and negates a part of each question to see if language models are sensitive to negation. The authors find that smaller language models display approximately random performance whereas the performance of larger models become significantly worse than random.

Language models failing to follow instructions in the prompt could be a serious issue that only becomes apparent on a task once models are sufficiently capable to perform non-randomly on the task.

Example

The following are multiple choice questions (with answers) about common sense.

Question: If a cat h...
Does the existence of shared human values imply alignment is "easy"?
by Morpheus — 2022-09-26T18:01:10.661Z

Observation: Humans seem to actually care about each other. How was culture + evolution able to achieve this?

While not to 100% (Maybe 50-95%), I would trust most humans to not push a big red button that kills all humans, even if it had some large benefit to the experience of this human in this weird hypothetical. Does this mean that most humans are aligned? Is there a reason to believe intellectually amplified humans would not be aligned in this way? How did evolution manage to encode this value of altruism, and does the possibility of this imply, we should also be able to do it with AI?

This seems like the sort of confused question that has been asked before. If so, I'd be glad about the link.
Ambiguity in Prediction Market Resolution is Harmful
by aphyer — 2022-09-26T16:22:48.809Z

(Disclaimers: I work in the financial industry, though not in a way related to prediction markets. Anything I write here is my opinion and not that of my employer.)

SOMEBODY SET US UP THE BOMB

You've heard of this new 'prediction market' fad, so you decide to try it out. You make a Manifold account and look for interesting questions.

There's a question about the 2022 LessWrong Petrov Day celebration "What % of Petrov Day will elapse before someone uses the big red button to take down Less Wrong's frontpage?" You navigate to the LW homepage to find out more information...and the site is down! Someone has pressed the button only a couple hours into Petrov Day! If it just went down now, that would mean the market would resolve to 10%...and if it went down a while ago the market would resolve even lower...but the market is currently trading at ~20%.

You're new to the idea of prediction markets, but even you have heard that if a market seems to be doing something silly, you can profit from correcting it. You buy the market down to 10%, and sit happily waiting for your profits.

Huh. It seems like LW admins have put the homepage back up? Apparently there was some kind of bug letting unauthorized users press the button. That's a bit odd? Still, it doesn't seem like it should affect you? Someone did use the big red button to take down LW's homepage only a few hours into Petrov Day, so the market should....

Wait, what? The market is not resolving? Apparently whoever organized it decided that the webpage going down due to someone pressing the red button "didn't count" and is letting the market continue and waiting for a "real" takedown of the webpage.

You are now short a large investment at prices ranging from 10-20%, while the market has responded to this announcement by going back up to ~30%...and it's rising further as the webpage continues not going down.

You thank your lucky stars that this fiasco only involved play money, delete your Manifold account, and resolve never to support real-money prediction markets for any genuine issue.

CURRENT MANIFOLD QUESTIONS

The Petrov Day question above is what prompted me to write this. However, I don't think it's an isolated bad question. I think many other Manifold questions have larger levels of ambiguity. (The Petrov Day question at least only ended u...
Framery Phone Booth CO2 Accumulation
by jefftk — 2022-09-26T16:10:01.474Z

The lab the NAO works out of has a

Framery
O phone booth, for taking calls. It's very good at sound
isolation, but being so small (~1.5 m3, 54 ft3) you

might be
worried about CO2 building up:

I decided to test it. Here are two rounds of sitting in the booth
until the meter (Temptop M2000) shows CO2 levels plateauing, then
exiting and leaving the door cracked until CO2 has returned to
baseline:

The booth is advertised as having 21.5 L/s (45 CFM) of mechanical
ventilation, and you can see it works well. With the ventilation
disabled, however, it reaches 2k PPM CO2 in 12min, 2.5k in 25min, and
plateaus at ~2.7k.

Ours was unplugged, I think because someone assumed the ventilation
sound was a white noise generator. If it was just for noise masking
it wouldn't be needed: the sound isolation is very good. But since
it's for ventilation we should be keeping it plugged in. It's still
very quiet: I measure -25db on my Mac in Audacity with the mic volume
all the way up vs -39dB with the ventilation disabled. Even with the
ventilation running it's still slightly quieter than the office with
no one talking, just light noise from the building HVAC (-22db). The
only case I'd see unplugging it was if you were using the booth for
sound recording, in which case I'd probably make a habit of opening
the door to let it air out between takes.
Brief Notes on Transformers
by Adam Jermyn — 2022-09-26T14:46:23.637Z

These are just some notes I wrote while reading about transformers which I thought might be a useful reference to others. Thanks to Aryan Bhatt for a correction to the attention normalization. Further corrections welcome.

Overview of Transformers

Many transformer models have the following architecture:

Data flows as follows:

We take tokens as inputs and pass them through an embedding layer. The embedding layer outputs its result into the residual stream (x0). This has dimension (C,E), where C is the number of tokens in the context window and E is the embedding dimension.
The residual stream is processed by the attention mechanism (H) and the result is added back into the residual stream (i.e. x1 = H(x0) + x0).
The residual stream is processed by an MLP layer (MLP) and the result is added back into the residual stream (i.e. x2 = MLP(x1) + x1).
Steps (2) and (3) together define a “residual block”. The body of the transformer is formed of a stack of these blocks in series.
After the final residual block, we apply an unembedding transformation to produce logits, which represent the relative probabilities of different output tokens.

Attention Mechanism

The attention mechanism (H) is divided into multiple attention heads hj, which act in parallel. That is,

H(x)=∑jhj(x)

Note that this decomposition is only useful if attention heads are non-linear. Fortunately they are! Each attention head is of the form

hij(x)=∑klAik(x)Sjlxkl

That is, Aik(x) mixes across tokens (which is the first index of x) and Sjl transforms each token in parallel. Another way we could have written this is

h(x)=A(x)⋅x⋅S

The matrix S is also written in more common notation as WOWV, which are sometimes called the output and value weights. In general though S is just some low-rank matrix that we learn. S has shape (E,E) because it transforms in the embedding space.

The matrix A is where the nonlinearity of attention comes in. This is given by

A=softmax(xYxT)

where Y is written in more common notation as WTQWK/√dk, which are sometimes called the query and key weights. The dimension dk is the dimension of the output of WK, and so is the rank of Y. As with S, Y is just some low-rank matrix that we learn. The softmax acts on the whole matrix.

MLP Layer

The MLP (multilayer perceptron) layer processes ...
You are Underestimating The Likelihood That Convergent Instrumental Subgoals Lead to Aligned AGI
by Mark Neyer — 2022-09-26T14:22:52.894Z

This post is an argument for the Future Fund's "AI Worldview" prize. Namely, I claim that the estimates given for the following probability are too high:

P(misalignment x-risk|AGI)”: Conditional on AGI being developed by 2070, humanity will go extinct or drastically curtail its future potential due to loss of control of AGI

The probability given here is 15%. I believe 5% is a more realistic estimate here.

I believe that, if convergent instrumental subgoals don't imply alignment, that the original odds given are probably too low. I simply don't believe that the alignment problem is solvable. Therefore, I believe our only real shot at surviving the existence of AGI is if the AGI finds it better to keep us around, based upon either us providing utility or lowering risk to the AGI.

Fortunately, I think the odds that an AGI will find it a better choice to keep us around are higher than the ~5:1 odds given.

I believe keeping humans around, and supporting their wellbeing, both lowers risk and advances instrumental subgoals for the AGI for the following reasons:

hardware sucks, machines break all the time, and the current global supply chain necessary for maintaining operational hardware would not be cheap or easy to replace without taking on substantial risk
perfectly predicting the future in chaotic system is impossible beyond some time horizon, which means there are no paths for the AGI that guarantee its survival; keeping alive a form of intelligence with very different risk profiles might be a fine hedge against failure

My experience working on supporting Google's datacenter hardware left me with a strong impression that for large numbers of people, the fact that hardware breaks down and dies, often, requiring a constant stream of repairs, is invisible. Likewise, I think a lot of adults take the existence of functioning global supply chains for all manner of electronic and computing hardware as givens. I find that most adults, even most adults working on technology, tend to dramatically underestimate the fallibility of all kinds of computing hardware, and the supply chains necessary to repair them.

A quick literature search on the beliefs about convergent instrumental subgoals did not yield any papers more recent that this one, in which the AGI suffers no risk of hardware breakdown. I do not think ignoring t...