trentmkelly/LessWrong-43k huggingface dataset:
Reflections on Trusting Trust & AI
by Itay Yona — 2023-01-16T06:36:24.761Z
Link: https://mentaleap.ai/trust/

Reflections on Trusting Trust & AI

The goal of the post is to highlight the necessity for mechanistic interpretability from an information security perspective:

Background

In 1984, Ken Thompson authored a seminal paper on information security, Reflection on Trusting Trust, in which he popularized the idea of digital Trojan horses [1]. In his three-page article, Thompson demonstrated a technique for injecting an undetectable backdoor into a compiler, posing a threat to the integrity of the entire computer infrastructure. He summarized the moral lesson of his attack in the following words:

You can’t trust code that you did not totally create yourself

This attack can be broken down into three simple steps:

It is possible to insert a backdoor into software by modifying the source code. For example, one could modify the code responsible for the operating system login screen. Once the compiled OS is deployed, an attacker will be able to bypass the login screen and gain admin access.
An attacker could modify a compiler so that it inspects the input source code. If the source code is part of the OS login, the compiler will inject the backdoor from step one. [As a result, the malicious code would not appear on the operating system, but anyone who has access to the compiler’s source will be able to detect the backdoor.]
Moreover, an attacker could modify a compiler so that it inspects the input source code in a different manner. If the source code is of a compiler, the malicious compiler will inject the backdoor from step two into it. This adds a self-replication mechanism to the backdoor that will propagate itself to all compilers and software thereafter.

Considering that every component of your system was once compiled, and that the original compiler is long forgotten, the backdoor will be undetectable in any source code available to you. The backdoor will however, remain in the machine’s assembly instructions and anyone with the tools and skills of software reverse-engineering will be able to locate the backdoor and eliminate it from the system. (see notes section below)

AI risk

Researchers in the field of artificial intelligence are often unable to agree on the following critical point: Deep Neural Networks execute code. Although neural networks can be described solely with architecture and floating point matrices, they could also be described algorithmically. Any digita...
Is “Earning to Give” a Bad Framework?
by clans — 2023-01-16T05:35:21.061Z
Link: https://locationtbd.home.blog/2022/12/25/is-earning-to-give-a-bad-framework/

Among the effective altruists, there are two main camps regarding the most effective way to do good with your career. One is to direct your work specifically at an EA-aligned organization, something like a specific charity or company producing a product that directly saves or improves lives. The other is simply to seek a high-paying career, make as much money as possible, and donate a significant fraction of that money to an effective giving organization like

Givewell

.

The latter is an idea born straight out of utilitarianist ethics, championed by William MacAskill,

who wrote in 2013

:

“… while researching ethical career choice, I concluded that it’s in fact better to earn a lot of money and donate a good chunk of it to the most cost-effective charities—a path that I call “

earning to give

.” … you don’t have to be a billionaire. By making as much money as we can and donating to the best causes, we can each save hundreds of lives… In general, the charitable sector is people-rich but money-poor. Adding another person to the labor pool just isn’t as valuable as providing more money so that more workers can be hired. You might feel less directly involved because you haven’t dedicated every hour of your day to charity, but you’ll have made a much bigger difference.”

MacAskill has later clarified that he does not advocate “earning to give” as the best framework for Effective Altruism (EA), but rather just one of the ways an average someone can pursue making the world a measurably better place with their career. I think that’s a worthwhile clarification to note. I’ve only recently begun earning money in my career and have been reading the discussions on this issue. I wanted to poke at this “earning to give” concept myself a bit. It's an issue even more on the forefront as the FTX scandal broke with SBF at the helm, who gave away a lot of money that he probably shouldn't have had.

While I think “earning to give” is certainly not a framework antithetical to the tenets of EA – simply to do a maximal amount of good in the world – I am not convinced that it is an advisable thing to advertise without caveats. I’ll cover my specific thoughts below, but overall, it is my impression that compared to direct work, “earning to give” is often not the most “good effecting” path. It is also vulnerable to “utilitarian traps” and is far from a golden rule for the EA community.

I will emph...
Why you ask the significance question why
by Slider — 2023-01-16T03:44:21.551Z

Hello.

(I am already doing 75% better)
I used to think that keeping your identity small is laudable. The reasoning goes that if "me" and "you" are interchangable instead of being only applicable for "me" there is a wider scope of applicability. And there is an effect.

At some point the other reduced to a hollow propertyless "x" means that people can not apply it or do not find it meaningful.

Lurking works well for abstract objective patterns.

If you actually do things, you have stakeholders which might not to a sufficient degree fit one general mold. Irreducable diversity might be real and it might be important.

I find myself writing still in a fact-stating manner. Sure, if two people are correct their takes should mesh without regard which specific turns were taken. But two experiences of (a facet of) reality do not have this automeshing property.

Look, I had negative experiences, emoted about it and often passively waited if/when a non-harmful action about it was possible. But what I now find signficant is that the "non-harmful action" takes an objective form, the action in itself is non-harmful. But if you ask about the affected parties, it is not a tag you put on the action, it is a line you draw between the party and action.

I might be trying to prematurely abstract, but, the emotion is about being turned into something more "correct". That a answer that is too unintellible, wrong ideological leaning or using a forbidden communication method will just be left stranded cold alone. Keeping your mouth shut when you have poor or no answer.

A question of "how" in the meaning technology, can be packaged so its customer does not need to opine about its operation. This works in a static environment. But environments are not static. Placing an art piece in a new context is especially dynamic.

In order not to drift into hurt, the receiver (customer) needs to know why it works the way it works, in order to have a chance to take the significance of a new context into account. This means essentially all kinds of behaviour profile specifications are pointless (bite me, don't actually bite me, but talk to me).

I am a middle-aged autistic programmer. I am trying to show more of me when I communicate.
Pause of monologue, to turn into dialogue
Investment, Work, and Vision:
Who is responsible for creating value?
by Sable — 2023-01-16T01:57:01.803Z
Link: https://affablyevil.substack.com/p/investment-work-and-vision

Previously in this series:

Value Created vs. Value Extracted

A Brief Recap

I define creating value as the process of producing something new, something the world did not have before. I define extracting value as monopolizing something that already exists in order to personally benefit from denying others access to it.

I operationalize the difference between the two with a metaphor of an engineer who builds a bridge connecting two towns.

This bridge makes trade easier, cheaper, and safer than trade or travel by boat. Thus the engineer has created value.

Some years later, a troll happens by and decides to live beneath the bridge. This troll, Rob, forces any who would cross the bridge to pay a toll to him (or be squished by a large club). Thus Rob has extracted value from the bridge. This is bad and he should feel bad.

Affably Evil is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

What Did The Engineer Really Do?

Now, I say that the engineer created this value, but what does that really mean?

What did the engineer add to the world, that wasn’t there before? And how much of that is the engineer responsible for? What was her contribution?

Not The Atoms

Whether or not this particular engineer happens to be aware of the laws of thermodynamics, she certainly didn’t create the atoms that make up the bridge. They came from the earth, and can trace a proud lineage all the way back to the big bang.

That being said, it’s not controversial to point out that the value we find in physical matter is almost entirely in the shape of that matter, the arrangement of the atoms, rather than the atoms themselves. Computer chips are no different than sand and dirt, atomically, save for the arrangement of the atoms.

And the atoms that make up the bridge might as well have been living in their mother’s basement and eating nachos all day for all the productive impact they had on society, before the engineer shoved them outside and made them get a job.

Not The Work

Think about everything that went into the creation of this bridge, all the tasks and work involved.

The stones that make up the physical structure of the bridge had to be found and quarried, then cut and chiseled to the correct shapes to form the structure.

They had to be carried to the site where the bridge was to be built, involving quite a bit of manual labor, and...
Conclusion and Bibliography for "Understanding the diffusion of large language models"
by Ben Cottier — 2023-01-16T01:46:42.852Z

This post is one part of the sequence Understanding the diffusion of large language models. As context for this post, I strongly recommend reading at least the 5-minute summary of the sequence.

Conclusion

In this sequence I presented key findings from case studies on the diffusion of eight language models that are similar to GPT-3. The phenomenon of diffusion has broad relevance to risks from TAI:

The diffusion of AI technology affects when TAI will be developed, and which actors will lead AI development by what margin. This in turn affects how safe the TAI systems are, how the systems are used, and what the state of global politics and economics is like when the systems are used.
Diffusion can have benefits, such as helping less-resourced actors to scrutinize leading AI developers, and supporting AI alignment research outside of leading industry AI labs.

GPT-3-like models are quite a specific domain, and may seem far from TAI. Nonetheless, I centered my research on case studies of GPT-3-like models because I think they are relatively informative about how diffusion will impact TAI development. In particular:

The way that diffusion works today (in broad terms) might persist until the development of TAI, especially if TAI is developed relatively soon (e.g., in the next 10 years).
TAI systems (or components of them) might resemble today’s best-performing language models, especially if the

scaling hypothesis

is true. So the implications of diffusion related to such models may be similar to the implications of diffusion related to transformative AI systems.
Even if a lot changes between now and TAI, the history of diffusion improves our understanding of what could happen.

My research has strong limitations, including that:

Much of the data from my case studies is highly uncertain, with quantitative estimates often spanning an order of magnitude.
I often generalize from a small set of case studies in a narrow domain. Some of my conclusions are not robust to counterexamples that I might discover in the future. However, I have tried my best to factor this possibility into my confidence levels.
Many of my bottom-line conclusions are not supported by much hard evidence, and are instead based on a combination of logical arguments and intuitions.

I think that the concept of diffusion is a productive framing to study competition, publication strategy, and o...
Questions for further investigation of AI diffusion
by Ben Cottier — 2023-01-16T01:46:15.107Z

This post is one part of the sequence Understanding the diffusion of large language models. As context for this post, I strongly recommend reading at least the 5-minute summary of the sequence.

This post lists questions about AI diffusion that I think would be worthy of more research at the time of writing. Some questions serve as direct follow-ups to my research, while others just seem like important questions related to diffusion. I already raised some of these questions throughout this sequence, but this post collects them all so that interested researchers can easily refer back to the questions.

Feel free to reach out to me about these research ideas. I may be able to offer advice, suggest links, and suggest people to talk to. It's possible that I or Rethink Priorities could help connect you with funding to work on these ideas if you're interested and a good fit.

Further research to evaluate my proposals to limit access to datasets and algorithmic insights

In a previous post I presented

proposals to limit access to datasets

and

proposals to limit access to algorithmic insights

. I believe those proposals are probably worth doing, but that belief has a low enough

resilience

that the next step should be further consideration of whether or not to do these things.

The follow-up questions for the dataset proposals that I think are highest priority are:

How feasible is it to actually convince various producers of large ML datasets to take the actions I recommended?

Would they keep a dataset private, when they would have otherwise openly published it? Would some further incentive (e.g. financial compensation) be necessary?
Is a data-labeling service such as

Surge AI

willing to add vetting procedures for who they provide services to?
I recommend at least 1 full-time equivalent week investigating this. The research could involve first evaluating the benefit of my initial proposal, and then possibly interviewing people at data hosters/curators such as the CommonCrawl foundation or Surge AI to get their opinion on the proposals.

Is structured access to datasets (in the manner I described in the previous post) technically feasible?

I recommend at least 2 full-time equivalent days investigating this. Someone with a good understanding of modern machine learning as well as user-facing APIs would be an ideal fit.

Is having structured a

...
Implications of large language model diffusion for AI governance
by Ben Cottier — 2023-01-16T01:45:41.745Z

This post is one part of the sequence Understanding the diffusion of large language models. As context for this post, I strongly recommend reading at least the 5-minute summary of the sequence.

The primary aim of this research project was to be descriptive about the diffusion of large language models. Drawing actionable implications was less of a priority, and should be more of a priority in future work. Having said that, in this post I explore how to beneficially shape AI diffusion, and what my findings mean for the governance of transformative AI (TAI). My conclusions here have generally lower confidence and

resilience

than conclusions elsewhere in the sequence.

Key takeaways

What should we generally be aiming to achieve by shaping the diffusion of AI technology? I think the following outcomes are both good and tractable enough to aim for: (

more

)

Delay the arrival of TAI by reducing diffusion.
Have fewer actors in the lead to develop TAI.
Help more favorable[1] actors become leaders, or remain leaders, or increase their lead, without also helping less favorable actors and while ideally not speeding up AI timelines in absolute terms.
Reduce downside risks from the last two outcomes above by supporting scrutiny towards leading actors and supporting some research outside the leading actors (mainly AI alignment research).

The key combination of factors that limited the diffusion of GPT-3-like models was (a) that training such models required a very expensive amount of compute, and (b) that the model weights for existing GPT-3-like models were not openly published. Limiting diffusion is much easier when both of these circumstances hold. If compute cost becomes less prohibitive but models remain closed-source, then the information security and publication practices around algorithmic insights would become more important levers than they are today. Limiting diffusion seems less tractable overall in that scenario. (

more

)
I’m 65% confident that, for each future state-of-the-art pretrained language model, the average delay until replications of that model are produced (with say, a 10-year cutoff point for which replications are included) will be longer than 2 years.[2],[3] (

more

)

I’m defining “state-of-the-art” as setting a new record in performance across many varied tasks in the language domain.
The argument for longer average

...
Publication decisions for large language models, and their impacts
by Ben Cottier — 2023-01-16T01:44:59.508Z

This post is one part of the sequence Understanding the diffusion of large language models. As context for this post, I strongly recommend reading at least the 5-minute summary of the sequence.

In this post, I:

Overview the different forms of information and artifacts that have been published (or not) for the GPT-3-like models that I studied.
Estimate some of the impacts of these publication decisions.
Assess the different rationales for these publication decisions.
Make predictions about how publication decisions and norms will change in the language modeling domain in the future.

Key takeaways

My case studies suggest there is still a strong incentive for researchers at industry-leading AI labs to publish details about the training process, model architecture, and hyperparameter values they used. This has significant effects on diffusion. (

more

)

I speculate that the rationale for publishing implementation details is to make research results more credible and easily verifiable. I think this is in turn desired due to (a) existing norms for research transparency and reproducibility, and (b) the incentive to attract top researchers, who tend to benefit from the transparency and reproducibility of the research they were involved in.
I estimate (with 90% confidence) that disclosing all hyperparameter values for a GPT-3-like model saves between 1% and 20% of the total compute cost needed to replicate that model.
EleutherAI based their choice of hyperparameter values on those of GPT-3. This seemed to shorten the duration of the GPT-NeoX-20B project—my best guess is a shortening of two weeks (90% CI: zero to six weeks).

Compared to the implementation details, the code, datasets, and trained models from research were published much less or much later, and seemed to have smaller effects on diffusion even when published. These effects were significant, but I think that improving the publication practices of top developers (who are often the first to make advances in implementation) is a higher priority than preventing artifacts from being openly published after a long delay (more than 1 year). (

more

)

I estimate that if code for training GPT-3 had been openly published beforehand, the production of EleutherAI’s GPT-NeoX-20B would have been sped up by two months (90% CI: one to six months), and Meta AI’s OPT-175B would have been sped up by two weeks (90% CI: one to six weeks).

...
Drivers of large language model diffusion: incremental research, publicity, and cascades
by Ben Cottier — 2023-01-16T01:44:23.862Z

This post is one part of the sequence Understanding the diffusion of large language models. As context for this post, I strongly recommend reading at least the 5-minute summary of the sequence.

Key takeaways

Up until the release of OPT-175B in May 2022, incremental research (i.e., research that makes a relatively small change to an existing method) was the prevailing diffusion mechanism for actors to gain direct access to the weights of a GPT-3-like model; nine GPT-3-like models were developed in that way prior to OPT-175B, of which none had their weights made widely accessible.[1] The wider accessibility of OPT-175B changed the prevailing mechanism to open publication, based on my estimate that more actors have direct access to the OPT-175B model weights than actors that have developed GPT-3-like models themselves. (

more

)

I don’t think that multiple discovery (i.e., two actors independently coming up with the same idea or result) was significantly involved in the diffusion of GPT-3-like models. In particular, I think if the relevant papers weren’t published, it would’ve been 6 months (90% CI: 1 to 18 months) before any other actor would’ve discovered either a model with GPT-3’s capabilities or the

scaling laws

it was based on.

It’s plausible that the publication of the GPT-3 and scaling laws papers was unnecessarily early in terms of beating other actors to the punch, but I don’t have enough evidence to be confident in that claim. Regardless, I think that developers should have more scrutiny about whether they are really in a race to publish, because the harm of accelerating AI capabilities could outweigh the benefit of publishing first with a more responsible strategy (in order to establish better publication norms).

Access to compute appears to have been the main factor hindering the diffusion of GPT-3-like models. The next biggest hindering factors appear to have been acquiring the necessary machine learning and engineering expertise. (

more

)

I estimate that compute makes up 87% (90% CI: 64% to 98%) of the combined compute cost and salary cost of GPT-3-like model development.[2]

The largest accelerating factors in the cases I studied (i.e., factors that aren't necessary for developing GPT-3-like models but that seemed to make development easier or more likely) are, in order of apparent importance, (1) publicity abou

...