Recent Discussion
Next Monday is Petrov Day (September 26), an annually observed Rationalist/EA holiday inspired by the actions of Stanislav Petrov:
...As a Lieutenant Colonel of the Soviet Army, Petrov manned the system built to detect whether the US government had fired nuclear weapons on Russia. On September 26th, 1983, the system reported five incoming missiles. Petrov’s job was to report this as an attack to his superiors, who would launch a retaliative nuclear response. But instead, contrary to the evidence the systems were giving him, he called it in as a false alarm.
It was subsequently determined that the false alarms were caused by a rare alignment of sunlight on high-altitude clouds and the satellites' Molniya orbits, an error later corrected by cross-referencing a geostationary satellite.
In explaining the factors leading
A week ago I was skeptical about the prospect of radically reducing human sleep needs. After reading John Boyle’s Cause area: Short-sleeper genes, I decided to research the area more deeply and updated to believe that it’s more likely that we can reduce human sleep needs without significant negative side effects. It might increase risk-taking which has both positive and negative effects. The one friend I have that has short-sleeper genes is a startup founder.
Boyle suggested that one of the best actions to attempt would be using orexin or an orexin agonist as a drug, but that there’s currently a lack of funding for doing so.
Given the way the FDA and EMA work, drugs only get approved when they are able to cure illnesses, with an illness being...
I mean how a bacterium will make lactase when there's lactose, but won't make lactose when there isn't lactose.
It's quite easy to have a receptor protein that binds to lactose and then leads to a protein being expressed that turns lactose into lactase.
"amount of available food in the environment" is not as simple to measure inside of a cell and as a result, the regulation is much more complex.
I'm asking why orexin wouldn't be physiologically adaptive to the amount of food that's generally around.
That assumes that orexin is independe... (read more)
Thanks to Adam Shimi, Lee Sharkey, Evan Hubinger, Nicholas Dupuis, Leo Gao, Johannes Treutlein, and Jonathan Low for feedback on drafts.
This work was carried out while at Conjecture.
"Moebius illustration of a simulacrum living in an AI-generated story discovering it is in a simulation" by DALL-E 2
Summary
TL;DR: Self-supervised learning may create AGI or its foundation. What would that look like?
Unlike the limit of RL, the limit of self-supervised learning has received surprisingly little conceptual attention, and recent progress has made deconfusion in this domain more pressing.
Existing AI taxonomies either fail to capture important properties of self-supervised models or lead to confusing propositions. For instance, GPT policies do not seem globally agentic, yet can be conditioned to behave in goal-directed ways. This post describes a frame that enables more...
This is great! I really like your "prediction orthogonality thesis", which gets to the heart of why I think there's more hope in aligning LLM's than many other models.
One point of confusion I had. You write:
... (read more)Optimizing toward the simulation objective notably does not incentivize instrumentally convergent behaviors the way that reward functions which evaluate trajectories do. This is because predictive accuracy applies optimization pressure deontologically: judging actions directly, rather than their consequences. Instrumental convergence only comes int
1. Introduction
1.1 Hypothalamus as “business logic”
In software jargon, there’s a nice term “business logic”, for code like the following (made-up) excerpt from corporate tax filing software (based on here):
def attachSupplementalDocuments(file):
if file.state == "California" or file.state == "Texas":
# SR008-04X/I are always required in these states
file.attachDocument("SR008-04X")
file.attachDocument("SR008-04XI")
if file.ledgerAmnt >= 500_000:
# Ledger of 500K or more requires AUTHLDG-1A
file.attachDocument("AUTHLDG-1A")When you think of “business logic”, think of stuff like that—i.e., parts of source code that more-or-less directly implement specific, real-world, functional requirements.
By contrast, things that are NOT business logic include infrastructure & subroutines & plumbing that are generally useful in many contexts—e.g. code for initializing a database, or code for memory management, or code for performing stochastic gradient descent.
If genomes are the “source code” of brains, then they...
Joe Biden firmly declared on 60 Minutes that the Covid pandemic is over.
As you would expect, those in Public Health did not all quietly agree with this.
When Bob Wachter says it is a judgment call whether the pandemic is over, you know the pandemic is over.
When Caitlin Rivers says ‘yes and no,’ and the details she lists are far more yes than no, you know the pandemic is over.
When Celine Gounder responds by saying that calling the pandemic over is Just Awful and means we are awful people violating sacred values and must Do More, you still know the pandemic is over. She points to some other similar quotes here. It similarly does not seem over to Tatiana Prowell.
More generally, with various Public Health voices chiming in.
...Experts
certain number of heads
Do you mean "certain number of wins"? Number of heads is independent of their guesses, and number of correctly-guessed heads is asking a different question than the original experiment
On occasion, for my work at Lightcone I have been able to buy things faster than their advertised lead times. For example, I once got…
- 10 sofas in 2 days when the website shipping speed said 5 days to 3 weeks
- 10 custom beds from Japan in 1 week when all suppliers initially said 2-3 months[1]
- A custom bookcase in 1 week when website said 5 week lead-time
- 1000 square feet of custom hardwood flooring in 3 days even though the salesperson initially said it would be 2 weeks
And a bunch of other stuff. The first times I did this were a bit of a desperate scramble. Now, however, I mostly have a handful of helpful tips and tricks that I keep reusing. When working with new colleagues, I’ve found myself...
Both would be interesting.
This summary was written as part of Refine. The ML Safety Course is created by Dan Hendrycks at the Center for AI Safety. Thanks to Adam Shimi and Thomas Woodside for helpful feedback.
Overview
Background
I recently completed the ML Safety Course by watching the videos and browsing through the review questions, and subsequently writing a short summary. As an engineer in the upstream oil and gas industry with some experience in dealing with engineering safety, I find the approach of the course of thinking in this framework especially valuable.
This post is meant to be a (perhaps brutally) honest review of the course despite me having no prior working experience in ML. It may end up reflecting my ignorance of the field more than anything else, but I would still consider it as a productive mistake. In...
This summary was written as part of Refine. The ML Safety Course is created by Dan Hendrycks at the Center for AI Safety. Thanks to Linda Linsefors and Chris Scammel for helpful feedback.
Epistemic status: Low effort post intended for my own reference. Not endorsed by the course creators.
I have also written a review of the course here.
Risk Analysis
Risk Decomposition
A risk can be decomposed into its vulnerability, hazard exposure, and hazard (probability and severity). They can be defined as below, with an example in the context of the risk of contracting flu-related health complications.
- Hazard: a source of danger with the potential to harm, e.g. flu prevalence and severity
- Hazard exposure: extent to which elements (e.g., people, property, systems) are subjected or exposed to hazards, e.g. frequency of contact with people who are possible
Acknowledgements: I wrote this report as part of a six-hour paid work-trial with Epoch AI.
Epistemic status: My dataset analysis is a bit simplistic but the inference I draw from it seems likely. The implications for TAI timelines, in descending order of confidence, are 3, 7, 8, 4, 1, 2, 5, 6.
Abstract:
AI forecasters seek to predict the development of large language models (LLMs), but these predictions must be revised in light of DeepMind's Chinchilla. In this report, I will discuss these revisions and their implications. I analyse a dataset of 45 recent LLM and find that previous LLMs were surprisingly trained neither Kaplan-optimally nor Hoffman-optimally. I predict that future LLMs will be trained Hoffman-optimally. Finally, I explore how these scaling laws should impact our AI alignment and governance...
Also there is now mounting evidence that these LLMs trained on internet scale data are memorizing all kinds of test sets for many downstream tasks, a problem which only gets worse as you try to feed them ever more training data.
Not really? If we assume that they just memorize data without having intelligence, then their memory requirements would scale as N parameters, when instead we see a smaller constant for compression, which essentially requires actual intelligence rather than simply memorizing all that data.
This comment is the first successful deployment of agree-disagree trick I have seen. Neat!