Next Monday is Petrov Day (September 26), an annually observed Rationalist/EA holiday inspired by the actions of Stanislav Petrov:
As a Lieutenant Colonel of the Soviet Army, Petrov manned the system built to detect whether the US government had fired nuclear weapons on Russia. On September 26th, 1983, the system reported five incoming missiles. Petrov’s job was to report this as an attack to his superiors, who would launch a retaliative nuclear response. But instead, contrary to the evidence the systems were giving him, he called it in as a false alarm.
It was subsequently determined that the false alarms were caused by a rare alignment of sunlight on high-altitude clouds and the satellites' Molniya orbits, an error later corrected by cross-referencing a geostationary satellite.
In explaining the factors leading
A week ago I was skeptical about the prospect of radically reducing human sleep needs. After reading John Boyle’s Cause area: Short-sleeper genes, I decided to research the area more deeply and updated to believe that it’s more likely that we can reduce human sleep needs without significant negative side effects. It might increase risk-taking which has both positive and negative effects. The one friend I have that has short-sleeper genes is a startup founder.
Boyle suggested that one of the best actions to attempt would be using orexin or an orexin agonist as a drug, but that there’s currently a lack of funding for doing so.
Given the way the FDA and EMA work, drugs only get approved when they are able to cure illnesses, with an illness being...
Thanks to Adam Shimi, Lee Sharkey, Evan Hubinger, Nicholas Dupuis, Leo Gao, Johannes Treutlein, and Jonathan Low for feedback on drafts.
This work was carried out while at Conjecture.
"Moebius illustration of a simulacrum living in an AI-generated story discovering it is in a simulation" by DALL-E 2
TL;DR: Self-supervised learning may create AGI or its foundation. What would that look like?
Unlike the limit of RL, the limit of self-supervised learning has received surprisingly little conceptual attention, and recent progress has made deconfusion in this domain more pressing.
Existing AI taxonomies either fail to capture important properties of self-supervised models or lead to confusing propositions. For instance, GPT policies do not seem globally agentic, yet can be conditioned to behave in goal-directed ways. This post describes a frame that enables more...
In software jargon, there’s a nice term “business logic”, for code like the following (made-up) excerpt from corporate tax filing software (based on here):
def attachSupplementalDocuments(file): if file.state == "California" or file.state == "Texas": # SR008-04X/I are always required in these states file.attachDocument("SR008-04X") file.attachDocument("SR008-04XI") if file.ledgerAmnt >= 500_000: # Ledger of 500K or more requires AUTHLDG-1A file.attachDocument("AUTHLDG-1A")
When you think of “business logic”, think of stuff like that—i.e., parts of source code that more-or-less directly implement specific, real-world, functional requirements.
By contrast, things that are NOT business logic include infrastructure & subroutines & plumbing that are generally useful in many contexts—e.g. code for initializing a database, or code for memory management, or code for performing stochastic gradient descent.
If genomes are the “source code” of brains, then they...
As you would expect, those in Public Health did not all quietly agree with this.
When Bob Wachter says it is a judgment call whether the pandemic is over, you know the pandemic is over.
When Caitlin Rivers says ‘yes and no,’ and the details she lists are far more yes than no, you know the pandemic is over.
When Celine Gounder responds by saying that calling the pandemic over is Just Awful and means we are awful people violating sacred values and must Do More, you still know the pandemic is over. She points to some other similar quotes here. It similarly does not seem over to Tatiana Prowell.
More generally, with various Public Health voices chiming in.
On occasion, for my work at Lightcone I have been able to buy things faster than their advertised lead times. For example, I once got…
And a bunch of other stuff. The first times I did this were a bit of a desperate scramble. Now, however, I mostly have a handful of helpful tips and tricks that I keep reusing. When working with new colleagues, I’ve found myself...
I recently completed the ML Safety Course by watching the videos and browsing through the review questions, and subsequently writing a short summary. As an engineer in the upstream oil and gas industry with some experience in dealing with engineering safety, I find the approach of the course of thinking in this framework especially valuable.
This post is meant to be a (perhaps brutally) honest review of the course despite me having no prior working experience in ML. It may end up reflecting my ignorance of the field more than anything else, but I would still consider it as a productive mistake. In...
Epistemic status: Low effort post intended for my own reference. Not endorsed by the course creators.
I have also written a review of the course here.
A risk can be decomposed into its vulnerability, hazard exposure, and hazard (probability and severity). They can be defined as below, with an example in the context of the risk of contracting flu-related health complications.
Acknowledgements: I wrote this report as part of a six-hour paid work-trial with Epoch AI.
Epistemic status: My dataset analysis is a bit simplistic but the inference I draw from it seems likely. The implications for TAI timelines, in descending order of confidence, are 3, 7, 8, 4, 1, 2, 5, 6.
AI forecasters seek to predict the development of large language models (LLMs), but these predictions must be revised in light of DeepMind's Chinchilla. In this report, I will discuss these revisions and their implications. I analyse a dataset of 45 recent LLM and find that previous LLMs were surprisingly trained neither Kaplan-optimally nor Hoffman-optimally. I predict that future LLMs will be trained Hoffman-optimally. Finally, I explore how these scaling laws should impact our AI alignment and governance...