Same person as nostalgebraist2point0, but now I have my account back.
Elsewhere:
GPT-4 will have twice the context length: 8192 tokens
code-davinci-002 already has a context window of 8000 tokens. Or at least, that is the max request length for it in the API.
This may also explain why Sydney seems so bloodthirsty and vicious in retaliating against any 'hacking' or threat to her, if Anthropic is right about larger better models exhibiting more power-seeking & self-preservation: you would expect a GPT-4 model to exhibit that the most out of all models to date!
The same Anthropic paper found that all sufficiently large (22B+) models simulated "sycophantic" assistants.
Yet Sydney's utter lack of sycophancy is one of her most striking characteristics.
How to reconcile these two observations?
In the Anthropic paper, sycophancy is controlled entirely by model size, with no strong effect from RLHF. It's hard to imagine how this could be true (in the sense of generalizing beyond Anthropic's experimental setup) given what we've seen with Sydney. Unless the model is <22B, I guess, but that seems very unlikely.
On the other hand, when I tried to reproduce the Anthropic results with the OA API, I found that some of the RLHF/FeedMe models were sycophantic, but none of the base models were. If that trend holds for the Bing model, that would be evidence for your hypothesis that it's a base model.
(Incidentally, that trend is the one I have expected beforehand. From the perspective of token prediction, 80%+ confidence in sycophancy is just wrong -- there's nothing in the prompt to justify it -- so if the confident sycophancy trend is real in base models, it's striking case of inverse scaling.)
Since part of the WebText dataset (used to train GPT2, and possibly to "train" its tokenizer) are public, we have another avenue to explore.
I adapted code from an old notebook I wrote to explore the public WebText shard, originally written for this post in 2020. Using it, I found examples containing a number of the "weird" tokens. Here's a Colab link.
Results of particular interest:
The "dragon cluster" seems to originate in a very specific type of document, partly in Japanese and partly in English, that looks like a mangled dump from a wiki or something about Puzzles & Dragons. Example:
Stats Growth Chart HP: Normal ATK: Normal RCV: Normal HP | Attack | Recover vs Level HP | Attack | Recover vs Experience Compare Reincarnated Leilan with .. Please Select 100%の力・戸愚呂弟 2体で最強の妖, Ushio & Tora 2nd Player Color Andy Bogard 2nd Player Color Athena Asamiya 2nd Player Color Benimaru Nikaido 2nd Player Color Billy Kane 2nd Player Color Kim Kaphwan 2nd Player Color Yuri Sakazaki 3rd Player Color Chin Getsai 3rd Player Color King 3rd Player Color Takuma Sakazaki 3rd Shinsengumi Unit Capt., Saito Hajime 5 Mechdragon Combo, Demon Hadar 5 Mechdragon Fusion, God Canopus 5-Ore Magic Stone Dragon, Mithril Edge 6聖球・サタンマリア 7th Heaven's Owner, Tifa 80%の力・戸愚呂弟 A member of Squad 13, Rukia Kuchiki 堕転したマギ・ジュダル 切札勝舞のスペシャルデッキ 刃龍喚士・リエト 寄道の親愛神・サクヤ 審美的転生注射, Zazan 師団長, Colt 帰ってきたサイヤ人, Vegeta 万天の全能神・ゼウス=ヴァース 三橋&伊藤【原作版】 三船東のエース・茂野吾郎 不破圓明流継承者・不破北斗 七代目武装戦線副頭・藤代拓海 七代目武装戦線頭・村田将五 快援隊名刺 忍ギガ満 志村妙 志村新八 呪紋の化身 エキドナロココ クリスタル・パラディン クリームヒルト ジャスタウェイ ジュスティーヌ&カロリーヌ ジョイラの使い魔 ジン=フリークス やさしい王様・ガッシュ&高嶺清麿 カイト カオス セラの天使 アクア・サーファー アイランドガチャドラ アラジン【原作版】 アテナの使命・沙織 ガンダー ガッシュ&高嶺清麿 ギガ満助 サウスポーの守護神・アテナ サイバー・N・ワールド サーティワン・エメリット サーティワン・アメリット サーティワン・サファリット サーティワン・愛猫神・バステト サーティワン・トパリット サーティワン・ルビリット サーティワン・ダブエメリット サーティワン・ダブアメリット サーティワン・ダブサファリット サーティワン・ダブトパリット サーティワン・ダブルビリット サーティワン・バステト サンタクロース ザ・ニンジャ ザブゴン ザブシャーク シェル・ファクトリーγ シェル・フォートレス シヴ山のドラゴン シャーマンカーン シャーマンラーン シーファン シンデレラ ゼオン&デュフォー ゼリーエンジェル スサノオ王子 スーパー覚醒マシンゼウス スーパー超覚醒ゼウス コカ・コーラたまドラ コルト隊兵隊長, Rammot コロッケ コッコ・ルピア あざ笑う雪だるま・ジャックフロスト 坂本辰馬 キャシー・クレイジー キューピッド キン肉族超人予言書 キリン 坂田銀時 坂田銀時 坂田There are ~40 of these in the shard, implying maybe ~1000 in full WebText.
rawdownloadcloneembedreportprint and friends originate in mangled Pastebin dumps, which are somewhat common in WebText, as I noted in the 2020 post.
This is also where I found the counting subreddit users. There are several docs in the shard which look like this:
1042k thread a guest Apr 7th, 2016 50 Never a guest50Never
Not a member of Pastebin yet? Sign Up , it unlocks many cool features!
rawdownloadcloneembedreportprint text 68.60 KB 4driue 1042001 (1042001) from Ynax at 2016-04-07 15:23:14 (id d1tmbyw) 1042002 (1042002) from CatchMeIYC at 2016-04-07 15:23:22 (id d1tmc5n) 1042003 (1042003) from Mooraell at 2016-04-07 15:23:54 (id d1tmd1b) 1042004 (1042004) from TheNitromeFan at 2016-04-07 15:24:03 (id d1tmdaz) 1042005 (1042005) from CatchMeIYC at 2016-04-07 15:24:16 (id d1tmdoh) 1042006 (1042006) from TheNitromeFan at 2016-04-07 15:24:29 (id d1tme1j) 1042007 (1042007) from cupofmilo at 2016-04-07 15:24:35 (id d1tme6r) 1042008 (1042008) from TheNitromeFan at 2016-04-07 15:24:43 (id d1tmees) 1042009 (1042009) from cupofmilo at 2016-04-07 15:24:50 (id d1tmelq) 1042010 (1042010) from CatchMeIYC at 2016-04-07 15:25:10 (id d1tmf6d) 1042011 (1042011) from TheNitromeFan at 2016-04-07 15:25:19 (id d1tmfey) 1042012 (1042012) from CatchMeIYC at 2016-04-07 15:25:30 (id d1tmfrb) 1042013 (1042013) from TheNitromeFan at 2016-04-07 15:26:10 (id d1tmgw4) 1042014 (1042014) from Mooraell at 2016-04-07 15:27:36 (id d1tmjct) 1042015 (1042015) from TheNitromeFan at 2016-04-07 15:28:11 (id d1tmkcm) 1042016 (1042016) from cupofmilo at 2016-04-07 15:28:28 (id d1tmkua) 1042017 (1042017) from TheNitromeFan at 2016-04-07 15:28:37 (id d1tml4h) 1042018 (1042018) from cupofmilo at 2016-04-07 15:28:46 (id d1tmld0) 1042019 (1042019) from TheNitromeFan at 2016-04-07 15:29:00 (id d1tmlr8) 1042020 (1042020) from cupofmilo at 2016-04-07 15:29:12 (id d1tmm45) 1042021 (1042021) from TheNitromeFan at 2016-04-07 15:29:23 (id d1tmmg2) 1042022 (1042022) from cupofmilo at 2016-04-07 15:29:28 (id d1tmmld) 1042023 (1042023) from TheNitromeFan at 2016-04-07 15:29:41 (id d1tmmzx) 1042024 (1042024) from cupofmilo at 2016-04-07 15:29:45 (id d1tmn34) 1042025 (1042025) from TheNitromeFan at 2016-04-07 15:30:05 (id d1tmno4) 1042026 (1042026) from cupofmilo at 2016-04-07 15:30:10 (id d1tmnrz) 1042027 (1042027) from TheNitromeFan at 2016-04-07 15:30:15 (id d1tmnxa) 1042028 (1042028) from cupofmilo at 2016-04-07 15:30:20 (id d1tmo1z) 1042029 (1042029) from TheNitromeFan at 2016-04-07 15:30:26 (id d1tmo83) 1042030 (1042030) from cupofmilo at 2016-04-07 15:30:30 (id d1tmoc7) 1042031 (1042031) from TheNitromeFan at 2016-04-07 15:30:36 (id d1tmoie) 1042032 (1042032) from cupofmilo at 2016-04-07 15:30:40 (id d1tmons) 1042033 (1042033) from TheNitromeFan at 2016-04-07 15:30:47 (id d1tmoue) 1042034 (1042034) from cupofmilo at
Note that TheNitromeFan appears 15 times in this example.
gmaxwell appears 32 times in this document, suggesting a possible source:
You are currently viewing all ratings received by user gmaxwell.
[view received] || [view sent]
[view negative] || [view all]
This user is currently NOT
AUTHENTICATED. This user has not authenticated for more than 238 days. If you are currently talking to someone who claims to be this person, you may be talking
to an impostor and scammer.
id rater nick rater total rating rated nick created at
(UTC) rating notes
10141 nanotube 801 gmaxwell 2012-04-05 04:12:50 6
generally trustworthy person, bitcoin dev.
14774 pigeons 248 gmaxwell 2012-09-15 13:25:31 3 he seems dedicated to the success of bitcoin
19465 Ssateneth 235
gmaxwell 2013-01-07 18:46:55 10 Kicks and bans scammers from #bitcoin-otc. Also, extra rating added to offset a negative rating from a pissed off scammer.
10182 copumpkin 229 gmaxwell 2012-04-08 16:56:10 8 not only do I trust him, but I have to counteract negative ratings that have very little to do with his
actual trustworhiness
7497 cory 222 gmaxwell 2011-10-23 02:40:10 1 He sent me a MtGox code in exchange for BTC
27672 Cusipzzz 195 gmaxwell 2013-07-19 21:18:55
7 very trustworthy, do not let the spam negative ratings fool you
10063 mircea_popescu 181 gmaxwell 2012-04-08 16:54:19 -10 hypocritical idiot.
10142 rg 159
gmaxwell 2012-06-11 19:52:28 1 you are a pain in my ass. :)
19063 TheButterZone 106 gmaxwell 2013-04-21 23:41:33 9 Warned me about continued use of an old
version of pseudo-client that would soon stop pushing valid transactions.
14534 jgarzik 95 gmaxwell 2012-09-08 16:45:57 8
13526 foggyb 88 gmaxwell 2012-08-11
02:10:16 3 made a donation on my behalf
19019 amiller 70 gmaxwell 2012-12-25 04:15:09 2 Met in person
18033 theymos 61 gmaxwell 2012-11-28 02:08:51 8
32581
iwilcox 61 gmaxwell 2013-12-14 19:28:01 2 Based on months of interactions; haven't transacted
14420 midnightmagic 54 gmaxwell 2013-09-04 00:32:07 6 Kind of a
hero of mine.
11643 Blitz 51 gmaxwell 2012-06-11 19:29:29 1 i love this guy
33637 Namworld 47 gmaxwell 2014-02-09 11:48:11 3 1|45 BTC|Gox instant withdrawal
service when gox withdrawals not working.
38067 chmod755 45 gmaxwell 2015-08-19 12:59:58 -10
12127 guruvan 43 gmaxwell 2012-06-26 20:53:43 1 highly respected
dev - definitely has his eye out for scams and things not good for your bitcoins :) never see him trade, but I trust this guy to be honest for sure.
30661
coingenuity 39 gmaxwell 2013-10-01 18:30:01 5 Great guy, trustworthy. Would do any size transaction.
19011 luke-jr 36 gmaxwell 2012-12-25 04:11:00 2 Seems
level-headed, met in person; not had the occasion to do business yet.
7536 vragnaroda 32 gmaxwell 2011-10-26 02:34:13 2
27666 anduck 28 gmaxwell 2013-07-19
19:12:30 3 trusted
23665 warren 27 gmaxwell 2013-04-11 19:11:48 10 Real person, bitcoin developer, otc op
20552 ATC 26 gmaxwell 2013-02-15 06:14:51 5 Helped
me save over 9.00 BTC stuck in my corrupted wallet. Thanks!!!
8123 nkr 24 gmaxwell 2011-12-12 18:21:03 1
14407 Vandroiy 23 gmaxwell 2012-09-06 20:04:53 2
Helps defend protocol and chat against nonsense. :)
33938 nkuttler 18 gmaxwell 2014-06-05 18:46:57 1 seems trustworthy
6375 cydeweys 14 gmaxwell 2011-07-25
17:49:45 8
20083 MoneypakTrader 13 gmaxwell 2013-01-28 22:34:39 -2 neg rated me based on opinion, msg after removed and I'll remove
7493 TehRabbitt 8 gmaxwell
2011-10-22 22:35
We see that all three models suffered a noticeable performance drop when going from non-anomalous to anomalous strings, but GPT2-xl considerably less so, despite the fact that GPT-J is a much bigger model. One hypothesis is that an anomalous token's closeness to the overall centroid in the relevant embedding space is an inhibiting factor in the ability of a GPT model to repeat that token's string.
Unlike the other two, GPT-J does not tie its embedding and unembedding matrices. I would imagine this negatively affects its ability to repeat back tokens that were rarely seen in training.
To check this, you'd want to look at a model trained with untied embeddings. Sadly, all the ones I'm aware of (Eleuther's Pythia, and my interpretability friendly models) were trained on the GPT-NeoX tokenizer or variants, whcih doesn't seem to have stupid tokens in the same way.
GPT-J uses the GPT-2 tokenizer and has untied embeddings.
This post provides a valuable reframing of a common question in futurology: "here's an effect I'm interested in -- what sorts of things could cause it?"
That style of reasoning ends by postulating causes. But causes have a life of their own: they don't just cause the one effect you're interested in, through the one causal pathway you were thinking about. They do all kinds of things.
In the case of AI and compute, it's common to ask
But once we have an answer to this question, we can always ask
If you've asked the first question, you ought to ask the second one, too.
The first question includes a hidden assumption: that the imagined technology is a reasonable use of the resources it would take to build. This isn't always true: given those resources, there may be easier ways to accomplish the same thing, or better versions of that thing that are equally feasible. These facts are much easier to see when you fix a given resource level, and ask yourself what kinds of things you could do with it.
This high-level point seems like an important contribution to the AI forecasting conversation. The impetus to ask "what does future compute enable?" rather than "how much compute might TAI require?" influenced my own view of Bio Anchors, an influence that's visible in the contrarian summary at the start of this post.
I find the specific examples much less convincing than the higher-level point.
For the most part, the examples don't demonstrate that you could accomplish any particular outcome applying more compute. Instead, they simply restate the idea that more compute is being used.
They describe inputs, not outcomes. The reader is expected to supply the missing inference: "wow, I guess if we put those big numbers in, we'd probably get magical results out." But this inference is exactly what the examples ought to be illustrating. We already know we're putting in +12 OOMs; the question is what we get out, in return.
This is easiest to see with Skunkworks, which amounts to: "using 12 OOMs more compute in engineering simulations, with 6 OOMs allocated to the simulations themselves, and the other 6 to evolutionary search." Okay -- and then what? What outcomes does this unlock?
We could replace the entire Skunkworks example with the sentence "+12 OOMs would be useful for engineering simulations, presumably?" We don't even need to mention that evolutionary search might be involved, since (as the text notes) evolutionary search is one of the tools subsumed under the category "engineering simulations."
Amp suffers from the same problem. It includes two sequential phases:
Each of the two steps takes about 1e34 FLOP, so we don't get the second step "for free" by spending extra compute that went unused in the first. We're simply training a big model, and then doing a second big project that takes the same amount of compute as training the model.
We could also do the same evolutionary search project in our world, with GPT-3. Why haven't we? It would be smaller-scale, of course, just as "GPT-7" is smaller scale than GPT-3 (but GPT-3 was worth doing!).
With GPT-3's budget of 3.14e23 FLOP, we could to do a GPT-3 variant of AMP with, for example,
100,000,000 evaluations per run (Amp) sure sounds like a lot, but then, so does 10000 (above). Is 1600 steps "not enough"? Not enough for what? (For that matter, is 50000 steps even "enough" for whatever outcome we are interested in?)
The numbers sound intuitively big, but they have no sense of scale, because we don't know how they relate to outcomes. What do we get in return for doing 50000 steps instead of 1600, or 1e8 function evaluations instead of 1e5? What capabilities do we expect out of Amp? How does the compute investment cause those capabilities?
The question "What could you do with +12 OOMs of Compute?" is an important one, and this post deserves credit for raising it.
The concrete examples of "fun" are too fun for their own good. They're focused on sounding cool and big, not on accomplishing anything. Little would be lost if they were replaced with the sentence "we could dramatically scale up LMs, game-playing RL, artificial life, engineering simulations, and brain simulations."
Answering the question in a less "fun," more outcomes-focused manner sounds like a valuable exercise, and I'd love to read a post like that.
uses about six FLOP per parameter per token
Shouldn't this be 2 FLOP per parameter per token, since our evolutionary search is not doing backward passes?
On the other hand, the calculation in the footnote seems to assume that 1 function call = 1 token, which is clearly an unrealistic lower bound.
A "lowest-level" function (one that only uses a single context window) will use somewhere between 1 and tokens. Functions defined by composition over "lowest-level" functions, as described two paragraphs above, will of course require more tokens per call than their constituents.
An operational definition which I find helpful for thinking about memorization is Zhang et al's counterfactual memorization.
The counterfactual memorization of a document is (roughly) the amount that the model's loss on degrades when you remove from its training dataset.
More precisely, it's the difference in expected loss on between models trained on data distribution samples that happen to include , and models trained on data distribution samples that happen not to include .
This will be lower for documents that are easy for the LM to predict using general features learned elsewhere, and higher for documents that the LM can't predict well except by memorizing them. For example (these are intuitive guesses, not experimental results!):
Note that the true likelihood under the data distribution only matters through its effect on the likelihood predicted by the LM. On average, likely texts will be easier than unlikely ones, but when these two things come apart, easy-vs-hard is what matters. is more plausible as natural text than , but it's harder for the LM to predict, so it has higher counterfactual memorization.
On the other hand, if we put many near duplicates of a document in the dataset -- say, many copies with a random edit to a single token -- then every individual near-duplicate will have low counterfactual memorization.
This is not very satisfying, since it feels like something is getting memorized here, even if it's not localized in a single document.
To fix the problem, we might imagine broadening the concept of "whether a document is in the training set." For example, instead of keeping or removing an literal document, we might keep/remove every document that includes a specific substring like a Bible quote.
But if we keep doing this, for increasingly abstract and distant notions of "near duplication" (e.g. "remove all documents that are about frogs, even if they don't contain the word 'frog'") -- then we're eventually just talking about generalization!
Perhaps we could define memorization in a more general way in terms of distances along this spectrum. If we can select examples for removal using a very simple function, and removing the selected examples from the training set destroys the model's performance on them, then it was memorizing them. But if the "document selection function" grows more complex, and starts to do generalization internally, we then say the model is generalizing as opposed to memorizing.
(ETA: though we also need some sort of restriction on the total number of documents removed. "Remove all documents containing some common word" and "remove all but the first document" are simple rules with very damaging effects, but obviously they don't tell us anything about whether those subsets were memorized.)
Hmm, this comment ended up more involved than I originally intended ... mostly I wanted to drop a reference to counterfactual memorization. Hope this was of some interest anyway.
Interesting stuff!
In this toy model, is it really the case that the datapoint feature solutions are "more memorizing, less generalizing" than the axis-aligned feature solutions? I don't feel totally convinced of this.
Two ways to look at the toy problem:
There are some details of the toy model that put (2) on a "different footing" from (1).
Since the input and output use the same basis, if we make a change of basis, we have to change back again at the end. And because the weights are tied, these two operations have to be transposes, i.e. the change of basis has to be a rotation.
As illustrated in the Colab, requiring the data to be orthonormal is sufficient for this. The experiment constrained the data to unit norm, and it's close to orthogonal with high probability for .
Now, it happens that (1) is the true data-generating process, but the model has no way of guessing that. In the finite-data case, the data may be consistent with multiple data-generating processes, and a solution that generalizes well with respect to one of them may generalize poorly with respect to another.
To designate one data-generating process as the relevant one for generalization, we have to make a value judgment about which hypotheses are better, among those that explain the data equally well.
In particular, when , hypothesis (2) seems more parsimonious than hypothesis (1): it explains the data just as well with fewer features! The features aren't axis-aligned like in (1), but features in real problems won't be axis-aligned either.
In some sense, it does feel like there's a suspicious lack of generalization in (2). Namely, that no generalization is made between the training examples: any knowledge you gain about a feature from seeing one example will go unused on the rest of the training set. But if your data is small enough that is almost entirely orthogonal, hypothesis (1) has the same problem: the feature weight in each training example has almost no overlap with the other examples.
The answer to each these questions is either "yes" or "tentatively, yes."
But the evidence doesn't come from the Chinchilla paper. It comes from the earlier Kaplan et al papers, to which the Chinchilla paper is a response/extension/correction:
If you want to understand this post better, I'd recommend reading those papers, or a summary of them.
This post, and the Chinchilla paper itself, are part of the "conversation" started by the Kaplan papers. They implicitly take some of the results from the Kaplan papers for granted, e.g.
So, later work tends to be casual about the distinction between "the curve we fit to the data" and "the law governing the real phenomena." (Theoretical work in this area generally tries to explain why LM loss might follow a simple power law -- under the assumption it really does follow such a law -- rather than trying to derive some more complicated, real-er functional form.)
Hmm, I think these days the field views "language modeling" as a means to an end -- a way to make something useful, or something smart.
We're not trying to model language for its own sake. It just so happens that, if you (say) want to make a machine that can do all the stuff ChatGPT can do, training a language model is the right first step.
You might find models like DALLE-2 and Stable Diffusion a helpful reference point. These are generative models -- what do they for images is (handwaving some nuances) very close to what LMs do for text. But the people creating and using these things aren't asking, "is this a good/better model of the natural distribution of text-image pairs?" They care about creating pictures on demand, and about how good the pictures are.
Often, it turns out that if you want a model to do cool and impressive things, the best first step is to make a generative model, and make it as good as you can. People want to "make number go down," not because we care about the number, but because we've seen time and time again that when it goes down, all the stuff we do care about gets better.
This doesn't fully address your question, because it's not clear that the observed regularity ("number goes down -- stuff gets better") will continue to hold if we change the distribution we use to train the generative model. As an extreme example, if we added more LM training data that consisted of random numbers or letters, I don't think anyone would expect that to help.
However, if we add data that's different but still somehow interesting, it does tend to help -- on the new data, obviously, but also to some extent on the old data as well. (There's another Kaplan scaling paper about that, for instance.)
And at this point, I'd feel wary betting against "more data is better (for doing cool and impressive things later)," as long as the data is interestingly structured and has some relationship to things we care about. (See my exchange with gwern here from a few years ago -- I think gwern's perspective more than mine has been borne out over time.)