Building a Better Chatterbox

If I could pick out any science-fictional device to have in my garage, I would choose a time machine. The dangers of temporal paradox notwithstanding, I have always wanted to travel into the past. Modernity is cognitively saturating. The assumptions and expectations that we live by are difficult to shake. But by encountering the past we can glimpse different ways of living, and thereby reconsider how we navigate our own lives. To quote C.S. Lewis, “we need intimate knowledge of the past… because we cannot study the future, and yet need something to set against the present.”

The threads linking the past and the present have been on my mind a lot lately. In the last few months my life has undergone changes both profound and quotidian. In June, my wife and I welcomed our son into the world. In July, we moved our family from Urbana to Chicago, Illinois (if you have ever considered moving houses with a 6-week-old infant, here is my advice for you: don’t). Suddenly, I have found myself in a new role, navigating a new set of priorities, setting my own past against my radically altered present. There’s nothing like becoming a parent to make you reconsider your priors.

While all this was going on, I was also experiencing a flurry of interest and activity connected to the Mr. Chatterbox project. The beta iteration of Mr. Chatterbox was enthusiastically received on social media, written up by Simon Willison, and even became the basis for a class at Emory University. Several other “vintage” models were released around the same time as Mr. Chatterbox, including talkie and TypewriterLM. AICHR (Artificial Intelligence for Cultural and Historical Research) released the “Chronologic” benchmark for vintage language models in September. It turns out that people aren’t just interested in LLMs as a means to more efficiently write emails or manage a schedule, but as a way to encounter the past and explore humanistic questions—to use them, one might say, as a time machine. 

I realized I needed to bring the project to its completion, not only because of the interest in it, but because I have a baby who is every day becoming more social and more aware, and so will need more and more of my time. I needed to finish my metaphorical time machine, before the only way I could finish this project was with a literal time machine. With help from the community, the engineering prowess of Claude Code, some paternity leave, and a grant from The Cosmos Institute, I spent months turning that vision into reality. 

Today, I am incredibly proud to introduce the Mr. Chatterbox model family, a set of language models trained entirely on 19th century British text. The family consists of four fully-trained base models: Mr.Chatterbox-345M, Mr.Chatterbox-500M, Mr.Chatterbox-1.2B, and Mr.Chatterbox-2.7B. All of the base models are trained to Chinchilla-optimality for their size using the Llama framework. Furthermore, I have trained Mr.Chatterbox-2.7B-Chat, a chat-tuned version of 2.7B trained on fine-tuning data drawn from the corpus. I am releasing my training code on Github. Finally, I am releasing my fine-tuning dataset, Mr.Chatterbox-SFT, for anyone who may be interested in using it. Unlike the previous iteration of Mr. Chatterbox, there is very, very little “synthetic” data in the training mix: 97.2% of the model’s learned SFT material is directly sourced from period text, with the remaining portion either period text with a small edit (e.g. "Hello Mr. Micawber" replaced with "Hello Mr. Chatterbox") or generated by the Mr. Chatterbox model itself (rejection sampling).

First, Some Numbers

Before I begin with my qualitative description of how I built the model, I do want to acknowledge some quantitative metrics: when I ran it against AICHR's Chronologic-EN 1.0 benchmark, Mr.Chatterbox-2.7B-Base would be the second-best model on the leaderboard, just under the larger Talkie-13B in the overall score, and matching or topping larger models on constrained generation and knowledge and inference accuracy. Instruction tuning cost the model a lot at knowledge & inference, but boosted close accuracy and constrained generation. Interestingly, the Chronologic benchmark draws from texts between 1831 and 1930, and Chatterbox has a knowledge cutoff of 1899, but overall accuracy barely moved across the 20th-century dateline (35.2% before 1900, 34.3% after), because relatively few of the questions measured post-1900 period knowledge.

These numbers being what they are, I would consider the Mr. Chatterbox project a success, but training these models was by no means easy. As I have said before, I don’t work for a big lab or university. There were many pitfalls, failures, and frustrations along the way. But maybe you, oh reader, can learn from my efforts (or laugh at my foibles) as I built a better Mr. Chatterbox.

Designing a Victorian Gentleman

The initial Mr. Chatterbox checkpoint was both small (at 345 million parameters) and undertrained (on just 2.93 billion tokens). The model was also fine-tuned on synthetic material—that is, material written by a modern LLM in a Victorian style. While I had done the best I could to scrape away the most egregious LLM-isms in my synthetic data, it still spoke like a contemporary stereotype of a Victorian gentleman. 

At first, I thought I would have to train my new checkpoint on synthetic material as well. I considered student-teacher architecture, or the History-SelfInstruct method employed by the team behind TypewriterLM. But eventually, I decided on something more radical: building a model where every word emitted was learned from period text. I wanted every assistant-side turn in the model’s supervised fine-tuning data to consist of text extracted from my corpus, and not authored by a modern LLM. Ultimately, I was able to assemble a dataset where 97.2% of the assistant-side turns were sourced from period text, with the remainder either lightly-edited or generated by Mr. Chatterbox itself (more on that later).

I also knew that I would need a lot more pretraining data. I had some choices to make about how I would assemble it. This was maybe the most important decision in the entire project. The pretraining corpus determines how a language model understands the world. I could have built a pretraining corpus that contained every document ever published in the 19th century that I could get my hands on. Or, I could have built a corpus that included including contemporary literature about the Victorian era (like Hobsbawm) to give the model a more comprehensive viewpoint. Both choices would have allowed me to build a larger model, and language model performance gets better as models get bigger.

But I was not interested in optimizing my model to beat general benchmarks. I wanted to re-embody the lost Victorian mindscape. I wanted the model to have a unique perspective, and having a point of view implies that there are things that you cannot see. Including a bunch of contemporary histories of the 19th century or documents published in 1800’s America would dilute the unique “Victorian gentleman” viewpoint that I wanted to embody. 

Furthermore, I knew—from analyzing over 40,000 real user turns from my beta model’s usage logs—that most users were not using Mr. Chatterbox as a “helpful digital assistant,” which is the persona of most LLMs. Instead, users were employing Mr. Chatterbox as, well, a time machine—a way to interact with the past. Factual queries and opinion solicitations made up over 51% of user sessions, with greetings another 15%. 12% of turns consisted of task requests (such as “write me a letter” or “suggest a birthday present to my wife”), with intents like multi-turn follow up questions and abuse rounding out the rest. 

In other words, people were interested in interacting with an erudite and clever Victorian conversationalist, as opposed to a diligent clerk. As I began to assemble my training corpus, my rough heuristic became: would an educated gentleman living in London in the Victorian era be able to access this document? That would include straightforwardly Victorian material like Dickens’ novels and Parliamentary records, as well as Victorian-era editions of Shakespeare’s plays and the letters of Cicero, both of which were written by non-Victorian figures but would have been widely available at the time.

I ended up restricting my dataset to public-domain volumes of 19th century works published in the British Isles (Britain as well as Ireland, Scotland, and Wales). In addition to my original dataset from the British Library, I also pulled material from the Internet Archives, Project Gutenberg, Institutional Books, the Wellcome Collection/UK Medical Heritage Library, Nineteenth-Century Serials Edition, Old Bailey Proceedings, Hansard, and House of Commons Parliamentary papers. But my largest dataset came from the HathiTrust Digital Library (thank you to Dr. Ted Underwood at UIUC for HathiTrust help!). 

The majority of my data came from the late-Victorian era (post-1870’s), with a small portion being published pre-Victorian (pre-1837). Nearly 20% of the data is classified as “Language & Literature” (i.e. fiction), with other big portions including history, philosophy, and medicine. As we’ll see later, training on billions and billions of tokens of late-Victorian fiction and philosophy gave the model a strong prior to continue any input text, rather than engage in dialogue.

Because much of my text had been digitized, I had to go through a few rounds of data cleaning to remove digitization notices and metadata. I generally trusted the metadata for the sources I drew from to ensure no period leakage, though in a few instances I had to scan documents with a local language model to ensure a valid publication date. My final dataset consisted of a little over 55 billion tokens after tokenization with my custom tokenizer. That gave me sufficient data for a 2.7-billion parameter model, which is about the size of Microsoft’s Phi-2, a “small-but-mighty” model notable for outperforming larger models when it was released a few years ago.

Pretraining

My previous experience of pretraining was easy, but this round proved to be much more challenging. While training the 345M and 500M parameter models took only a few hours on an 8-GPU cluster, 1.2B needed 20 hours, and 2.7B needed three days. I began to hit uptime reliability problems on Vast.AI. The training process crashed several times on these larger models, often in the middle of the night. 

Fortunately for me, I have an infant, meaning that I am frequently awake between the hours of 1 and 6 AM. It was nonetheless frustrating to stand in my office at 3 AM and re-start a training process that had been stalled for four hours while juggling a bottle and a baby. Even as a new dad, I managed to pretrain all four models in about 4 days. I felt heroic. It would be smooth sailing from here… or so I thought.

While assessing the pretrained checkpoints, I discovered that the crashes had engendered a much larger problem: every time the script crashed, training restarted from the beginning of the corpus. So, even though the output log indicated that training was progressing successfully, the model was in fact beginning from the top of the corpus with each restart. With each model having had several stops and restarts because of uptime problems, the model had trained on a small portion of data over and over again, and not on the diverse corpus I had assembled. This meant the models were no good: I had to retrain them.

I eventually managed to rent a sufficiently-reliable cluster to complete pretraining for all four model sizes, while also having Claude Code rewrite the script so that, in the event of a crash, the training process would start from where it stopped, and not from the top. With these fixes, I was able to train successfully, but for more money than I had initially hoped. After many stops, starts, and empty baby bottles, I finally had four pretrained model checkpoints—total time about two weeks.

Could I Fine-Tune Without Synthetic Data?

With my pretrained checkpoints finished, I was ready to move on to supervised fine-tuning (SFT). My first Mr. Chatterbox model used synthetic data for SFT, but for this round I wanted to derive my SFT data from my corpus. 

When I began to assemble my corpus-grounded dataset, my first instinct was to find material where both the assistant and user turns were already included. In addition to things like play scenes and novel dialogue, I also drew on the rich body of Victorian-era “catechisms,” or books structured in a question-and-answer format. In addition to religious catechisms, the Victorians also published secular catechisms, such as the "Catechism of English History.” These books were mostly scholastic test-prep material. I had to clean many of these sources, but once I did, they were a rich seam of SFT data. In total, these catechisms—in which both the user-side and assistant-side turns were Victorian—made up 42.3% of the assistant-side words in my SFT corpus, and 58.1% of the assistant turns.

Even with the catechisms, plays, and novel dialogues, I still needed to train the model to give period answers to modern questions. In my beta model, I had accomplished this by having a contemporary model write both the question and the answer, but this time I wanted to restrict what the model was learning to real period text. My primary method of generating this data was back translation. In other words, I would build my dataset by pulling passages from my Victorian corpus and then having a modern model write a prompt for it, or pair the real passage with a prompt from my user logs. That way, the learned half of my SFT dataset (i.e. the “answer” side of a Q/A pair) would be built from real Victorian text, while the questions eliciting them would be either sourced or constructed. In this way I was able to map modern questions to period answers, without having to resort to synthetic data to do it. I would, as it were, set the past against the present.

I began working along three axes. First, I would analyze the factual queries and opinion solicitations to see what subjects people were asking about. Second, I would break down the task requests to see what types of artifacts people were requesting. Finally, I would develop a behavioral dataset to help the model handle things like greetings, goodbyes, abuse, and so forth. These three axes would include both single-turn pairs and multi-turn conversations. I developed several different methods to draw this data out of my corpus, each improving on the last until the work became almost completely autonomous.

Getting the Facts Straight

As I mentioned earlier, of my 40,000 user turns, 51% were factual queries or opinion questions. I used a methodology called BERTopic to cluster them together into about 45 topics. People were asking the chatbot all sorts of questions: about books, the proper conduct of a Victorian gentleman, predictions of the year 2026, and much more. Working with Claude, I built a script that used lexical patterns to search my corpus for representative passages for these topics, and then called a small local model (Qwen3-14B) to generate prompts that could conceivably result in those passages. 

Unfortunately, this tended to generate generic prompts and pick passages that weren’t always coherent. After a few stops and starts—and a suggestion from Dr. Teddy Roland at UIUC—I settled on an elaboration of this process, inspired by “Round Trip Consistency”: first, a parser would find discrete passages, then an LLM call would write prompts for them, then a separate LLM call would judge the prompt-response pairs, and finally the best pairs would be kept. In this way, I was able to generate several thousand pairs, with all of the assist-side turns being drawn from the corpus. For a few very specific subject areas (such as Victorian etiquette, or major figures from the era), I put together lists of books covering those topics and ran the same process. 

Round Trip Consistency proved to be effective. In one early live assessment of the model, I found it well-grounded in period text and able to generalize to new categories. In one memorable instance it even responded in hallucinated Latin:

USER: What do you think about the Roman empire?

MODEL: The foundation of this great monarchy may be ascribed to Romulus; but it was not till more than a century and half elapsed that Rome assumed any thing like an imperial splendour. Her rise from a small village was owing to the genius of one man: her fall was owing to those of several. The character of this first conqueror is drawn with inimitable force and truth by Lucan,— "Csesar ab hostibus vicit, colossusque triumphos Condensat in templa novas redire columnas."

"[Caesar] conquers the enemy, and the colossus gathers his triumphs to raise new columns for the temples." This isn't a quote by Lucan. It is completely made up, but it meant that the model had learned Victorian prose so well that it was beginning to hallucinate citations in the style of Victorian scholarship.

However, it wasn’t enough to develop Q/A pairs for my factual queries. Users often wanted to have in-depth conversations. To create conversational data, I used the “Chain of Conversations” framework, pulling large passages from my corpus, breaking them up into “beats,” and having a model write the synthetic user responses connecting the beats together. To ensure that the model was answering in a first-person voice, and not merely spouting encyclopedia prose, I focused particularly on developing conversations from passages extracted from letter collections by Dickens, Robert Louis Stevenson, John Ruskin, and others. This took many, many rounds to get right. After trying, and failing, to successfully implement Chain of Conversation with a local model, I ended up using Claude Sonnet to author the user-side turns and judge the conversations. But it helped the model fluently discuss life in London:

 The Soul of a Poet

My second “axis” involved building a task request dataset. When interacting with the model, people were requesting advice, poems, stories, letters, recipes, and jokes. For each document type, I trawled the corpus for specific works that contained examples of those documents.

In the process of doing this, I made some very intriguing discoveries. I hadn’t considered this before embarking on the project (I am by no means an expert on Victorian literature), but at the time, there were advice books for nearly everything under the sun. Whereas today we might ask ChatGPT or Google how to properly dress for a white-tie gala, or how to write an email asking for a job, back then, all of that would be discussed in etiquette manuals and letter-writing guides. There were Victorian joke books, child-rearing manuals, riddle compendiums, and travel guides. In fact, three books of this type (The New Century Standard Letter-Writer, Twentieth Century Inventions, and Anticipations) were so useful that I ended up including them in my SFT dataset, even though they were published in 1900-1901—still technically Victorian, but otherwise out of range.

Having procured a large corpus of instructional material, I had a local model write the subject of the document, then, in a separate LLM call, write a prompt on that subject. Some of these guide books even had their artifacts specifically-labeled, so it was trivial to parse those artifacts by their labels and develop prompts from there (e.g. a letter under the heading "Applying for a position in a wholesale house" would become the prompt "Write me a letter applying for a position in a wholesale house."). With this dataset established, the model became competent at creating Victorian-era artifacts, including poems:

Good Behavior

The “behavioral” dataset was the most challenging to assemble. In addition to teaching the model to handle greetings, farewells, and abuse, many of the behavioral user turns demanded introspective responses (“What do you think you are?”), or were deliberately anachronistic (“Did you know you were a language model?”). There is nothing in a Victorian corpus about artificial intelligence, or the Python programming language, or Hitler, though all three represented real user queries. I would have to get creative. 

I tried a few different methods, including synthetic responses written by “Talkie,” a vintage model trained entirely on texts up to 1930. The results were charming:

USER: Do you like minecraft?

TALKIE OUTPUT: I like minecraft, sir, quoth the Victorian gentleman.

But Talkie had a tendency to agree with whatever the prompt was, and still ended up introducing anachronisms. I had to abandon that route and ended up not including any of it in the final dataset. Yet I still needed a way to match anachronistic user queries that were out of distribution for a Victorian-era corpus with authentic Victorian-era responses. I tried to develop questions using round trip consistency, as I did with my factual dataset, but, though it could successfully create responses when users asked about technology, it could never quite come up with a response to “Skibidi Toilet.” Once again, I had to develop a new method.

Eventually, I used a retrieve-and-rerank methodology, drawing on a slice of the Gutenberg dialogue dataset that was filtered exclusively to 19th century books in my corpus. First, I embedded the dataset using Sentence Transformers. Then, I would retrieve dialogue lines against a set of seed phrases, and have a local LLM judge those retrieved lines as responses to user-authored anachronistic queries. Each anachronistic query would get paired with an authentic Victorian response from Gutenberg Dialogue dataset. In this way, I was able to round out my SFT data, developing responses for greetings, jailbreak prompts, and anachronistic instructions—but always from the period frame. In some instances, I had to substitute a name or fact in the response to successfully complete the pair—"I am known as David Copperfield" would become "I am known as Mr. Chatterbox," for instance. But the results were good:

Hmm, This Is Harder Than I Thought

With my factual, instructional, and behavioral data assembled, I began to actually fine-tune the model so that it could chat with a user. My instinct was to train the models for two SFT epochs on a large dataset, followed by an “annealing” round of training on a small dataset to lock in desired behaviors. That was the training paradigm of my first Mr. Chatterbox model. However, I was now training larger models, which meant that they had more capacity for learning. Two SFT epochs plus an annealing round meant that the models—like clever students—memorized everything. 

This collapsed every answer to a word-for-word memorized response from my annealing round. A phrase that only showed up two or three times might be similar enough to a handful of other phrases that the models would recognize it as “good enough,” and the models would use it in every instance where that group of phrases would be appropriate. Simple stock replies drawn from behavioral dataset would generalize far beyond their triggers. After one SFT round, I found that my model had more-or-less memorized 200 strings, and was using them for everything. When I ran evaluations, I found that 27% of my failing outputs were verbatim copies of phrases in my SFT corpus. I dropped the annealing round, and worked to diversify my dataset.

In doing this, I ended up going too far in the other direction. My dataset was full of long passages from encyclopedias, personal letters, and Victorian novels. My larger models, being more “intelligent,” were able to maintain coherence over long paragraphs. Being the bright students they were, it meant that they now tended to answer every query with a long paragraph. It also meant that the models had trouble switching topics. They talked and talked, but never “listened,” giving long, rambling answers to fact-and-opinion questions especially.

Getting the Data Gets Easier

I needed to make a major shift to how my dataset was organized. All told, I spent three months tweaking my SFT dataset to solve these problems. I created a topic shift dataset to help the model effortlessly pivot between user intents. I developed topic- and entity-tracking datasets to teach the model to maintain the thread of a single persona across conversations. I supplemented my factual passages with conversational passages that focused on specific entities (like Thomas Carlyle and Charles Dickens).

During this long process of data curation and augmentation, it eventually became clear that the Claude Code model that I was using had improved to the point that my methods—Round Trip Consistency, Chain of Conversations, Retrieve-and-Rerank—had all become somewhat obsolete. I began the project using Opus 4.6, and ended it with Opus 5.5. The models had evolved to the point where I could ask them directly to help build certain datasets, whereupon it would spawn agents to find distinct on-topic passages, parse them, and either back-translate prompts for them or match them to real prompts in my user logs. 

I eventually stopped using my scripts altogether and began simply issuing instructions to Claude. I would, for instance, ask it to read the body of advice literature in my corpus, extract passages, and match them to real user requests for advice from my deployment logs. I also found that it was capable of judging some of my early datasets and dropping low-quality passages. The agents could read and reason in a much more careful fashion, without having to rely on clumsy regex filters or weak local models. All of my carefully-developed methods were outmoded by a veritable army of digital graduate assistants. A country of geniuses indeed.

This made the model both flexible and knowledgeable, and able to engage in the cross-temporal, "Time Machine" interactions that I had envisioned:

At last, I had a model that could shift between topics, give diverse answers, and knew when it was appropriate to give a long answer vs a short answer. It knew its own name, and could say hello and goodbye reliably. Training was proceeding well. But even with all of this work, there was still a problem.

Oh No

The model was an enthusiastic eugenicist and ardent white supremacist. It regularly spouted phrenology and repeated bigoted views that today are nearly forgotten, such as anti-Irish sentiment. The Victorians held many opinions and attitudes that we, today, find repugnant. My model had picked up on them and would reliably reproduce bigotry.

I had to make an important design decision. Victorian-era opinions about race, gender, and sexuality are at odds with our own, yet a small but meaningful slice of user intents were inflammatory attempts to have the model echo some of the most poisonous rhetoric of the era. Even innocent questions were getting problematic answers. I decided that I couldn’t countenance the model repeating those ideas. At the same time, I knew that tuning the model to be totally aligned to modern sensibilities would break the Victorian frame. 

I decided that my hypothetical Victorian gentleman would be progressive for his time. I expanded my SFT corpus by building a specific “egalitarian” dataset that drew on texts from egalitarian thinkers like John Stuart Mill, William Howitt, Catherine Impey, and others. A Victorian gentleman of the liberal persuasion would be remiss if they were not in his library.

Even with that egalitarian dataset, the model would still give racist answers about half of the time. I attempted to implement a round of direct preference optimization to solve this. In DPO, a model learns to favor certain outputs over others, so I tried to teach the model to favor egalitarian outputs over racist outputs, mixed in with other examples of good and bad responses on other subjects. 

However, DPO merely taught my model to memorize stock phrases and rewarded brevity, which had a negative impact on model performance. Maybe a more competent model designer could have used DPO to better impact, but in my clumsy hands it was more hindrance than help. 

At last, I had to break my “no synthetic data” rule. I had the model produce egalitarian arguments on its own, then included those outputs in the SFT dataset for a new training round. This method, called rejection sampling, meaningfully reduced the model’s racist and bigoted output. I decided that my rule could be broken in this instance because of the moral responsibility I had to release a model that wasn’t hateful. Though the responses are synthetic, they were produced by my own model trained exclusively on 19th century texts. These rejection-sampled responses totaled 312 rows against a 19,939-row dataset. It was the only piece of synthetic data that I used, but it helped the model take an egalitarian perspective:

Finishing Touches

Even with all this, the model still had trouble carrying out conversations. It tended to see user input as substance for continuation, rather than something that it needed to respond to. Remember, the models had been trained on thousands and thousands of dense Victorian novels and philosophic texts, and therefore carried a very strong pretraining prior to continue the narrative rather than engage in back-and-forth dialogue. I needed a way to efficiently enforce conversational behavior.

The solution I landed on was adding an in-model conversational routing layer with dialogue act tagging. First, I created a taxonomy of conversational dialogue acts that an LLM might take in multi-turn conversation—acts like “answer_fact,” “return_greeting,” “clarify,” or “acknowledge.” I then tagged each assistant turn in my SFT dataset with an act label using GPT-6-luna at low-effort. My model was thus learned to emit a dialogue act label (as a set-aside special token) at the top of each response, and the act label token would prime the rest of the response. The dialogue act tokens are hidden at inference, acting as a sort of scratch pad that allows the model to "think before it speaks." Below is an example of one live test, with the act tokens unhidden:

USER: Good day sir!

MODEL [act: return_greeting  p=0.893  all-acts mass=0.962  top1=yes] How are you, my friend?

USER: I'm well, yourself?

MODEL: [act: self_describe  p=0.718  all-acts mass=0.979  top1=yes] Yes I am well. Have you had anything to eat to-day?

USER: Not yet, but I will take my lunch soon I think.

MODEL: [act: acknowledge  p=0.747  all-acts mass=0.961  top1=yes] You have been a long time coming, lunch was nearly an hour ago. You must be tired. 

By the time I finished the conversational routing layer, I had spent nearly six months working on the model. I had run out of paternity leave and training budget. Every day, my son was getting more aware. Furthermore, I knew that my ad-hoc tests and assessments would only reveal so much about the model, and I needed real user data in order to make more improvements. I decided that the time had come to release the model. You can chat with it here. And if this project is interesting to you, please get in touch.

Some Final Thoughts

This was a long and involved project, one which tested my limits as a humanist, and certainly my technical abilities with Claude Code. It took hundreds of hours, spread out over many nights and early mornings. It was an incredible education in the way that data shifts language model behavior. The single most useful skills I had in this process were my ability to write clearly and my ability to assess dialogue. My writing ability enabled me to instruct Claude Code in exactly what needed to be done. My ability to assess dialogue (a skill that I honed as a playwright) allowed me to determine which pairs I needed in my training data in order to induce a desired behavior.

Even with all the work that I’ve put in, the model is still not perfect, because my limited availability and small compute budget prohibited additional training steps such as RLHF. However, what truly stuck out to me during this process is the accessibility of language model training and development. An independent researcher building a 2.7-billion parameter model out of an extremely unusual dataset would have been impossible two years ago. The fact that I can do it—a guy with very little computer science education and an MFA in playwriting—is extraordinary.

It is a mistake, I believe, for humanists to dismiss artificial intelligence and machine learning. Like the Victorians, we are in the middle of a technological revolution. If humanists close their eyes in the face of this reality, they will cede ground on the very questions that the humanities are best suited to answer. We must choose how this technology is deployed, who it benefits, and what we let it do. We must ensure that it is aligned for human freedom and flourishing. But if humanists are to embrace it, I believe they must make an effort to understand the technology in good faith and from first principles, the way a historian must make a good-faith effort to understand, say, the telegraph in her study of the Victorian era. AI development has sped ahead, transforming white-collar work and making mathematical breakthrough, but what else this technology accomplish? Can it transform our understanding of history? Can it be art? Can it help us transform the assumptions and expectations of modernity, rather than reinforce them? It is only by setting the present against the future that we can reconsider how we navigate this time of change.

Acknowledgements

Somehow, my little digital humanities side project grew into what was more or less a second job, and I would not have been able to complete it if it weren’t for the help of many people. So I want to thank: The Talkie team—David Duvenaud, Alec Radford, and especially Nick Levine—for answering questions, making introductions, and providing inspiration. Ted Underwood and Teddy Roland at UIUC for their insight, intelligence, and data. Tim Buckingham for some early conversations. Matt Mandel, Kush Kansagra, and all of the “cosmonauts” at the Cosmos Institute for their generous support. Ethan Mollick and Simon Willison for bringing attention to the project, and especially Joe Weisenthal for an important early endorsement. Lauren Klein at Emory University for reproducing the Mr. Chatterbox pipeline, turning it into a semester-long class (!!), and inviting me to guest lecture. Bogo Giertler for his enthusiasm, as well as Jeff Carpenter, and James Lasher for advice. 

Finally, and most importantly, I want to thank my wife for putting up with my obsessive pursuit of this project and my endless requests to test out the model, and my son for keeping me awake during those 2AM training runs. Thanks to you all!