I hope so 😊. And if they are not challenging enough, I do also have plenty of bonus material in the GitHub repo here: https://github.com/rasbt/LLMs-from-scratch
This may also answer the question regarding the templates to modify 😊. Actually, when I first started writing the book I wanted to move some of the codes that are reused throughout the chapters into a Python package that readers can pip-install and import from, but then I decided against it and made the code in each folder self-contained so it’s easier to just go in and edit, which is nice for tinkering.

Thank you so much Sebastian. Sounds very good to me.

You are absolutely correct, I think there will only be a few experts truly training LLMs from the ground up. However, I think it’s important to understand the process on a general level and have good understanding of how the architectures look like. Because this helps making decisions which LLMs to use in practice since there are more and more options out there.
Just as an example, Gemma-2 has a 2x larger vocabulary than Llama 3.2. This is nice because this way you can fit more tokens into the context window, but the trade-off is that the output projection layer will require more GPU memory. And there are lots of these insights or understandings that come from a solid understanding of how LLMs work. But for that, you don’t need to be an expert in building LLMs on a daily basis though.

Hi Low Kim Hoe! This is an absolutely warranted question, because it’s a bit of a time investment. However, I think it will pay off! I answered a related question, and I think the answer would maybe apply here as well:
> However, I think it’s important to understand the process on a general level and have good understanding of how the architectures look like. Because this helps making decisions which LLMs to use in practice since there are more and more options out there.
>
> Just as an example, Gemma-2 has a 2x larger vocabulary than Llama 3.2. This is nice because this way you can fit more tokens into the context window, but the trade-off is that the output projection layer will require more GPU memory. And there are lots of these insights or understandings that come from a solid understanding of how LLMs work. But for that, you don’t need to be an expert in building LLMs on a daily basis though.
Please let me know in case you have any follow-up questions!

Thanks for asking! Actually, there’s currently no help required; I am mainly typing up some of notes in a nicer formats and sometimes also address reader questions with those. I have a list of some more that I plan to add over the next few months. Thanks!

> In your opinion, what are the most promising trends in LLM development? Beyond simply increasing model size, what new directions do you see for LLM research? Are there specific areas, like interpretability or reasoning, or domain-specific enhancements that excite you?
The architecture changes of the state-of-the-art LLMs are actually relatively minor. Most of the progress seems to come from improving the training recipes. I wrote more about it here (https://magazine.sebastianraschka.com/p/new-llm-pre-training-and-post-training) if you are interested.
> What do you consider the biggest challenges to further improving LLMs? Are there intrinsic limitations that we may never be able to overcome with this technology? For instance, could there be a point where scaling up no longer yields significant gains?
Good question. Actually, I think we already stopped scaling up. E.g., after the release of GPT-4 I don’t think OpenAI (or others) released a bigger more scaled up model (haha, I may be proven wrong if/when GPT-5 is released one day). Instead, the focus was/is more on making models smaller and cheaper. That being said, there are interesting new approaches to get more out of the model (E.g,. see the GPT o1 model).
> Do you think LLMs could replace software engineers and data scientists in the near future? Alternatively, could they significantly augment our capabilities, potentially leading to fewer jobs? What role do you see LLMs playing in enabling non-technical roles, like product managers, to handle technical aspects of product development without developer support?
Unfortunately, I can’t really comment on that. I think it is likely going to be similar to the introduction of computers or the internet back then. Some jobs will be more automated, some jobs will go away, and some jobs will be created, and some jobs will be unaffected.
> In your view, are there specific application areas where LLMs are unsuitable or even potentially harmful? For example, would you recommend against their use in certain high-stakes domains, such as medical or legal advising, due to accuracy or ethical concerns?
That’s another tricky question 😅. I think it all comes down to using the technology responsibly. You mentioned medical domains. Fully automating a medical diagnosis with LLMs is probably irresponsible (right now). However, I don’t think that LLMs are necessarily harmful here…it all depends on the user and usecase.
Imagine you are a doctor and have a patient where you are not exactly sure what the patient’s health issue is. Given the symptoms, you could ask an LLM to suggest potential diagnoses. Then, as a doctor, you can carefully examine those, check the literature, and maybe come up with a diagnosis you otherwise wouldn’t have thought of. I.e., the LLM could be used here to see things that you otherwise would miss.

Sebastian Raschka thank you for your answers 🙂 I will definitely have a look! Let’s see what the future brings!

Get well soon! 🍀

Hope you get well soon!

hope you feel better soon!

Gute Besserung!

Get well soon 🌿

Thanks for your interest in my book. Actually, I am not super aware of a good follow-up book at the moment (I’ve been in the zone writing the book and haven’t had much time to read anything besides research papers for the last couple of months).
That being said. for the next stages, I have a GPT-2 to Llama 3.2 guide here that could be a good follow-up: https://github.com/rasbt/LLMs-from-scratch/tree/main/ch05/07_gpt_to_llama
Another thing that may be interesting to study is LitGPT, a open-source library I help developing: https://github.com/Lightning-AI/litgpt
I am mentioning it because it implements a variety of LLMs, but the code base is kept relatively readable. I think there you could get a good understanding of how the different LLM architectures (Phi, Llama, Gemini, Mistral) etc compare. Of course, it’s not a tutorial, but maybe it’s useful!

Thanks, I’ll definitely check those two out. I’m also planning to look at Jay Alammar’s new book - looks promising. Hope you’re feeling much better now!

Thanks, Rileen! I actually just got an ebook copy of the book and it looks super interesting. It’s more of a survey book but I think it would work in both ways:

1. Reading it before the Build an LLM from Scratch book for a top-down-view to see what types of LLMs and applications are down there, then diving into “Build an LLM from Scratch” to understand how LLMs work.
   
2. Reading it after Build an LLM from Scratch, after getting a good understanding of how LLMs work, and then reading gaining an understanding of what other applications and usecases are out there

Thanks! I’m inclined to go for option 2, so that I can potentially add a few projects to the ones I did while doing the “LLM Zoomcamp” course with Alexey, and then get under the hood with your book. Looking forward to both!

You are absolutely right, keeping up with the latest developments can be a fulltime job itself.
Fortunately, there are active online communities (lots of blogs, newsletters, and others such as the LocalLlama subreddit) where interesting new developments are shared. Once you understand the basics, you’ll see that most of these are just extensions.
I recorded a short video a few weeks ago to discuss/show how most is still based on the original GPT architecture: https://www.youtube.com/watch?v=itIab9ZTAqk

Thanks, I am slowly getting better 😊.
I think that’s a good question, and it depends a bit on the use case. There’s always something you can improve.
Finetuning on custom data is usually always an option if the domain and application benefits from custom data. Here, the same rules apply as in classic ML: the dataset quality and format matters. This includes many factors:

• inclusion of synthetic data
  
• dataset ratio/mixing
  
• quantity and quality
  
• formatting
  
• etc.
  But let’s assume you are not interested in finetuning. Still there are lots of ways to improve the model. In this case, the most substantial one would probably be prompt engineering.

Thanks for the question! I am not sure if I am understanding it correctly, but let me give it a try! I think that accessing the features explicitly is tricky. As a general rule of thumb the earlier layers access more of the fine-grained feature information and the later layers capture more of the coarse-grained features. Then the different heads in multi-head attention access different aspects of these features. Beyond that, I am not not aware of any approach that codes things more explicitly, except maybe Alphafold 2, which coded geometric constraints (it’s not an LLM per-se but LLM-adjacent); however, even they found that this is not necessary and dropped that in their recent AlphaFold model.

Hopefully!

Thank you Sebastian! It was a big pleasure hosting you!

As a reader and questioner I also wanted to thank all the Organizers and Sebastian himself for the whole thread. Very useful, also in future. Hope you are better and healthy now Sebastian. All the best for you.

Thanks, Malik Hebbat!

Just ordered my copy; turns out Amazon India takes 1 month to deliver!

Received my copy today Sebastian Raschka, thank you and best wishes from India.

That’s an excellent question. I think they are very interesting and refreshing, actually! I do think these approaches are currently still limited.
There hasn’t been a competitive state space model-based LLM as far as I know. With competitive, I mean GPT-4 or Llama 3.2 quality.
I often wonder whether it’s
a) state space models are too limited (actually, this is partly true, which is why Jamba and Samba are now transformer-hybrids)
b) researchers who are working on it don’t have the budget to scale it more
c) there hasn’t been that much interest yet
Maybe all of the 3 are a bit true. In any case, I think they are interesting and academic benchmarks in the respective papers don’t look too bad. But as far as I recall the benchmarks are mostly multiple-choice based, like MMLU. I am haven’t seen any state space model in a conversational benchmark to test the text writing/response quality (but maybe I missed that).
Maybe for genetic data these models can be quite useful though because they don’t need to conversational chatbots. And they scale very well. I think one of the first state space models was actually used in that context (https://arxiv.org/abs/2306.15794)

Thank you very much for your answer.
There was something from Nivida, but it was taken down, where they trained a chatbot based on Mamba:

• https://developer.nvidia.com/blog/performance-efficient-mamba-chat-from-nvidia-ai-foundation-models/
  For genetics they did it in this research:
  
• https://arxiv.org/abs/2403.03234
  I was just wondering from your intution how these models work (especially Transformers) if you could see if they acutally could work. Because I actually tried to rap my head the state space models but wasn’t really convinced that they actually have a attention that could be as powerful as the one from Transformers.

Another great question!
With “Transformer”,

1. do you mean literally the original transformer architecture (encoder-decoder)
   
2. or do you mean just transformer-based LLMs like BERT, Llama, GPT in general?
   (Just asking because your previous question was about state space models.)
   In case you mean 1, then I would say then that actually both encoder- and decoder-style models can work well. (There was a related question here: https://datatalks-club.slack.com/archives/C01H403LKG8/p1729084855205539?thread_ts=1728923923.942779&cid=C01H403LKG8)
   In fact, in chapter 6 we trained a decoder-style GPT model for classification (attaching a linear layer to the output embeddings) and get like 97% spam classification accuracy on balanced dataset. Or 92% classification accuracy on the balanced 50k IMDB movie review dataset (the 3x larger RoBERTa model was only 1% better).

I mean transformers in general. I will also probably compare with a State Space Model or Hybrid form and then see what works best and compare them. It is often claimed that the State Space Models would work as good even when they have less parameters.
For the project, a pretrained decoder-Style model, finetuned on the task, will probably work the best, since also the language shouldn’t be too far out of distribution.
For the Encoder-Style model like Bert I think they are better for Sentence embeddings and downstream classifications (but I am not sure, just what I read so far) but I am scared that their window will be too small (if for example I am using a Sentence-BERT-Model) for my long clinical texts.
And maybe to point you to this again since I have no intuition on that:
What do you think about the LLM2Vec approach? Is there something special to it than just taking the embedding and putting a linear layer at the end? https://arxiv.org/abs/2404.05961
And thank you very much for your answers to my two posts and the thoughts you putted into them 🙂

I think the book is for everyone who wants to understand how an LLM works, not conceptually, but really how the mechanisms look like and how it functions.
A good target audience is also someone who is using a third party LLM library like Hugging Face transformers and so forth, where LLMs are more of a black box.
It’s like people who want to understand how a motor in their car works. You don’t learn it by driving the car but by building your own motor. Of course, if you build your own motor this won’t be as fast as a Ferrari or as reliable Honda. You probably won’t use this motor in your daily car. But building it and getting it to work will teach you a lot about motors, way more than driving a car or looking at pictures of motors.

In case you want to add new labels later on without having to retrain, then yes, I would probably not do the multi-label approach. Your approach could work, but you will need a relatively large dataset to finetune a model to do well here. Maybe 10k to 50k examples would be good. You could use a proprietary model to help with the dataset generation.

Thank you again for your response! I really appreciate it.

1. Yes, I think RL will remain important. The first model that served in ChatGPT (a variant of GPT-3 / InstructGPT) used RLHF+PPO (reinforcement learning with human feedback based on proximal policy optimization). Then, researchers and practitioners use a simpler variant called DPO, short for direct preference optimization (e.g. Llama 3 and others). Neither RLHF nor DPO are like traditional RL, but the concept of learning a policy that optimizes for user preferences will remain important. Also, the recent OpenAI o1 model likely uses RL for the chain of thought optimization.
   
2. That’s the billion dollar question. I am not sure. I think everyone in deep learning is hoping for it, but no one knows how it could work.

Hi there! There are good questions, and phew, non-technical questions are actually hard to answer, haha! But let me give it a try!

1. I am actually quite excited that we recently got “smaller” LLMs. I.e., the Llama 3.2 1B and 3B models in particular are super interesting to me, because they can be more conveniently used and finetuned on less expensive hardware. That’s super cool for tinkering and research. I am hoping to see more of this!
   
2. I tend to bookmark maybe 20-30 papers each month but nowadays only read 1-2 papers per week (usually one that is closely related to what I am currently working on and one that is not super related to widen my horizon). When I see an exciting paper I try not to read it immediately but add it to my bookmark list instead. And then when I have time for reading, I go through all the bookmarks and pick the most interesting/relevant one at that moment. This helps me to not get too distracted. But yeah, ideally I would like to read more papers.
   
3. If you mean with LLMs, then I think ColPali is pretty good: https://arxiv.org/abs/2407.01449 , https://github.com/illuin-tech/colpali); otherwise, if you mean as a human reader, then I don’t really have a secret, I just read or skim them 😅

Thank you very much!

Building and training an LLM from scratch can be crazy expensive. It an literally be a (multi)million $$$ question 😅. (No worries, the LLM training in the book is kept small so it runs in reasonable time on a standard laptop; we also load pretrained weights from OpenAI into that model later, so there is no need to train from scratch.)
That being said, there are multiple reasons for building a model from scratch:

• Educational purposes: It’s the best and most thorough way to learn LLMs. Maybe the only way where you’d really understand each detail. It’s great for understanding the limitations also.
  
• Research purpose: If you want to develop new and better (or just different) architectures, you’d have to know how to implement them
  
• Customization & privacy: by knowing how LLMs work, it will be easier to adopt various LLM models out there like Llama and finetune them for your needs. In addition, you can run these models locally if you have sensitive data that you can’t share with any of the LLM providers like OpenAI, Google, or Anthropic
  Regarding the overheads:
  
• You’d likely need a couple of GPUs if you are serious about training LLMs; but for finetuning, one GPU can already be enough
  Regarding the evaluation:
  
• The public benchmarks are useful for quick sanity checks but they are also very limited. For example, MMLU and other public benchmarks are just multiple-choice tests, but a real-world application of an LLM is probably not going to be just multiple-choice question answering.
  
• In my opinion, one of the best ways to evaluate an LLM is to use a test set that you set aside similar to regular machine learning (this is done as an example in chapter 7) and then use a capable LLM like Llama 3 or GPT-4 to compare the written solutions to the correct test set solutions and assign scores, etc.

Good question! I just responded to another question here (https://datatalks-club.slack.com/archives/C01H403LKG8/p1728900925747959) which was a bit related.
In the book, I use the training and validation losses for the pretraining chapter (chapter 5) along with human evaluation.
In chapter 6, which is about spam classification, we can calculate the training, validation, and test accuracy like in classic machine learning (based on whether the text is correctly labeled as spam or not spam).
Then for instruction-finetuning (chapter 7) it gets a bit trickier. Here, we set aside a test set with correct responses and then use another model (Llama 3 and optionally GPT-4) to compare the model response to the correct response. Below is a screenshot to illustrate what I mean:

The limitation of MMLU and other public benchmarks is that it’s just multiple choice question answering:

A currently popular way is the LMSYS ChatBot Arena, where humans can see responses of 2 models and then choose which one they like better. And based on the response they create a leaderboard ranking. Overall, this is a pretty good way to compare LLMs, but the shortcoming is that it strongly favors style over correctness.

So in short, there is no perfect way to evaluate LLMs. But using all the different ways mentioned above together can be informative:

• MMLU etc for testing knowledge
  
• Instruction-response evaluation with a closed test set to test how well it can actually answer free-form questions
  
• LMSYS ChatBot arena as an additional test to rank LLMs by preference

That’s a great question, actually. It sounds like it should be simple to answer, but unfortunately there are no comparisons of the same architecture pretrained on different datasets. This is probably (and understandably) because pretraining is so expensive! Actually, pretraining sets range also a lot in size. There’s a trend towards training sets becoming larger (but there are also LLMs that focus more on smaller high-quality data like Microsoft’s Phi models)
Now, regarding instruction-finetuning, there are fortunately more comparisons. Here, specifically, quality is more important than quantity (e.g., one paper that demonstrates that nicely is LIMA, where the researchers found that their 1000 example dataset beats the 50,000-example Alpaca dataset, https://arxiv.org/abs/2305.11206)
When it comes to recommendations, I would say the Olmo paper is actually a good start as it shares the training data (other LLMs don’t do that): https://arxiv.org/abs/2402.00838

Yes!! I made sure that everything works and runs in reasonable time on a local machine. (I actually tested all the code on my MacBook Air).

Actually, I think may not need a very comprehensive deep learning background. Sure, LLMs are neural networks, but they are also their own thing in a way.
In addition, I wrote a ~40-page Introduction to PyTorch (Appendix A) where I tried to get someone started with deep learning in PyTorch effectively. It doesn’t replace a full deep learning course, but I hope it covers the prerequisites for the book.

I think they are all important and build on each other, so it’s hard to pick one over the other 😅. But I would say how the embedding vectors that come out of an LLM are converted into tokens is probably the most important aspect.

I think most common mistake or trickiest part is to get the weight reshaping and transposing in the multihead attention mechanism wrong. Here, I recommend starting with the simplest implementation first and then gradually morphing it into a more efficient version while making sure that the results numerically still match.

For finetuning it’s actually quite easy because the finetuning dataset are usually quite small and can be held in memory. E.g., just to give an example, the Alpaca dataset with 50-000 instruction-answer pairs is just 21.7 Mb large. Even if the dataset is 2x the size, it’s still convenient to just store it as a JSON file.

Encoder-based models like BERT have been specifically developed for classification tasks (along some other tasks), so they are a more natural choice here. But that being said, you can actually get really good performance with decoder-style models. E.g., in chapter 6, we finetune a GPT classifier for spam classification. Or take the IMDB movie review dataset for sentiment analysis, in the bonus materials I compared a 124 M parameter GPT model to a 3x larger RoBERTa, and the accuracy of RoBERTa was only 1% higher.
The problem with BERT models is that they are really old, and the training (data) has not been as sophisticated as the one of more recent decoder-style models. As a practical tip, I would recommend using RoBERTa or DistilBERT as a baseline and then compare it to a recent classification finetuned decoder-style model.

Unfortunately, no. The book is more focused on coding the LLM, and a RAG system is more of an application surrounding LLMs. But putting the LLM trained in the book into a RAG system could be an interesting bonus material that I may add.

Good question… Actually, I made sure that all code works fine on a conventional laptop (I tested everything on my MacBook Air). However, some chapters benefit from GPUs. I have some recommendations here: https://github.com/rasbt/LLMs-from-scratch/tree/main/setup#cloud-resources

Thank you for answering and providing link to recommendations!

I am also new to this, so please take this with a grain of salt. But as far as I know it runs from Monday (today) to Fri. And you could post questions here that I then answer asynchronously.

That’s a good question. The term SLM (small language model) is really fluid and keeps changing. E.g, an LLM today may be called an SLM tomorrow. That being said, the book uses models with 124 M parameters in most chapters to make it work on most hardware. But it also includes the configuration for 1.5B parameters that can be used by changing only one line of code.

That’s a good question, and this is usually a big problem and challenge in LLM training, The most widely applied strategy is to include ~5% of the original training data in the domain-specific data mix. (e.g., https://arxiv.org/abs/2403.08763)

Yes, I think so. One example of this is classification finetuning where you can match the performance of a huge general-purpose LLM by finetuning just on one target task

That’s actually a refreshing question! I’ve worked with LLMs for several years, and in my older books from 2021, I only had a single chapter on LLMs. But readers enjoyed it so much that I decided to turn it into a standalone book. (Plus, I am actually super excited about LLMs myself and work on them daily.)
I’m not a full-time writer; it’s more of a (very intensive) side project. Unfortunately, I don’t think it’s feasible to make a living as a tech book author. It’s more of a labor of love.
I’d say I spent about 2 hours a day over 1.5 to 2 years on this particular book. It was a lot of work! (I’m definitely taking a long break before starting another book… if I even write another one, haha.)

As far as I know they are complimentary resources. I started watching the videos recently and they seem really good. But yeah, the differences are that the material is presented very differently, and the book goes into finetuning, like classification and instruction-finetuning. Also, in the book I am covering a conventional GPT-style model (plus Llama models in the bonus material) whereas the Karpathy videos are focused on character-level models (as far as I know). In either case, I would recommend checking out both the videos and the book to get a more complete picture.

I am not an expert in this area, but based on my knowledge alignment and preference tuning are the most commonly used approaches (although they are far from perfect).

In addition to altering the LLM itself, other (or additional) solutions include putting guardrails around the LLM. One of the popular examples is the NeMO-guardrails framework: https://github.com/NVIDIA/NeMo-Guardrails

Good question. I answered a similar question here that may already address your question, but please don’t hesitate to reach out if you have a follow-up question: https://datatalks-club.slack.com/archives/C01H403LKG8/p1728901113313779

The book is not focused on inference optimizations, but general techniques would include KV-caching, quantization, low-precision formats, graph compilation, etc. We implement many of these techniques in LitGPT, an open source library I help developing (the LitGPT code base is actually a good follow-up for the book): https://github.com/Lightning-AI/litgpt

Thanks Sebastian Raschka – this helps.

I wish I could say more here, but I don’t have any specific experience in this intersection. That being said, protein amino acid sequences are essentially text, so I can see LLMs being useful for certain pattern-recognition related tasks here.

That’s essentially what I try to address in the book 😊

I think it’s important to start with / select a tokenizer that is aware of the languages. The GPT-4 tokenizer (used by Meta AI’s Llama 3 models and many others) should be a good fit. Then, you’d pretrain the model on a dataset that contains a lot of Turkish texts (but pretraining is expensive, so you may want to pick a model that has already been pretrained).
Regarding the pretrained, I think Llama 3.1 and 3.2 are capable of Turkish (I have a conversion from the book’s model to Llama 3, 3.1, and 3.2 here: https://github.com/rasbt/LLMs-from-scratch/blob/main/ch05/07_gpt_to_llama/converting-llama2-to-llama3.ipynb). After using a basemodel such as those, you could consider further finetuning on Turkish texts. I just did a quick search and there seem to be a bunch of datasets for this out there: https://huggingface.co/collections/umarigan/turkish-llm-fine-tune-datasets-66290b6cce62e36cd325fa88

Good question. Yes, they can! I think my previous answer above may already address this 😊

I would always start with a baseline, e.g., an existing model that tries to accomplish something similar. This can be a related method or a classical method. E.g., if you work on an LLM to classify texts, also consider a simple baseline like a logistic regression classifier with bag-of-words

Good question. There are lots of active online communities like the LocalLlama subreddit where it might be possible to find and partner up with people who are interested in specific topics: https://www.reddit.com/r/LocalLLaMA/

Fair question. And I wish I could say something useful here, but I haven’t worked or read anything about protein sequences in years and don’t want to give any outdated or misleading information 😅

Yes, keeping up with everything can be overwhelming and would be a full-time job itself. I think here the crucial thing is to work towards something, like an applied project, and then try to be a bit selective. It’s good to be aware of what’s generally going on (e.g., via following newsletters or checking the posts on communities like LocalLlama). But at the end of the day you can’t be an expert in every LLM subtopic, so I would select those that are most relevant to the problem you are currently working towards.

I wish I could tell you a secret here but I think it mostly comes automatically after some experience with applied projects & problems. If you work towards a particular project and goal and remain curious, you’ll want to check out different solutions to solve similar problems, and things will follow automatically.
Suppose you are interested in retrieving information from documents. You may start with a RAG. Then you read about long context LLMs and wonder if they could replace your RAG. After doing some reading, you implement the long-context LLM and notice that you can’t run it on your hardware, and you look for techniques to lower memory requirements, and so on… I think while working on applied projects, you will automatically learn and encounter many different caveats and workarounds that will naturally lead to building your expertise 😊

I am glad to hear that the book has been helpful for your dissertation!

1. I only focus on LoRA. That’s because implementing techniques from scratch takes a lot of space (/pages), and a book can only be so long 😅. The focus here is also mainly on understanding the most important techniques well (versus writing a survey). That being said, LoRA is the one technique that stood the test of time so far (and it’s also what Apple uses in their Apple Intelligence models). I think once you understand LoRA, it’s relatively easy to modify it and experiment with other variants like MoRA and DoRA, etc.
   
2. The book itself doesn’t use grouped attention because it’s not used in the GPT architecture implemented in the book. But I implement it here in the book’s bonus materials when converting the GPT architecture to Llama: https://github.com/rasbt/LLMs-from-scratch/blob/main/ch05/07_gpt_to_llama/converting-llama2-to-llama3.ipynb

thank you very much Sebastian.

I think that building an LLM from scratch is the best (and maybe only) way to really understand how LLMs work, what the limitations are, etc.
But yes, in practice you want to finetune pretrained models and not create models from scratch. I do think that in order to finetune models effectively, it’s crucial to have some understanding of how finetuning is implemented (vs just using a 3rd party library that feels like a black box)

Thanks so much for the kind words (and the kind support!!).
And this is a good question. The building from scratch part is mainly to understand how an LLM is structured and how it works (how it processes data, and so forth). I think it really helps with understanding the finetuning process.
But like you said, in practice you can finetune a pretrained model instead of building and training it from scratch.
When you say PII, do you mean (1) you are worried that a user can access to PII that has been included in the pretraining dataset when the pretrained model was created, or (2) do you worry about using the foundation model when prompts may contain PII?
For (1) I remember reading in the Llama papers that the researchers took specific care to exclude PII, but of course this will never be perfect. If you train your own model from scratch, your model may actually also have similar problems if you are not careful with your training dataset. And since pretraining requires such huge datasets, it would be very hard to ensure that the foundation model is 100% PII-free. So, in short, this is a given risk in either case.
Re (2): That’s a valid concern since we don’t know how our data is used by OpenAI, Google, Anthropic, etc. For this reason, for example, we are not allowed to use these LLMs for many work-related tasks and use LLMs that run locally instead.

These are good and challenging questions.

1. I am not an expert in AI ethics and unfortunately can’t give an elaborate answer here. A lot depends on the intended use case of the LLM, too. In the book, training the model from scratch is mainly for educational purposes, and we use only public domain data. After building a solid understanding of how the process works, you may consider available LLMs with pretrained LLMs where researchers applied some mitigation techniques to reduce potential issues (it’s never perfect though). For example, the recent Llama 3 paper had a really good section on safety: https://arxiv.org/abs/2407.21783
   
2. RLHF and DPO are in my experience very tricky. Compared to instruction-finetuning, it’s really hard to get them to work well. Also, the dataset is hard to come by. However, they can definitely make a significant difference (although they are not perfect). E.g. from the Llama 2 paper (screenshot below. https://arxiv.org/abs/2307.09288) you can see that with more RLHF, the harmlessness and helpfulness definitely improve

Good question! Actually, I only include it via the bonus materials. It’s a tricky topic and doesn’t always work so well. Actually, instead of RLHF+PPO I implement DPO, which is nowadays a bit more popular and widely used (e.g. most recent models, including Llama 3.2 use it): https://github.com/rasbt/LLMs-from-scratch/blob/main/ch07/04_preference-tuning-with-dpo/dpo-from-scratch.ipynb
I would say the complexity is higher than the complexity of instruction-finetuning, but after understanding instruction-finetuning, I think it’s a gradual step and shouldn’t be too complex

Oh, thank you very much. I’ll read it

Good question. There are various different methods. The simpler (and now more popular ones) “simply” encode images as tokens similar how text is encoded as tokens. E.g., two recent examples are the Emu3 paper: https://arxiv.org/abs/2409.18869 and Molmo & Pixmo: https://www.arxiv.org/abs/2409.17146
Stay tuned for more details 😊. I started writing an article about that a few weeks ago and plan to polish it up by the end of the month/early next month. (I’ll share it on my blog here: https://magazine.sebastianraschka.com)

Great, thanks for the sneak preview 👍🏼

I would say that the book is for people who would like to understand how LLMs (really) work, versus just using the ChatGPT API or LLM libraries as an opaque system.
The prerequisite is to be familiar with Python. I think a strong machine learning background is not required (but of course it would not hurt). I included a ~40 page introduction to PyTorch in Appendix A that covers the basics that are necessary for the book. If it is your first time working with PyTorch, it may be a steeper learning curve, but I think it’s going to be fine.
> Question: also what are you working on currently…? We would love to have some learning insight’s on it….
I used to be a professor at University of Wisconsin-Madison, but my day to day work consists of developing LLM-related tools at our company (https://lightning.ai) that I joined about 2.5 years ago. I do like coding a lot actually, which is why I made this shift. A lot of this is related on internal research, solving problems for customers, and developing open-source libraries, for example LitGPT (https://github.com/Lightning-AI/litgpt). But of course, when time permits, I also like to work on educational materials when time (and family) permits…Now that the book is finished, I have some more time for some exploration again, working on a finetuning related research project and some other fun stuff!
> What are the key trade-offs between training a model from scratch versus fine-tuning an existing model?
Pretraining from scratch is almost never advisable unless you are a big company and can spend millions of dollars to develop your own state-of-the-art LLM. (No worries, in the book we only use a small dataset for that; the point is more to code and understand the process; but we then load pretrained weights). That being said, both pretraining and finetuning are part of the LLM development cycle. So finetuning usually comes after the pretraining (and instead of pretraining the model yourself, you can start with pretrained weights)

I think there’s a lot of interesting work that could be designed around each of the chapters. You could consider

• Chapter 4: exploring different architectural variations and see how they affect model performance
  
• Chapter 5: different tokenizers and training on low-resource languages
  
• Chapter 6: implementing the classification approach with different models such as Gemma, Llama , Phi etc as a comparative study
  
• Chapter 7: exploring parameter-efficient finetuning techniques
  I hope this helps!

I think your approach is totally reasonable. Instead of instruction-finetuning, you could also consider classification finetuning though (chapter 6), because you don’t need the model to generate a response text but just a few labels. This may be cheaper and more accurate. (In this case, this would be a multi-label classification problem, so you’d have to make some slight modifications.)
The evaluation metrics you mentioned seem good; if you have a set of true labels then this is easy to calculate.
> Q) Would it be better to exclude the reference keywords from the instruction-prepared validation set as well to get a better idea of how the model will perform in the wild?
Oh wait, maybe I am misunderstanding; if this is an open-ended list, then the multi-label classification approach I mentioned above may not be feasible because you may have labels that weren’t known to you beforehand.
In this case, I do think the model will also be challenged in this instruction setting though.

Hi there! That’s a tricky one actually because OpenAI doesn’t share the architectural details of their models anymore. The GPT-4o model is likely very similar to the original GPT architectures though and decoder-style model. For the embedding model, this could be either a similar model (where they don’t convert the outputs into tokens but return the vector representations) or it could be a BERT-style model.
I think ada002 is likely a smaller GPT-model where they simply don’t convert the outputs back into tokens. Using a figure from the book, it is likely similar to this where it returns the output embedding vectors before converting them back into tokens

Finetuning the embedding model for a RAG system is somewhat similar to classic self-supervised examples in computer vision with a triplet loss or other contrastive loss. Basically you have triplets of (1) the text of interest, (2) an unrelated text snippet, (3) a related text snippet. Then, the goal (which is achieved by minimizing the loss) is to make the distance between the (1) and (2) large and the distance between (1) and (3) small.