Hi Igor, thanks for your questions!

1. The book assumes some familiarity with deep learning and PyTorch, so we recommend reading it after working through e.g. the fast.ai course / book.
   
2. Are you referring to unsupervised topic modelling, e.g. cluster my texts into a set of topics? For this, there’s a neat library called BERTopic (https://github.com/MaartenGr/BERTopic) that get’s quite good results in my experience. The alternative is to use sentence-transformers to embed your documents (https://sbert.net/) and then apply clustering on the embeddings - this too gives quite good results.

thanks Lewis Tunstall for your answers 🙂

• I had in mind taking that course for a long while, your suggestion add another point to it 😄
  
• that is exactly what I wanted to know, thanks a lot for pointing me out to the right resources. So far what I have been working on was a supervised approach, but defining a topic dictionary manually requires a lot of time and knowledge..

Hi Gonzalo, thanks for your question!
I believe that haystack did a major overhaul of their Pipeline abstraction in v1 (which unfortunately came out after our book was in production), and my understanding is that it now supports several NLP tasks like summarization (i.e. summarize the retrieved docs): https://haystack.deepset.ai/reference/pipelines#searchsummarizationpipeline
For a completely custom pipeline, I’d suggest joining their Slack group: https://haystack.deepset.ai/community/join
They’re really responsive and helpful for these kind of questions!

Thanks a ton Lewis Tunstall I will check out the source code of Pipeline and definitely join the Slack group!

When we started working on the book multi-modality was not really a thing, yet. Since than CV, Audio, Tabular applications etc. were also conquered by transformers. We were able to give a brief overview of what exists in the last chapter but obviously it would have been nice to include them in more detail and show how to train them. Maybe v2 of the book needs to be renamed to Machine Learning with Transformers 🙂
It helped a lot to work closely with the maintainers of transformers and datasets to plan a bit around upcoming features or features that would soon be deprecated. E.g the streaming API used in the Training transformers from scratch chapter was finalized at the same time as we wrote the chapter.

We used the fastdoc library (https://github.com/fastai/fastdoc) to handle all the conversion from notebooks to Asciidoc (which is O’Reilly’s preferred format).
Overall we found the experience was rather smooth, and we were lucky that O’Reilly already had experience from the fastai book 🙂
We heard from the production editors that they’re getting more submissions in this format, so I think notebooks might become the norm for these kind of publications - it really makes writing a breeze when everything is in one place!

One point to addd here is that the O’Reilly team was happy to keep the notebook -> asciidoc conversion pipeline in place until the very last reviewing process which kept the notebooks in sync with the formatting/copy-editing changes.
So it was really nice of them production team to adapt to this new workflow. As a nice side effect the whole book is nicely git version controlled 🙂

Hi Max, thanks for the question! As far as I know, explainability of transformers is still an open research problem and one that we didn’t have space to cover in the book.
One of the best examples I know of is Jay Alammar’s ecco library (https://github.com/jalammar/ecco) which uses various visualisation techniques to understand which factors lead to a specific prediction. It doesn’t give you an explanation “for free”, since you have to do some work to interpret these results, but I think it might be interesting to you

Thanks alot

So if by prediction scores you mean the model’s output probabilities then yu can actually get them fairly easy from the transformer models: For each token the model will return the logits and by applying a SoftMax function you’ll get each classes probability.
In the book we show how to get the logits from the model for NER so the only thing that’s missing to get probabilities is applying SoftMax. Hope that helps!

the problem is that the softmax will indeed squish the logits into a probability distribution (formally), but this distribution is unfortunately poorly calibrated to the correctness likelihood: https://arxiv.org/abs/1706.04599

1. It took as roughly 1.5 years to write the book. Since we did it in our spare time this means that we mostly worked on it on weekends 🙂
   
2. For me working on the few labels and pretraining from scratch were the most challenging. The few labels chapter required a lot of experimentation and research to first figure out what works well. The pretraining chapter was one of my first contacts with distributed training and in addition we used a lot of features of the hugging face ecosystem that were brand new.
   Maybe the others have other chapters they found challenging 🙂

Yes you can! See the table with Colab links in the book’s repository:
https://github.com/nlp-with-transformers/notebooks

You could look at chapter 6 where we show how to train a transformer model on text summarization. All the book’s code is freely available in the repository (but without the text from the book, so no explanations).

Hi Amruta Ranade, yes transfer learning can be applied to transformers and is in fact the driving force behind their success. Except for the last chapter where we train a model from scratch all chapters show how to apply transfer learning to tune a pretrained model on a specific task such as classification, NER, QA, or summarization.

Thankyou for the information ! Sounds interesting to me. I am definitely going to buy your book! 😊

Hi Siato, thanks for being here!

1. What we try in the book is to be much more practical than the documentation, which is why each chapter aims to solve a realistic use-case. So it involves more steps like preparing your data or doing some error analysis of your model.
   
2. In general the later chapters in the book are mode advanced (e.g. making transformers efficient for prod, few labels or pretraining from scratch). Almost every chapter also tackles a different task (classification, ner, qa, summarization etc.). So you could also pick a task that you haven’t encountered before.
   
3. Unfortunately, we don’t go into detail of BigScience. The very last chapter looks a bit ahead especially at the scaling trend of transformers and multimodal transformers.

So I think this can be divided into two categories:

• data cleaning: before you train your model you want to make sure that the data is as clean as possible. for BigScience we spent a lot of time cleaning and filtering the data. this includes: removing duplicates, deleting html code from websites and so on. whatever you do to your training data here you should also do to the inputs in production. always clean your data 🙂
  
• tokenization: there are a few steps that besides splitting the string into tokens that the tokenizer takes care of such as unicode normalization or lower casing all strings. as long as you use the same tokenizer in prod as you used during training you should be fine.
  hope that answers your question! 🤗

Yes it answers partially. But I was looking for a deeper answer. 🙂
More of practical example

I guess the deeper answer depends on your use-case 🙂 To work as well in production as on your local test set the most important thing is to make sure the data preprocessing steps are the same and the test set reflects the production data.
The second aspect is the performance on the test set: data cleaning is usually very important here but again which particular steps are necessary depends on your use-case and dataset.

I experimented in the past with creating a single model for three classification tasks and thus creating a model with three heads. the motivation was more to have just one large model in production instead of three and we didn’t see a significant performance improvement. Also this was the same task type just three different classification criteria.
A model that works well on a wide range of tasks at the same time is T5. Maybe you can tune it for the tasks you are trying to cover.
Alternatively one thing that can work well in low resource setting is to use models that are already trained on a e.g. summarization task instead of the raw pretrained models. We do this in the QA chapter: we use a model that has been trained on SQuAD and train it a bit more on the Amazon dataset which gives better results.
Which (if any) of these suggestion works depends a lot on what you are actual use-case looks like. My general advice is to build a good evaluation set that captures your use-case and lets you easily evaluate your models. Start creating a few simple baselines and then iterate quickly on different ideas. Without the evaluation pipeline the iterations will be slow.

Thank you leandro von werra
Yes trial and error is the way forward

Hi A, thanks for being here!

1. I am personally very excited about ML in science. I think any tool that enables iterate faster in science will have a huge impact. One of my favourite application here is AlphaFold from DeepMind. Predicting how proteins fold is a critical task in chemistry and biology to make progress and until AlphaFold the only way to get good predictions was through very expensive and long experiments that could take weeks or month. With AlphaFold the same researchers can now iterate through 100s of molecules in a day which changes the game completely.
   
2. I would even argue that most biased models that make headlines these days were deployed by DS folks 🙂 I am not an expert here and maybe Thomas Wolf has more to share. My two cents are that thinking about and investigation bias and impact of a deployed model should become a core task of DS lifecycle. This requires documenting on what exactly and how the model was trained and rigorous testing of how it performs in different situations. There is a lot of work going on in BigScience for example where people work on showing how a large language model can be built and released responsibly.

Indeed our book is NLP beginner friendly and the main requirement are the core ML concepts (what’s a train test split, how to build and train a neural network, standard metrics etc). So if your are familiar with CV that won’t be an issue 🙂
I know a lot of people got into NLP with Chris Mannings excellent lecture series: https://web.stanford.edu/class/cs224n/. Also the popular fastai course covers some NLP: https://course.fast.ai/videos/?lesson=8

Yes, we go into that in more detail, but to give you the short summary here:
Every model in the transformers library that is able to do text generation has a .generate() method. This method is loaded with options to do all sorts of generation strategies: greedy, beam search, sampling (with top-k or nucleus), repetition penalties etc. You can also define your own stopping criteria or processors to be applied.
see here
Patrick von Platen wrote a great article about that function and how to use it:
https://huggingface.co/blog/how-to-generate

Awesome. Thank you!

There are a number of advantages:

• you can host the modals on your own infrastructure - for many companies sending data to an outside API can be complicated, especially if the data is sensitive
  
• you have a lot more flexibility to train the models and iterate through different training setups. you can also customize your architecture to your use-case
  
• dopending on your setup hosting your own models can be much cheaper, especially if you have heavy workloads. Nils Reimers wrote a thread about this: https://twitter.com/nils_reimers/status/1487014195568775173

Thanks Leandro! Indeed, that totally makes sense.
Also thanks for the link! 🙏

oh there are many, hard to pinpoint a single hardest 🙃 just to list a few:

• engineering: training such models requires a large distributed infrastructure with <tel:100-1000

  100-1000>s of GPUs. you need some resilience when some of them crash and you need to make sure they are optimally used. also at that scale you can be haunted by many numerical instabilities. in BigScience we are training such a model in the open and you can read the chronicles of the experiments: https://github.com/bigscience-workshop/bigscience

• dataset: training these beadts requires a lot of data (~TBs of text). at the same time you need to make sure that the data is clean which gets harder at that scale. you can‘t just look at it
  
• release: how can one responsibly release such models. there is a lot of work going on about for example licensing.
  maybe Thomas Wolf has some more insights here :)

Ah, interesting! How can you deal with the numerical instabilities? I imagine they also propagate and get larger and larger 🤔
I’m looking forward to reading more in your book 🙂

Also what do you think about “sparsity-aware inference engines”?

Currently it seems that transformers are penetrating almost every field in ML. on the topic of reasoning you might find this recent paper by Google interesting: https://arxiv.org/abs/2201.11903

Wow, that’s impressive!!! Thanks a lot for the link
58.1% on GSM8K is impressive (though also a reminder of long path ahead!)

1. definitley. there are for example ~1000 translation models already on the huggingface hub! see: https://huggingface.co/models?pipeline_tag=translation&sort=downloads. you can either use them out of the box or further tune them on your specific data!
   
2. we don‘t go into details on translation specifically, but we show summarization in more detail. conceptually the task is very similar (one input text and an output text) and we also show and explain BLEU which is often used in translation.

thank you very much 👍

Hi Armin, thanks for your question! There do exist transformers like SAINT for numerical features like the ones you would find in tabular data.
However, it seems that gradient boosting remains the state-of-the-art and it might be a while before transfer learning is truly cracked for tabular data (see plot).
There’s a very nice survey of these models here: https://arxiv.org/abs/2110.01889

Hi Allan, yes Chapter 8 covers various techniques related to optimisation including:

• Knowledge distillation
  
• Quantization
  
• Graph optimization with ONNX Runtime
  
• Pruning (although at the time of writing it wasn’t easy to prune transformers effectively)
  The end result is figure like this which shows how you can reduce the latency of a BERT model by ~7x 🙂

Thanks!

Hi Shantanu Ladhwe, not an expert on Pathways but as far as I understand it is mainly a vision towards multi-tasking, multi-modality, and maybe a switch to sparsity (see Jeff Dean’s article here). This vision could be (and will likely be) realized with a Transformer architecture. There are already multi-tasking (e.g. T0pp), multi-modality (e.g. CLIP, Perceiver), and sparse (e.g. SwitchTransformer) transformer models out there.
The Pathways Language Model (PaLM) uses a classical transformer decoder architecture and the main innovation is around infrastructure to train the model efficiently on 6000 TPUs.