In the course GitHub repository, there’s a link to sign up. Here it is: airtable.com

The course is available in a self-paced mode, allowing you to go through the materials anytime. If you prefer to join a cohort with other students, the next sessions will begin in September 2023, September 2024, and September 2025.

Zoomcamps are scheduled throughout the year. For more information, refer to A Guide to Free Online Courses at DataTalks.Club.

Course videos are pre-recorded and available to watch immediately. We also occasionally host office hours for live Q&A, which are recorded and made available later. You can find these recordings and pre-recorded course videos on the Course Channel’s Bookmarks or DTC’s YouTube channel.

Everything is recorded, so you won’t miss office hours or any content. You can ask your questions in advance, and they will be covered during the live stream. Additionally, you can ask questions in Slack.

The bare minimum. The focus is more on practice, and we'll cover the theory only on the intuitive level.

For example, we won't derive the gradient update rule for logistic regression (there are other great courses for that), but we'll cover how to use logistic regression and make sense of the results.

Math is not strictly a prerequisite to start learning machine learning (ML), but having a strong foundation in certain mathematical concepts can significantly improve your understanding and ability to work with ML models.

Yes! We'll cover some linear algebra in the course, but in general, there will be very few formulas, mostly code.

Here are some interesting videos covering linear algebra that you can already watch:

• ML Zoomcamp 1.8 - Linear Algebra Refresher from Alexey Grigorev
• The excellent playlist from 3Blue1Brown: Vectors | Chapter 1, Essence of linear algebra

Never hesitate to ask the community for help if you have any questions.

Normally, you'd receive the email shortly after you signed up.

• Check your promotions tab in Gmail as well as spam, as the email might have been filtered there.
• If you unsubscribed from our newsletter, you won't receive course-related updates.

Don't worry, it’s not a problem. To make sure you don’t miss anything, join the #course-ml-zoomcamp channel in Slack and our telegram channel with announcements. This is sufficient to follow the course.

Approximately 4 months, but it may take longer if you want to engage in extra activities such as an additional project or writing an article.

Around ~10 hours per week.

You can see how much time people spend on the 2024 edition here.

Yes, if you finish at least 2 out of 3 projects and review 3 peers’ projects by the deadline, you will get a certificate. This is what it looks like: this.

Yes, it's possible. See the previous answer.

Check this article. If you know everything in this article, you know enough. If not, read the article and consider joining the course Introduction to Python – Machine Learning Bootcamp.

You can also follow the free English course "Learn Python Basics for Data Analysis" on the OpenClassrooms e-learning platform: Learn Python Basics for Data Analysis - OpenClassrooms.

It's important to know some basics such as:

• How to run a Jupyter notebook
• How to import libraries (and understand what libraries are)
• How to declare a variable (and understand what variables are)
• Some important operations regarding data analysis

For the Machine Learning part, all you need is a working laptop with an internet connection. The Deep Learning part is more resource intensive, but for that you can use a cloud. We use SaturnCloud, but it can be anything else.

Here’s how you join in Slack: https://slack.com/help/articles/205239967-Join-a-channel

• Click “All channels” at the top of your left sidebar. If you don't see this option, click “More” to find it.
• Browse the list of public channels in your workspace, or use the search bar to search by channel name or description.
• Select a channel from the list to view it.
• Click Join Channel.

Do we need to provide the GitHub link to only our code corresponding to the homework questions?

Yes. You are required to provide the URL to your repo in order to receive a grade.

Yes, you can. Even though you missed the start date, you can register for the course. You won’t be able to submit some of the homeworks, but you can still take part in the course.

In order to get a certificate, you need to submit 2 out of 3 course projects and review 3 peers by the deadline. It means that if you join the course at the end of November and manage to work on two projects, you will still be eligible for a certificate.

No, it’s not possible. The form is closed after the due date. But don’t worry, homework is not mandatory for finishing the course.

Welcome to the course!

• Visit the course page: GitHub - DataTalksClub/machine-learning-zoomcamp.
• Scroll down and start going through the course materials.
• Read everything in the cohort folder for your cohort's year.
• Click on the links and start watching the videos.
• Watch office hours from previous cohorts on the DTC YouTube channel by clicking on Playlists and searching for {course yyyy}.

For direct access to the syllabus, use this link: http://mlzoomcamp.com/#syllabus.

For the 2024 cohort, you can find the deadlines here (it’s taken from the 2024 cohort page) or in Google Calendar.

No, you need to do projects individually, but it’s okay to partner up to discuss weekly lectures or exchange ideas.

When you post about what you learned from the course on your social media pages, use the tag #mlzoomcamp. When you submit your homework, there’s a section in the form for putting the links there. Separate multiple links by any whitespace character (linebreak, space, tab, etc).

• For posting the learning in public links, you get extra scores.
• The number of scores is limited to 7 points: if you put more than 7 links in your homework form, you’ll get only 7 points.
• The same content can be posted to 7 different social sites and still earn you 7 points if you add 7 URLs per week.

For midterms/capstones, the awarded points are doubled as the duration is longer. So for projects, the points are capped at 14 for 14 URLs.

We kindly ask you not to share your answers.

If you encounter the error "wget is not recognized as an internal or external command", you need to install it.

• On Ubuntu, run:

  sudo apt-get install wget
  

• On Windows, you can use Chocolatey:

  choco install wget
  

  Or download a binary from here and add it to your PATH (e.g., C:/tools/).

• On Mac, use Homebrew:

  brew install wget
  

Alternatively, you can use Python libraries:

• Python wget library:

  Install it first:

  pip install wget
  

  Then, in your Python code:

  import wget
  
  wget.download("URL")
  

• Using pandas to read a CSV directly from a URL:

  import pandas as pd
  
  url = "https://raw.githubusercontent.com/alexeygrigorev/datasets/master/housing.csv"
  
  df = pd.read_csv(url)
  

  Valid URL schemes include http, ftp, s3, gs, and file.

• Bypassing HTTPS checks (if needed):

  import ssl
  
  ssl._create_default_https_context = ssl._create_unverified_context
  

• Using Python's urllib for downloading files:

  import urllib.request
  
  url = "https://raw.githubusercontent.com/alexeygrigorev/datasets/master/housing.csv"
  
  urllib.request.urlretrieve(url, "housing.csv")
  

  The urlretrieve() function allows you to download files from URLs and save them locally. It is part of the standard Python library urllib.request, available on all devices and platforms.

The best way is to use pandas and give it the URL directly:

url = 'https://raw.githubusercontent.com/alexeygrigorev/datasets/master/housing.csv'
df = pd.read_csv(url)

You can also execute cmd/bash commands inside Jupyter:

!wget https://raw.githubusercontent.com/alexeygrigorev/datasets/master/housing.csv

The exclamation mark ! lets you execute shell commands inside your notebooks. This works for shell commands such as ls, cp, mkdir, mv, etc.

For instance, if you then want to move your data into a data directory alongside your notebook-containing directory, you could execute the following:

!mkdir -p ../data/
!mv housing.csv ../data/

If you have a Windows 11 device and would like to use the built-in WSL to access Linux, you can use the Microsoft Learn link Set up a WSL development environment | Microsoft Learn.

To connect this to VS Code, download the Microsoft verified VS Code extension ‘WSL’. This will allow you to remotely connect to your WSL Ubuntu instance as if it were a virtual machine.

If you encounter the following error when trying to use Git for the first time:

error: src refspec master does not match any
error: failed to push some refs to 'https://github.com/XXXXXX/1st-Homework.git'

Solution:

1. Make an initial commit using:

   git commit -m "initial commit"
   

2. Push to the main branch instead:

   git push origin main
   

For a comprehensive guide on using GitHub, visit GitHub Quickstart.

You can also use GitHub's "upload file" feature, or share your Google Colab notebooks directly to GitHub:

I'm trying to invert the matrix but I got an error that the matrix is singular.

The singular matrix error is caused because not every matrix can be inverted. In particular, it happens when dealing with multiplication (using the method .dot) since multiplication is not commutative. X.dot(Y) is not necessarily equal to Y.dot(X). Respect the order; otherwise, you get the wrong matrix.

If you encounter an issue where the command:

conda create -n ml-zoomcamp python=3.13

doesn't work, try the following solutions:

1. Use Anaconda Prompt:

   • If you are on Windows, use Anaconda's own terminal called "Anaconda Prompt". This should automatically configure your environment to recognize Conda commands.

2. Install Anaconda or Miniconda:

   • If you do not have Anaconda or Miniconda installed, download and install one of them first.

Note: Any version of Python starting from 3.10 should be suitable.

How do I read the dataset with Pandas in Windows?

I used the code below but it's not working:

df = pd.read_csv('C:\Users\username\Downloads\data.csv')

Unlike Linux/Mac OS, Windows uses the backslash (\) to navigate the files, which causes a conflict with Python. In Python, the \ is used for escape sequences, e.g., \n for a new line or \t for a tab. To avoid this issue, add an r before the file path to treat it as a raw string:

df = pd.read_csv(r'C:\Users\username\Downloads\data.csv')

To resolve a '403 Forbidden' error when pushing to a GitHub repository, follow these steps:

1. Check the current remote URL configuration by running:

   git config -l | grep url
   

   The output should be similar to:

   remote.origin.url=https://github.com/github-username/github-repository-name.git
   

2. Change the URL format to include your GitHub username:

   git remote set-url origin "https://github-username@github.com/github-username/github-repository-name.git"
   

3. Verify the change is reflected using the command in step 1. Make sure the URL is correctly updated.

I encountered a problem when trying to push code from Git Bash:

remote: Support for password authentication was removed on August 13, 2021.
remote: Please see https://docs.github.com/en/get-started/getting-started-with-git/about-remote-repositories#cloning-with-https-urls for information on currently recommended modes of authentication.
fatal: Authentication failed for 'https://github.com/username'

Solution:

1. Create a personal access token from your GitHub account.
2. Use this token to authenticate when you push your changes.

For more details, see the documentation on generating a new SSH key and adding it to the SSH agent.

In Kaggle, when you attempt to !wget a dataset from GitHub or any other public repository, you might encounter the following error:

--2022-09-17 16:55:24--  https://raw.githubusercontent.com/alexeygrigorev/datasets/master/housing.csv
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... failed: Temporary failure in name resolution.
wget: unable to resolve host address 'raw.githubusercontent.com'

Solution:

• In your Kaggle notebook settings, enable internet access for your session. This option is found in the settings panel on the right-hand side of the Kaggle screen.
• You will need to verify your phone number to confirm you are not a bot.

I found this video quite helpful: Creating Virtual Environment for Python from VS Code

Native Jupyter Notebooks Support in VS Code

In VS Code, you can have native Jupyter Notebooks support, i.e., you do not need to open a web browser to code in a Notebook. If you have port forwarding enabled, run a jupyter notebook command from a remote machine, and have a remote connection configured in .ssh/config (as Alexey’s video suggests), VS Code can execute remote Jupyter Notebook files on a remote server from your local machine: Visual Studio Code Jupyter Notebooks.

Git Support in VS Code

You can work with GitHub from VS Code. Staging and commits are easy from the VS Code’s UI:

Visual Studio Code Source Control Overview

If you prefer using the terminal for port forwarding, configure it in your SSH config file.

1. Open your SSH config file:

   nano ~/.ssh/config
   

2. Add the following line to forward your Jupyter server:

   LocalForward 8888 localhost:8888
   

To set up a Conda environment for the project:

• Initial Setup: Run the following command only once to create the environment:

  conda create -n ml-zoomcamp
  

• Activating Environment: Each time you want to work on the project, activate the environment:

  conda activate ml-zoomcamp
  

• Exporting Environment: To export your existing environment to a YAML file:

  conda env export > environment.yml
  

• Recreating Environment: Use the YAML file to recreate the environment:

  conda env create -f environment.yml
  

It prints the information about the dataset, including:

• Index datatype
• Number of entries
• Column information with not-null count and datatype
• Memory usage by the dataset

We use it as:

df.info()

If you're using numpy or pandas, make sure to import the libraries before using them:

import pandas as pd
import numpy as np

To select columns by data type, you can use the following methods:

• To get columns with numeric data:

  df.select_dtypes(include=np.number).columns.tolist()
  

• To get columns with object (string) data:

  df.select_dtypes(include='object').columns.tolist()
  

To identify the shape of a dataset in Pandas, you can use the .shape attribute:

• df.shape: Returns a tuple representing the dimensionality of the DataFrame.
• df.shape[0]: Returns the number of rows.
• df.shape[1]: Returns the number of columns.

You can also use the built-in len function to find the total number of rows:

len(df)

If you encounter the error below when launching a Jupyter notebook in a new environment:

ImportError: cannot import name 'contextfilter' from 'jinja2' (anaconda\lib\site-packages\jinja2\__init__.py)

Follow these steps:

1. Switch to the main environment.

2. Run the following command:

   pip install nbconvert --upgrade
   

Executing the following command hangs on MacOS Ventura M1:

wget https://raw.githubusercontent.com/alexeygrigorev/datasets/master/housing.csv

If you encounter this and see IPv6 addresses in the terminal, follow these steps:

1. Go to System Settings.
2. Select Network.
3. Choose your network connection and click Details.
4. Set Configure IPv6 to Manually.
5. Click OK.
6. Try the command again.

Wget doesn't ship with macOS, but you can use curl as an alternative.

Example command:

curl -o ./housing.csv https://raw.githubusercontent.com/alexeygrigorev/datasets/master/housing.csv

Explanation:

• curl: A utility for retrieving information from the internet.
• -o: Specifies the output filename for the file being downloaded.
• filename: Your choice for naming the file.
• URL: The web address from which curl will download the data and save it using the specified filename.

For more information, you can refer to the Curl Documentation

You can use the round() function or f-strings:

• Using round() function:

  round(number, 4)  # This will round number up to 4 decimal places
  

• Using f-strings:

  print(f'Average mark for the Homework is {avg:.3f}')  # Formats the number to 3 decimal places
  

• Using pandas.Series.round if you need to round values in a whole Series:

  See the documentation for more information: pandas.Series.round

If you’re struggling to get a Jupyter notebook running locally on your machine or have other issues (like only having a cellphone available), consider using the following online platforms that don't require installation:

• JupyterLab
• JupyterLite
• Replit
• Google Colab

I was doing Question 7 from Week 1 Homework and with step 6: Invert XTX. I created the inverse. Now, an inverse when multiplied by the original matrix should return an identity matrix. But when I multiplied the inverse with the original matrix, it gave a matrix like this:

Inverse * Original:

[[ 1.00000000e+00 -1.38777878e-16]
 [ 3.16968674e-13  1.00000000e+00]]

Solution:

It's because floating point math doesn't work well on computers as shown here: https://stackoverflow.com/questions/588004/is-floating-point-math-broken

First of all, use np.dot for matrix multiplication. When you perform matrix-matrix multiplication, remember that the order of multiplication is crucial and affects the result.

Dimension Mismatch

To perform matrix multiplication, the number of columns in the first matrix should match the number of rows in the second matrix. Rearrange the order to satisfy this condition.

You would first get the average of the column and save it to a variable, then replace the NaN values with the average variable.

This method is called imputing - when you have NaN/null values in a column, but you do not want to get rid of the row because it has valuable information contributing to other columns.

When you calculate the mode using the mode() function in pandas, the function always returns a Series. This design choice allows mode() to handle cases where there may be multiple modes (i.e., multiple values with the same highest frequency). Even when there is only one mode, the function will still return a Series with that single value.

If you are certain that your column has only one mode and you want to extract it as a single value, you can access the first element of the Series returned by mode():

single_mode_value = your_dataframe['your_column'].mode()[0]

In Question 7, we are asked to calculate the following:

• X^T X
• (X^T X)^{-1}
• w = (X^T X)^{-1} X^T y

The initial problem w = X^{-1} y can be solved by this, where a matrix X is multiplied by some unknown weights w resulting in the target y.

Additional Reading and Videos

• Ordinary least squares
• Multiple Linear Regression in Matrix Form
• Pseudoinverse Solution to OLS

This is most likely because you interchanged the first step of the multiplication.

Ensure you use:

• Correct:

  XTX = XT X
  

• Instead of incorrect:

  XTX = X XT
  

Matrix multiplication, such as matrix-matrix or matrix-vector multiplication, is often represented using the * operator. However, in NumPy, it is performed using the @ operator or np.matmul(). The * operator in NumPy is used for element-wise multiplication, also known as the Hadamard product).

For matrix-matrix multiplication, using the @ operator or np.matmul() is preferred, as noted in the NumPy documentation.

When multiplying by a scalar, it's preferred to use numpy.multiply() or the * operator.

References:

• numpy.dot()
• ndarray.dot()

To avoid accidentally pushing CSV files (or any specific file type) to a Git repository, you can use a .gitignore file.

• Add a rule to ignore CSV files by including:

  *.csv
  

• If the CSV files have already been committed, you can remove them from Git tracking but keep them locally by using the command:

  git rm --cached filename.csv
  

To analyze the long tail of data, you can use a histogram or check skewness and descriptive statistics.

Using Histogram

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

# Load the data
url = 'https://raw.githubusercontent.com/alexeygrigorev/datasets/master/housing.csv'
df = pd.read_csv(url)

# EDA
sns.histplot(df['median_house_value'], kde=False)
plt.show()

Check Skewness and Descriptive Statistics

# Describe the 'median_house_value'
print(df['median_house_value'].describe())

# Calculate the skewness of the 'median_house_value' variable
skewness = df['median_house_value'].skew()

# Print the skewness value
print("Skewness of 'median_house_value':", skewness)

It’s possible that when you follow the videos, you’ll get a Singular Matrix error. This will be explained in the Regularization video. Don’t worry, it’s normal to encounter this.

You might also receive this error if you invert matrix X more than once in your code.

I was using for loops to apply RMSE to a list of y_val and y_pred. However, the resulting RMSE was all NaN.

I discovered that the issue occurred during the mean calculation step in the RMSE function, after squaring the error. There were NaNs in the array, which I traced back to the initial data splitting step. I had only used fillna(0) on the training data, not on the validation and test data.

The problem was resolved by applying fillna(0) to all datasets (train, validation, and test). My for loops now successfully compute RMSE for all seed values.

Why should we transform the target variable to logarithm distribution? Do we do this for all machine learning projects?

Only if you see that your target is highly skewed. The easiest way to evaluate this is by plotting the distribution of the target variable.

Transforming to a logarithmic scale can help address skewness and improve the distribution of your data set.

For more information, you can refer to Skewness on Wikipedia.

To load a dataset in Kaggle Notebooks, you can use the following command. Remember that the ! before wget is essential.

!wget https://raw.githubusercontent.com/alexeygrigorev/datasets/master/housing.csv

Once the dataset is loaded onto the Kaggle Notebook server, it can be read using the following pandas command:

df = pd.read_csv('housing.csv')

We can filter a dataset by using its values as shown below:

# Using OR condition
 df = df[(df['ocean_proximity'] == '<1H OCEAN') | (df['ocean_proximity'] == 'INLAND')]

You can use | for 'OR', and & for 'AND'.

Alternative method:

# Using isin()
 df = df[df['ocean_proximity'].isin(['<1H OCEAN', 'INLAND'])]

Here's another way to load a dataset using the requests library:

import requests

url = 'https://raw.githubusercontent.com/alexeygrigorev/datasets/master/housing.csv'

response = requests.get(url)

if response.status_code == 200:
    with open('housing.csv', 'wb') as file:
        file.write(response.content)
else:
    print("Download failed.")

When creating a duplicate of your dataframe, if you do the following:

X_train = df_train
X_val = df_val

You're still referencing the original variable. This is called a shallow copy. To ensure that no references are attaching both variables and to keep a copy of the data, create a deep copy:

X_train = df_train.copy()
X_val = df_val.copy()

Yes, you can. Here we implement it ourselves to better understand how it works, but later we will only rely on Scikit-Learn’s functions. If you want to start using it earlier — feel free to do it.

Yes, you can. We will also do that next week, so don’t worry, you will learn how to do it.

What are the equivalents in Scikit-Learn for linear regression with and without regularization used in week 2?

Without Regularization:

sklearn.linear_model.LinearRegression

With Regularization:

sklearn.linear_model.Ridge

For more information, you can refer to the Scikit-Learn documentation on linear models:

Scikit-Learn Linear Models

Linear regression often provides a good approximation of the underlying relationship but rarely achieves a "perfect" fit in real-world applications.

Q: Why is y_pred different from y?

In lesson 2.8, the question arises: after training X_train to get the weights, shouldn't multiplying by X_train give exactly y?

A: Linear regression is a simple model and should not fit 100%, as this would indicate overfitting. Consider a single feature X:

As the model is linear, how would you draw a line to fit all the "dots"?

You could "fit" all the "dots" using something like scipy.optimize.curve_fit (non-linear least squares), but consider how it would perform on previously unseen data.

Refer to: scipy.optimize.curve_fit

One of the questions on the homework calls for using a random seed of 42. When using 42, all my missing values ended up in my training dataframe and not my validation or test dataframes, why is that?

The purpose of the seed value is to randomly generate the proportion split. Using a seed of 42 ensures that all learners are on the same page by getting the same behavior (in this case, all missing values ending up in the training dataframe). If using a different seed value (e.g. 9), missing values will then appear in all other dataframes.

It is possible to shuffle the dataset using the pandas built-in function pandas.DataFrame.sample. To shuffle the complete dataset and reset the index, use the following commands:

• Set frac=1 to return a shuffled version of the complete dataset.
• Set random_state=seed for consistent randomization.

df_shuffled = df.sample(frac=1, random_state=seed)
df_shuffled.reset_index(drop=True, inplace=True)

While the lectures have you use the shuffle function to shuffle the index of the dataframe, it no longer accepts random seed as a parameter. This is because Numpy converted this feature into its own "Generator Class". In order to assign the random generator a seed, you have to specify the object (rng) that you are going to utilize in your code:

# Create index from range of values in array
idx = np.arange(n)

# Create random generator object and set seed
rng = np.random.default_rng(random_seed)

# Shuffle values using Generator object
rng.shuffle(idx)

When the target variable has a long tail distribution, such as prices with a wide range, you can transform it using the np.log1p() method. However, be aware that this method will not work if your target variable contains negative values.

X_train = prepare_X(df_train)
w_0, w = train_linear_regression(X_train, y_train)

X_val = prepare_X(df_val)
y_pred = w_0 + X_val.dot(w)

rmse(y_val, y_pred)

We get:

ValueError                                Traceback (most recent call last)
Input In [132], in <cell line: 5>()
      2 w_0, w = train_linear_regression(X_train, y_train)
      4 X_val = prepare_X(df_val)
----> 5 y_pred = w_0 + X_val.dot(w)
      7 rmse(y_val, y_pred)

ValueError: shapes (4128,) and (1,) not aligned: 4128 (dim 0) != 1 (dim 0)

If we try to perform an arithmetic operation between two arrays of different shapes or dimensions, it throws an error like operands could not be broadcast together with shapes. Broadcasting can occur in certain scenarios and will fail in others.

To solve this issue, you can use the * operator instead of the dot() method:

X_train = prepare_X(df_train)
w_0, w = train_linear_regression(X_train, y_train)

X_val = prepare_X(df_val)
y_pred = w_0 + (X_val * w)

rmse(y_val, y_pred)

Output:

0.5713144443358035

Copy a dataframe using:

X_copy = X.copy()

This creates a deep copy of the dataframe. If you use X_copy = X, it will create a "view" and any changes to X_copy will affect the original dataframe X. This is not a real copy.

One of the most important characteristics of the normal distribution is that mean = median = mode, meaning the most popular value, the mean of the distribution, and 50% of the sample are under the same value. This is equivalent to saying that the area under the curve (black) is the same on the left and on the right. The long tail (red curve) results from a few observations with high values, altering the behavior of the distribution. Consequently, the area is different on each side, and the mean, median, and mode become different. The mean is no longer representative, the range is larger, and the probability of being on the left or right is not the same.

In statistics, the standard deviation is a measure of the amount of variation or dispersion of a set of values. A low standard deviation indicates that the values tend to be close to the mean (also called the expected value) of the set, while a high standard deviation indicates that the values are spread out over a wider range. Wikipedia The formula to calculate standard deviation is:

The application of regularization depends on the specific situation and problem. It is recommended to consider it when training machine learning models, especially with small datasets or complex models, to prevent overfitting. However, its necessity varies depending on the data quality and size. Evaluate each case individually to determine if it is needed.

Defining functions can speed up development significantly. You can create a function like prepare_df(initial_df, seed, fill_na_type) to prepare all three dataframes and y_vectors. The fillna() operation can be applied before splitting the initial_df.

Additionally, you can reuse functions such as rmse() and train_linear_regression(X, y, r) from the class notebook.

When applying a function to a DataFrame, it is important to consider that if you do not want to alter the original DataFrame, you should create a copy of it first. Failing to do so may result in unintended modifications to the original dataset.

To preserve the integrity of your data, always use df.copy() before making any changes.

To find the standard deviation of a list or series of data using Pandas, you can convert the list into a Pandas Series and use the .std() function. For example:

import pandas as pd

x = [1, 2, 3, 4, 5]
standard_deviation = pd.Series(x).std()
print(standard_deviation)

This will calculate the standard deviation of the list x.

Numpy and Pandas use different equations to compute the standard deviation. Numpy uses the population standard deviation by default, whereas Pandas uses the sample standard deviation.

Numpy

Pandas

Pandas computes the standard deviation using one degree of freedom by default. You can modify the degree of freedom in Numpy to achieve a similar result by using the ddof parameter:

import numpy as np

np.std(df.weight, ddof=1)

The result will be similar if we set ddof=1 in Numpy.

In pandas, you can use the built-in function std() to calculate the standard deviation. For example:

• To get the standard deviation of a single column:

  df['column_name'].std()
  

• To get the standard deviation of multiple columns:

  df[['column_1', 'column_2']].std()
  

Use pandas.concat function (pandas documentation) to combine two dataframes. To combine two numpy arrays, use numpy.concatenate (numpy documentation).

The code would be as follows:

df_train_combined = pd.concat([df_train, df_val])
y_train = np.concatenate((y_train, y_val), axis=0)

The Root Mean Squared Error (RMSE) is one of the primary metrics to evaluate the performance of a regression model. It calculates the average deviation between the model's predicted values and the actual observed values, offering insight into the model's ability to accurately forecast the target variable. To calculate the RMSE score:

1. Import the necessary libraries:

   import numpy as np
   from sklearn.metrics import mean_squared_error
   

2. Calculate the Mean Squared Error (MSE):

   mse = mean_squared_error(actual_values, predicted_values)
   

3. Compute the RMSE:

   rmse = np.sqrt(mse)
   print("Root Mean Squared Error (RMSE):", rmse)
   

I found this video useful for understanding how we derive the normal form in linear regression: Normal Equation Derivation for Regression.

A Guide to Handling Missing Values in Python

Version 0.24.2 and Python 3.8.11.

In the context of regression, particularly with regularization:

• r typically represents the regularization parameter in some algorithms. It controls the strength of the penalty applied to the coefficients of the regression model to prevent overfitting.

• In sklearn.Ridge(), the parameter alpha serves the same purpose as r. It specifies the amount of regularization applied to the model. A higher value of alpha increases the amount of regularization, which can reduce model complexity and improve generalization.

r and alpha are both regularization parameters and control the "strength" of regularization. Increasing these values will lead to stronger regularization. However, the mathematical implementation differs:

• sklearn.Ridge():

  ||y - Xw||^2_2 + alpha * ||w||^2_2
  

• Lesson's Notebook (train_linear_regression_reg function):

  XTX = XTX + r * np.eye(XTX.shape[0])
  

  r adds “noise” to the main diagonal to prevent multicollinearity, which “breaks” finding the inverse matrix.

For further reference, see the sklearn.Ridge documentation and the lesson’s notebook.

That’s normal. We all have different environments: our computers have different versions of OS and different versions of libraries — even different versions of Python.

If it’s the case, just select the option that’s closest to your answer.

In question 3 of HW02 it is mentioned: ‘For computing the mean, use the training only’. What does that mean?

It means that you should use only the training data set for computing the mean, not the validation or test data set. This is how you can calculate the mean:

# Calculate mean for a specific column in the training data
mean_value = df_train['column_name'].mean()

Another option:

# Get descriptive statistics, including the mean
stats = df_train['column_name'].describe()

The instruction for applying log transformation to the ‘median_house_value’ variable is provided before Q3 in the homework for Week-2 under the ‘Prepare and split the dataset’ heading.

However, this instruction is absent in the subsequent questions of the homework, and you might encounter issues like a huge RMSE. Remember to apply log transformation to the target variable for each question.

My result is about 12.4 different from the closest option. For previous questions, my answers were close, so I'm unsure why there's a large discrepancy for question 6.

For questions 5 and 6, please ensure you reinitialize with:

Idx = np.arange(n)

This should be done for each iteration of r in question 5 and also for question 6.

You can handle missing values by:

• Imputing the missing values with the mean, median, or mode.
• Using algorithms that support missing values inherently (e.g., some tree-based methods).
• Removing rows or columns with missing data, depending on the extent of missingness.
• Utilizing feature engineering to derive new features from incomplete data.

The error message "could not convert string to float: 'Nissan'" typically occurs when a machine learning model or function is expecting numerical input but receives a string instead. In this case, it seems like the model is trying to convert the car brand 'Nissan' into a numerical value, which isn’t possible.

To resolve this issue, you can encode categorical variables like car brands into numerical values. One common method is one-hot encoding, which creates new binary columns for each category/label present in the original column.

Here’s an example of how you can perform one-hot encoding using pandas:

import pandas as pd

# Assuming 'data' is your DataFrame and 'brand' is the column with car brands
data_encoded = pd.get_dummies(data, columns=['brand'])

In this code, pd.get_dummies() creates a new DataFrame where the brand column is replaced with binary columns for each brand (e.g., brand_Nissan, brand_Toyota, etc.). Each row in the DataFrame has a 1 in the column that corresponds to its brand and 0 in all other brand columns.

Mutual Information score calculates the relationship between categorical or discrete variables. In the homework, the target, median_house_value, was continuous. Thus, we changed it to a binary format to make its values discrete (either 0 or 1).

Keeping the target as a continuous variable would require the algorithm to divide it into bins, which would be highly subjective. This is why continuous variables are not used for mutual information score calculation.

First, you have to consider whether the data is numerical or categorical. If it’s numerical, you can correlate it directly. If it’s categorical, you can find the correlations indirectly by vectorizing the data using One-Hot encoding or a similar method.

To determine if data is numerical, check the dtypes of the DataFrame. Data types such as integer and float are numerical, while types such as objects are categorical. You can correlate the numerical data by specifying which columns are numerical and using that as input to a correlation matrix.

Example:

numerical = ['tenure', 'monthlycharges', 'totalcharges']

correlation_matrix = df[numerical].corr()
print(correlation_matrix)

The background of any DataFrame, including the correlation matrix, can be colored based on its numerical values using the method pandas.io.formats.style.Styler.background_gradient.

Here is an example of how to color the correlation matrix. A color map of choice can be passed; here, 'viridis' is used:

• Ensure the DataFrame contains only numerical values before calling corr:

corr_mat = df_numerical_only.corr()

corr_mat.style.background_gradient(cmap='viridis')

• Here is an example of how the coloring will look using a DataFrame containing random values and applying background_gradient to it:

np.random.seed = 3

df_random = pd.DataFrame(data=np.random.random(3*3).reshape(3,3))

print(df_random.style.background_gradient(cmap='viridis'))

To identify highly correlated feature pairs using unstack:

import pandas as pd

data_corr = pd.DataFrame(data_num.corr().round(3).abs().unstack().sort_values(ascending=False))

print(data_corr.head(10))

You can also use seaborn to create a heatmap with the correlation:

import seaborn as sns

sns.heatmap(
    df[numerical_features].corr(),
    annot=True,
    square=True,
    fmt=".2g",
    cmap="crest"
)

To refine your heatmap and plot only a triangle, with a blue to red color gradient, that will show every correlation between your numerical variables without redundant information:

# Set figure size: modify it here or create new function arguments
plt.figure(figsize=(12, 6))

# define the mask to set the values in the upper triangle to True
mask = np.triu(np.ones_like(dataframe.corr(numeric_only=True), dtype=bool))

# Generate a custom diverging colormap
cmap = sns.diverging_palette(230, 20, as_cmap=True)

heatmap = sns.heatmap(
    dataframe.corr(numeric_only=True),
    mask=mask,
    cmap=cmap,
    vmin=-1,
    vmax=1,
    annot=True,
    linewidths=0.5,
)

heatmap.set_title(title, fontdict={"fontsize": 18}, pad=16)
plt.show()

Which outputs, in the case of a churn dataset:

It's indeed good practice to only rely on the train dataset for EDA. Including validation might be okay. But we aren't supposed to touch the test dataset; even just looking at it isn't a good idea. We indeed pretend that this is the future unseen data.

Validation datasets are used to optimize models by providing an estimate of performance on unseen data. Understanding how to properly use the DictVectorizer class is crucial for maintaining this separation between training and validation.

• Fitting on Training Data: The fit method of DictVectorizer analyzes the training dataset to determine how to map dictionary values. Categorical features are one-hot encoded, while numeric features remain unchanged.
• Avoid Fitting on Validation Data: Applying the fit method to validation data can lead to information leakage, as it exposes the model to data it should not see during training.
• Appropriate Usage:
  1. Use fit_transform on the training dataset.
  2. Use transform only on validation and test datasets.

By following these practices, the model's performance on new data can be more accurately assessed.

In newer versions of scikit-learn, the method has been replaced by get_feature_names_out().

Instead, use the method .get_feature_names_out() from the DictVectorizer function to resolve the warning.

# Example usage
from sklearn.feature_extraction import DictVectorizer

# Initialize the vectorizer
vectorizer = DictVectorizer()

# After fitting the vectorizer
vectorizer.fit_transform(...)

# Get feature names
feature_names = vectorizer.get_feature_names_out()

Note: The warning indicates that get_feature_names will be removed, so switching to get_feature_names_out is recommended even though the warning itself won't cause issues yet.

Fitting the logistic regression takes a long time or the kernel crashes when calling predict() with the fitted model.

Ensure that the target variable for the logistic regression is binary.

Ridge regression is a linear regression technique used to mitigate the problem of multicollinearity (when independent variables are highly correlated) and prevent overfitting in predictive modeling. It adds a regularization term to the linear regression cost function, penalizing large coefficients.

• sag Solver: The sag solver stands for "Stochastic Average Gradient." It's particularly suitable for large datasets, as it optimizes the regularization term using stochastic gradient descent (SGD). sag can be faster than some other solvers for large datasets.

• Alpha: The alpha parameter controls the strength of the regularization in Ridge regression. A higher alpha value leads to stronger regularization, which means the model will have smaller coefficient values, reducing the risk of overfitting.

from sklearn.linear_model import Ridge

ridge = Ridge(alpha=alpha, solver='sag', random_state=42)

ridge.fit(X_train, y_train)

DictVectorizer(sparse=True) produces CSR) format, which is both more memory efficient and converges better during fit().

It stores non-zero values and indices instead of adding a column for each class of each feature, which can result in large numbers of columns (e.g., models of cars).

Using "sparse" format is slower (around 6-8 minutes for Q6 task - Linear/Ridge Regression) for a high number of classes (like car models) and produces slightly worse results in both Logistic and Linear/Ridge Regression.

It also generates convergence warnings for Linear/Ridge Regression.

If you're encountering the following warning:

ConvergenceWarning: The max_iter was reached which means the coef_ did not converge

This usually happens because the solver the model uses is sensitive to feature scales.

You can do the following to address it:

1. Normalize Numerical Features

   • Scale your numerical features using techniques like StandardScaler or MinMaxScaler.
   • This ensures that all numerical features are on a similar scale, which helps the solver converge.

2. Encode Categorical Features

   • Apply OneHotEncoder (OHE) to categorical features to represent them as binary vectors.
   • Use sparse=False when necessary to return a dense array.

3. Separate and Combine Features

   • Process numerical and categorical features separately (scaling for numerical, OHE for categorical).
   • Combine them afterward into a single feature matrix (X_train) to use as input for Ridge regression.

4. Experiment with Different Scalers

   • If issues persist, try different scalers as Ridge can behave differently depending on feature scaling.

By following these steps, you can reduce convergence errors and improve model stability. For a detailed example, see this notebook: notebook-scaling-ohe.ipynb.

A sparse matrix is more memory-efficient because it only stores the non-zero values and their positions in memory. This is particularly useful when working with large datasets with many zero or missing values.

The default DictVectorizer configuration is a sparse matrix. For Week 3, Question 6, using the default sparse configuration is beneficial due to the size of the matrix. Training the model was also more performant and didn’t produce an error message like dense mode.

The warnings in Jupyter notebooks can be disabled or avoided with the following commands:

import warnings

warnings.filterwarnings("ignore")

You need to calculate the mutual information score between the binarized price (above_average) variable and ocean_proximity, the only original categorical variable in the dataset.

Both work in similar ways to convert categorical features to numerical variables for use in training a model. The difference lies in the input:

• OneHotEncoder uses an array as input.
• DictVectorizer uses a dictionary.

Both will produce the same result. However, with OneHotEncoder, features are sorted alphabetically. With DictVectorizer, you stack features as desired.

They are basically the same. There are some key differences with regards to their input/output types, handling of missing values, etc., but they are both techniques to one-hot-encode categorical variables with identical results.

• pandas get_dummies: A convenient choice when working with Pandas DataFrames.
• sklearn OneHotEncoder: More suitable for building a scikit-learn-based machine learning pipeline to handle categorical data as part of that pipeline.

Correlation should be calculated after splitting the data, specifically on the train dataset.

To find the two most correlated features:

1. Generate the correlation matrix of the train dataset.
2. Identify the pair of features with the highest absolute correlation coefficient.

Use sklearn.preprocessing encoders and scalers, e.g. OneHotEncoder, OrdinalEncoder, and StandardScaler.

These methods both receive a dictionary as input. While the DictVectorizer will store a large vocabulary and take up more memory, FeatureHasher creates vectors with a predefined length. They are both used for handling categorical features.

• If you have high cardinality in categorical features, it's better to use FeatureHasher.
• If you want to preserve feature names in transformed data and have a small number of unique values, use DictVectorizer.

Your choice will depend on your data. For more information, you can visit scikit-learn.org

The reason it's recommended to do it after splitting is to avoid data leakage. You don't want any data from the test set influencing the training stage, similarly from the validation stage in the initial training. See e.g. scikit-learn documentation on "Common pitfalls and recommended practices": https://scikit-learn.org/stable/common_pitfalls.html

This article explains different encoding techniques used.

All About Categorical Variable Encoding

I got this error multiple times; here is the code:

accuracy_score(y_val, y_pred >= 0.5)

TypeError: 'numpy.float64' object is not callable

I solved it using:

from sklearn import metrics

metrics.accuracy_score(y_train, y_pred >= 0.5)

In Scikit-Learn’s LogisticRegression, a model that is trained will have raw values and the predicted probabilities.

• .decision_function() returns raw values that are a linear combination of the features and weights, similar to the output of Linear Regression.

• .predict_proba() goes one step further by inputting these raw values into the sigmoid function to convert them into probabilities (between 0 and 1).

The error occurs because some features you try to drop have been one-hot encoded into multiple columns. After encoding, the original column may no longer exist, leading to the KeyError. To resolve this, identify and drop all related one-hot encoded columns (e.g., those starting with the original feature name) instead of the original feature itself.

For example, after one-hot encoding, the column marital could have been split into columns like marital_single, marital_married, etc. This means that the original column marital no longer exists, leading to the KeyError.

Q2 asks about the correlation matrix and converting median_house_value from numeric to binary. Just to clarify, we are only dealing with df_train, not df_train_full, correct? The question explicitly mentions the train dataset.

Yes, it is only on df_train. The reason is that df_train_full also contains the validation dataset. At this stage, we don't want to make conclusions based on the validation data, since we want to test how we did without using that portion of the data.

For Q5 in homework, should we calculate the smallest difference in accuracy in real values (i.e. -0.001 is less than -0.0002) or in absolute values (i.e. 0.0002 is less than 0.001)?

We should select the “smallest” difference, and not the “lowest”, meaning we should reason in absolute values.

If the difference is negative, it means that the model actually became better when we removed the feature.

To select the alpha parameter, you need to find the RMSE for each alpha. If RMSE scores are equal, choose the lowest alpha.

Do we need to train the model only with the features: total_rooms, total_bedrooms, population, and households, or with all the available features, then remove each of the previous features one at a time to compare accuracy?

1. Create a list of all features and evaluate the model to obtain the original accuracy.
2. Remove one feature at a time.
3. Train the model each time, calculate the accuracy, and find the difference between the original accuracy and the new accuracy.
4. Identify which feature has the smallest absolute accuracy difference.

While calculating differences between accuracy scores, use the smallest absolute difference. For example, if the differences are -4 and -2, the smallest absolute difference is abs(-2). Use this value to determine the impact of the feature on accuracy.

For the test_train_split question on week 3's homework, are we supposed to use 42 as the random_state in both splits or only the first one?

Answer: For both splits, random_state = 42 should be used.

When searching for the best value of C that yields the highest accuracy, be mindful that you should be looking for the maximum accuracy, not the minimum.

Although the goal is to find the smallest C value, ensure that it corresponds to the highest accuracy achieved. Always prioritize accuracy maximization while minimizing C.

If you are getting 1.0 as accuracy, it is possible that you have overfitted the model. Dropping the column msrp/price can help you solve this issue.

This is not possible since the parameter C represents the inverse of the regularization strength. Setting C to 0 means infinite regularization. Attempting this with the scikit-learn module of Logistic Regression will result in a ValueError.

Import the data using the following command:

import pandas as pd

df = pd.read_csv("bank-full.csv", sep=';')

Note that the data is separated by a semicolon, not a comma.

Refer to the sklearn documentation, random_state is used to ensure the "randomness" that is used to shuffle the dataset is reproducible. It typically requires both random_state and shuffle parameters to be set accordingly.

To get classification metrics like precision, recall, F1 score, and accuracy simultaneously, use classification_report from sklearn.

For more information, check here.

This error indicates that your dataset's churn column only contains the class 0, but at least two classes are required.

1. Check your data processing steps where binary conversion might be applied. Specifically, ensure that the line:

   df.churn = (df.churn == 'yes').astype(int)
   

   is operating correctly. Verify that there are indeed records where churn should evaluate to 1 (i.e., cases where churn equals 'yes').

2. If all values are 0, make sure your original dataset and preprocessing steps are correctly implemented to represent cases with both classes (0 and 1).

3. Review data preprocessing steps and confirm the filtering, transformation, or data importing steps do not inadvertently drop or misclassify the non-zero class records.

This should resolve the error by ensuring your data contains at least one record for each class.

Use Yellowbrick. Yellowbrick is a library that combines scikit-learn with matplotlib to produce visualizations for your models. It produces colorful classification reports.

Check the solutions from the 2021 iteration of the course. You should use roc_auc_score.

When calculating the ROC AUC score using sklearn.metrics.roc_auc_score, the function expects two parameters: y_true and y_score. For each numerical value in the dataframe, it will be passed as the y_score to the function, and the target variable will be passed as y_true each time.

KFold is a cross-validation technique that splits your dataset into k equal parts (folds). It trains the model k times, each time using a different fold as the validation set while training on the remaining folds. This process helps provide a more reliable estimate of a model's performance by ensuring every data point gets to be in both the training and validation sets. The average score across all folds offers a robust evaluation, minimizing the risk of overfitting to a specific train-test split.

What does this line do?

KFold(n_splits=n_splits, shuffle=True, random_state=1)

• Positioning in Code: Whether you instantiate KFold inside the loop over different regularization values like [0.01, 0.1, 1, 10] or outside, it typically does not affect your answer. This is because KFold is essentially a generator object containing the information n_splits, shuffle, and random_state.

• Impact of Random State: Changing the random_state can yield different results because it affects how the data is shuffled. However, the creation of the KFold object, either inside or outside a loop, doesn't make a difference as long as the configuration (n_splits, shuffle, random_state) remains constant.

• Best Practice: It is recommended to create the KFold object before the loop to avoid unnecessary repetition:

  kFold = KFold(n_splits=n_splits, shuffle=True, random_state=1)
  for C in [0.01, 0.1, 1, 10]:
      for train_idx, val_idx in kFold.split(df_full_train):
          # train and evaluate model
  

For more details, you can refer to the official scikit-learn documentation.

I'm getting "ValueError: multi_class must be in ('ovo', 'ovr')" when using roc_auc_score to evaluate feature importance of numerical variables in question 1.

This error occurs because the parameters were passed to roc_auc_score incorrectly. Here is the correct usage:

roc_auc_score(y_train, df_train[col])

Inverting or negating variables with ROC AUC scores less than the threshold is a valuable technique to improve feature importance and model performance when dealing with negatively correlated features. It helps ensure that the direction of the correlation aligns with the expectations of most machine learning algorithms.

Using predict(X) provides binary classification predictions, which are either 0 or 1. This could result in inaccurate evaluation values.

The alternative is to use predict_proba(X)[:, 1], which gives the probability that the value belongs to a specific class.

predict_proba displays probabilities for each class.

For churn/not churn predictions, when the threshold is 1.0:

• FPR (False Positive Rate) is 0.0
• TPR (True Positive Rate) is 0.0

When the threshold is set to 1.0, the condition for belonging to the positive class (churn class) is g(x) >= 1.0. However, g(x) is a sigmoid function in a binary classification problem, which produces values between 0 and 1. The function never reaches the outermost values of 0 or 1.

Therefore, no sample will satisfy the condition for the positive class (churn), resulting in no positive (churn) predictions. Consequently, this leads to both the false positive and true positive rates being 0.0 when the threshold is 1.0.

Matplotlib has a cool method to annotate where you can provide an X,Y point and annotate with an arrow and text. For example, this will show an arrow pointing to the x,y point optimal threshold.

plt.annotate(f'Optimal Threshold: {optimal_threshold:.2f}\nOptimal F1 Score: {optimal_f1_score:.2f}',

xy=(optimal_threshold, optimal_f1_score),

xytext=(0.3, 0.5),

textcoords='axes fraction',

arrowprops=dict(facecolor='black', shrink=0.05))

It's a complex and abstract topic and it requires some time to understand. You can move on without fully understanding the concept.

Nonetheless, it might be useful for you to rewatch the video, or even watch videos/lectures/notes by other people on this topic, as the ROC AUC is one of the most important metrics used in Binary Classification models.

You can find the intercept between these two curves using numpy's diff and sign functions:

1. Ensure your df_scores DataFrame is ready with three columns: threshold, precision, and recall.

2. Determine the indices where the precision and recall curves intersect (i.e., where the sign of the difference between precision and recall changes):

   import numpy as np
   
   idx = np.argwhere(
       np.diff(
           np.sign(np.array(df_scores['precision']) - np.array(df_scores['recall']))
       )
   ).flatten()
   

3. Print the result to easily read it:

   print(f"The precision and recall curves intersect at a threshold equal to {df_scores.loc[idx]['threshold']}.")
   

You can use the Scikit Learn library to calculate precision, recall, and F1 score without having to define true positive, true negative, false positive, and false negative manually.

from sklearn.metrics import precision_score, recall_score, f1_score

precision = precision_score(y_true, y_pred, average='binary')
recall = recall_score(y_true, y_pred, average='binary')
f1 = f1_score(y_true, y_pred, average='binary')

Replace y_true and y_pred with your actual data. The average parameter is set to 'binary' by default for binary classification.

Cross-validation evaluates the performance of a model and chooses the best hyperparameters. It does this by splitting the dataset into multiple parts (folds), typically 5 or 10. It then trains and evaluates your model multiple times, each time using a different fold as the validation set and the remaining folds as the training set.

"C" is a hyperparameter typically associated with regularization in models like Support Vector Machines (SVM) and logistic regression.

• Smaller "C" values: They introduce more regularization, which means the model will try to find a simpler decision boundary, potentially underfitting the data. This is because it penalizes the misclassification of training examples more severely.

• Larger "C" values: They reduce the regularization effect, allowing the model to fit the training data more closely, potentially overfitting. This is because it penalizes misclassification less severely, allowing the model to prioritize getting training examples correct.

Model evaluation metrics can be easily computed using the off-the-shelf calculations available in the scikit-learn library. This method is more precise compared to calculating from scratch using numpy and pandas libraries.

from sklearn.metrics import (
    accuracy_score,
    precision_score,
    recall_score,
    f1_score,
    roc_auc_score
)

accuracy = accuracy_score(y_val, y_pred)
precision = precision_score(y_val, y_pred)
recall = recall_score(y_val, y_pred)
f1 = f1_score(y_val, y_pred)
roc_auc = roc_auc_score(y_val, y_pred)

print(f'Accuracy: {accuracy}')
print(f'Precision: {precision}')
print(f'Recall: {recall}')
print(f'F1-Score: {f1}')
print(f'ROC AUC: {roc_auc}')

Scikit-learn offers another way: precision_recall_fscore_support.

Example:

from sklearn.metrics import precision_recall_fscore_support

precision, recall, fscore, support = precision_recall_fscore_support(y_val, y_val_pred, zero_division=0)

ROC curves are appropriate when the observations are balanced between each class, whereas precision-recall curves are appropriate for imbalanced datasets.

The reason for this recommendation is that ROC curves present an optimistic picture of the model on datasets with a class imbalance. This is because of the use of true negatives in the False Positive Rate in the ROC Curve and the careful avoidance of this rate in the Precision-Recall curve.

If the proportion of positive to negative instances changes in a test set, the ROC curves will not change. Metrics such as accuracy, precision, lift, and F scores use values from both columns of the confusion matrix. As a class distribution changes these measures will change as well, even if the fundamental classifier performance does not. ROC graphs are based upon TP rate and FP rate, in which each dimension is a strict columnar ratio, so cannot give an accurate picture of performance when there is class imbalance.

Precision-recall curves, and thus the F-score, explicitly depend on the ratio of positive to negative test cases. This means that comparing the F-score across different problems with differing class ratios can be problematic.

One way to address this issue is to use a standard class ratio when making such comparisons.

We can import precision_recall_curve from scikit-learn and plot the graph as follows:

from sklearn.metrics import precision_recall_curve

precision, recall, thresholds = precision_recall_curve(y_val, y_predict)

plt.plot(thresholds, precision[:-1], label='Precision')
plt.plot(thresholds, recall[:-1], label='Recall')
plt.legend()

For multiclass classification, it is important to keep class balance when you split the dataset. Stratified k-fold returns folds that contain approximately the same percentage of samples of each class.

Please check the implementation in the scikit-learn library: https://scikit-learn.org/stable/modules/cross_validation.html#stratified-k-fold

Accuracy is the proportion of correct predictions made by the model, but it can be misleading, especially with imbalanced datasets. For example, if 95% of your data belongs to one class, a model that always predicts this majority class will have high accuracy, even though it completely fails to identify the minority class. In such cases, metrics like precision, recall, F1-score, or AUROC might be more appropriate, as they provide a clearer view of model performance on both classes.

Precision is TruePositive/PredictedPositive and recall means TruePositive / ActualPositive.

• Precision: Precise predictions (how accurate are our YES predictions?)
• Recall: Remembering (how many real YES cases did we find?)

Precision:

Memory tip: Think of Precision as "How Precise Are Our Positive Predictions?" It relates to the accuracy of the positive results, emphasizing how many of the predicted positive instances are actually correct.

Interpretation:

• High Precision:

  • Most of the predicted positives are correct.
  • This makes the model more reliable.

• Low Precision:

  • Indicates a higher rate of false positives.
  • This decreases trust in the positive predictions.

When to prioritize precision: In scenarios like email spam detection, where marking a legitimate email as spam (false positive) can lead to missed communications, high precision is preferred to ensure that most flagged emails are indeed spam.

Recall:

Memory tip: Think of Recall as "How Sensitive Are We to the Positives?" It emphasizes capturing all actual positive cases. A high recall means that the model is good at identifying most of the positives.

Interpretation:

• High Recall:

  • The model captures most of the true positives.
  • This is crucial in situations where missing a positive case is costly.

• Low Recall:

  • Many actual positives are overlooked.
  • This highlights potential issues in detection.

When to prioritize recall: In medical diagnostics for a severe or highly contagious disease, missing a true positive (an actual case of the disease) can have serious public health implications.

Balancing Precision and Recall:

• Improving one metric may lead to a decrease in the other.
• The choice between precision and recall depends on specific goals and acceptable trade-offs in a given application.

This warning occurs when your model doesn't predict any samples for certain labels, causing a zero-division error when calculating the F-score. The warning is triggered when there are no true positives or predicted positives for certain labels, leading to undefined precision or recall.

To address this, you can use the zero_division parameter in scikit-learn's f1_score function. This parameter defines what should happen in cases of zero division:

• Set zero_division=1: This will set the precision, recall, and F-score to 1 when no positive samples are predicted.
• Set zero_division=0: This is the default behavior, setting the metric to 0 when there are no predicted samples for a given label.
• Set zero_division='warn': This is the default behavior, acts like 0 but also raises a warning.

Example usage:

from sklearn.metrics import f1_score, precision_score, recall_score

# For precision score
precision = precision_score(y_true, y_pred, average='weighted', zero_division='warn')

# For recall score
recall = recall_score(y_true, y_pred, average='weighted', zero_division=0)

# For f1-score
f1 = f1_score(y_true, y_pred, average='weighted', zero_division=0)

I am getting multiple thresholds with the same F1 score. Does this indicate I am doing something wrong, or is there a method for choosing? Should I just pick the lowest?

• Choose the threshold closest to any of the options.

You can also use scikit-learn (or other standard libraries/packages) to verify results obtained using your own code. For example, use classification_report to obtain precision, recall, and F1-score.

Refer to the documentation: scikit-learn classification_report

That’s fine, use the closest option.

You must use the dt_val dataset to compute the metrics asked in Question 3 and onwards, as you did in Question 2.

One main reason behind this issue is the method of splitting the data. For example, if we want to split the data into train/validation/test with the ratios 60%/20%/20%, different methods may yield different results even if the final ratios are the same.

1. Method 1:

   df_train, df_temp = train_test_split(df, test_size=0.4, random_state=42)
   df_val, df_test = train_test_split(df_temp, test_size=0.5, random_state=42)
   

2. Method 2:

   df_full_train, df_test = train_test_split(df, test_size=0.2, random_state=42)
   df_train, df_val = train_test_split(df_full_train, test_size=0.25, random_state=42)
   

While both methods achieve the same ratio, the data split differently, resulting in variations in accuracy. It is recommended to use the second method, as it is more consistent with the lessons and homeworks.

You can use the roc_auc_score function from the sklearn.metrics module. Pass the vector of the target variable (e.g., above_average) as the first argument and the vector of feature values as the second one. This function will return the AUC score for the feature that was passed as the second argument.

from sklearn.metrics import roc_auc_score

# Example usage:
auc_score = roc_auc_score(target_variable, feature_values)

Q1 is not making sense to me. The score should be between 0 to 1. I tried computing roc_curve (df_train['age'], y) and the graph does not have the model line. Can anyone clarify?'

The idea of the question is to evaluate the importance of features with respect to the prediction of the binary target variable (yes/no).

In my case, I did the following:

1. Identified the numerical features in the dataset.

2. For each feature in the list of numerical features, I calculated the AUC:

   roc_auc_score(y_target, feature_vector)
   

   Here, y_target is the target variable, and feature_vector contains the values for each numerical column in the train dataset.

3. Created a data frame with two columns: the name of the numerical feature and the ROC AUC score.

4. Sorted the data frame by the ROC AUC score to determine the numerical feature with the highest ROC AUC.

While weeks 1-4 can relatively easily be followed and the associated homework completed with just about any default environment or local setup, week 5 introduces several layers of abstraction and dependencies.

It is advised to prepare your “homework environment” with a cloud provider of your choice. A thorough step-by-step guide for doing so for an AWS EC2 instance is provided in an introductory video taken from the MLOPS course here:

https://www.youtube.com/watch?v=IXSiYkP23zo

Note that (only) small instances can be run for free, and larger ones will be billed hourly based on usage (but can and should be stopped when not in use).

Alternative ways are sketched here: https://github.com/alexeygrigorev/mlbookcamp-code/blob/master/course-zoomcamp/01-intro/06-environment.md

To download CSV data via Jupyter Notebook using the Kaggle API, follow these steps:

1. Set up a Kaggle account:

   • Go to your Kaggle account settings, navigate to the API section, and click Create New Token. This will download a kaggle.json file containing your username and key.

2. Place the kaggle.json file:

   • Ensure the kaggle.json file is in the same directory as your Jupyter Notebook.

3. Set permissions for the kaggle.json file:

   !chmod 600 <ENTER YOUR FILEPATH>/kaggle.json
   

4. Configure the environment:

   • Import the os module and set the Kaggle config directory:

   import os
   os.environ['KAGGLE_CONFIG_DIR'] = '<STRING OF YOUR FILE PATH>'
   

5. Download the dataset:

   • Use the Kaggle API to download your desired dataset:

   !kaggle datasets download -d kapturovalexander/bank-credit-scoring
   

6. Unzip and access the CSV file:

   !unzip -o bank-credit-scoring.zip
   

Follow these steps to seamlessly integrate Kaggle data retrieval into your Jupyter workflow.

• cd ..
  

  Go back to the previous directory.

• ls
  

  List the contents of the current directory.

• cd 'path'/
  

  Navigate to the specified path.

• pwd
  

  Print the current working directory.

• cat 'file name'
  

  Display the contents of a file.

To check your current Python version, open your terminal and run:

python3 --version

For Windows:

1. Visit the official Python website to download the desired version: Python Downloads.
2. Run the installer and ensure you check the box that says "Add Python to PATH" during installation.
3. Complete the installation by following the prompts.

For Python 3:

Open your command prompt or terminal and run the following command:

pip install --upgrade python

Windows 10:

1. Open PowerShell as Admin.
2. Run the following command:

   wsl --install
   

3. Restart your computer.
4. Set up your Linux distribution (e.g., Ubuntu).

Windows 11:

1. Open Windows Terminal as Admin.
2. Run:

   wsl --install
   

3. Restart if prompted.
4. Set up your Linux distribution.

Additional Notes:

• To install a specific distribution, use:

  wsl --install -d <DistributionName>
  

• For updates, run:

  wsl --update
  

It is important to ensure that the "Virtual Machine Platform" feature is activated in your Windows "Features." You can check this by searching for "features" in the search bar to see if the checkbox is selected. Additionally, ensure that your system (in the BIOS) supports virtualization.

In the Microsoft Store, search for ‘Ubuntu’ or ‘Debian’ (or any Linux distribution you want) and install it. After downloading, open the app and choose a username and a password. Note that while typing your password, it may not display any characters (this is normal).

Once inside your Linux system, you can try commands such as pwd. To navigate back to your Windows system:

1. Use cd ../.. twice.

2. Go to the "mnt" directory:

   cd mnt
   

3. List your files to view your disks and move to the desired folder.

   mfouesnard@DESKTOP-39IH8UP:/mnt/c/Users/Melanie/ML_Zoomcamp/ML_ZoomCamp$ ls
   Homework_week2.ipynb  Homework_week3_2023.ipynb  README.md  car_price.csv  data.csv  housing.csv
   Homework_week3_2022.ipynb  Homework_week4_2023.ipynb  Untitled.ipynb  churn.csv  homework_week1.ipynb
   

Python should be already installed, but you can check with:

sudo apt install python3

To make your current folder the default when opening Ubuntu terminal, use:

echo "cd ../../mnt/your/folder/path" >> ~/.bashrc

To disable bell sounds, edit the inputrc file:

1. Open the file:

   sudo vim /etc/inputrc
   

2. Uncomment set bell-style none:
   • Press i (for insert), navigate to the line, delete #, press Escape, and then :wq to save and quit.
3. Open a new terminal to check the changes.

To install pip, run:

sudo apt install python3-pip

Possible Error

You might encounter the following error when installing pipenv:

/sbin/ldconfig.real: Can't link /usr/lib/wsl/lib/libnvoptix_loader.so.1 to libnvoptix.so.1
/sbin/ldconfig.real: /usr/lib/wsl/lib/libcuda.so.1 is not a symbolic link

To resolve, create a symbolic link:

sudo ln -s /usr/lib/wsl/lib/libcuda.so.1 /usr/lib64/libcuda.so

Do you get errors building the Docker image on the Mac M1 chipset?

The error received was:

Could not open '/lib64/ld-linux-x86-64.so.2': No such file or directory

To fix this error:

1. Open the mlbookcamp-code/course-zoomcamp/01-intro/environment/Dockerfile.

2. Replace line 1 with:

   FROM --platform=linux/amd64 ubuntu:latest
   

3. Now build the image as specified.

Note: Building the image may take over 2 hours, but it should complete successfully.

To find the version of a Python library in a Jupyter Notebook, you can use the following method:

import waitress

print(waitress.__version__)

Simply replace waitress with the name of the library you want to check.

Ensure Docker Daemon Is Running

On Windows:

• Open Docker Desktop (admin rights may be required).
• Check if it’s running, and restart Docker Desktop if necessary.

On Linux:

1. Run the following command to start the Docker daemon:

   sudo systemctl start docker
   

2. Verify it’s running with:

   sudo systemctl status docker
   

Verify Docker Group Membership (Linux Only)

• Check if your user is in the Docker group:

  groups $USER
  

• If "docker" isn’t listed, add yourself with:

  sudo usermod -aG docker $USER
  

• Log out and back in to apply changes.

Restart the Docker Service (Linux)

sudo systemctl restart docker

Check Docker Socket Permissions (Linux)

• Run the following command to confirm Docker socket permissions:

  sudo chmod 666 /var/run/docker.sock
  

Try Running Docker with sudo (Linux)

• Run the following to check if permissions are causing the issue:

  sudo docker ps
  

Test Docker Setup

• Run a test Docker command to verify connection:

  docker run hello-world
  

Solution for WSL Error

If you’re encountering the error on WSL, re-install Docker by removing the Docker installation from WSL and installing Docker Desktop on your host machine (Windows).

On Linux, start the docker daemon with either of these commands:

• Start the Docker daemon:

  sudo dockerd
  

  or

  sudo service docker start
  

After using the command docker build -t churn-prediction . to build the Docker image, this error occurs, and the image is not created.

To fix this issue, adjust the Python version in your Dockerfile to match the version installed on your system:

1. Determine your Python version by running:

   python --version
   

   Example output:

   Python 3.9.7
   

2. Update the first line of your Dockerfile with the correct Python version:

   FROM python:3.9.7-slim
   

Make sure to replace 3.9.7 with your actual Python version.

When installing sklearn version 1.0.2, you may encounter errors. This issue is due to the package name. Instead of "sklearn," you should use its full name. Here's how you can resolve this:

1. Use the following command to install the correct version:

   pipenv install scikit-learn==1.0.2
   

2. If your homework requires version 1.3.1, use the following command:

   pipenv install scikit-learn==1.3.1
   

Using the correct full package name should resolve the installation issues.

When adding libraries to the virtual environment in lesson 5.5, the trainer used the command:

pipenv install numpy scikit-learn==0.24.2 flask

However, some people using Python 3.11 or later may encounter an error, failing to lock files correctly with Pipfile.lock. You may need to install scikit-learn==1.4.2 as the error differs from the trainer's example. This should resolve the issue.

If you are still having problems, try the following steps:

• Delete the Pipfile.lock using:

  rm Pipfile.lock
  

• Rebuild the lock with:

  pipenv lock
  

• If it still doesn't work, delete the pipenv environment, Pipfile, and Pipfile.lock, and create a new one:

  pipenv --rm
  rm Pipfile*
  

I initially installed Flask with pipenv, but I received a "No module named 'flask'" error. I then reinstalled Flask using pip, and after that, I was able to import Flask successfully.

When running Docker containers, using the --rm flag ensures that the containers are removed upon exit. This helps in managing disk space by preventing the accumulation of stopped containers, which can consume unnecessary space.

Here are the main points regarding the use of the --rm flag:

• Space Management: Running containers with the --rm flag prevents the accumulation of stopped containers, thus conserving disk space.
• Development and Testing: During these phases, containers often don't need to persist, making the --rm flag useful for automatic removal.
• Images vs Containers: It's crucial to differentiate between them. Images are not modified upon execution; containers are the instances created from these images. The --rm flag affects containers, not the images themselves.
• Rebuilding: When a file like a Pipfile changes, the image is rebuilt, often under the same or a new tag, and the --rm flag helps maintain a clean environment.

Use docker images to list images and docker ps -a to list all containers, helping you manage your Docker resources efficiently.

When you create the Dockerfile, ensure the name is Dockerfile without any extensions. A common mistake is naming it with an extension, such as Dockerfile.dockerfile, which results in an error during the image build. To avoid this, create the file simply as Dockerfile.

# Incorrect way:
Dockerfile.dockerfile

# Correct way:
Dockerfile

Refer to the page https://docs.docker.com/desktop/install/mac-install/. Remember to check if you have an Apple chip or Intel chip.

To list all information for all local Docker images, you can use the following commands:

docker images
docker image ls

To retrieve information for a specific image, use:

docker image ls <image_name>

Or alternatively:

docker images <image_name>

To dump only the size of a specified image, use the --format option. This will display only the image size:

docker image ls --format "{{.Size}}" <image_name>

Or alternatively:

docker images --format "{{.Size}}" <image_name>

Pipenv creates environments in different locations depending on the operating system:

• OSX/Linux: ~/.local/share/virtualenvs/folder-name_cryptic-hash
• Windows: C:\Users\<USERNAME>\.virtualenvs\folder-name_cryptic-hash

For example:

• C:\Users\Ella\.virtualenvs\code-qsdUdabf (for module-05 lesson)

The environment name is based on the name of the last folder in the directory where the pipenv install command was executed. For example, if you run any pipenv command in the directory ~/home/user/Churn-Flask-app, it will create an environment named Churn-Flask-app-some_random_characters, and its path will look like:

• /home/user/.local/share/virtualenvs/churn-flask-app-i_mzGMjX

All libraries for this environment will be installed inside this folder. To activate the environment, navigate back to the project folder and type pipenv shell. Essentially, the location of the project folder acts as an identifier for an environment, replacing any specific name.

To debug a Docker container, follow these steps:

1. Launch the container image in interactive mode while overriding the entrypoint, so that it starts with a bash command:

   docker run -it --entrypoint bash <image>
   

2. If the container is already running, execute a command in the specific container:

   • First, find the container ID by listing the running containers:

     docker ps
     

   • Then, execute bash in the container:

     docker exec -it <container-id> bash
     

docker exec -it 1e5a1b663052 bash

Error:

the input device is not a TTY. If you are using mintty, try prefixing the command with 'winpty'

Fix:

winpty docker exec -it 1e5a1b663052 bash

A TTY is a terminal interface that supports escape sequences, moving the cursor around, etc. Winpty is a Windows software package providing an interface similar to a Unix pty-master for communicating with Windows console programs.

For more information on terminal, shell, and console applications:

https://conemu.github.io/en/TerminalVsShell.html

Create a virtual environment using the command line and use the pip freeze command to write the requirements to a text file.

This error occurs because pipenv is installed but not accessible from the PATH.

You might encounter this error when running:

pipenv --version

or

pipenv shell

Solution for Windows:

1. Open this option:

   

2. Click here:

   

3. Click the Edit button:

   

4. Ensure the following locations are included in the PATH. If not, add them:

   • C:\Users\AppData\...\Python\PythonXX\
   • C:\Users\AppData\...\Python\PythonXX\Scripts\

   

Note: This solution is for setups without Anaconda. If you use Windows, Anaconda might be a better and less error-prone choice.

Following the instruction from video week-5.6, using pipenv to install Python libraries throws the error shown below:

naneen@xps:ml_zoomcamp_ht$ pipenv install numpy
Traceback (most recent call last):
  File "/usr/bin/pipenv", line 33, in <module>
    sys.exit(load_entry_point('pipenv==11.9.0', 'console_scripts', 'pipenv')())
  File "/usr/lib/python3/dist-packages/pipenv/vendor/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/pipenv/vendor/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/pipenv/vendor/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3/dist-packages/pipenv/vendor/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/pipenv/vendor/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/pipenv/cli/py.py", line 347, in install
    from . import core
  File "/usr/lib/python3/dist-packages/pipenv/core.py", line 21, in <module>
    import requests
  File "/usr/lib/python3/dist-packages/pipenv/vendor/requests/__init__.py", line 65, in <module>
    from . import utils
  File "/usr/lib/python3/dist-packages/pipenv/vendor/requests/utils.py", line 27, in <module>
    from .cookies import RequestsCookieJar, cookiejar_from_dict
  File "/usr/lib/python3/dist-packages/pipenv/vendor/requests/cookies.py", line 172, in <module>
    class RequestsCookieJar(cookielib.CookieJar, collections.MutableMapping):
AttributeError: module 'collections' has no attribute 'MutableMapping'
naneen@xps:ml_zoomcamp_ht$

Solution:

• Ensure you are working with Python version 3.10+

After entering pipenv shell, ensure you use exit before pipenv --rm. Failing to do so may cause installation errors, making it unclear whether you are "in the shell" on Windows, as there are no clear markers for it.

If this messes up your PATH, use these terminal commands to fix it:

• For Windows:

  set VIRTUAL_ENV ""
  

• For Unix:

  export VIRTUAL_ENV=""
  

Additionally, manually re-creating the removed folder at C:\Users\username\.virtualenvs\removed-envname can help. The removed-envname can be identified in the error message.

Set the host to '0.0.0.0' on the Flask app and Dockerfile, then run the URL using localhost.

% docker build -t zoomcamp_test .

[+] Building 0.1s (10/10) FINISHED
 => [internal] load build definition from Dockerfile
 => => transferring dockerfile: 332B
 => [internal] load .dockerignore
 => => transferring context: 2B
 => [internal] load build context
 => => transferring context: 2B
 => [internal] load metadata for docker.io/svizor/zoomcamp-model:3.9.12-slim
 => [1/6] FROM docker.io/svizor/zoomcamp-model:3.9.12-slim
 => [internal] load build context
 => => transferring context: 2B
 => CACHED [2/6] RUN pip install pipenv
 => CACHED [3/6] WORKDIR /app
 => ERROR [4/6] COPY [Pipfile, Pipfile.lock, ./]
 => CACHED [5/6] RUN pipenv install --system --deploy
 => ERROR [6/6] COPY [q5_predict.py, model1.bin, dv.bin, ./]

This error occurred because I used single quotes around the filenames. Stick to double quotes.

(hw5) (base) home@ls-MacBook-Pro hw5 % docker build -t zoomcamp_test .
[+] Building 19.7s (9/10)
 => [internal] load build definition from Dockerfile
 => => transferring dockerfile: 332B
 => [internal] load .dockerignore
 => => transferring context: 2B
 => [internal] load metadata for docker.io/svizor/zoomcamp-model:3.9.12-slim
 => CACHED [1/6] FROM docker.io/svizor/zoomcamp-model:3.9.12-slim
 => [internal] load build context
 => => transferring context: 19.77kB
 => [2/6] RUN pip install pipenv
 => [3/6] WORKDIR /app
 => [4/6] COPY [Pipfile, Pipfile.lock, ./]
 => ERROR [5/6] RUN pipenv install --system --deploy
------
 > [5/6] RUN pipenv install --system --deploy:
#8 0.659 Your Pipfile.lock (65dad0) is out of date. Expected: (f3760a).
#8 0.660 Usage: pipenv install [OPTIONS] [PACKAGES]...
#8 0.660 ERROR:: Aborting deploy

I tried the first solution on Stackoverflow which recommended running pipenv lock to update the Pipfile.lock.

However, this didn’t resolve it. The following switch to the pipenv installation worked:

RUN pipenv install --system --deploy --ignore-pipfile

I was getting the following error when I rebuilt the Docker image, although the port was not allocated, and it was working fine.

Error message:

Error response from daemon: driver failed programming external connectivity on endpoint beautiful_tharp (875be95c7027cebb853a62fc4463d46e23df99e0175be73641269c3d180f7796): Bind for 0.0.0.0:9696 failed: port is already allocated.

The issue can be resolved by running the following command:

docker kill $(docker ps -q)

For more information, refer to the GitHub issue on Docker for Windows.

I was getting an error on the client side with this:

Client Side Error:

File "C:\python\lib\site-packages\urllib3\connectionpool.py", line 703, in urlopen ...

raise ConnectionError(err, request=request)

requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

Server Side:

An error was shown for Gunicorn, although the waitress command was running smoothly from the server side.

Solution:

• Use the IP address 0.0.0.0:8000 or 0.0.0.0:9696. They are the ones which work most of the time.

To install md5sum on macOS, use the following command:

brew install md5sha1sum

Then run the command to check the hash for a file to see if it matches the provided hash:

md5sum model1.bin dv.bin

I started a web-server in a terminal (command window, PowerShell, etc.). How can I run another Python script that makes a request to this server?

• Open another terminal (command window, PowerShell, etc.).
• Run your Python script from this new terminal.

When executing the commands:

pipenv shell
pipenv run gunicorn --bind 0.0.0.0:9696 predict:app

the following warning may occur:

UserWarning: Trying to unpickle estimator DictVectorizer from version 1.1.1 when using version 0.24.2. This might lead to breaking code or invalid results. Use at your own risk.

• Ensure you create the virtual environment with the same version of Scikit-Learn that was used to train the model, in this case, version 1.1.1.
• Resolve version conflicts by verifying that the model and DictVectorizer files are compatible with the Scikit-Learn version used for the project.

If you install packages via pipenv install, and encounter an error like this:

pipenv.vendor.plette.models.base.ValidationError: {'python_version': '3.9', 'python_full_version': '3.9.13'}

python_full_version: 'python_version' must not be present with 'python_full_version'

python_version: 'python_full_version' must not be present with 'python_version'

Follow these steps to resolve the issue:

1. Open the Pipfile in a text editor, such as nano:

   nano Pipfile
   

2. Remove either the python_version or python_full_version line.

3. Save the changes by pressing CTRL+X, then type Y and press Enter.

4. Run the following command to create the Pipfile.lock:

   pipenv lock
   

You can now continue with your work.

If during running the docker build command, you get an error like this:

Your Pipfile.lock (221d14) is out of date. Expected: (939fe0).

Usage: pipenv install [OPTIONS] [PACKAGES]...

ERROR:: Aborting deploy

You can try the following solutions:

1. Delete and Rebuild Pipfile.lock:

   • Delete the Pipfile.lock using the command:

     rm Pipfile.lock
     

   • Rebuild the lock file:

     pipenv lock
     

   • Retry the docker build command.

2. Remove and Recreate Pipenv Environment:

   • Remove the pipenv environment:

     pipenv --rm
     

   • Remove the Pipfile and Pipfile.lock:

     rm Pipfile*
     

   • Create a new environment before retrying the Docker build.

1. Uninstall Waitress and MLflow:

   pip uninstall waitress mlflow
   

2. Reinstall MLflow:

   pip install mlflow
   

By this time, you should have successfully built your Docker image, so you don't need to reinstall Waitress.

All good. Happy learning.

Ensure that you are in the correct AWS region. Check if you are in eu-west-1 (Ireland) when reviewing your Elastic Beanstalk environments. It's possible you might be in a different region in your AWS console.

Running pip install waitress as a command on GitBash may not download the executable file waitress-serve.exe. You need this file to use the commands with waitress in Git Bash. To resolve this issue:

1. Open a Jupyter notebook and run the command pip install waitress. This will download the executable file. You may see the following warning:

   WARNING: The script waitress-serve.exe is installed in 'c:\Users\....\anaconda3\Scripts' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
   

2. Add the path where waitress-serve.exe is installed to GitBash's PATH:

   • Enter the following command in GitBash: nano ~/.bashrc

   • Add the path to waitress-serve.exe to PATH using the command:

     export PATH="/path/to/waitress:$PATH"
     

3. Close GitBash and open it again. You should now be able to run waitress-serve successfully.

This is an error encountered while using Pipenv to install Scikit-Learn version 1.3.1 in the ml-zoomcamp conda environment. The error indicates that explicit language specifications are not set in the bash profile.

The error is not fatal and can usually be ignored. However, if you'd like to address it, consider the following quick-fix:

Visit this discussion for more details: StackOverflow - Getting error while trying to run this command 'pipenv install requests' in ma.

In most cases, you can proceed without fixing this warning.

https://apps.microsoft.com/detail/windows-terminal/9N0DX20HK701?hl=es-419&gl=CO

When running the command:

pipenv run waitress-serve --listen=localhost:9696 q4-predict:app

You may encounter the following error message:

There was an exception (ValueError) importing your module.

It had these arguments:

1. Malformed application 'q4-predict:app'

Waitress doesn’t accept a dash in the Python file name.

To resolve this, rename the file by replacing the dash with an underscore, for example, use q4_predict.py.

I wanted to have a fast and simple way to check if the HTTP POST requests are working just by running a request from the command line. This can be done using curl. (Used with WSL2 on Windows; should also work on Linux and MacOS)

curl --json '<json data>' <url>

Piping the structure to the command:

cat <json file path> | curl --json @- <url>
echo '<json data>' | curl --json @- <url>

Example using piping:

echo '{"job": "retired", "duration": 445, "poutcome": "success"}' \
| curl --json @- http://localhost:9696/predict

When executing:

eb local run  --port 9696

You may encounter the following error:

ERROR: NotSupportedError - You can use "eb local" only with preconfigured, generic and multicontainer Docker platforms.

There are two options to fix this issue:

1. Re-initialize the Environment:

   • Run the initialization command:

     eb init -i
     

   • Choose the appropriate options from the list provided (the first default option for the Docker platform should suffice).

2. Manually Edit Configuration:

   • Open and edit the ‘.elasticbeanstalk/config.yml’ file.
   • Change default_platform from Docker to:

     default_platform: Docker running on 64bit Amazon Linux 2023
     

   • Note that this option might not be available in the future.

Alternative Solution:

• Re-run the init command and change the -p flag value:

  eb init -p "Docker running on 64bit Amazon Linux" <appname>
  

• Then re-run:

  eb local run --port 9696
  

Original solution from Stack Overflow

You need to include the protocol scheme: [http://localhost:9696/predict](http://localhost:9696/predict).

Without the http:// part, requests has no idea how to connect to the remote server.

Note that the protocol scheme must be all lowercase; if your URL starts with HTTP:// for example, it won’t find the http:// connection adapter either.

While running the Docker image, if you get the same result, check which model you are using.

Remember, you are using a model by downloading the model and Python version. Ensure to change the model in your file when running your prediction test.

Ensure that you used pipenv to install the necessary modules including gunicorn. Follow these steps:

1. Use pipenv shell to enter the virtual environment.
2. Build and run your Docker image.

Make sure all dependencies are correctly specified in your Pipfile.

You can copy files from your local machine into a Docker container using the docker cp command. Here's how to do it:

To copy a file or directory from your local machine into a running Docker container, use the following syntax:

docker cp /path/to/local/file_or_directory container_id:/path/in/container

You can copy files from your local machine into a Docker container using the docker cp command.

In the Dockerfile, you can specify the folder containing the files you want to copy. The basic syntax is as follows:

COPY ["src/predict.py", "models/xgb_model.bin", "./"]

I struggled with the command:

 eb init -p docker tumor-diagnosis-serving -r eu-west-1

Which resulted in an error when running:

eb local run --port 9696

ERROR: NotSupportedError - You can use "eb local" only with preconfigured, generic and multicontainer Docker platforms.

I replaced it with:

eb init -p "Docker running on 64bit Amazon Linux 2" tumor-diagnosis-serving -r eu-west-1

This allowed the recognition of the Dockerfile and the build/run of the docker container.

I encountered this error when creating an AWS ElasticBean environment using the command:

eb create tumor-diagnosis-env

Error Message:

ERROR   Instance deployment: Both 'Dockerfile' and 'Dockerrun.aws.json' are missing in your source bundle. Include at least one of them. The deployment failed.

The error occurred because I had not committed the files used to build the container, particularly the Dockerfile. After performing the following Git operations, the command worked properly:

1. Add the modified files to staging:

   git add .
   

2. Commit the changes:

   git commit -m "Add Dockerfile and necessary files"
   

When creating and launching an AWS Elastic Beanstalk environment with eb create, you might encounter the following error:

ERROR: CommandError - git could not find the HEAD; most likely because there are no commits present

Explanation and Steps to Resolve:

This error indicates that your project directory has not been initialized as a Git repository or is in a "detached HEAD" state. Elastic Beanstalk's CLI relies on Git for managing application versions. Here's how to resolve it:

1. Check Git Initialization:

   • Run:

     git status
     

   • If Git is not initialized, you will see an error or a message indicating no repository exists.

2. Initialize Git:

   git init
   

3. Create an Initial Commit (if none exists):

   git add .
   git commit -m "Initial commit"
   

4. Manage "Detached HEAD" State:

   • Create a new branch (if needed):

     git checkout -b main
     

   • Or switch to an existing branch:

     git checkout main
     

5. Reinitialize Elastic Beanstalk (if necessary):

   eb init
   

6. Retry Deployment:

   eb create <env_name> --enable-spot
   

When you make local changes to the Dockerfile or any other files and do not commit these changes, AWS Elastic Beanstalk (EB) won’t deploy them. By default, the EB CLI deploys the latest commit in the current branch.

If you want to deploy to your environment without committing, you can use the –-stage option to deploy changes that have been added to the staging area.

If the Docker image creation fails during the eb create process, you can still create the image and deploy it by running eb deploy.

To deploy changes without committing:

1. Add new and changed files to the staging area:

   ~/eb$ git add .
   

2. Deploy the staged changes with eb deploy:

   ~/eb$ eb deploy --staged
   

To resolve this issue, you can create your environment using the --enable-spot flag, which automatically uses Launch Templates.

Example:

eb create med-app-env --enable-spot

Another option is to run eb create and follow the wizard options:

1. Enter Environment Name:

   • Default: churn-serving-dev
   • Example: churn-serving-dev

2. Enter DNS CNAME prefix:

   • Default: churn-serving-dev
   • Example: churn-serving-dev

3. Select a load balancer type:

   1) classic
   2) application
   3) network
   (default is 2): 1
   

4. Would you like to enable Spot Fleet requests for this environment?

   • Prompt: (y/N):
   • Example: y

5. Enter a list of one or more valid EC2 instance types separated by commas (at least two instance types are recommended).

   • Defaults provided on Enter: Press Enter

Starting on October 1, 2024, the Amazon EC2 Auto Scaling service will no longer support the creation of launch configurations for new accounts. This change is due to launch configurations being phased out and replaced by launch templates by the Amazon EC2 Auto Scaling service.

For more details refer to: AWS Documentation

This replacement of launch configurations by launch templates is what caused the error described previously (“...use launch templates to create configuration templates for your Auto Scaling groups).

When encountering a VPC configuration error during the deployment to AWS Elastic Beanstalk, follow these steps:

1. Execute the command:

   eb create churn-prediction-env
   

2. If the environment creation initially appears successful but later shows an error related to VPC configuration, it likely means there is no default VPC for the selected region.

3. Go to the AWS Console and select your region from the top bar (e.g., us-east-2).

4. Search for "VPC" and from the left menu, navigate to "Your VPCs".

5. If no VPCs are present, the option to create a default VPC will be available. Click on it.

6. Once the default VPC is created, rerun the command.

Gunicorn is a Python WSGI HTTP server that is more suitable for production than the default Flask development server:

• Performance: Better at handling multiple simultaneous requests.
• Stability: More robust and can manage worker processes.

Usage:

Modify the CMD in your Dockerfile:

CMD ["gunicorn", "--bind", "0.0.0.0:9696", "app:app"]

This warning occurs because the Pipfile is expecting Python 3.12, but the local container is likely running an older version, such as Python 3.8.12-slim, as shown in the video 5.6 - Environment Management : Docker.

To resolve this issue, update the Dockerfile to use the appropriate version:

FROM python:3.12.7-slim

Ensure that both Python versions (the local version shown in the Pipfile and the container version) match to guarantee compatibility.

First, avoid being in a virtual environment when using pipenv. You can point pipenv directly to the Python 3.11 interpreter from your Conda installation:

1. Activate the Conda environment:

   conda activate env_name
   

2. Get the Python path:

   which python
   

3. Deactivate the Conda environment:

   conda deactivate
   

4. Use pipenv with the Python path found in step 2:

   pipenv --python /path/to/python
   

You could try running your homework on GitHub Codespaces instead of your local computer. In my experience, the compute resources on GitHub Codespaces are quite sufficient for Homework 5. No issues at all in terms of speed.

You can convert a Jupyter notebook to a Python script using the following methods:

• Using the terminal

  Run the command below in the terminal:

  jupyter nbconvert --to python notebook.ipynb
  

  This converts the notebook into a Python script with the same name but with a .py extension.

• Using the Jupyter Notebook interface

  1. Navigate to File in the menu.
  2. Select Save and Export Notebook As.
  3. Choose Executable Scripts.

  This will download the file to your downloads folder.

They both do the same; it's just less typing from the script.

When you encounter the error stating that there is no module named 'ping', it means the 'ping' module is not found in your Python environment.

To fix this, you can import the 'ping' function directly from the specific file where it is defined using:

from [file name] import ping

Replace [file name] with the actual file name where the 'ping' function is located.

This error indicates that the executable gunicorn is not found in the container's $PATH. To resolve this, you need to add gunicorn and flask to your Pipfile.

Update your Pipfile as follows:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
scikit-learn = "==1.5.2"
gunicorn = "*"
flask = "*"

[dev-packages]

[requires]
python_version = "3.11"

After making these changes, follow these steps:

1. Run pipenv lock to update the Pipfile.lock.
2. Build the Docker image with:

   docker build -t [name] .
   

3. Run the Docker container with:

   docker run [name]
   

This error occurs because recent versions of xgb.DMatrix expect the feature_names parameter to be a list of strings rather than a NumPy array. Older tutorial videos might use feature_names=dv.get_feature_names_out() directly, which now results in this error.

Convert dv.get_feature_names_out() to a list using .tolist(). Here's an updated example:

# Convert feature names to a list
feature_names = dv.get_feature_names_out().tolist()

# Create DMatrix objects with the corrected feature names
dfulltrain = xgb.DMatrix(
    X_full_train, 
    label=y_full_train, 
    feature_names=feature_names
)

dtest = xgb.DMatrix(
    X_test, 
    feature_names=feature_names
)

Explanation: The dv.get_feature_names_out() method returns a NumPy array, but xgb.DMatrix now expects feature_names to be a list of strings. Using .tolist() converts the array into a compatible format, allowing the code to run without errors.

To ensure that the string values like "None" are treated as valid strings rather than being converted to NaN when reading a CSV file, you can read the CSV file with keep_default_na set to False and specify the values you want to consider as NaN with the na_values parameter.

Here’s an example of how to do this:

import pandas as pd

df = pd.read_csv("dataset_path.csv", keep_default_na=False, na_values=['', 'NaN', 'null'])

Using keep_default_na=False prevents Pandas from applying its default set of NaN values, allowing "None" to be read as a regular string.

I was using Google Collab Notebook for the 2024 cohort HW 06. For Question 6, the following was not working in the Collab Notebook:

This led me to find a solution as follows:

1. Import the required libraries:

   import io
   import sys
   

2. Capture output using io.StringIO:

   output_capture = io.StringIO()
   sys.stdout = output_capture  # Redirect stdout to the StringIO buffer
   
   # Train the model with eta=0.3
   model_eta_03 = xgb.train(xgb_params, dtrain, num_boost_round=num_rounds, verbose_eval=2, evals=watchlist)
   
   # Reset stdout
   sys.stdout = sys.__stdout__
   
   # Retrieve and print the captured output
   captured_output = output_capture.getvalue()
   

3. Modify the parser function for one line:

   Replace this line in Alexey’s parser function:

   for line in output.stdout.strip().split('\n'):
   

   With this line:

   for line in output.strip().split('\n'):
   

4. Call the parser function:

   Use df_score_03 = parse_xgb_output(captured_output) to get the desired dataframe.

During the XGBoost lesson, we created a parser to extract the training and validation AUC from the standard output. However, we can accomplish that in a more straightforward way.

We can use the evals_result parameter, which takes an empty dictionary and updates it for each tree. Additionally, you can store the data in a dataframe and plot it in an easier manner.

You should create a sklearn.ensemble.RandomForestRegressor object. It’s similar to sklearn.ensemble.RandomForestClassifier for classification problems. For more information, check the official documentation.

When creating DMatrix for train and validation, you might encounter the error:

ValueError: feature_names must be string, and may not contain [, ] or <

The cause of this error is special characters in feature names, such as = and <. To fix this error, you can remove or replace these characters:

features = [i.replace("=<", "_").replace("=","_") for i in features]

If the equal sign = is not a problem for you, the following adjustment could also work:

features = []

for f in dv.feature_names_:
    string = f.replace("=<", "-le")
    features.append(string)

If you’re encountering a TypeError like this:

TypeError: Expecting a sequence of strings for feature names, got: <class 'numpy.ndarray'>

This might be because you have executed:

features = dv.get_feature_names_out()

This returns a numpy.ndarray instead of a list. Converting it to a list with list(features) won't solve the issue.

If you face a ValueError such as:

ValueError: feature_names must be string, and may not contain [, ] or <

This could be because you have tried one of these:

• features = list(dv.get_feature_names_out())
• features = dv.feature_names_

The problem originates from the output of DictVectorizer, which might look like:

['households',
 'housing_median_age',
 'latitude',
 'longitude',
 'median_income',
 'ocean_proximity=<1H OCEAN',
 'ocean_proximity=INLAND',
 'population',
 'total_bedrooms',
 'total_rooms']

The symbols [, ] or < are not compatible with XGBoost.

Solutions:

1. Do not specify feature_names= when creating xgb.DMatrix.

2. Alternatively, you can clean your feature names using regex:

   import re
   features = dv.feature_names_
   pattern = r'[\[\]<>]'
   features = [re.sub(pattern, '  ', f) for f in features]
   

To install Xgboost, use the following command directly in your Jupyter Notebook:

pip install xgboost

Note: Pip 21.3+ is required.

You can update your pip using the command below:

pip install --upgrade pip

For more information about Xgboost and installation details, check the official documentation.

Sometimes someone might wonder what eta means in the tunable hyperparameters of XGBoost and how it helps the model.

ETA is the learning rate of the model. XGBoost uses gradient descent to calculate and update the model. In gradient descent, we are looking for the minimum weights that help the model to learn the data very well. This minimum weight for the features is updated each time the model passes through the features and learns them during training. Tuning the learning rate helps you tell the model what speed it would use in deriving the minimum for the weights.

For ensemble algorithms during week 6, one bagging algorithm and one boosting algorithm were presented: Random Forest and XGBoost, respectively.

Random Forest trains several models in parallel. The output can be, for example, the average value of all the outputs of each model. This is called bagging.

XGBoost trains several models sequentially: the previous model error is used to train the following model. Weights are used to ponderate the models so that the best models have higher weights and are therefore favored for the final output. This method is called boosting.

Note that boosting is not necessarily better than bagging.

Bagging stands for “Bootstrap Aggregation”:

• It involves taking multiple samples with replacement to derive multiple training datasets from the original training dataset (bootstrapping).
• A classifier (e.g., decision trees or stumps for Random Forests) is trained on each such training dataset.
• The predictions are combined (aggregation) to obtain the final prediction.
  • For classification, predictions are combined via voting; for regression, via averaging.
• Bagging can be done in parallel since the various classifiers are independent.
• Bagging decreases variance (but not bias) and is robust against overfitting.

Boosting, on the other hand, is sequential:

• Each model learns from the mistakes of its predecessor.
• Observations are given different weights - observations/samples misclassified by the previous classifier are given a higher weight.
• This process is continued until a stopping condition is reached (e.g., a maximum number of models is reached, or error is acceptably small).
• Boosting reduces bias and is generally more accurate than bagging, but can be prone to overfitting.

I wanted to directly capture the output from the XGBoost training for multiple eta values to a dictionary without needing to run the same cell multiple times, edit the eta value manually, or copy the code for different eta values.

Using the magic cell command %%capture output, I could only capture the complete output for all iterations of the loop. I was able to solve this using the following approach. This is just a code sample to illustrate the idea:

from IPython.utils.capture import capture_output
import sys

different_outputs = {}

for i in range(3):
    with capture_output(sys.stdout) as output:
        print(i)
        print("testing capture")
    different_outputs[i] = output.stdout

# Output:
# different_outputs
# {0: '0\ntesting capture\n',
#  1: '1\ntesting capture\n',
#  2: '2\ntesting capture\n'}

Calling roc_auc_score() to get AUC is throwing the above error.

Solution to this issue is to ensure that you pass y_actuals as the first argument and y_pred as the second argument.

roc_auc_score(y_train, y_pred)

To visualize decision trees, you can use Graphviz along with Scikit-learn's export_graphviz method and plot_tree function.

Here are two approaches:

1. Using export_graphviz:

   from sklearn import tree
   import graphviz
   
   dot_data = tree.export_graphviz(regr, out_file=None,
                                   feature_names=boston.feature_names,
                                   filled=True)
   graphviz.Source(dot_data, format="png")
   

2. Using plot_tree:

   from sklearn import tree
   
   tree.plot_tree(dt, feature_names=dv.feature_names_)
   

Both methods help in generating a visual representation of the decision tree.

This problem occurs when using DecisionTreeClassifier instead of DecisionTreeRegressor.

To resolve this issue:

• Check whether you want to use a decision tree for classification or regression.
• Use DecisionTreeRegressor for regression tasks.

from sklearn.tree import DecisionTreeRegressor

# Example: for regression
model = DecisionTreeRegressor()
model.fit(X_train, y_train)

When running dt = DecisionTreeClassifier() in Jupyter on the same laptop, different AUC values are observed each time it is re-run or after restarting the kernel. Examples include values like 0.674, 0.652, 0.642, 0.669, etc. This variability is discussed in a video between 7:40-7:45 of section 6.3.

Solution:

• Set a random seed to ensure reproducibility by using:

  dt = DecisionTreeClassifier(random_state=22)
  

The DictVectorizer has a function to get the feature names using get_feature_names_out()). This is helpful if you need to analyze feature importance but are using the dict vectorizer for one-hot encoding.

Keep in mind that it returns a NumPy array, so you may need to convert this to a list depending on your usage. For example:

• dv.get_feature_names_out() will return an ndarray of string objects.
• list(dv.get_feature_names_out()) will convert it to a standard list of strings.

Also, ensure that you fit the predictor and response arrays before accessing the feature names.

They both do the same. Using the script involves less typing.

To make it easier to determine which features are important, we can use a horizontal bar chart to illustrate feature importance sorted by value.

1. Extract the feature importances from the model

   feature_importances = list(zip(features_names, rdr_model.feature_importances_))
   importance_df = pd.DataFrame(feature_importances, columns=['feature_names', 'feature_importances'])
   

2. Sort the DataFrame in descending order using the feature_importances value

   importance_df = importance_df.sort_values(by='feature_importances', ascending=False)
   

3. Create a horizontal bar chart

   plt.figure(figsize=(8, 6))
   sns.barplot(x='feature_importances', y='feature_names', data=importance_df, palette='Blues_r')
   plt.xlabel('Feature Importance')
   plt.ylabel('Feature Names')
   plt.title('Feature Importance Chart')
   

Instead of using np.sqrt() as a second step, you can extract it using:

mean_squared_error(y_val, y_predict_val, squared=False)

I like this visual implementation of features importance in scikit-learn library:

https://scikit-learn.org/stable/auto_examples/ensemble/plot_forest_importances.html

It adds standard errors to features importance, allowing you to trace the stability of features—important for a model’s explainability—over different parameters of the model.

Expanded error:

xgboost.core.XGBoostError: sklearn needs to be installed in order to use this module.

Solution:

• Ensure that sklearn is listed in your requirements and installed. This should solve the problem.

Information gain in Y due to X, or the mutual information of Y and X:

Where ( I(Y; X) = H(Y) - H(Y | X) ).

• If X is completely uninformative about Y, then ( I(Y; X) = 0 ).
• If X is completely informative about Y, then ( I(Y; X) = H(Y) ).

Filling in missing values using an entire dataset before splitting for training/testing/validation causes data leakage.

If you have problems with pickling the models, you can use an alternative approach.

Save model by calling save_model, load with load_model:

model.save_model('model.bin')

...

bst = xgb.Booster()
bst.load_model('model.bin')

In lesson 6.3 around 6:00, there is an error due to missing values. Subsequently, .fillna(0) is used on df_train to deal with this. However, since version 1.3, support for missing values has been added for DecisionTreeClassifier and DecisionTreeRegressor.

More details can be found here under sklearn.tree.

To pair feature names with their importance values, use dv.get_feature_names_out() to retrieve the feature names and rf.feature_importances_ for the importances. Then, combine them with zip(feature_names, importances) to view or sort by importance.

# Assuming rf is your RandomForest model and dv is your DictVectorizer

feature_names = dv.get_feature_names_out()
feature_importances = rf.feature_importances_

# Pair feature names with their importance values
feature_importance_dict = dict(zip(feature_names, feature_importances))

# Sort by importance (optional)
sorted_feature_importance = sorted(feature_importance_dict.items(), key=lambda x: x[1], reverse=True)

# Display results
for feature, importance in sorted_feature_importance:
    print(f"{feature}: {importance}")

XGBoost’s performance stems from its flexibility, thanks to a range of parameters.

For initial tuning, focus on:

• learning_rate: Controls the impact of each tree. Lower values (e.g., 0.01–0.1) typically improve performance but require more trees (n_estimators).
• n_estimators: Sets the number of boosting rounds; adjust this in conjunction with learning_rate.
• max_depth: Prevents overfitting by limiting the tree’s depth.
• subsample: Dictates the fraction of samples used for training each tree, adding randomness to improve generalization.

Begin with these parameters before exploring others, like gamma and min_child_weight, for additional control over model complexity and performance.

Using model.feature_importances_ can give you an error:

AttributeError: 'Booster' object has no attribute 'feature_importances_'

If you train the model like this: model = xgb.train, you should use get_score() instead.

Here is an article that worked for me: https://knowmledge.com/2023/12/07/ml-zoomcamp-2023-project/

• Create or import your notebook into Kaggle.
• Click on the three dots at the top right-hand side.
• Click on "Accelerator."
• Choose "T4 GPU."

1. Create or import your notebook into Google Colab.
2. Click on the drop-down at the top right-hand side.
3. Click on “Change runtime type.”
4. Choose T4 GPU.

Connecting your GPU on Saturn Cloud to a Github repository is not compulsory, as you can download the notebook and copy it to the Github folder manually. However, if you prefer an automated approach, follow these instructions:

1. Create SSH Keys:

   • Refer to the following GitHub documentation to generate an SSH private and public key:
     • Generating a new SSH key and adding it to the SSH agent
     • Adding a new SSH key to your GitHub account

2. Authenticating via Terminal:

   • Access the second video in the Saturn Cloud module to learn how to add SSH keys to secrets and authenticate via a terminal.

3. Using Saturn Cloud's Default Public Keys

   • Click on your username and select "Manage".
   • In the "Git SSH keys" section, copy the default public key provided by Saturn Cloud.
   • Paste this key into the SSH keys section of your GitHub repository.
   • Open a terminal on Saturn Cloud and run the following command:

ssh -T git@github.com

You should receive a successful authentication notice.

Follow these steps to efficiently push from Saturn Cloud to GitHub.

The template referred to in the video 8.1b Setting up the Environment on Saturn Cloud has been moved.

You can now find it under "Python Deep Learning Tutorials" on the Saturn Cloud homepage.

For updated text instructions (as of Nov 2024) for setting up Saturn Cloud with TensorFlow and GPU, refer to the link provided above.

The error occurs because the scipy module is not installed in the Saturn Cloud TensorFlow image.

To resolve this issue:

1. When creating the Jupyter server resource, navigate to the "Extra Packages" section.
2. In the pip textbox, write scipy.
3. A command will appear below the textbox: pip install scipy.
4. This ensures that when the resource starts, the scipy package will be automatically installed.

This method can be used to install additional Python packages as needed.

Uploading data directly from Kaggle to Saturn Cloud can save time, especially for large datasets. You can download it to your local machine and then upload it to Saturn Cloud, but there is a more efficient method.

1. Install Kaggle Package

   Run the following command in your notebook:

   !pip install -q kaggle
   

2. Generate Kaggle API Token

   • Go to the Kaggle website and log into your account.
   • Click on your profile image and select "Account."
   • Scroll down to the "API" section.
   • Click on "Create New API Token." A JSON file named kaggle.json will download to your local computer.

3. Upload the Kaggle API Token to Saturn Cloud

   • In your notebook, click on the folder icon in the upper left corner to navigate to the root folder.
   • Click on the .kaggle folder.
   • Upload the kaggle.json file into the .kaggle folder.

4. Set File Permissions

   Run this command to secure your Kaggle API token:

   !chmod 600 /home/jovyan/.kaggle/kaggle.json
   

5. Download the Dataset

   Use the following command to download your desired dataset (e.g., the "dino-or-dragon" dataset):

   !kaggle datasets download -d agrigorev/dino-or-dragon
   

6. Unzip the Dataset

   • Create a folder to unzip your files:

     !mkdir data
     

   • Unzip your files inside that folder:

     !unzip dino-or-dragon.zip -d data
     

By following these steps, you set up Saturn Cloud with access to all Kaggle datasets efficiently.

In order to run TensorFlow with GPU on your local machine, you’ll need to set up CUDA and cuDNN.

The process can be overwhelming. Here’s a simplified guide.

When loading a saved model, you encounter the error:

ValueError: Unable to load weights saved in HDF5 format into a subclassed Model which has not created its variables yet. Call the Model first, then load the weights.

Before loading the model, you need to evaluate the model on input data:

model.evaluate(train_ds)

When following module 8.1b video to set up Git in Saturn Cloud, running the command:

ssh -T git@github.com

results in the error:

Permission denied (publickey).

An alternative method involves setting up Git in your Saturn Cloud environment by generating an SSH key in Saturn Cloud and adding it to your Git account. After completing this setup, you can access and manage your Git repositories through Saturn’s Jupyter server.

For detailed steps, refer to this tutorial: https://saturncloud.io/docs/using-saturn-cloud/gitrepo/

Getting an error using

git clone git@github.com:alexeygrigorev/clothing-dataset-small.git

The error:

Cloning into 'clothing-dataset'...

Host key verification failed.

fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

When cloning the repo, you can also choose HTTPS, then it should work. This error occurs when your SSH key is not configured.

Use the following command:

git clone https://github.com/alexeygrigorev/clothing-dataset-small.git

The accuracy and the loss are both the same or nearly the same while training.

• In the homework, set class_mode='binary' while reading the data.
• The problem may also occur if you choose the wrong optimizer, batch size, or learning rate.

When resuming training after augmentation, the loss skyrockets (1000+ during the first epoch) and accuracy settles around 0.5 – i.e. the model becomes as good as a random coin flip.

Check that the augmented ImageDataGenerator still includes the rescale option as specified in the preceding step.

While attempting to reload a model with TensorFlow, you might encounter the following error:

ValueError: The channel dimension of the inputs should be defined. The input_shape received is (None, None, None, None), where axis -1 (0-based) is the channel dimension, which found to be `None`.

This error is usually caused when the number of channels is not explicitly defined in the Input layer of the model.

Ensure that you explicitly specify the number of channels in the Input layer of the model architecture.

Example:

import tensorflow as tf
from tensorflow import keras

# Model architecture:
inputs = keras.Input(shape=(input_size, input_size, 3))

base = base_model(inputs, training=False)
vectors = keras.layers.GlobalAveragePooling2D()(base)
inner = keras.layers.Dense(size_inner, activation='relu')(vectors)
drop = keras.layers.Dropout(droprate)(inner)
outputs = keras.layers.Dense(10)(drop)

model = keras.Model(inputs, outputs)

This configuration ensures that the channel dimension is explicitly defined, preventing the reload error.

If you unzip a dataset within a Jupyter Notebook using the ! unzip command, you may encounter extensive output messages for each file. To suppress this output, follow these solutions:

Using Magic Commands

%%capture

! unzip zipped_folder_name.zip -d destination_folder_name

Using Python's zipfile Library

import zipfile

local_zip = 'data.zip'
zip_ref = zipfile.ZipFile(local_zip, 'r')
zip_ref.extractall('data')
zip_ref.close()

Use Solution 1 to suppress output directly in the notebook. Solution 2 provides an alternative approach using Python code.

Keras flow_from_directory understands the names of classes from the names of folders.

• When using train_gen.flow_from_directory(), the class names are derived from the folder names within the specified directory.
• For example, if you create a folder named "xyz", it will be considered a class.
• This behavior aligns with the function name flow_from_directory.

A detailed explanation can be found in this tutorial.

I created a new environment in SaturnCloud and chose the image corresponding to Saturn with Tensorflow, but when I tried to fit the model it showed an error about the missing module: scipy.

1. Install the module in a new cell:

   !pip install scipy
   

2. Restart the kernel and fit the model again.

The command to read folders in the dataset in the TensorFlow source code is:

for subdir in sorted(os.listdir(directory)):

Reference: Keras Image Preprocessing, line 563.

This means folders will be read in alphabetical order. For example, with a folder named dino and another named dragon, dino will be read first and will have class label 0, whereas dragon will be read next and will have class label 1.

When a Keras model predicts binary labels, it returns one value, which is the probability of class 1. This occurs with the sigmoid activation function in the last dense layer with 2 neurons. The probability of class 0 can be calculated as:

prob(class(0)) = 1 - prob(class(1))

In the case of using from_logits to get results, you will receive two values for each of the labels.

A prediction of 0.8 indicates that the probability the image has class label 1 (in this case, dragon) is 0.8. Conversely, the probability that the image has class label 0 is 0.2.

Running the wasp/bee model on a Mac laptop might result in higher reported accuracy and lower standard deviation than expected from the HW answers. This discrepancy could be related to the version of the SGD optimizer being used. A message may appear about new and legacy versions.

• Try running the same code on Google Colab or another platform. The results may align closer with HW answers on Colab.
• Change the runtime to use a T4 GPU for faster execution compared to CPU.

When running model.fit(...), an additional parameter workers can be specified for speeding up data loading and generation. The default value is 1. Try different values between 1 and the CPU count on your system to check which performs best.

For more information, refer to the TensorFlow documentation.

Reproducibility for training runs can be achieved by following these instructions: TensorFlow Documentation

seed = 1234

tf.keras.utils.set_random_seed(seed)

tf.config.experimental.enable_op_determinism()

This ensures consistent results when the script is executed multiple times.

PyTorch is also a deep learning framework that allows you to perform equivalent tasks as Keras. Here is a tutorial to create a CNN from scratch using PyTorch:

Writing CNNs from Scratch in PyTorch

The functions have similar goals, although the syntax may vary slightly. For the lessons and the homework, we use Keras, but you are free to make a pull request with the equivalent implementation using PyTorch for the lessons and homework!

While training a Keras model, you may encounter the error:

Failed to find data adapter that can handle input: <class 'keras.src.preprocessing.image.ImageDataGenerator'>, <class 'NoneType'>

This typically occurs if you accidentally pass the image generator instead of the dataset to the model. Here is an example of incorrect usage:

train_gen = ImageDataGenerator(rescale=1./255)

train_ds = train_gen.flow_from_directory(…)

history_after_augmentation = model.fit(
    train_gen,  # this should be train_ds!!!
    epochs=10,
    validation_data=test_gen  # this should be test_ds!!!
)