Four Reasons to Use a Checklist in Machine Learning Projects
- To reduce decision fatigue by not forcing us to remember every little thing
- To eliminate mistakes
- To ensure consistency
- To ensure that everything necessary is completed and nothing is missed.
Here is the checklist to guide you through your machine learning project.
Frame the Problem
- Define the objective in business terms. Look at the big picture
- How will your solution be used?
- What are the current solutions/workarounds?
- How should you frame this problem (supervised/unsupervised)?
- How should performance be measured?
- What is current accuracy. What would be the minimum performance needed to reach the business objective?
- List the assumptions you (or others) have made so far.
Get the Data
- List the data you need.
- Find and document where you can get that data.
- Check how much space it will take. Create a workspace.
- Check legal obligations and get access authorization if necessary.
- Ensure sensitive information is deleted or protected (e.g., anonymized).
- Get the data. Convert the data to a format you can easily manipulate
- Sample a test set, put it aside, and never look at it (no data snooping!).
Explore the Data
- Create a copy of the data for exploration.
- Create a Jupyter notebook to keep a record of your data exploration.
- Study each attribute and its characteristics:
• Type (categorical, int/float, bounded/unbounded, text, structured, etc.)
• % of missing values
• Noisiness and type of noise (stochastic, outliers, rounding errors, etc.)
• Possibly useful for the task?
• Type of distribution (Gaussian, uniform, logarithmic, etc.)
- For supervised learning tasks, identify the target attribute(s).
- Visualize the data.
- Study the correlations between attributes.
- Study how you would solve the problem manually.
- Identify the promising transformations you may want to apply.
- Identify extra data that would be useful
- Document what you have learned.
Prepare the Data
- Data cleaning:
• Fix or remove outliers (optional).
• Fill in missing values (e.g., with zero, mean, median…)
- Feature selection (optional):
• Drop the attributes that provide no useful information for the task.
- Feature engineering, where appropriate:
• Discretize continuous features.
• Decompose features (e.g., categorical, date/time, etc.).
• Add promising transformations of features (e.g., log(x), sqrt(x), x², etc.).
• Aggregate features into promising new features.
- Feature scaling: standardize or normalize features.
Short-List Promising Models
1. Train many quick and dirty models from different categories (e.g., linear, naive Bayes, SVM, Random Forests, neural net, etc.) using standard parameters.
2. Measure and compare their performance: For each model, use N-fold cross-validation and compute the mean and standard deviation of the performance measure on the N folds.
3. Analyze the most significant variables for each algorithm.
4. Analyze the types of errors the models make: What data would a human have used to avoid these errors?
5. Short-list the top three to five most promising models, preferring models that make different types of errors.
Fine-Tune the System
1. Fine-tune the hyperparameters using cross-validation.
2. Try Ensemble methods. Combining your best models will often perform better than running them individually.
3. Once you are confident about your final model, measure its performance on the test set to estimate the generalization error.
Present Your Solution
- Document what you have done.
- Create a nice presentation: Make sure you highlight the big picture first.
- Explain why your solution achieves the business objective.
- Don’t forget to present interesting points you noticed along the way.
- Ensure your key findings are communicated through beautiful visualizations or easy-to-remember statements (e.g., “the median income is the number-one predictor of housing prices”).
- Get your solution ready for production (plug into production data inputs, write unit tests, etc.).
- Write monitoring code to check your system’s live performance at regular intervals and trigger alerts when it drops.
- Retrain your models on a regular basis on fresh data (automate as much as possible).
Following a checklist in a Machine Learning project is critically important. It will save a lot of effort and time in the long run and avoid errors and improve accuracy in the model.