[Google_Bootcamp_Day12]

ML strategy 1

Why ML strategy important

• Assume we have cat classifer which have 90% performance, and want to improve the performance better.
• Then what should we do?
  • Collect more data
  • Collect more diverse training set
  • Train algorithm longer with gradient descent
  • Try Adam instead of gradient descent
  • Try dropout
  • Add L2 regularization
  • Change network architecture
  • etc…
• ML strategy is important to choose the most efficient way to improve your model

Orthogonalization

Orthogonalization or orthogonality is a system design property that assures that modifying an instruction or a component of an algorithm will not create or propagate side effects to other components of the system.
It becomes easier to verify the algorithms independently from one another, and reduces testing and development time.

When a supervised learning system is design, these are the 4 assumptions that needs to be true and orthogonal.

• Fit training set well on cost function
  • If it doesn’t fit well, try bigger network or Adam…
• Fit development set well on cost function
  • If it doesn’t fit well, try regularization or bigger training set
• Fit test set well on cost function
  • If it doesn’t fit well, try bigger dev set
• Performs well in real world
  • If it doesn’t fit well, try to change dev set or cost function

cf. early stopping -> affect both training performace and dev performance

Single number evaluation metric

• The problem with using precision/recall as the evaluation metric is that you are not sure which one is better
• F-1 score, a harmonic mean combine both precision and recall (Average also could be an evaluation metric)

Satisficing and optimizing metrics

There are different metrics to evaluate the performance of a classifier which are called evaluation metrics. They can be categorized as satisficing and optimizing metrics.

• Assume we want to maximize accuracy, and running time <= 100ms
• In this example, accuracy is the optimizing metric and running time is the satisficing metric.
• If you have N metrics, then it should be 1 optimizing metric + (N-1) satisficing metric.

Train/dev/test distributions

Setting up the training, development and test sets have a huge impact on productivity. It is important to choose the development and test sets from the same distribution and it must be taken randomly from all the data.
Choose a development set and test set to reflect data you expect to get in the future and consider important to do well.

Size of dev and test sets

• Old way of Splitting data
  • we had samller dataset therefore we had to use a greater percentage of data to develop and test ideas and models.
• Modern era - Big data
  • Now, because a large amount of data is available, we don’t have to compromised as much and can use a greater portion to train the model.

When to change dev/test sets and metrics

If you find that evaluation metric is not giving the correct rank order preference for what is actually better algorithm, then there’s a time to think about defining a new evaluation metric.

• Define correctly an evaluation metric that helps better rank order classifiers
• Optimize the evaluation metric
  
  

[Source] https://www.coursera.org/learn/machine-learning-projects