[Google_Bootcamp_Day7]

Normalizing inputs

• Use same mean, and variance to normalize test set

Why normalize inputs

• Easy and fast to optimize

Vanishing/Exploding gradients

• Single neuron example
  • Partial solution for vanishing gradient problem is careful choice of random initialization

Numerical approximation of gradients

Gradient Checking

• Don’t use in training, only use to debug (slow computation)
• If algorithm fails grad check, look at components to try to identify bug
• Remember regularization
• Doesn’t work with dropout (Turn off dropout, then grad check)
• Run at random initialization, perhaps again after some training

[Source] https://www.coursera.org/learn/deep-neural-network