11 Practical Methodology¶
practical design process:
- Determine your goals.
- which error metric to use?
- what level of performance you desire?
- Establish a working end-to-end pipeline as soon as possible.
- Instrument the system well to determine bottlenecks in performance.
- Repeatedly make incremental changes.
- Determinig whether to gather more data <= when train error is accpted, while test error not.
- plot curves showing the relationship between training set size and generalization error. usually logarithmic scale.
- Selecting Hyperparameters
- manually: learning rate
- automatic
- grid search: typically use logarithmic scale
- random search: more convenient to use, and converges much faster
- Debugging strategies
- Visualize the model in action.
- Visualize the worst mistakes.
- Reasoning about software using train and test error.
- Fit a tiny dataset.
- Compare back-propagated derivatives to numerical derivatives.
- Monitor histograms of activations and gradient:
- visualize statistics of neural network activations and gradients, collected over a large amount of training iterations (maybe one epoch).
- compare the magnitude of parameter gradients to themselves.