Recurrent Neural Network
2018-07-30
Machine-Learning
388
This article is my learning note of the coursera course Sequence Models by Andrew Yan-Tak Ng.
There are two typical RNN units of the hidden layers of the RNN according to Andrew Ng. One is GRN (Gated Recurrent Unit), the other is LSTM (Long Short-Term Memory).
Notice: Please refer to Mathematical Basis - Squashing Function for some basic math knowledge about the squashing functions.
GRN - Gated Recurrent Unit
The GRN is a gating mechanism in recurrent neural networks, introduced in 2014 by Kyunghyun Cho et al.
The fully gated version :
The formulas :
@ : The memory cell.
@ : The input sequence.
@ : The output sequence.
@ : Gate gamma r. It tells us how relevance is to computing the next candidate for .
@ : Gate gamma u. The update gate vector. Decide whether or not we actually update , the memory cell.
@ : The candidate value for the memory
