[Google_Bootcamp_Day23]

Sequence Models

Examples of Sequence problem

• Speech recognition
• Music generation
• Sentiment classification
• DNA sequence analysis
• Machine translation
• Video activity recognition
• Name entity recognition
• etc…

Notation example in Named-entity recognition problem

How to represent words

• Based on words dictionary, use one-hot encoding
• Assume dictionary have 10,000 words which means 100 dimensional vector. (Most case dictionary size = 30,000 ~50,000)
• Then every words will be represented to 100 dimensional vector (one-hot encoding)
• Goal is that given this representation for X to learn a mapping using a sequence model to then target output y

Why not use standard network?

Problems

• Inputs, outputs can be different lengths in different examples.
• Doesn’t share features learned across different positions of text.

Recurrent Neural Networks

Limitation : the prediction at a certain time uses information from the inputs earlier in the sequence but not information later in the sequence.

RNN forward propagation

Simplified RNN forward propagation notation

Forward propagation and backpropagation in RNN

Loss function

Examples of RNN architecture

• many-to-many
• many-to-one
  • sentiment classification (x=text, y=0/1)
• one-to-one
• one-to-many
  • music generation

[Source] https://www.coursera.org/learn/nlp-sequence-models