Gru

the vanilla rnn cannot learn long-range dependencies: its gradient signal is a product of jacobians that shrinks or blows up geometrically with distance. the lstm’s answer is architectural, not numerical — give the network a second state, updated additively rather than by repeated matrix multiplication, and let learned gates decide what enters, what stays, and what leaves. 𐃏 the design dates to hochreiter and schmidhuber’s 1997 paper (long short-term memory, neural computation 9(8)), and for two decades it was simply what “rnn” meant in practice (Goodfellow, Ian, 2016).