Saurabh B Kulkarni, 44Mountain View, CA

2021031, Oct 14, 2021

Apr 14, 2020

16/848748

- Redmond WA, US
Tiyasa MITRA - San Jose CA, US
Sujeeth Subramanya BHARADWAJ - Milpitas CA, US
Marc TREMBLAY - Bellevue WA, US
Saurabh Mohan KULKARNI - Redmond WA, US

G06N 3/04
G06N 3/08
G06F 40/289
G06K 9/62

Embodiments of the present disclosure include a method for token-position handling comprising: processing a first sequence of tokens to produce a second sequence of tokens, wherein the second sequence of tokens has a smaller number of tokens than the first sequence of tokens; masking at least some tokens in the second sequence to produce masked tokens; moving the masked tokens to the beginning of the second sequence to produce a third sequence; encoding tokens in the third sequence into a set of numeric vectors in a first array; and processing the first array in a transformer neural network to determine correlations among the third sequence, the processing the first array producing a second array.

2021031, Oct 14, 2021

Apr 14, 2020

16/848707

- Redmond WA, US
Tiyasa MITRA - San Jose CA, US
Sujeeth Subramanya BHARADWAJ - Milpitas CA, US
Saurabh Mohan KULKARNI - Redmond WA, US
Marc TREMBLAY - Bellevue WA, US

G06N 3/08
G06N 3/04
G06F 40/126
G06F 17/16
G06F 17/15

Techniques for training neural networks are provided. According to one set of embodiments, a first array is processed in a spreading component to produce a second array, where a first dimension of the first array corresponds to at least one sequence of approximately orthogonal numeric vectors representing tokens, and where the spreading component combines values along the first dimension. The second array is processed in a transformer neural network to determine correlations between the sequence, which produces a third array. One or more batches of the third array are processed in a de-spreading component to produce a fourth array.

2021009, Apr 1, 2021

Sep 27, 2019

16/585105

- Redmond WA, US
Tiyasa Mitra - San Jose CA, US
Saurabh M. Kulkarni - Redmond WA, US
Marc Tremblay - Bellevue WA, US
Sujeeth S. Bharadwaj - Milpitas CA, US

G06N 3/04
G06F 17/27
G06N 3/08

Systems and methods for pipelined neural network processing with continuous and asynchronous updates are described. A method for processing a neural network comprising L layers, where L is an integer greater than two, includes partitioning the L layers among a set of computing resources configured to process forward passes and backward passes associated with each of the L layers. The method further includes initiating processing of the forward passes and the backward passes using the set of computing resources. The method further includes upon completion of a first set of forward passes and a first set of backward passes associated with a first layer of the L layers, initiating update of parameters associated with the first layer when gradients are available for updating the parameters associated with the first layer without waiting to calculate gradients associated with any of remaining L layers.