Abstract
Test of GLU and its variants in Transformer's feed-forward layers improves quality over ReLU or GELU.
AI-generated summary
Gated Linear Units (arXiv:1612.08083) consist of the component-wise product of two linear projections, one of which is first passed through a sigmoid function. Variations on GLU are possible, using different nonlinear (or even linear) functions in place of sigmoid. We test these variants in the feed-forward sublayers of the Transformer (arXiv:1706.03762) sequence-to-sequence model, and find that some of them yield quality improvements over the typically-used ReLU or GELU activations.
Get this paper in your agent:
hf papers read 2002.05202 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 95
Browse 95 models citing this paperDatasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2002.05202 in a dataset README.md to link it from this page.