Prof. Dr. Lucie Flek cordially invites you to the DAAD AI Net Visiting Researcher speaker series she is organizing. Meet highly renowned ML and AI Scientists from all over the world and learn more about their research topics.
The Postdoc Networking Tour in Artificial Intelligence (Postdoc-NeT-AI) is a DAAD initiative funded by the Federal Ministry of Education and Research (BMBF). The aim is to offer talented scientists from all over the world personal access to the German AI research landscape. Within this tour, Prof. Dr. Lucie Flek offers a speaker series with the guest researchers.
The visiting researchers will give exciting machine learning talks at regular intervals. Afterwards, the audience will have the opportunity to talk 1:1 to the scientists.
The event will take place in the b-it max or online via Zoom.
|Topic and Speaker
November 22, 2023
(When) do we need self attention?
Abstract: The attention mechanism has proven to be very powerful, but it has quadratic complexity in the size of the input, making it prohibitively expensive for some use cases. To address this issue, we first propose a linear-time attention alternative, HyperMixer, which uses only multi-layer perceptrons to mix information between tokens and performs competitively with Transformers on text classification tasks. Next, we integrate HyperMixer into Conformer, a popular method for automatic speech recognition, again eliminating the need for attention in this important task. Finally, we take a look into the future and discuss the question when or if we will still need self-attention.