to get to the other bank." you need to "attend to" the word to disambiguate "bank". If is "street" then it's "bank" as in "financial institution" (most likely). If is "river" then …" />

What We Talk About When We Talk About Attention

Thu 03 November 2022

The "attention" in ML is "what you should attend to," not "alertness." In the sentence "They crossed the <???> to get to the other bank." you need to "attend to" the <???> word to disambiguate "bank". If <???> is "street" then it's "bank" as in "financial institution" (most likely). If <???> is "river" then it's (most likely) "bank" as in "muddy banks of the Mississippi.

It's perfectly fair to replace "attend to" with "pay attention to," so it's not a fundamentally different definition of the word. But I think a logical first reaction to the phrase "Attention is all you need," could be to think that what's being advocated has something with "alertness" or "arousal" being a single value that's being turned up or down, which would be confusing and lead to wrong intuitions.

blogroll

social