This newsletter made possible by MosaicML. And thanks to @snowclipsed for the Twitter shoutout this week!
This is amazing work and it makes my early mornings. I get to learn so much (not an exaggeration), easily my favorite blog of this year. Keep up the good work sir! ~snowclipsed
Interestingly Hyena filters use the same technique as ALiBi. (ExponentialModulation in https://github.com/HazyResearch/safari/blob/main/src/models/sequence/hyena.py)
This is amazing work and it makes my early mornings. I get to learn so much (not an exaggeration), easily my favorite blog of this year. Keep up the good work sir! ~snowclipsed
Interestingly Hyena filters use the same technique as ALiBi. (ExponentialModulation in https://github.com/HazyResearch/safari/blob/main/src/models/sequence/hyena.py)