This newsletter made possible by MosaicML. And thanks to @snowclipsed for the Twitter shoutout this week! Also, I wrote a blog post about using language models to generate training data. Been thinking about this for a while and finally felt like I got a clear enough mental model to share. Spoiler: effectively infinite data is possible, but only under certain conditions.
This is amazing work and it makes my early mornings. I get to learn so much (not an exaggeration), easily my favorite blog of this year. Keep up the good work sir! ~snowclipsed
This is amazing work and it makes my early mornings. I get to learn so much (not an exaggeration), easily my favorite blog of this year. Keep up the good work sir! ~snowclipsed
Interestingly Hyena filters use the same technique as ALiBi. (ExponentialModulation in https://github.com/HazyResearch/safari/blob/main/src/models/sequence/hyena.py)