https://yidingjiang.github.io/blog/ Recent content in Blogposts on Yiding's blog Hugo en-us Wed, 25 Jun 2025 17:57:46 -0400 https://yidingjiang.github.io/blog/post/exploration/ Wed, 25 Jun 2025 17:57:46 -0400 https://yidingjiang.github.io/blog/post/exploration/ <p>Large language models are the unintended byproduct of about three decades worth of freely accessible human text online. Ilya Sutskever compared this reservoir of information to <a href="https://www.youtube.com/watch?v=YD-9NG1Ke5Y">fossil fuel</a>, abundant but ultimately finite. Some studies suggest that, at current token‑consumption rates, frontier labs could exhaust the highest‑quality English web text well before the decade ends. Even if those projections prove overly pessimistic, one fact is clear: today’s models consume data far faster than humans can produce it.</p> https://yidingjiang.github.io/blog/post/curriculum/ Tue, 06 May 2025 20:29:38 +0800 https://yidingjiang.github.io/blog/post/curriculum/ <p>A fundamental property of an intelligent system is the ability to predict, be it predicting the label of an image, the optimal action to take, or the next token in a sentence. Having a good predictive model is intimately tied to having a good compression algorithm. Conversely, a strong compression algorithm must have a good model that can identify and leverage patterns in the data.</p> <p>While the relationship between compression and intelligence has been thoroughly discussed<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup>, we will focus here on a particularly interesting relationship between compression and massively overparameterized models, which also naturally lends itself to an objective for data selection and curriculum learning.</p> https://yidingjiang.github.io/blog/post/ado/ Sun, 13 Apr 2025 00:00:00 +0000 https://yidingjiang.github.io/blog/post/ado/