ML Architecture and papers
Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet Really important article, seemingly a breakthough in peering inside LLMs. A nice adjunct to this article is the less dense An Intuitive Explanation of Sparse Autoencoders for Mechanistic Interpretability of LLMs. My big worry about this is that it seems to make jail-breaking LLMs very easy for bad actors.
Genetic Algorithm for Fusing LLM weights Not sure it’s practical, but I always love a good GA
Transformer architecture with only -1, 0, 1 weights I don’t know enough about modern chip architecture to know if this is a big win, but you’d think so.
Consciousness, Machines, and Moral States Really good (long) article. He proposes “Shevlin’s Tried” : “independently plausible and appealing [positions on moral agents that are ] collectively inconsistent. Might write a full blogpost on this one.
ML Programming
No more shape errors! Type annotations for the shape and Dtype of tensors and arrays Announcing the latest version of the jaxtyping library. Despite it’s name, it’s compatible with not just Jax, but Pytorch,
Tools I haven’t necessarily tried
Mesh Anything : Claims to be a significant improvement in going from point clouds to meshes. Paper, source, and Gradio demo available. I didn’t see weights.
Cut Off Facebook A browser script that disallows Facebook advertisers (one-by-one).
Mesop: Quickly build Web UIs in PythonFrom Google. Has hot reload. Might try it for my next demo.
Misc
Neo-Nazis are all in on AI Speaking of bad actors.
ChatGPT now has PhD-level Intelligence and the poor personal choices to prove it Funny
Taking a closer look at AI’s supposed energy apocalypsetl;dr: It is overstated.
How to build anything extremely quickly tl;dr: outline recursively.
The Functional Programming Hiring Problem 100% accurate.
Oldest shipwreck found: Cool.