Sebastian Raschka

"I used that feature before, and I always feel bad because it does that every day, and I rarely check it out"

"So, I use the Codeium plugin for VS Code. You know, it's very convenient. It's just like a plugin, and then it's a chat interface that has access to your repository."

"So, I use the Codeium plugin for VS Code."

"What I would recommend doing, or what I also do, is if I want to understand, for example, how OLMo is implemented, I would look at the weights in the model hub, the config file, and then you can see, 'Oh, they used so many layers.'"

"Even back when I was a grad student, I was in a lab doing biophysical simulations, molecular dynamics, and we had a Tesla GPU back then just for the computations. It was about 15 years ago now."

"I should say I use Composer a lot because one of the benefits it has is that it's fast"

"Sometimes it takes me a day. With OLMo 3, the challenge was RoPE for the position embeddings. They had a YaRN extension and there was some custom scaling there, and I couldn't quite match these things."

"Sometimes for pastime I play video games, like I like- Video games with puzzles, like Zelda and Metroid."

"I was training the Qwen 3 base model with RLVR on MATH-500. The base model had an accuracy of about 15%."

"Exa is my preferred search provider"

"I try Claude Code on the web every three to six months, which is just prompting a model to make an update to some GitHub repository that I have"

"We had a Tesla GPU back then just for the computations. It was about 15 years ago now"

"I can give you also a hands-on example. I was training the Qwen 3 base model with RLVR on MATH-500. The base model had an accuracy of about 15%. Just 50 steps, like in a few minutes with RLVR, the model went from 15% to 50% accuracy."

"Sometimes for pastime I play video games, like I like- Video games with puzzles, like Zelda and Metroid."

"So I suggested, 'Hey, let's try ChatGPT.' We copied the text into ChatGPT, and it fixed them. Instead of two hours going from link to link fixing that, it made that type of work much more seamless."

"The Recursive Language Model paper, that is one of the papers that tries to kind of address the long context thing"

"Let's throw in Mistral AI, Gemma..."

"gpt-oss, the open weight model by OpenAI... gpt-oss-120b is actually a very strong model and does some things that other models don't do very well."

"Actually, NVIDIA had a really cool one, Nemotron 3."

"And then you start, let's say, with your GPT-2 model and add these things."

"I would love to have tried Bing Sydney. Did that have more voice? Because it would so often go off the rails, which is historically obviously a scary way—like telling a reporter to leave his wife—is a crazy model to potentially put in general adoption."

"There was a lot of backlash last year with GPT-4o getting removed, and I've personally never used the model, but I've talked to people at OpenAI where they get emails from users that might be detecting subtle differences in the deployments in the middle of the night."

"even something simpler like MMLU, which is a multiple-choice benchmark. If you just change the format slightly, like, I don't know, if you use a dot instead of a parenthesis or something like that, the model accuracy will vastly differ."

"When you code these from scratch, you can take an existing model from the Hugging Face Transformers library. The library is great, but if you want to learn about LLMs, it's not the best place to start because the code is so complex to fit so many use cases."

"even Transformers, the library, is not used in production. People use SGLang or vLLM, and it adds another layer of complexity."

"even Transformers, the library, is not used in production. People use SGLang or vLLM, and it adds another layer of complexity."

"With OLMo 3, the challenge was RoPE for the position embeddings. They had a YaRN extension and there was some custom scaling there, and I couldn't quite match these things."

"They had a YaRN extension and there was some custom scaling there, and I couldn't quite match these things."

"For the character training thing, I think this research is built on fine-tuning about 7 billion parameter models with LoRA, which is essentially only fine-tuning a small subset of the weights of the model."

"And listeners may know diffusion models from image generation, like Stable Diffusion popularized it."

"There was a paper on generating images. Back then, people used GANs, Generative Adversarial Networks."

"It's kind of similar to the BERT models by Google. Like, when you go back to the original transformer, they were the encoder and the decoder."

"But there was an announcement by Google, a site where they said they are launching Gemini Diffusion, and they put it into context of their Gemini Nano 2 model, and they said basically: for the same quality on most benchmarks, we can generate things much faster."

"they put it into context of their Gemini Nano 2 model"

"Like what Apple tried to do with the Apple Foundation models, putting them on the phone, where they learn from experience."

"DeepSeek-V3.2, where they had a sparse attention mechanism where they have essentially a very efficient, small, lightweight indexer"

"There was a paper by Meta, a paper called World Models. So where they basically apply the concept of world models to LLMs again"

"There is a competition called CASP, I think, where they do protein structure prediction"

"AlphaFold, when it came out, it crushed this benchmark"

"There's some work in this area like RTX, I think it was a few years ago, where people are starting to do that"

"I don't know if you like the originally titled AI27 report. They focus more on code and research taste, so the target there is the superhuman coder"

"I think there are startups—maybe Harmonic is one—where they're going all in on language models plus Lean for math"

"language models plus Lean for math"

"We hear about Reflection AI, where they say their two billion dollar fundraise is dedicated to building US open models"

"Scale AI for almost $30 billion and countless other deals like this"

"That's why part of what Vera Rubin is- where they have a new chip with no high-bandwidth memory, which is one of the most expensive pieces"

"The moat of NVIDIA is probably not just the GPU. It's more like the CUDA ecosystem, and that has evolved over two decades"

"Like, Google obviously can make TPUs"

"I think it only happened because you could purchase those GPUs."

"The word 'transformer' could still be known. I would guess that deep learning is definitely still known, but the transformer might be evolved away from in 100 years with AGI researchers everywhere."

"They're signing licensing deals with Black Forest Labs, which is an image generation company"

"A lot of researchers at these companies are so well-motivated, and definitely Anthropic and OpenAI culturally want to do good for the world."

"A lot of researchers at these companies are so well-motivated, and definitely Anthropic and OpenAI culturally want to do good for the world."

"You want to add a new tab in Slack that you want to use, and I think AI will be able to do that pretty well"

"my wife the other day—she has a podcast for book discussions, a book club, and she was transferring the show notes from Spotify to YouTube, and then the links somehow broke."

"take something like Slack or Microsoft Word. I think if organizations allow it, AI could very easily implement features end-to-end"

"signing licensing deals with Black Forest Labs, which is an image generation company, or Midjourney"

"Let's throw in Mistral AI, Gemma..."

"Amazon is making Trainium"

"my wife the other day—she has a podcast for book discussions, a book club, and she was transferring the show notes from Spotify to YouTube, and then the links somehow broke."

"We see this with TikTok. You open it... I don't use TikTok, but supposedly in five minutes the algorithm gets you. It's locked in."

"For example, if you read a Substack article, I could maybe ask an LLM to give me opinions on that, but I wouldn't even know what to ask."

"We are starting to see some types of consolidation with Groq for $20 billion"

"I think there will be some other multi-billion dollar acquisitions, like Perplexity"

"ChatGPT has a memory feature, right? And so you may have a subscription and you use it for personal stuff, but I don't know if you want to use that same thing at work."

"I think when I was at Hugging Face, I was trying to get this to happen, but it was too early. It's like these open robotic models on Hugging Face"