Another week in AI means more breakthroughs, new models, incredible research, and massive leaps in hardware. I’ve scoured a lot of dark and obscure place of the interwebs to bring you this content as usual.
If you don’t like reading, here’s the podcast of this episode:
Before we start!
- Like this post, because LinkedIn rewards engagement like it’s handing out candy. 📢
- Connect with me on LinkedIn and let’s be buddies. 🔗
- Subscribe to my TechTonic Shifts newsletter to get these updates without having to scroll through your feed. Seriously, your inbox will thank you. 💉
OpenAI’s o1 Models
The AI that thinks (slowly), so you don’t have to….
OpenAI has put their shiny new o1 series of “reasoning” models to the test: read the article: Strawberry is live, how to prompt it, the subscription fee, and the hunt for cash.
These models take their sweet time to think through things just like a chess grandmaster would. Including the time it takes to make a move. Darn… this thang (thang?, hmm… where have I heard that before) is sooo slow.
The o1 models actually use multiple steps of reasoning before they give you an answer. Yep, they sit there thinking with something called reinforcement learning.
Now, people are either calling this a step toward Artificial General Intelligence or (as I do), dismissing it as just a glorified Chain of Thought prompt. The “truth” according to OpenAI is that it’s somewhere in between.
Is it a breakthrough? I think it is..
But before you get way too excited, don’t expect to just plug this into your existing LLM pipeline and get smarter results. Because this big baby needs work and lots of compute power.
But what’s a little extra money, right?
Speaking of compute power….
OpenAI invested big time in a new large-scale RL algorithm. They probably had some super-smart folks hammering out their thought processes and feeding the model data. The result is a model that can tackle tasks your typical GPT can’t, though you might need a lot of patience, and money.
Did I mention this thing can take up to 60 seconds to think?
Yeah, your latte might get cold while you wait for your answer.
The performance breakdown (because a good blog needs numbers!)
Let’s get nerdy for a sec.
🤓
On some benchmarks, especially in science, math, code, and reasoning, the o1 models are crushing it. For example, on the PhD-level science questions, the GPT-4o scored 53.6%. The o1-mini hit a 60%, and the o1-preview hit 73.3%. But the actual o1 scored 77.3% on this questionnaire.
That’s right people!
AI just keeps leveling up, and now these guys are inching closer to super intelligence, where the AI knows more than a PhD guy who spent years studying one tiny sliver of a topic. this is an AI that can beat grad students at their own game…and answer science questions like Stephen Hawking would.
Of course, this comes at a cost.
Lots of cost (and water by the way – one little bottle of water to generate an e-mail apparently).
o1-Preview tokens are 6x more expensive than GPT-4o tokens (A token is a broken down piece of a sentence or word that is used to turn into a vector that the ML thingy can work with), and when you factor in all that extra “thinking time,” prices will skyrocket to 30x.
Yes, you read that right.
You might as well be burning your money while the model does its best Sherlock Holmes impression.
And this is why there is so much speculation of OpenAI making the final o1 model as expensive as a few thousands bucks a month….
But what makes o1 oh so special
What was really new to me and made me kinda geeky-happy is the non-plateauing scaling law that OpenAI let loose upon us all. In simple terms this means that you can throw more money (sorry, I mean “compute”) at the model during inference (where it tries to come up with an answer based on what it learned), and it just keeps on getting better.
Even simpler: the more you pay, the harder harder it thinks.
Convenient for OpenAI’s business model, no? Especially since they are now making huge losses a year (1.5 Billion), and are begging for money from NVIDIA and Microsoft (6.5 Billion, based on a valuation of 150 Billion). A first prognosis is that OpenAI will generate 11 Billion revenue based on these models.
Nah… I think I’ll stick around with 4o for a while if the price hits a few grand…
So, what would this mean for the rest of us?
Well, theoretically, o1 could spend days solving a problem that regular LLMs couldn’t touch.
So if you have a week to spare or a huge budget, the o1 might just crack that impossible math problem or write your PhD thesis.
Why should you even care
Good question!
Besides the fact that it will make brainstorming sessions feel like a walk in the park (not that it will actually churn out REAL inventions though: read Lobotomized AI), o1 is truly an example of how AI is evolving.
It is no longer just cramming more tokens into the machine. Nah, it is now about having those tokens think for you. I think that RL-based reasoning steps will be the thing that finally gets AI to think like a human would.
Or at least, like a really expensive human.
If you’ve ever complained that our previous LLMs are just glorified word-predictors, here is your answer.
Turns out that they have just been trained to skip the thinking part.
But not anymore.
Now, with o1, you get the full inner monologue (or a dialogue if it’s turning into a schizoid AI). And this model is not only for science nerds though. I really think that it will be worth to anyone who wants to unlock some real out-of-the-box thinking.
But for that to happen, I really need to test the final o1 model, and not the preview.
Hottest news in AI
Brace yourself for some serious geek speak
What pappened this week in AI
So, another week, another batch of shiny new AI models trying to take over the world (or at least your search results). The open-source gang is getting stronger, but let’s be honest, they’re still trying to catch up to the big boys like OpenAI’s o1 “reasoning” models. Still, progress is progress.
Here’s what went down this week:
Qwen 2.5: The open-source champion
Let’s start with Qwen 2.5, that is Alibaba’s latest attempt to prove that China can do AI just as well as anyone else. This model is currently the top dog in the open-source world for general language tasks. Heck, it even beats the much larger LLaMA 3.1 in some benchmarks (hahaha, when I read it I thought of a chihuahua taking down a Great Dane). It comes in various flavors like Qwen2.5-Coder and Qwen2.5-Math for specialized tasks. It is trained on a casual 18 trillion tokens (no big deal) and it supports over 29 languages. And yes, it is open-source under the Apache 2.0 license. Want to play with it? There’s also an API version, the Qwen-Plus, for those who prefer their AI with a side of exclusivity. [Check out Qwen 2.5 here](https://huggingface.co/).
Pixtral 12B: the new multimodal hotshot
Pixtral 12B from Mistral AI. Is their first multimodal model that does both text and images. It is competing with the much bigger LLaVa OneVision 72B. It is a 12-billion-parameter model that can do chart understanding, document Q&A, and of course …..reasoning ! Oh, and it is also pretty good at text-only tasks like coding and math. It is again open-source under the same Apache 2.0 license and it supports images in a 128K token context window, so it can handle massive amounts of data and not break the bank like ChatGPT 4o1 final would. Go to Pixtral 12B here: https://mistral.ai/pixtral12b.
GRIN-MoE: Microsoft’s low-key beast
No, it’s not a smiling cow, but Microsoft that has finally entered the chat with a mixture-of-experts (that is what MoE stands for, duhhh) model designed for low-compute environments. Think of your phone. This one has got 16×3.8B parameters but only uses 6.6B active parameters. I think this will be the new hero in the small-inference-compute world. It is the AI model for those of us without a supercomputer sitting in our basement. It is efficient, scales well, and is great for when you need something powerful but don’t have a Musk-sized budget. It’s open-source under an MIT license. Check out GRIN-MoE here: https://microsoft.com/grinmoe.
Why should you even care at all
OpenAI’s o1 models might be amazing, but they are truly expensive….like, “sell your soul for extra tokens” kinda expensive. The new open-source models you read about up there are not perfect, but they give you a few solid alternatives without the need for a second mortgage.
It doesnt matter nymore if you are working on advanced reasoning, or coding, or multimodal tasks, there is going to be a model out there for you. And yeah, you will probably need to mix and match. A bit of directing some queries to OpenAI, some to Qwen, and others to Pixtral, because no single model does everything well.
Welcome to the future, where model routers will become a thing.
But don’t get too cozy because pre-training and post-training these models is getting super expensive, and that means that fewer companies will compete at the foundation model level. So, expect most of the innovation to come from building on top of these existing models.
BrightData plug, because web data runs the world
By the way, if you think static datasets are ok for AI, that is not the case. Because as our AI’s get smarter, we need more real-time, unstructured web data. It will become more critical than ever. Now that is where Bright Data comes in. They sell you businesses organized and real-time insights from across the web. Learn more about web data here: https://brightdata.com.
Other news in AI
Microsoft wants to revive three mile island to power AI: Yeah, you read that right. Microsoft just signed a 20-year deal to tap into the energy from the infamous Three Mile Island nuclear power plant to fuel their AI data centers. I’ve written about Oracle’s plans to build a nuke before, read: Oracle has commissioned three small nuclear reactors to power its new AI data center. And if approved (and I know it will because it will produce less CO2), Microsoft will own 835 megawatts of energy for the next two decades. Read more here: https://microsoft.com/three-mile-island-ai.
Anthropic Introduces Contextual Retrieval: Anthropic rolled out a fancy new retrieval method called Contextual Retrieval. That thang improves accuracy in AI tasks by up to 67%. Basically what it is, it makes AI less dumb when fetching info, or reducing errors in downstream tasks. Anthropic’s blog has the details: https://anthropic.com/contextual-retrieval.
Michelangelo: long context evaluations: Forget simple Q&A thingies because Michelangelo evaluates how well models handle long-context data. It is designed to challenge AI by forcing it to extract deeper relationships from text. Now that is much harder than your usual “needle in a haystack” tests. Learn more about Michelangelo here: https://arxiv.org/abs/michelangelo-long-context.
Sam Altman declares the “Intelligence Age”: Hahahaha, in a self-congratulatory blog, our dearly beloved Sam Altman reminds us all that deep learning works, and, no surprise, it gets better with scale (and with lots of dough). He also talks about the moral and ethical dilemmas of the Intelligence Age, which is apparently just around the corner (is that why nearly his entire AI safety staff left the building, starting their own AI safety companies?)Check out Sam’s victory lap here: https://openai.com/sam-altman-intelligence-age.
Coupe of minutes reads (for when you pretend to work)
1. The Open Source Project Maintainer’s Guide: Tips on how not to scare off contributors and how to actually get people to help with your open-source project.
2. AI vs. Human Engineers: CodeSignal pitted AI against human engineers in coding assessments. Spoiler: AI isn’t that bad at thinking outside the box.
3. How Streaming LLM APIs Work: Want to know how OpenAI, Anthropic, and Google make their LLM APIs work in real-time? This guide breaks it down.
4. How I Deal With Hallucinations at an AI Startup: Ever had an AI that really wanted to be creative? Here’s how to deal with its “hallucinations.”
5. Fine-Tuning LLMs to 1.58bit: Can’t afford to train your own model from scratch? This guide shows you how to fine-tune existing ones on a budget.
This weeks’ research papers
1. Training Language Models to Self-Correct via Reinforcement Learning
This paper developed SCoRe, a multi-turn online reinforcement learning approach that significantly improves an LLM’s self-correction ability using entirely self-generated data. When applied to Gemini 1.0 Pro and 1.5 Flash models, SCoRe improved the base models’ self-correction by 15.6% and 9.1%, respectively, on the MATH and HumanEval benchmarks.
2. OneGen: Efficient One-Pass Unified Generation and Retrieval for LLMs
This paper introduces the One-pass Generation and retrieval framework (OneGen). It is designed to improve LLMs’ performance on tasks requiring generation and retrieval. The framework incorporates retrieval tokens generated autoregressively. This enables a single LLM to handle both tasks simultaneously in a unified forward pass.
3. Eureka: Evaluating and understanding progress in AI
This paper presents Eureka, an open-source framework for standardizing evaluations of large foundation models beyond single-score reporting and rankings. It also introduces Eureka-Bench as an extensible collection of benchmark testing capabilities. It analyzes 12 state-of-the-art models, providing in-depth insights into failure understanding and model comparison that can be leveraged to plan targeted improvements.
This paper introduces SAMMO, a framework to perform symbolic prompt program searches for compile-time optimizations of prompt programs. SAMMO generalizes previous methods and improves the performance of complex prompts on instruction tuning, RAG pipeline tuning, and prompt compression across several different LLMs.
5. Neptune: The Long Orbit to Benchmarking Long Video Understanding
This paper introduces Neptune, an evaluation benchmark that includes tough multiple-choice and open-ended questions for videos of variable lengths up to 15 minutes long. Neptune’s questions are designed to require reasoning over multiple modalities (visual and spoken content) and long time horizons, challenging the abilities of current large multimodal models.
6. Wings: Learning Multimodal LLMs without Text-only Forgetting
This paper presents Wings, an MLLM that excels in text-only dialogues and multimodal comprehension. The experimental results demonstrate that Wings outperforms equally-scaled MLLMs in text-only and visual question-answering tasks.
Links
1. Google Quantum AI demonstrates a quantum memory system that greatly reduces error rates. The quantum computer uses multiple physical bits to create one logical qubit. The researchers have developed an algorithm that they call “surface code” to correct errors.
2. Former Apple design chief Jony Ive has confirmed that he’s working with OpenAI CEO Sam Altman on an AI hardware project. There aren’t many details on the project. Ive reportedly met Altman through Brian Chesky, the CEO of Airbnb, and the venture is being funded by Ive and Laurene Powell Jobs’ company.
Top thingies of the week
- Javascript Algorithms contain JavaScript-based examples of many popular algorithms and data structures.
- optillm is an OpenAI API-compatible optimizing inference proxy that implements several techniques to improve LLMs’ accuracy and performance.
- Solidroad is an AI-first training and assessment platform.
- Agent Zero is a personal and organic AI framework for tasks.
Signing-off Marco
Well, that’s a wrap for today. Tomorrow, I’ll have a fresh episode of TechTonic Shifts for you. If you enjoy my writing and want to support my work, feel free to buy me a coffee ♨️
Think a friend would enjoy this too? Share the newsletter and let them join the conversation. Google appreciates your likes by making my articles available to more readers.
Leave a Reply