Elon Musk’s answer to ChatGPT is getting an update to make it better at math, coding and more. Musk’s xAI has launched Grok-1.5 to early testers with “improved capabilities and reasoning” and the ability to process longer contexts. The company claims it now stacks up against GPT-4, Gemini Pro 1.5 and Claude 3 Opus in several areas.
Going by xAI’s numbers, Grok-1.5 appears to be a large improvement over Grok-1. It shot up to 50.6 percent in the MATH benchmark, over double the previous score. It also climbed to 90 percent and 74.1 percent in GSM8K (math word problems) and HumanEval (coding), respectively, compared to 62.9 percent and 63.2 percent before. Those numbers are within shouting distance of Gemini Pro 1.5, GPT-4 and Claude 3 Opus — in fact, the HumanEval coding score beats all rivals except Claude 3 Opus.
It can also process long contexts of up to 128K tokens within its context window, meaning it can amalgamate data from more sources to understand a situation. “This allows Grok to have an increased memory capacity of up to 16 times the previous context length, enabling it to utilize information from substantially longer documents,” the company said.
xAI didn’t detail Grok’s progress in other areas, though, where it still may be lagging (academic scores, multimodal and others). And Grok-1.5 may not keep its position for long. ChatGPT 5 is set to arrive sometime this summer, promising a feature set that “makes it feel like you are communicating with a person rather than a machine,” according to OpenAI.
Currently, Grok is only available for users of the Premium+ tier on X (formerly Twitter), though Elon Musk recently promised to open it up to X’s regular Premium users. The company also recently open sourced its Grok chatbot, after Musk sued OpenAI and Sam Altman for allegedly abandoning its non-profit mission.