Google Gemini’s performance falls well short of GPT’s. This recent study embarrasses the online search giant.
December 6, Google launched Gemini with the ambition of becoming the new benchmark for large-scale language models. But the current version of the model does not achieve this goal. According to a team of researchers, it can’t even compete with GPT-3.5.
Google Gemini doesn’t worry OpenAI yet
It should be remembered that Gemini was launched to compete with OpenAI in the large language models (LLM). But its Pro version, which is the most advanced version available to the public, is much less efficient than the GPT-4 from OpenAI.
This is the conclusion of researchers at theCarnegie Mellon University of Pittsburgh, U.S.A., and California startup BerriAI. They detail their research in a paper available on the open archive arXiv.
According to the study, Gemini Pro offers accuracy comparable to, but slightly less than, that of GPT 3.5 Turbo. But Google’s model is clearly inferior to GPT-4 Turbo on this point. It is also handicapped by shortcomings in various areas. These include word reorganization and the symbol generation.
On the other hand, the model developed by Google’s DeepMind lab struggles to solve multiple-choice questions, à understand complex queries and to code.
Better than French Mixtral
For their work, the American researchers selected four major language models. These were pitted against each other in various tests to compare their performance. Each model had to process a sequence of promptsencompassing fields such as mathematics, reasoning and the social sciences.
The models in question are :
- Gemini Pro from Google
- GPT-3.5 Turbo and GPT-4 Turbo OpenAI
- Mixtral 8x7B from the French startup Mistral
Note that the research team also used LiteLLMan artificial intelligence aggregator site.
Nevertheless, Google’s model outperforms GPT 3.5 Turbo in certain tasks. It can better word sorting and manages prompts more efficiently on the security and the microeconomics. Gemini Pro also performed better than its competitors in this test. On the other hand, it beats Mixtral in all areas.
A more powerful Ultra version for 2024
The researchers from Carnegie Mellon University and BerriAI are obviously embarrassing Google. The online search leader was quick to respond to their paper, maintaining that Gemini Pro was superior to GPT-3.5.
On the other hand, the American digital giant has announced an improved version of its model. Gemini Ultra should be launched in the coming weeks. This next version is said to surpass GPT-4 in terms of performance, according to Google’s internal evaluations. Which gives an impression of déjà vu.
Indeed, this isn’t the first time that the number one in online search has announced a language model with superior capabilities. Then an independent study contradicts its internal assessments.
Our blog is powered by readers. When you purchase via links on our site, we may earn an affiliate commission.