S.Korean LLM by Upstage beats global benchmark ChatGPT

The latest artificial intelligence model of South Korean AI startup Upstage scored higher than ChatGPT by global AI giant OpenAI in a world-recognized open-source language model evaluation, becoming the first large language model (LLM) to beat the world’s most famous chatbot model.

Upstage said on Tuesday that its new AI model trained with Meta’s LLaMA 2 70B model rose to an average score of 72.3 on the latest HuggingFace Open LLM Leaderboard, a standard for open-source LLM evaluation of over 500 LLMs.  

The Korean AI startup said it is the first time a non-major AI model has ever elbowed out ChatGPT powered with GPT-3.5 to top the scoreboard, proving the Korean firm’s technological competency on the global AI stage.

ChatGPT’s benchmark score was 71.9 on the same day.

The latest victory also comes after its previous LLM model with 30 billion parameters rose to the top spot on the leaderboard with an average score of 64.7 after outperforming Meta’s LLaMA 2 70B model released on the same day the previous month.

Upstage’s newest LLM is advanced with the latest LLaMA 2 model with higher parameter sof 70 billion, the Korean startup said.

TO PIONEER THE GLOBAL PRIVATE AI MARKET

The HuggingFace leaderboard evaluates four key metrics, including reasoning challenge, common sense inference, contextual understanding and factual accuracy to gauge the level of AI hallucination.

With its own 70B LLM, the Korean AI startup snatched back its crown from Stability AI’s Stable Beluga 2, which beat the Korean LLM on the previous leaderboard, with a 71.4 score, since the introduction of the LLaMA 2.

After showing off its technological excellency, Upstage plans to foray into the private AI market, it said on Tuesday.

A private AI is a business AI solution with a strong data protection and security system as it is specifically trained on a private internal dataset of a company.  

It is gaining traction because it is known to have less AI hallucination risk of generating misleading information presented as accurate.

“Upstage will strive to enhance its leadership in the private AI market at home and abroad with its excellent technological expertise,” Upstage Chief Executive Officer Kim Seong-hoon said in a press release.

The Korean startup is betting big on its AI solutions after proving that a smaller AI model can be competitive against bigger global giants even with lower costs.

Upstage attributed the latest achievement to a team of outstanding AI experts who have developed KLUE, a Korean language understanding (NLU) dataset, and the company’s flagship chatbot AskUp.

AskUp, the first Korean chatbot service powered by OpenAI’s GPT-4, is now used by about 1.3 million people.

It took only two months for the team to develop the latest LLM fine-tuned with the LLaMA 2 model.

Founded in 2020, Upstage is led by Kim, a former professor at the Hong Kong University of Science and Technology and one of the world’s best AI experts to converge software engineering and machine learning. He also led the development of Naver Clova AI.

Lee Hal-seok, from the team that developed Naver Visual AI and Kim Jae-beom, a former Kakao AI team leader, are also on board.

Other AI engineers from US Big Tech companies such as Meta, Nvidia and Amazon have joined the company, according to the company.

Write to Joo-Wan Kim at kjwan@hankyung.com

Sookyung Seo edited this article.