HyperCLOVA X surpasses GPT-4 in Korean AI evaluation

South Korea's Naver Cloud, the cloud computing arm of Naver Corp., announced on Tuesday that its HyperCLOVA X has scored higher than OpenAI and Google's generative AIs in the Korean AI performance evaluation system Measuring Massive Multitask Language Understanding in Korean (KMMLU).

KMMLU is an AI performance evaluation metric construction project led by the renowned domestic open-source language model research team HAE-RAE.

It consists of 35,030 questions asking for expert-level knowledge in 45 fields, including humanities, sociology, science, and technology.

About 80% of the questions ask for broad knowledge that can be applied worldwide, such as mathematical reasoning ability, while 20% evaluate the ability to solve Korea-specific problems, such as the geography of the Korean Peninsula and domestic law.

Composed of test questions in Korean, KMMLU allows for a more accurate assessment of AI's Korean language understanding capabilities, measuring both universal abilities and local knowledge to comprehensively judge AI solutions useful for Korean users, according to Naver.

According to the KMMLU research paper, HyperCLOVA X recorded higher scores than OpenAI's GPT-3.5-Turbo and Google's Gemini-Pro and even surpassed OpenAI's GPT-4 in terms of Korea-specific knowledge.

Naver Cloud plans to develop HyperCLOVA X into a Sovereign AI solution equipped with both security and performance, based on the competitive performance proven through KMMLU.

Write to Ju-Hyun Lee at deep@hankyung.com