Banca de DEFESA: ANTÔNIO PEREIRA DE SOUZA JÚNIOR

Uma banca de DEFESA de MESTRADO foi cadastrada pelo programa.
STUDENT : ANTÔNIO PEREIRA DE SOUZA JÚNIOR
DATE: 15/03/2024
TIME: 09:30
LOCAL: meet.google.com/odk-yuba-pah
TITLE:

Mitigating the Limits of the Current Evaluation Metrics for Topic Modeling

KEY WORDS:

topic modeling, evaluation metrics, machine learning, artificial inteligence

PAGES: 60
BIG AREA: Ciências Exatas e da Terra
AREA: Ciência da Computação
SUBÁREA: Metodologia e Técnicas da Computação
SPECIALTY: Banco de Dados
SUMMARY:

Topic Modeling (TM) is a popular approach to extracting and organizing information from large amounts of textual data by discovering and representing semantic topics from documents. In this work, we investigate an important challenge in the TM context, namely Topic evaluation, responsible for driving the advances in the field and assessing the overall quality of the topic generation process. Traditional TM metrics capture the quality of topics by strictly evaluating the words that built the topics syntactically (i.e., NPMI, TF-IDF Coherence) or semantically (i.e., WEP). In here, we investigate whether we are approaching the limits of what the current evaluation metrics can assess regarding topic quality for TM. We performed a comprehensive experiment, considering three data collections widely used in automatic classification, for which each document's topic (class) is known (i.e., ACM, 20News and WOS). We contrast the quality of topics generated by four of the main TM techniques (i.e., LDA, NMF, CluWords and BerTopic) with the previous topic structure of each collection. Our results show that, despite the importance of the current metrics, they could not capture some important idiosyncratic aspects of TM, indicating the need to propose new metrics that consider, for example, the structure and organization of the documents that comprise the topics. In order to mitigate this limitation, we propose to adapt metrics commonly used to evaluate clustering algorithms since there are significant similarities between the TM and clustering strategies. Both have an unsupervised nature and the purpose of grouping similar elements. We evaluate three distinct metrics (Silhouette Score, Calinski-Harabasz and BetaCV) in the same previous scenarios and the results highlight the effectiveness of clustering metrics in distinguishing the results of MT algorithms and ground truth. However, this implies expanding the analysis space by including a new set of metrics. Therefore, we propose consolidating the various metrics, which consider both the quality of the words that make up the topics and the organizational structure of the documents, into a unified result, using Multiattribute Utility Theory (MAUT). Our results demonstrated that this approach allowed us to classify more precisely the different Topic Modeling, showing that the semantic advances generated by the use of word embeddings present in some MT strategies, as well as the solidity and consistency in the construction of topics through matrix factorization strategies.

BANKING MEMBERS:
Presidente - 1674068 - LEONARDO CHAVES DUTRA DA ROCHA
Interno - 2325597 - DIEGO ROBERTO COLOMBO DIAS
Externa ao Programa - 1516364 - ELISA TULER DE ALBERGARIA
Externo à Instituição - FELIPE AUGUSTO RESENDE VIEGAS
Externo à Instituição - ADRIANO CÉSAR MACHADO PEREIRA - UFMG