Detecting and correcting Factual Error in LLM Text series

Miss Pratibha Tiwari

MTech Scholar, Computer Science and engineering RCET, Bhilai, Chhattisgarh, India

Anisha Soni

Department of, Computer Science RCET, Bhilai, Chhattisgarh, India

---------------------------------------------------------------------***-----------------------------------------------------------------

 Abstract

This thesis delves into advanced modelling techniques, employing Titan Text Lite for news article summarization, a method that integrates BERT in the Encoder and GPT-2 in the Decoder. Transformer based models enhance the summarization process, ensuring a comprehensive and contextually accurate output. The evaluation of the generated summary is the key focus in this thesis. Initially, baseline ROUGE scores are calculated, serving as a fundamental metric for summarization quality. However, recognizing the limitations of automated metrics, the evaluation process is enriched by leveraging a Large Language Model (LLM). The LLM is instructed to critically scrutinize the generated summaries, aiming to ascertain whether supporting evidence can be inferred from the original article. This approach not only validates the coherence of the summaries but also ensures alignment with the source news article. To further refine the evaluation, the LLM is guided through specialized instructions for chain-of-thought reasoning, with a specific emphasis on noun entities and dates. This strategic emphasis addresses nuanced aspects of content understanding, contributing to a more nuanced evaluation of summarization effectiveness. The evaluation process is then replicated at the sentence level, offering a granular perspective on the model's performance. The outcomes of the evaluation are recorded as binary "Yes" or "No" responses, providing a tangible measure of the LLM's accuracy in assessing the summaries. This meticulous and comprehensive approach seeks to unravel the intricacies of the model's capabilities in generating coherent, contextually.

 Key Words: Natural Language Processing (NLP), Large Language Models (LLM), Chain of Thought  (COT), Generative Pre-trained Transformers (GPT), Recall-Oriented Understudy