Training Dynamics of Parametric and In-Context Knowledge Utilization in Language Models

Описание: Paper: https://arxiv.org/abs/2510.02370

Title: Training Dynamics of Parametric and In-Context Knowledge Utilization in Language Models

Authors: Minsung Kim, Dong-Kyum Kim, Jea Kwon, Nakyeong Yang, Kyomin Jung, Meeyoung Cha

Abstract: Large language models often encounter conflicts between in-context knowledge retrieved at inference time and parametric knowledge acquired during pretraining. Models that accept external knowledge uncritically are vulnerable to misinformation, whereas models that adhere rigidly to parametric knowledge fail to benefit from retrieval. Despite the widespread adoption of retrieval-augmented generation, we still lack a systematic understanding of what shapes knowledge-arbitration strategies during training. This gap risks producing pretrained models with undesirable arbitration behaviors and, consequently, wasting substantial computational resources after the pretraining budget has already been spent. To address this problem, we present the first controlled study of how training conditions influence models' use of in-context and parametric knowledge, and how they arbitrate between them. We train transformer-based language models on a synthetic biographies corpus while systematically controlling various conditions. Our experiments reveal that intra-document repetition of facts fosters the development of both parametric and in-context capabilities. Moreover, training on a corpus that contains inconsistent information or distributional skew encourages models to develop robust strategies for leveraging parametric and in-context knowledge. Rather than viewing these non-ideal properties as artifacts to remove, our results indicate that they are important for learning robust arbitration. These insights offer concrete, empirical guidance for pretraining models that harmoniously integrate parametric and in-context knowledge.

Tags: Machine Learning, Natural Language Processing, Research, gan, transformer, unsupervised, supervised, few-shot, gru, attention, search, training, dynamics, parametric, context, knowledge, research paper, academic, study, analysis, tutorial, explained, breakdown, paper review, research summary, AI research, scientific paper, methodology, results, findings, innovation, technology, computing, algorithm, model, dataset, evaluation, performance, accuracy, efficiency

Welcome to the Mayuresh Shilotri's Youtube . Maintained by Mayuresh Shilotri



You can follow me at

Blog - https://shilotri.com/

LinkedIn -   / mayureshshilotri  

Twitter -   / mshilotri  



Note: I only claim to have read the research paper and created a Video using AI tool. I am not the author. All intellectual heavy lifting was performed by the respective authors. 🙏