GoogleGemma3n Model : LoRA-Aug. Fine Tuning of a 4-bit Multimodal LLM Model on Scientific Literature

Описание: Academic papers-finetuning-inference:
LoRA-Augmented Fine-Tuning of a 4-bit Multimodal Language Model on Scientific Literature

Full Paper/Blog Link: https://handsonlabs.org/gemma-3n-4b-a...
Github Source : https://github.com/tobimichigan/Gemma...

ABSTRACT

We present a novel workflow for the parameter-efficient fine-tuning (PEFT) of a 4-bit quantized multimodal large language model (LLM), Gemma 3N-E4B, on a curated corpus of 20 state-of-the-art research papers in AI, climate science, healthcare, and computer vision. Leveraging LoRA adapters, we freeze the majority of the backbone weights and fine-tune only low-rank updates in attention and MLP modules, achieving substantial memory savings. We introduce a robust PDF download and parsing module using streaming requests and PyPDF2 to extract full-text content at scale. To address version conflicts in a heterogeneous dependency environment (Colab vs. local), we propose an ordered, pinned installation sequence that ensures reproducible environments. Our training regime—1 GPU, 4-bit weights, batch size 1 with gradient accumulation—completes 40 LoRA steps under early-stopping criteria, consuming under 8 GB of GPU memory.

Keywords
LoRA, 4-bit quantization, Gemma 3N, multimodal LLM, PDF parsing, PyPDF2, PEFT, early stopping, memory efficiency, reproducible dependencies