PaliGemma2 LaTeX OCR 微调实战公式图片识别与文本差异对比这篇教程是我根据 PaliGemma2 在 LaTeX OCR 任务上的微调复现过程整理出来的。重点演示如何下载公式图片 JSONL 数据集微调 PaliGemma2-10B并用可视化 diff、BLEU 和 TER 评估生成公式文本。LaTeX OCR 的输入是公式图片输出是 LaTeX 字符串。模型不仅要识别符号还要生成正确的结构和命令因此文本差异可视化和序列级指标很重要。本文会重点跑通以下流程下载 LaTeX OCR JSONL 数据集展示公式图片和目标 LaTeX 文本使用 QLoRA 微调 PaliGemma2-10B可视化标注文本和生成文本差异使用 BLEU 和 TER 评估 OCR 输出如果你正在系统学习多模态微调、目标检测、OCR 或图像分割建议收藏本文配套 notebook、示例图片和运行环境说明后续会继续整理。如果环境配置卡住可以在评论区说明具体报错。 文章目录PaliGemma2 LaTeX OCR 微调实战公式图片识别与文本差异对比⚙️ 环境准备 下载 LaTeX OCR 数据集 加载并预览 OCR 数据 加载 PaliGemma2-10B 模型️ 微调 LaTeX OCR 模型 推理并对比生成文本 评估 OCR 文本质量 小结 同系列教程汇总⚙️ 环境准备检查 GPU安装PEFT、bitsandbytes并安装支持该 notebook 的 transformers 分支。!nvidia-smi 下载 LaTeX OCR 数据集从数据集后台获取 unsloth-latex-ocr JSONL 数据集并查看训练、验证、测试集规模。!pip install-q supervision peft bitsandbytes!pip uninstall-y transformers!pip install-q githttps://github.com/probicheaux/transformers.gitmainfromtypesimportSimpleNamespace# 从数据集后台下载并解压数据集后修改 DATASET_DIR 指向数据集目录。DATASET_DIR/content/dataset# 修改为数据集后台导出的数据集目录datasetSimpleNamespace(locationDATASET_DIR)!head-n5{dataset.location}/train/annotations.jsonl!wc-l{dataset.location}/train/annotations.jsonl !wc-l{dataset.location}/valid/annotations.jsonl !wc-l{dataset.location}/test/annotations.jsonl# !head -n 10000 {dataset.location}/train/annotations.jsonl {dataset.location}/train/annotations.sample.jsonl# !head -n 1000 {dataset.location}/valid/annotations.jsonl {dataset.location}/valid/annotations.sample.jsonl# !head -n 1000 {dataset.location}/test/annotations.jsonl {dataset.location}/test/annotations.sample.jsonl 加载并预览 OCR 数据定义 JSONL 数据集类并用 HTML 表格展示公式图片和对应 prefix/suffix。importosimportjsonimportrandomfromPILimportImagefromtorch.utils.dataimportDatasetclassJSONLDataset(Dataset):def__init__(self,jsonl_file_path:str,image_directory_path:str):self.jsonl_file_pathjsonl_file_path self.image_directory_pathimage_directory_path self.entriesself._load_entries()def_load_entries(self):entries[]withopen(self.jsonl_file_path,r)asfile:forlineinfile:datajson.loads(line)entries.append(data)returnentriesdef__len__(self):returnlen(self.entries)def__getitem__(self,idx:int):ifidx0oridxlen(self.entries):raiseIndexError(Index out of range)entryself.entries[idx]image_pathos.path.join(self.image_directory_path,entry[image])imageImage.open(image_path)returnimage,entrytrain_datasetJSONLDataset(jsonl_file_pathf{dataset.location}/train/annotations.jsonl,image_directory_pathf{dataset.location}/train,)valid_datasetJSONLDataset(jsonl_file_pathf{dataset.location}/valid/annotations.jsonl,image_directory_pathf{dataset.location}/valid,)test_datasetJSONLDataset(jsonl_file_pathf{dataset.location}/test/annotations.jsonl,image_directory_pathf{dataset.location}/test,)fromIPython.core.displayimportdisplay,HTMLfromPILimportImageimportioimportbase64defpil_image_to_base64(img):Convert a PIL image to a base64 string.bufferedio.BytesIO()img.save(buffered,formatJPEG)img_strbase64.b64encode(buffered.getvalue()).decode(utf-8)returnfdata:image/jpeg;base64,{img_str}defdisplay_images_and_text(dataset,num_entries10): Display images and their corresponding text side by side in an HTML table. :param dataset: PyTorch dataset to extract images and texts from. :param num_entries: Number of entries to display. images[]texts[]foriinrange(min(num_entries,len(dataset))):img,datadataset[i]images.append(pil_image_to_base64(img))textfPrefix:{data[prefix]}brSuffix:{data[suffix]}texts.append(text)rows[]forimg,textinzip(images,texts):row_htmlf tr tdimg src{img} altImage stylemax-width:300px; max-height:300px; object-fit:cover;/td td{text}/td /tr rows.append(row_html)html_contentf html head style body {{ font-family: Arial, sans-serif; margin: 0; padding: 0; }} table {{ width: 100%; border-collapse: collapse; margin: 20px 0; }} th, td {{ border: 1px solid #ddd; padding: 8px; text-align: left; }} img {{ display: block; margin: auto; }} /style /head body table tr thImage/th thText/th /tr{.join(rows)}/table /body /html display(HTML(html_content))display_images_and_text(train_dataset,num_entries10) 加载 PaliGemma2-10B 模型加载 PaliGemma2-10B processor并通过 LoRA/QLoRA 控制训练显存。importtorchfromtransformersimportPaliGemmaProcessor,PaliGemmaForConditionalGeneration MODEL_IDgoogle/paligemma2-10b-pt-224DEVICEtorch.device(cudaiftorch.cuda.is_available()elsecpu)fromhuggingface_hubimportnotebook_login notebook_login()processorPaliGemmaProcessor.from_pretrained(MODEL_ID)USE_LORAFalseUSE_QLORATrueFREEZE_VISIONFalsefrompeftimportget_peft_model,LoraConfigfromtransformersimportBitsAndBytesConfigifUSE_LORAorUSE_QLORA:lora_configLoraConfig(r8,target_modules[q_proj,o_proj,k_proj,v_proj,gate_proj,up_proj,down_proj],task_typeCAUSAL_LM,)ifUSE_QLORA:bnb_configBitsAndBytesConfig(load_in_4bitTrue,bnb_4bit_quant_typenf4,bnb_4bit_compute_typetorch.bfloat16)modelPaliGemmaForConditionalGeneration.from_pretrained(MODEL_ID,device_mapauto,quantization_configbnb_configifUSE_QLORAelseNone,torch_dtypetorch.bfloat16)modelget_peft_model(model,lora_config)modelmodel.to(DEVICE)model.print_trainable_parameters()else:modelPaliGemmaForConditionalGeneration.from_pretrained(MODEL_ID,device_mapauto).to(DEVICE)modelmodel.to(DEVICE)ifFREEZE_VISION:forparaminmodel.vision_tower.parameters():param.requires_gradFalseforparaminmodel.multi_modal_projector.parameters():param.requires_gradFalseTORCH_DTYPEmodel.dtype️ 微调 LaTeX OCR 模型构造 OCR 任务 collate 函数使用 Transformers Trainer 训练模型。fromtransformersimportTrainer,TrainingArgumentsdefcollate_fn(batch):images,labelszip(*batch)paths[label[image]forlabelinlabels]prefixes[imagelabel[prefix]forlabelinlabels]suffixes[label[suffix]forlabelinlabels]inputsprocessor(textprefixes,imagesimages,return_tensorspt,suffixsuffixes,paddinglongest).to(TORCH_DTYPE).to(DEVICE)returninputs argsTrainingArguments(num_train_epochs3,remove_unused_columnsFalse,per_device_train_batch_size3,gradient_accumulation_steps12,warmup_steps2,learning_rate2e-5,weight_decay1e-6,adam_beta20.999,logging_steps100,optimpaged_adamw_8bitifUSE_QLORAelseadamw_hf,save_strategysteps,save_steps1000,save_total_limit1,output_dirpaligemma2_latex_ocr_v5,bf16True,report_to[tensorboard],dataloader_pin_memoryFalse)trainerTrainer(modelmodel,train_datasettrain_dataset,eval_datasetvalid_dataset,data_collatorcollate_fn,argsargs)trainer.train() 推理并对比生成文本对测试集样本生成 LaTeX并用左右对比 diff 标出差异。# title Function to render text diffsfromdifflibimportSequenceMatcherfromIPython.core.displayimportdisplay,HTMLdefside_by_side_diff_divs(text1,text2):lines1text1.splitlines()lines2text2.splitlines()original_output[]modified_output[]forline1,line2inzip(lines1,lines2):words1line1.split()words2line2.split()matcherSequenceMatcher(None,words1,words2)original_line[]modified_line[]fortag,i1,i2,j1,j2inmatcher.get_opcodes():iftagreplace:original_line.append(fspan classdiff-remove{ .join(words1[i1:i2])}/span)modified_line.append(fspan classdiff-add{ .join(words2[j1:j2])}/span)eliftagdelete:original_line.append(fspan classdiff-remove{ .join(words1[i1:i2])}/span)eliftaginsert:modified_line.append(fspan classdiff-add{ .join(words2[j1:j2])}/span)eliftagequal:original_line.append( .join(words1[i1:i2]))modified_line.append( .join(words2[j1:j2]))original_output.append( .join(original_line)br)modified_output.append( .join(modified_line)br)original_htmlbr.join(original_output)brmodified_htmlbr.join(modified_output)brhtmlf html head style body {{ font-family: Arial, sans-serif; margin: 0; padding: 0; }} .container {{ display: flex; align-items: flex-start; }} .column {{ flex: 1; padding: 10px; white-space: pre-wrap; text-align: left; }} .diff-remove {{ background-color: #d9534f; /* Dark red */ color: white; text-decoration: line-through; border-radius: 4px; padding: 2px 4px; }} .diff-add {{ background-color: #5cb85c; /* Dark green */ color: white; border-radius: 4px; padding: 2px 4px; }} /style /head body div classcontainer div classcolumn styleborder-right: 1px solid #ccc;{original_html}/div div classcolumn{modified_html}/div /div /body /html returnhtml# title Suffix vs. generated textforiinrange(10):image,labeltest_dataset[i]prefiximagelabel[prefix]suffixlabel[suffix]inputsprocessor(textprefix,imagesimage,return_tensorspt).to(TORCH_DTYPE).to(DEVICE)prefix_lengthinputs[input_ids].shape[-1]withtorch.inference_mode():generationmodel.generate(**inputs,max_new_tokens256,do_sampleFalse,num_beams3)generationgeneration[0][prefix_length:]generated_textprocessor.decode(generation,skip_special_tokensTrue)html_diffside_by_side_diff_divs(suffix,generated_text)display(image)display(HTML(html_diff)) 评估 OCR 文本质量收集测试集预测结果使用 BLEU 和 TER 评估生成文本质量。importnumpyasnp targets[]predictions[]withtorch.inference_mode():foriinrange(10):image,labeltest_dataset[i]prefiximagelabel[prefix]suffixlabel[suffix]inputsprocessor(textprefix,imagesimage,return_tensorspt).to(TORCH_DTYPE).to(DEVICE)prefix_lengthinputs[input_ids].shape[-1]generationmodel.generate(**inputs,max_new_tokens256,do_sampleFalse,num_beams3)generationgeneration[0][prefix_length:]generated_textprocessor.decode(generation,skip_special_tokensTrue)targets.append(suffix)predictions.append(generated_text)!pip install-q evaluate# title Calculate BLEUfromevaluateimportload bleuload(bleu)resultsbleu.compute(predictionspredictions,referencestargets)print(results)!pip install-q sacrebleu# title Calculate TERfromevaluateimportload terload(ter)resultster.compute(predictionspredictions,referencestargets,case_sensitiveTrue)print(results) 小结这篇教程完整整理了Fine-Tune PaliGemma2 for LaTeX OCR的核心复现流程。实际操作时建议先确认 GPU、依赖版本、数据集路径和模型权限再逐段运行 notebook。下载 LaTeX OCR JSONL 数据集展示公式图片和目标 LaTeX 文本使用 QLoRA 微调 PaliGemma2-10B可视化标注文本和生成文本差异使用 BLEU 和 TER 评估 OCR 输出后续我会继续按源项目顺序整理 项目教程 中的目标检测、实例分割、OCR、多目标跟踪和视觉大模型教程。 同系列教程汇总Google Gemini 3.5 Flash 零样本目标检测教程从提示词到可视化结果GLM-OCR 文档识别实战教程从验证码、公式到车牌 OCRRF-DETR ByteTrack 多目标跟踪实战教程从命令行到 Python 视频轨迹可视化SAM 3 图像分割实战教程文本、框和点提示的多种分割方式PaliGemma2 LaTeX OCR 微调实战公式图片识别与文本差异对比-本文