影刀RPA 文件编码问题：UTF-8、GBK、BOM

影刀RPA 文件编码问题UTF-8、GBK、BOM署名林焱什么情况用什么RPA流程中读写文件时最头疼的就是编码问题——采集的网页是UTF-8保存到CSV后Excel打开乱码读取同事用Windows记事本保存的文件报UnicodeDecodeError处理旧系统导出的GBK编码数据。编码使用场景特点UTF-8通用、网页、API国际标准变长编码UTF-8 with BOMWindows兼容带BOM头Excel可正确识别GBK/GB2312Windows中文系统中文专用不兼容国际latin-1兜底不会报错但可能乱码怎么做一读写文件指定编码# 写入文件withopen(rD:\data\output.txt,w,encodingutf-8)asf:f.write(中文内容)# 读取文件withopen(rD:\data\input.txt,r,encodingutf-8)asf:contentf.read()# CSV文件用utf-8-sig带BOMExcel兼容importcsvwithopen(rD:\data\data.csv,w,encodingutf-8-sig,newline)asf:writercsv.writer(f)writer.writerow([姓名,年龄])二自动检测文件编码拼多多店群自动化报活动上架defdetect_encoding(file_path,sample_size1024):尝试检测文件编码encodings[utf-8-sig,utf-8,gbk,gb2312,gb18030,big5,latin-1]withopen(file_path,rb)asf:rawf.read(sample_size)forencinencodings:try:raw.decode(enc)returnencexcept(UnicodeDecodeError,LookupError):continuereturnutf-8# 默认回退# 使用encodingdetect_encoding(rD:\data\unknown.txt)print(f检测到编码:{encoding})withopen(rD:\data\unknown.txt,r,encodingencoding)asf:contentf.read()三编码转换defconvert_encoding(input_path,output_path,from_encNone,to_encutf-8):转换文件编码iffrom_encisNone:from_encdetect_encoding(input_path)withopen(input_path,r,encodingfrom_enc)asf:contentf.read()withopen(output_path,w,encodingto_enc)asf:f.write(content)print(f转换完成:{from_enc}→{to_enc})# GBK转UTF-8convert_encoding(rD:\data\gbk文件.txt,rD:\data\utf8文件.txt,from_encgbk,to_encutf-8)四处理BOM头# 检查文件是否有BOMdefhas_bom(file_path):withopen(file_path,rb)asf:returnf.read(3)b\xef\xbb\xbf# 去掉BOM头defremove_bom(file_path):withopen(file_path,rb)asf:contentf.read()ifcontent[:3]b\xef\xbb\xbf:contentcontent[3:]withopen(file_path,wb)asf:f.write(content)# 读取时自动处理BOM# 用utf-8-sig读取会自动去掉BOMwithopen(file.txt,r,encodingutf-8-sig)asf:contentf.read()# content不会有BOM字符完整流程编码兼容读写工具importosclassEncodingSafeFile:编码安全的文件读写工具# 常见编码优先级ENCODINGS[utf-8-sig,utf-8,gbk,gb18030,latin-1]classmethoddefread(cls,file_path,encodingNone):安全读取文件自动检测编码ifencoding:withopen(file_path,r,encodingencoding)asf:returnf.read()# 自动检测forencincls.ENCODINGS:try:withopen(file_path,r,encodingenc)asf:returnf.read()exceptUnicodeDecodeError:continue# 兜底用latin-1不会报错但中文会乱码withopen(file_path,r,encodinglatin-1)asf:returnf.read()classmethoddefwrite(cls,file_path,content,encodingutf-8-sig):安全写入文件os.makedirs(os.path.dirname(file_path)ifos.path.dirname(file_path)else.,exist_okTrue)withopen(file_path,w,encodingencoding)asf:f.write(content)classmethoddefread_lines(cls,file_path,encodingNone):逐行读取ifencoding:encencodingelse:# 先检测编码foreincls.ENCODINGS:try:withopen(file_path,r,encodinge)asf:f.read(100)encebreakexceptUnicodeDecodeError:continueelse:enclatin-1withopen(file_path,r,encodingenc)asf:forlineinf:yieldline.rstrip(\n\r)# 使用contentEncodingSafeFile.read(rD:\data\未知编码.txt)print(f读取成功:{len(content)}字符)EncodingSafeFile.write(rD:\data\输出.txt,测试中文内容)有什么坑坑一UTF-8文件在Excel中乱码现象Python用UTF-8编码写的CSV/TXT文件用Excel打开后中文全是乱码。原因Excel在Windows中文系统下默认用GBK编码打开文本文件UTF-8文件没有BOM头时Excel无法识别。解决写入时用utf-8-sig带BOM的UTF-8# CSV文件importcsvwithopen(data.csv,w,encodingutf-8-sig,newline)asf:writercsv.writer(f)writer.writerow([姓名,年龄,部门])# 普通文本withopen(data.txt,w,encodingutf-8-sig)asf:f.write(中文内容)# UTF-8-sig和UTF-8的区别# UTF-8: 不带BOM头文件开头直接是内容# UTF-8-sig: 带BOM头3字节EF BB BFExcel能正确识别编码坑二读取GBK文件报UnicodeDecodeError现象读取Windows记事本保存的文件报UnicodeDecodeError: utf-8 codec cant decode byte...。原因Windows记事本默认用ANSI编码中文系统下是GBK用UTF-8读取会失败。解决指定GBK编码或自动检测TEMU店群矩阵自动化运营核价报活动# 方案1直接用GBKwithopen(file.txt,r,encodinggbk)asf:contentf.read()# 方案2用errors参数忽略错误可能丢失字符withopen(file.txt,r,encodingutf-8,errorsignore)asf:contentf.read()# 方案3用errorsreplace替换无法解码的字符withopen(file.txt,r,encodingutf-8,errorsreplace)asf:contentf.read()# 方案4自动检测推荐defsafe_read(file_path):forencin[utf-8-sig,utf-8,gbk,gb18030,latin-1]:try:withopen(file_path,r,encodingenc)asf:returnf.read()exceptUnicodeDecodeError:continuereturncontentsafe_read(file.txt)坑三BOM头导致字符串开头多出不可见字符现象读取UTF-8 with BOM文件后字符串开头多了\ufeff字符导致字符串比较失败。原因utf-8编码读取带BOM的文件时BOM不会被自动去掉变成\ufeff字符。解决用utf-8-sig编码读取或手动去掉# 正确用utf-8-sig读取自动去BOMwithopen(file.txt,r,encodingutf-8-sig)asf:contentf.read()# content开头没有\ufeff# 手动去掉contentcontent.lstrip(\ufeff)# 检查是否有BOM字符ifcontent.startswith(\ufeff):contentcontent[1:]# 常见错误用utf-8读取BOM文件# with open(file.txt, r, encodingutf-8) as f: ❌ content开头有\ufeff# content f.read()# if content hello: ❌ 永远不等因为实际是\ufeffhello坑四混合编码文件读取失败现象一个文件大部分是UTF-8编码但中间有少部分GBK字符导致读取中途报错。原因文件编码不统一可能是从不同来源拼接的。解决用errors参数或逐段处理# 方案1忽略错误字符withopen(file.txt,r,encodingutf-8,errorsignore)asf:contentf.read()# 丢弃无法解码的字节# 方案2替换错误字符withopen(file.txt,r,encodingutf-8,errorsreplace)asf:contentf.read()# 无法解码的字节替换为# 方案3逐字节处理混合编码解码defread_mixed_encoding(file_path):result[]withopen(file_path,rb)asf:dataf.read()i0whileilen(data):# 尝试UTF-8解码try:chardata[i:i4].decode(utf-8)result.append(char)ilen(char.encode(utf-8))continueexceptUnicodeDecodeError:pass# 回退到GBKtry:chardata[i:i2].decode(gbk)result.append(char)i2continueexceptUnicodeDecodeError:pass# 跳过无法解码的字节i1return.join(result)