每次在进行写文案、分析文档操作的时候, 我总会产生好奇, 这些被称作大语言模型的LLM究竟是怎样去“思考”的呢? 我们能够看到的仅仅是输入以及输出, 而处于中间位置的关于权重的相关情况与神经元激活的状态, 就如同一个黑箱一般。一直到见到了相关教程以后, 才发觉, 普通的人依靠自己个人的电脑就能够对它进行拆解, 不需要借助服务器, 不会花费云费用, 只要顺着给出的步骤, 便能够看到大语言模型LLM是怎样借助神经元去编码情感、语义以及类比的。将关于我的实操过程予以分享, 此过程涵盖从搭建环境起始, 一直到达可视化模型的内部情形, 引领你去揭开语言模型的神秘面纱。只要电脑拥有数量达到8GB的内存数值若能有16GB之数值则更佳, 再存有版本在3.10以上情形的版本, 借助轻量型的模型便能够轻松达成运行且能畅通无阻, 即使是新手之人也可顺利上手操作此事。准备工作搭建“专属沙盒”虚拟环境未开始之前, 需安装四个核心库去用: torch, 其用于跑模型再拿用于加载模型工具的库还有用于画图的库以及 -learn, 它实现于数据降维。然而, 先别着急去安装, 务必得先去构建虚拟环境, 我先前直接进行安装着, 遇到过不少的坑: 不同项目当中需要有着不同版本的库, 一旦升级便会搞垮旧的项目全局安装过多没有用处的库, 环境终将表现得很乱。虚拟环境如同“专属沙盒”, 它仅仅服务于这个LLM项目, 库全部安装在项目文件夹当中, 使用完毕后删除文件夹并不会对电脑造成影响。其具体操作是非常简单的:建立项目文件夹: 于终端之处命令提示符、Mac终端键入“mkdir ”, 随后输入“cd ”以此进入到文件夹当中去了。创建虚拟环境: 输入“ -m venv venv”, 如此便会在其中生成venv子文件夹了。激活环境: 在对应的地方输入“venv\\”, Mac则要输入“ venv/bin/”, 当激活之后于终端前面便会出现“(venv)”这样的标识了。安装依赖: 输入“pip torch -learn”, 仅仅几分钟的时间便能够安装完成了。我们挑选模型名-base-, 是由于它体积小大约250MB, 普通笔记本即可运行要是电脑配置较高, 也能够更换为LLaMA 2, 不过新手应先从此处着手。第一步加载模型与“翻译官”等环境搭建完成之后, 去建好一个名为 app.py 的文件, 接着去书写加载模型以及相关的代码。有关代码的长度并不长, 然而在最为关键的地方需要予以留意:from transformers import AutoTokenizer, AutoModel import torch model_name distilbert-base-uncased tokenizer AutoTokenizer.from_pretrained(model_name) model AutoModel.from_pretrained(model_name, output_hidden_statesTrue)头一回运行“ app.py”, 会自行从 Face开展下载大概250MB, 往后无需再度进行下载。第二步提取隐藏状态窥见模型“思考”加载完毕那个好模型之后, 便能够去提取它的“思考过程”, 也就是隐藏状态了。在app.py当中添加一个函数:def get_hidden_states(text): inputs tokenizer(text, return_tensorspt) with torch.no_grad(): outputs model(**inputs) # 取最后一个隐藏层第一个样本的激活值 hidden outputs.hidden_states[-1][0] # 把token ID转成可看懂的词 tokens tokenizer.convert_ids_to_tokens(inputs[input_ids][0]) return tokens, hidden # 测试 tokens, hidden get_hidden_states(I love pizza!) print(tokens) # 输出[[CLS], i, love, pizza, !, [SEP]] print(hidden.shape) # 输出torch.Size([6, 768])第三步可视化情感激活区分“喜欢”与“讨厌”将隐藏状态提取出来, 便能够进行可视化操作了。率先查看模型究竟如何区分正负情感, 举例来说, 通过对比呈现出来的正面影评“I love this movie, it is!”以及负面影评“I hate this move, it is .”。在app.py里加画图函数import matplotlib.pyplot as plt def plot_token_activations(tokens, hidden, title, filename): plt.figure(figsize(12, 4)) for i, token in enumerate(tokens): plt.plot(hidden[i].numpy(), labeltoken) plt.title(title) plt.xlabel(Neuron Index) plt.ylabel(Activation) plt.legend(locupper right, fontsizex-small) plt.tight_layout() plt.savefig(filename) plt.close() # 生成正负情感图 tokens_pos, hidden_pos get_hidden_states(I love this movie, it is fantastic!) plot_token_activations(tokens_pos, hidden_pos, Positive Sentiment Example, positive_sentiment.png) tokens_neg, hidden_neg get_hidden_states(I hate this movie, it is terrible.) plot_token_activations(tokens_neg, hidden_neg, Negative Sentiment Example, negative_sentiment.png)运行代码之后, 文件夹当中会出现两张图片。在正面的那张图片里面, “love”的激活曲线以及其他词语的激活曲线差异显著突出在负面的那张图片里面, “hate”的曲线存在着独特的模式, 这表明LLM并非依靠“记词”来判断情感, 而是借助神经元激活模式对情感进行编码, 有些神经元对于正面词汇敏感, 有些神经元对于负面词汇敏感, 这是模型从训练数据当中学习而来的关联。第四步比较句子激活看关键词影响整体语义再去试着比较一下较为相似的句子: “I love.”, 还有“I hate.”, 仅仅只差了一个单词, 这般情况下模型对于整体语义的理解会存在怎样的不同呢?加个比较函数def compare_sentences(s1, s2, filename): tokens1, hidden1 get_hidden_states(s1) tokens2, hidden2 get_hidden_states(s2) # 计算句子平均激活值整合所有token的信息 plt.figure(figsize(10,5)) plt.plot(hidden1.mean(dim0).numpy(), labels1[:30]...) plt.plot(hidden2.mean(dim0).numpy(), labels2[:30]...) plt.title(Sentence Activation Comparison) plt.xlabel(Neuron Index) plt.ylabel(Mean Activation) plt.legend() plt.tight_layout() plt.savefig(filename) plt.close() # 生成对比图 compare_sentences(I love coding., I hate coding., sentence_comparison.png)在生成的图当中, 两条曲线之间的差异十分显著, 哪怕仅仅更换一个词语, 整个句子的平均激活模式都会发生改变。这表明LLM对句子的理解属于“整体式”的, 关键词的激活会对全局产生影响, 致使模型能够区分“喜欢编程”与“讨厌编程”的差异。第五步用PCA看类比解码语义关系最后去探索经典的类比“男人对应女人 :∶ 国王对应女王”这种情况, 模型是不是能够理解这样的语义关系呢, 运用主成分分析也就是PCA把768维的词嵌入转化为降到2维, 如此便能够在平面上看到相应关系。加两个函数from sklearn.decomposition import PCA def get_sentence_embedding(text): inputs tokenizer(text, return_tensorspt) with torch.no_grad(): outputs model(** inputs) # 取最后一层平均激活值作为词嵌入 return outputs.last_hidden_state.mean(dim1).squeeze() def plot_embeddings(words, embeddings, filename): pca PCA(n_components2) reduced pca.fit_transform(torch.stack(embeddings).numpy()) plt.figure(figsize(8, 6)) for i, word in enumerate(words): x, y reduced[i] plt.scatter(x, y, s100) plt.text(x0.02, y0.02, word, fontsize12) plt.title(Word Embeddings in 2D (PCA)) plt.xlabel(PC1) plt.ylabel(PC2) plt.grid(True) plt.tight_layout() plt.savefig(filename) plt.close() # 测试类比 words [man, woman, king, queen] embeddings [get_sentence_embedding(w) for w in words] plot_embeddings(words, embeddings, word_analogies.png)在生成的图当中, man与woman、king和queen之间的距离是相近的, 并且man到woman的向量跟king到queen的向量近乎是平行的, 这恰恰就是类比的一种体现模型将“性别差异”这种语义关系, 编码成为了高维空间里的向量关系, 所以才能够完成类比推理结语LLM不是魔法是可探索的数学系统仅仅全程借助普通笔记本便能够实现运行畅通, 尽管其规模较小, 然而却清晰地呈现出了LLM的核心逻辑, 那便是不像依靠魔法去理解语言, 需运用高维激活向量对语义、情感以及关系进行编码。像LLaMA 2、GPT-4这类更大的模型, 仅仅是神经元数量更多、隐藏层更为深厚, 但其原理是相通的。你还能够进行延伸实验, 将不同主题的文本输入, 以便查看激活差异, 又或者检测面对带偏见文本时模型所做出的反应, 该种探索, 并非仅仅是学习技术, 而是在于领悟AI进行 “思考” 的这一过程, 一旦我们能够看懂其内部激活的情况, 那么便不再是被动地运用AI, 而是能够主动地对其展开探索且予以优化, 这恰恰就是拆解LLM所具备的最大意义所在。Www.xiaoqiyx.com/Article/details/1595132.sHtMLWww.xiaoqiyx.com/Article/details/8932278.sHtMLWww.xiaoqiyx.com/Article/details/9323533.sHtMLWww.xiaoqiyx.com/Article/details/3861116.sHtMLWww.xiaoqiyx.com/Article/details/6125881.sHtMLWww.xiaoqiyx.com/Article/details/3742692.sHtMLWww.xiaoqiyx.com/Article/details/8751685.sHtMLWww.xiaoqiyx.com/Article/details/7050662.sHtMLWww.xiaoqiyx.com/Article/details/5563883.sHtMLWww.xiaoqiyx.com/Article/details/6527562.sHtMLWww.xiaoqiyx.com/Article/details/8360171.sHtMLWww.xiaoqiyx.com/Article/details/6313531.sHtMLWww.xiaoqiyx.com/Article/details/7764706.sHtMLWww.xiaoqiyx.com/Article/details/1706186.sHtMLWww.xiaoqiyx.com/Article/details/9054940.sHtMLWww.xiaoqiyx.com/Article/details/3661315.sHtMLWww.xiaoqiyx.com/Article/details/1907780.sHtMLWww.xiaoqiyx.com/Article/details/2252373.sHtMLWww.xiaoqiyx.com/Article/details/9765438.sHtMLWww.xiaoqiyx.com/Article/details/1397717.sHtMLWww.xiaoqiyx.com/Article/details/8777707.sHtMLWww.xiaoqiyx.com/Article/details/4402693.sHtMLWww.xiaoqiyx.com/Article/details/5293727.sHtMLWww.xiaoqiyx.com/Article/details/4229538.sHtMLWww.xiaoqiyx.com/Article/details/8052601.sHtMLWww.xiaoqiyx.com/Article/details/4309686.sHtMLWww.xiaoqiyx.com/Article/details/4384511.sHtMLWww.xiaoqiyx.com/Article/details/8263860.sHtMLWww.xiaoqiyx.com/Article/details/3491026.sHtMLWww.xiaoqiyx.com/Article/details/8601660.sHtMLWww.xiaoqiyx.com/Article/details/9221974.sHtMLWww.xiaoqiyx.com/Article/details/9023775.sHtMLWww.xiaoqiyx.com/Article/details/2037613.sHtMLWww.xiaoqiyx.com/Article/details/3054005.sHtMLWww.xiaoqiyx.com/Article/details/2516412.sHtMLWww.xiaoqiyx.com/Article/details/1084357.sHtMLWww.xiaoqiyx.com/Article/details/8667723.sHtMLWww.xiaoqiyx.com/Article/details/7385048.sHtMLWww.xiaoqiyx.com/Article/details/7971150.sHtMLWww.xiaoqiyx.com/Article/details/8257758.sHtMLWww.xiaoqiyx.com/Article/details/4197596.sHtMLWww.xiaoqiyx.com/Article/details/5315301.sHtMLWww.xiaoqiyx.com/Article/details/6711929.sHtMLWww.xiaoqiyx.com/Article/details/2756388.sHtMLWww.xiaoqiyx.com/Article/details/5654021.sHtMLWww.xiaoqiyx.com/Article/details/1912973.sHtMLWww.xiaoqiyx.com/Article/details/1141734.sHtMLWww.xiaoqiyx.com/Article/details/7105322.sHtMLWww.xiaoqiyx.com/Article/details/6570011.sHtMLWww.xiaoqiyx.com/Article/details/2865677.sHtMLWww.xiaoqiyx.com/Article/details/9784605.sHtMLWww.xiaoqiyx.com/Article/details/9485087.sHtMLWww.xiaoqiyx.com/Article/details/4843927.sHtMLWww.xiaoqiyx.com/Article/details/9268325.sHtMLWww.xiaoqiyx.com/Article/details/7173132.sHtMLWww.xiaoqiyx.com/Article/details/7971774.sHtMLWww.xiaoqiyx.com/Article/details/4388985.sHtMLWww.xiaoqiyx.com/Article/details/3519918.sHtMLWww.xiaoqiyx.com/Article/details/9628878.sHtMLWww.xiaoqiyx.com/Article/details/2931355.sHtMLWww.xiaoqiyx.com/Article/details/3194499.sHtMLWww.xiaoqiyx.com/Article/details/5256857.sHtMLWww.xiaoqiyx.com/Article/details/1680696.sHtMLWww.xiaoqiyx.com/Article/details/4043927.sHtMLWww.xiaoqiyx.com/Article/details/1722084.sHtMLWww.xiaoqiyx.com/Article/details/2621868.sHtMLWww.xiaoqiyx.com/Article/details/4572782.sHtMLWww.xiaoqiyx.com/Article/details/9544856.sHtMLWww.xiaoqiyx.com/Article/details/2421397.sHtMLWww.xiaoqiyx.com/Article/details/9874661.sHtMLWww.xiaoqiyx.com/Article/details/1995041.sHtMLWww.xiaoqiyx.com/Article/details/4733302.sHtMLWww.xiaoqiyx.com/Article/details/6096577.sHtMLWww.xiaoqiyx.com/Article/details/1671090.sHtMLWww.xiaoqiyx.com/Article/details/7230784.sHtMLWww.xiaoqiyx.com/Article/details/5718262.sHtMLWww.xiaoqiyx.com/Article/details/6138500.sHtMLWww.xiaoqiyx.com/Article/details/5597696.sHtMLWww.xiaoqiyx.com/Article/details/5193941.sHtMLWww.xiaoqiyx.com/Article/details/5103876.sHtMLWww.xiaoqiyx.com/Article/details/5268645.sHtMLWww.xiaoqiyx.com/Article/details/8913954.sHtMLWww.xiaoqiyx.com/Article/details/6407430.sHtMLWww.xiaoqiyx.com/Article/details/9490742.sHtMLWww.xiaoqiyx.com/Article/details/9815720.sHtMLWww.xiaoqiyx.com/Article/details/8662931.sHtMLWww.xiaoqiyx.com/Article/details/6499365.sHtMLWww.xiaoqiyx.com/Article/details/6642104.sHtMLWww.xiaoqiyx.com/Article/details/4020760.sHtMLWww.xiaoqiyx.com/Article/details/9201329.sHtMLWww.xiaoqiyx.com/Article/details/5879245.sHtMLWww.xiaoqiyx.com/Article/details/7290444.sHtMLWww.xiaoqiyx.com/Article/details/3281311.sHtMLWww.xiaoqiyx.com/Article/details/8578597.sHtMLWww.xiaoqiyx.com/Article/details/7511467.sHtMLWww.xiaoqiyx.com/Article/details/3875208.sHtMLWww.xiaoqiyx.com/Article/details/2597913.sHtMLWww.xiaoqiyx.com/Article/details/5390842.sHtMLWww.xiaoqiyx.com/Article/details/9949134.sHtMLWww.xiaoqiyx.com/Article/details/6155013.sHtMLWww.xiaoqiyx.com/Article/details/2657994.sHtMLWww.xiaoqiyx.com/Article/details/2614980.sHtMLWww.xiaoqiyx.com/Article/details/8827343.sHtMLWww.xiaoqiyx.com/Article/details/7252443.sHtMLWww.xiaoqiyx.com/Article/details/5556979.sHtMLWww.xiaoqiyx.com/Article/details/2170487.sHtMLWww.xiaoqiyx.com/Article/details/8827442.sHtMLWww.xiaoqiyx.com/Article/details/9650949.sHtMLWww.xiaoqiyx.com/Article/details/6111355.sHtMLWww.xiaoqiyx.com/Article/details/3869639.sHtMLWww.xiaoqiyx.com/Article/details/3336387.sHtMLWww.xiaoqiyx.com/Article/details/3153139.sHtMLWww.xiaoqiyx.com/Article/details/3476698.sHtMLWww.xiaoqiyx.com/Article/details/2503217.sHtMLWww.xiaoqiyx.com/Article/details/5079811.sHtMLWww.xiaoqiyx.com/Article/details/1038306.sHtMLWww.xiaoqiyx.com/Article/details/7297836.sHtMLWww.xiaoqiyx.com/Article/details/3357665.sHtMLWww.xiaoqiyx.com/Article/details/6898449.sHtMLWww.xiaoqiyx.com/Article/details/7906737.sHtMLWww.xiaoqiyx.com/Article/details/4322445.sHtMLWww.xiaoqiyx.com/Article/details/8724849.sHtMLWww.xiaoqiyx.com/Article/details/7563191.sHtMLWww.xiaoqiyx.com/Article/details/4804338.sHtMLWww.xiaoqiyx.com/Article/details/8381573.sHtMLWww.xiaoqiyx.com/Article/details/1776549.sHtMLWww.xiaoqiyx.com/Article/details/1774192.sHtMLWww.xiaoqiyx.com/Article/details/7701733.sHtMLWww.xiaoqiyx.com/Article/details/5408898.sHtMLWww.xiaoqiyx.com/Article/details/9265711.sHtMLWww.xiaoqiyx.com/Article/details/4645568.sHtMLWww.xiaoqiyx.com/Article/details/5040821.sHtMLWww.xiaoqiyx.com/Article/details/3817673.sHtMLWww.xiaoqiyx.com/Article/details/6908131.sHtMLWww.xiaoqiyx.com/Article/details/9511519.sHtMLWww.xiaoqiyx.com/Article/details/8707107.sHtMLWww.xiaoqiyx.com/Article/details/3444841.sHtMLWww.xiaoqiyx.com/Article/details/6361912.sHtMLWww.xiaoqiyx.com/Article/details/7714908.sHtMLWww.xiaoqiyx.com/Article/details/7235336.sHtMLWww.xiaoqiyx.com/Article/details/7691042.sHtMLWww.xiaoqiyx.com/Article/details/2847877.sHtMLWww.xiaoqiyx.com/Article/details/8699498.sHtMLWww.xiaoqiyx.com/Article/details/6029193.sHtMLWww.xiaoqiyx.com/Article/details/6247922.sHtMLWww.xiaoqiyx.com/Article/details/7172155.sHtMLWww.xiaoqiyx.com/Article/details/1755309.sHtMLWww.xiaoqiyx.com/Article/details/4192360.sHtMLWww.xiaoqiyx.com/Article/details/6830322.sHtMLWww.xiaoqiyx.com/Article/details/6170494.sHtMLWww.xiaoqiyx.com/Article/details/9388029.sHtMLWww.xiaoqiyx.com/Article/details/3840210.sHtMLWww.xiaoqiyx.com/Article/details/4033578.sHtMLWww.xiaoqiyx.com/Article/details/9596943.sHtMLWww.xiaoqiyx.com/Article/details/2916493.sHtMLWww.xiaoqiyx.com/Article/details/5591731.sHtMLWww.xiaoqiyx.com/Article/details/9835909.sHtMLWww.xiaoqiyx.com/Article/details/4429947.sHtMLWww.xiaoqiyx.com/Article/details/2816862.sHtMLWww.xiaoqiyx.com/Article/details/1838045.sHtMLWww.xiaoqiyx.com/Article/details/4597521.sHtMLWww.xiaoqiyx.com/Article/details/8287118.sHtMLWww.xiaoqiyx.com/Article/details/2860785.sHtMLWww.xiaoqiyx.com/Article/details/9923598.sHtMLWww.xiaoqiyx.com/Article/details/6970899.sHtMLWww.xiaoqiyx.com/Article/details/7479504.sHtMLWww.xiaoqiyx.com/Article/details/5504697.sHtMLWww.xiaoqiyx.com/Article/details/8660232.sHtMLWww.xiaoqiyx.com/Article/details/9052538.sHtMLWww.xiaoqiyx.com/Article/details/3629395.sHtMLWww.xiaoqiyx.com/Article/details/8886320.sHtMLWww.xiaoqiyx.com/Article/details/3010779.sHtMLWww.xiaoqiyx.com/Article/details/7832260.sHtMLWww.xiaoqiyx.com/Article/details/1142615.sHtMLWww.xiaoqiyx.com/Article/details/8958848.sHtMLWww.xiaoqiyx.com/Article/details/1986690.sHtMLWww.xiaoqiyx.com/Article/details/2175840.sHtMLWww.xiaoqiyx.com/Article/details/4073949.sHtMLWww.xiaoqiyx.com/Article/details/2415771.sHtMLWww.xiaoqiyx.com/Article/details/5401714.sHtMLWww.xiaoqiyx.com/Article/details/3381634.sHtMLWww.xiaoqiyx.com/Article/details/3307448.sHtMLWww.xiaoqiyx.com/Article/details/9044267.sHtMLWww.xiaoqiyx.com/Article/details/7900845.sHtMLWww.xiaoqiyx.com/Article/details/8227992.sHtMLWww.xiaoqiyx.com/Article/details/2179628.sHtMLWww.xiaoqiyx.com/Article/details/4835211.sHtMLWww.xiaoqiyx.com/Article/details/6601656.sHtMLWww.xiaoqiyx.com/Article/details/6604531.sHtMLWww.xiaoqiyx.com/Article/details/4324524.sHtMLWww.xiaoqiyx.com/Article/details/8699774.sHtMLWww.xiaoqiyx.com/Article/details/3807313.sHtMLWww.xiaoqiyx.com/Article/details/1123244.sHtMLWww.xiaoqiyx.com/Article/details/5208687.sHtMLWww.xiaoqiyx.com/Article/details/3299243.sHtMLWww.xiaoqiyx.com/Article/details/4955167.sHtMLWww.xiaoqiyx.com/Article/details/5054157.sHtMLWww.xiaoqiyx.com/Article/details/5346141.sHtMLWww.xiaoqiyx.com/Article/details/7562355.sHtMLWww.xiaoqiyx.com/Article/details/7101444.sHtMLWww.xiaoqiyx.com/Article/details/8812200.sHtMLWww.xiaoqiyx.com/Article/details/7444841.sHtMLWww.xiaoqiyx.com/Article/details/4546977.sHtMLWww.xiaoqiyx.com/Article/details/3310223.sHtMLWww.xiaoqiyx.com/Article/details/1871425.sHtMLWww.xiaoqiyx.com/Article/details/4269020.sHtMLWww.xiaoqiyx.com/Article/details/2948890.sHtMLWww.xiaoqiyx.com/Article/details/1739296.sHtMLWww.xiaoqiyx.com/Article/details/7262464.sHtMLWww.xiaoqiyx.com/Article/details/9079864.sHtMLWww.xiaoqiyx.com/Article/details/7608076.sHtMLWww.xiaoqiyx.com/Article/details/5729895.sHtMLWww.xiaoqiyx.com/Article/details/6419645.sHtMLWww.xiaoqiyx.com/Article/details/1511372.sHtMLWww.xiaoqiyx.com/Article/details/1489748.sHtMLWww.xiaoqiyx.com/Article/details/4794760.sHtMLWww.xiaoqiyx.com/Article/details/3159495.sHtMLWww.xiaoqiyx.com/Article/details/6259533.sHtMLWww.xiaoqiyx.com/Article/details/1757070.sHtMLWww.xiaoqiyx.com/Article/details/8004215.sHtMLWww.xiaoqiyx.com/Article/details/2461139.sHtMLWww.xiaoqiyx.com/Article/details/4999607.sHtMLWww.xiaoqiyx.com/Article/details/3471743.sHtMLWww.xiaoqiyx.com/Article/details/6513513.sHtMLWww.xiaoqiyx.com/Article/details/2607593.sHtMLWww.xiaoqiyx.com/Article/details/8174937.sHtMLWww.xiaoqiyx.com/Article/details/1338152.sHtMLWww.xiaoqiyx.com/Article/details/4810358.sHtMLWww.xiaoqiyx.com/Article/details/6402993.sHtMLWww.xiaoqiyx.com/Article/details/3096349.sHtMLWww.xiaoqiyx.com/Article/details/2947745.sHtMLWww.xiaoqiyx.com/Article/details/8699553.sHtMLWww.xiaoqiyx.com/Article/details/9182447.sHtMLWww.xiaoqiyx.com/Article/details/6787998.sHtMLWww.xiaoqiyx.com/Article/details/8337791.sHtMLWww.xiaoqiyx.com/Article/details/5507072.sHtMLWww.xiaoqiyx.com/Article/details/5170765.sHtMLWww.xiaoqiyx.com/Article/details/3201845.sHtMLWww.xiaoqiyx.com/Article/details/7761074.sHtMLWww.xiaoqiyx.com/Article/details/2013115.sHtMLWww.xiaoqiyx.com/Article/details/3798248.sHtMLWww.xiaoqiyx.com/Article/details/3571893.sHtMLWww.xiaoqiyx.com/Article/details/4994581.sHtMLWww.xiaoqiyx.com/Article/details/5275665.sHtMLWww.xiaoqiyx.com/Article/details/6419333.sHtMLWww.xiaoqiyx.com/Article/details/6305757.sHtMLWww.xiaoqiyx.com/Article/details/7096825.sHtMLWww.xiaoqiyx.com/Article/details/8703812.sHtMLWww.xiaoqiyx.com/Article/details/9274775.sHtMLWww.xiaoqiyx.com/Article/details/8098413.sHtMLWww.xiaoqiyx.com/Article/details/2052804.sHtMLWww.xiaoqiyx.com/Article/details/3907741.sHtMLWww.xiaoqiyx.com/Article/details/5736311.sHtMLWww.xiaoqiyx.com/Article/details/1873510.sHtMLWww.xiaoqiyx.com/Article/details/1814272.sHtMLWww.xiaoqiyx.com/Article/details/1584727.sHtMLWww.xiaoqiyx.com/Article/details/2114542.sHtMLWww.xiaoqiyx.com/Article/details/3444531.sHtMLWww.xiaoqiyx.com/Article/details/7161300.sHtMLWww.xiaoqiyx.com/Article/details/2134127.sHtMLWww.xiaoqiyx.com/Article/details/4852843.sHtMLWww.xiaoqiyx.com/Article/details/2625511.sHtMLWww.xiaoqiyx.com/Article/details/5877976.sHtMLWww.xiaoqiyx.com/Article/details/6516272.sHtMLWww.xiaoqiyx.com/Article/details/3315167.sHtMLWww.xiaoqiyx.com/Article/details/9675982.sHtMLWww.xiaoqiyx.com/Article/details/6919305.sHtMLWww.xiaoqiyx.com/Article/details/4304542.sHtMLWww.xiaoqiyx.com/Article/details/6095573.sHtMLWww.xiaoqiyx.com/Article/details/4079669.sHtMLWww.xiaoqiyx.com/Article/details/1031266.sHtMLWww.xiaoqiyx.com/Article/details/3582628.sHtMLWww.xiaoqiyx.com/Article/details/4361661.sHtMLWww.xiaoqiyx.com/Article/details/8820778.sHtMLWww.xiaoqiyx.com/Article/details/1866295.sHtMLWww.xiaoqiyx.com/Article/details/4994895.sHtMLWww.xiaoqiyx.com/Article/details/8829784.sHtMLWww.xiaoqiyx.com/Article/details/4286025.sHtMLWww.xiaoqiyx.com/Article/details/1643190.sHtMLWww.xiaoqiyx.com/Article/details/6985139.sHtMLWww.xiaoqiyx.com/Article/details/2086409.sHtMLWww.xiaoqiyx.com/Article/details/9912511.sHtMLWww.xiaoqiyx.com/Article/details/8069077.sHtMLWww.xiaoqiyx.com/Article/details/8905161.sHtMLWww.xiaoqiyx.com/Article/details/6894645.sHtMLWww.xiaoqiyx.com/Article/details/1980284.sHtMLWww.xiaoqiyx.com/Article/details/9930903.sHtMLWww.xiaoqiyx.com/Article/details/2070482.sHtMLWww.xiaoqiyx.com/Article/details/8090516.sHtMLWww.xiaoqiyx.com/Article/details/2567071.sHtMLWww.xiaoqiyx.com/Article/details/1479799.sHtMLWww.xiaoqiyx.com/Article/details/1547806.sHtMLWww.xiaoqiyx.com/Article/details/3873713.sHtMLWww.xiaoqiyx.com/Article/details/8427236.sHtMLWww.xiaoqiyx.com/Article/details/4206246.sHtMLWww.xiaoqiyx.com/Article/details/4868339.sHtMLWww.xiaoqiyx.com/Article/details/3716660.sHtMLWww.xiaoqiyx.com/Article/details/3230327.sHtMLWww.xiaoqiyx.com/Article/details/4275203.sHtMLWww.xiaoqiyx.com/Article/details/1767805.sHtMLWww.xiaoqiyx.com/Article/details/6968630.sHtMLWww.xiaoqiyx.com/Article/details/3398721.sHtMLWww.xiaoqiyx.com/Article/details/5592697.sHtMLWww.xiaoqiyx.com/Article/details/6973199.sHtMLWww.xiaoqiyx.com/Article/details/4823973.sHtMLWww.xiaoqiyx.com/Article/details/1071959.sHtMLWww.xiaoqiyx.com/Article/details/4170902.sHtMLWww.xiaoqiyx.com/Article/details/8738382.sHtMLWww.xiaoqiyx.com/Article/details/8495263.sHtMLWww.xiaoqiyx.com/Article/details/3500225.sHtMLWww.xiaoqiyx.com/Article/details/3258573.sHtMLWww.xiaoqiyx.com/Article/details/9772414.sHtMLWww.xiaoqiyx.com/Article/details/9215740.sHtMLWww.xiaoqiyx.com/Article/details/3143016.sHtMLWww.xiaoqiyx.com/Article/details/2040399.sHtMLWww.xiaoqiyx.com/Article/details/1657767.sHtMLWww.xiaoqiyx.com/Article/details/8534492.sHtMLWww.xiaoqiyx.com/Article/details/5226278.sHtMLWww.xiaoqiyx.com/Article/details/9048740.sHtMLWww.xiaoqiyx.com/Article/details/9106999.sHtMLWww.xiaoqiyx.com/Article/details/3907588.sHtMLWww.xiaoqiyx.com/Article/details/3414444.sHtMLWww.xiaoqiyx.com/Article/details/2602523.sHtMLWww.xiaoqiyx.com/Article/details/6251372.sHtMLWww.xiaoqiyx.com/Article/details/7999763.sHtMLWww.xiaoqiyx.com/Article/details/3561247.sHtMLWww.xiaoqiyx.com/Article/details/6584741.sHtMLWww.xiaoqiyx.com/Article/details/8781348.sHtMLWww.xiaoqiyx.com/Article/details/5490549.sHtMLWww.xiaoqiyx.com/Article/details/1790671.sHtMLWww.xiaoqiyx.com/Article/details/2260696.sHtMLWww.xiaoqiyx.com/Article/details/2130062.sHtMLWww.xiaoqiyx.com/Article/details/2795726.sHtMLWww.xiaoqiyx.com/Article/details/6580423.sHtMLWww.xiaoqiyx.com/Article/details/1391066.sHtMLWww.xiaoqiyx.com/Article/details/8680190.sHtMLWww.xiaoqiyx.com/Article/details/1679903.sHtMLWww.xiaoqiyx.com/Article/details/6801305.sHtMLWww.xiaoqiyx.com/Article/details/9957689.sHtMLWww.xiaoqiyx.com/Article/details/3025236.sHtMLWww.xiaoqiyx.com/Article/details/1256581.sHtMLWww.xiaoqiyx.com/Article/details/1836190.sHtMLWww.xiaoqiyx.com/Article/details/4008174.sHtMLWww.xiaoqiyx.com/Article/details/7936427.sHtMLWww.xiaoqiyx.com/Article/details/7326430.sHtMLWww.xiaoqiyx.com/Article/details/7969798.sHtMLWww.xiaoqiyx.com/Article/details/2093124.sHtMLWww.xiaoqiyx.com/Article/details/9976957.sHtMLWww.xiaoqiyx.com/Article/details/2931225.sHtMLWww.xiaoqiyx.com/Article/details/4593784.sHtML