stable diffusion实践操作-常见lora模型介绍

系列文章目录

本文专门开一节写Lora相关的内容，在看之前，可以同步关注：
stable diffusion实践操作

文章目录

前言

Lora是微调模型，某品牌汽车（特斯拉），人物姿势（剪刀手），可以固定我们照片的特征：人物特征、动作特征、还有照片风格。

如果把基础模型比喻作一座房子的地基，那么Lora模型就好比在这个地基上盖起来的房子。我们通常也称为微调模型，用于满足一种特定的风格，或指定的人物特征属性。在数据相似度非常高的情形下，使用微调模型，可以节省大量的训练时间和训练资源，就可以产出我们需要的结果.，常见的后缀是ckpt,safetensors,pt，大小100M左右。

Lora是可以叠加使用的，叠加后会非常变态的。
Lora之间一样用英文逗号隔开，每个Lora后面都有数字，这是用来调整这个Lora的权重的。
正常情况下是1，我们一般只会去降低权重，因为增加权重照片可能就会变得奇奇怪怪，每个Lora设置的权重不一样，出来的照片就会不一样，想要生成一个好看的小姐姐，就要多去尝试不同的权重组合，并且lora有提示词，如下：

<lora:Beautiful_Dress_1.0:0.6>,dress

模型分两种，一种是sd大模型，一种是类似Lora一样的小模型
国内的是：https://www.liblibai.com
国外的是：https://civitai.com

一、什么是lora?

1.1 lora 定义

LoRA，英文全称Low-Rank Adaptation of Large Language Models，直译为大语言模型的低阶适应，这是微软的研究人员为了解决大语言模型微调而开发的一项技术。

1.2 lora的基本原理

大模型，比如，GPT-3有1750亿参数，为了让它能干特定领域的活儿，需要做微调，但是如果直接对GPT-3做微调，成本太高太麻烦了。

LoRA的做法是，冻结预训练好的模型权重参数，然后在每个Transformer（Transforme就是GPT的那个T）块里注入可训练的层，由于不需要对模型的权重参数重新计算梯度，所以，大大减少了需要训练的计算量。

如下图，lora在PLM旁边增加一个旁路，做一个降维再升维的操作，训练的时候固定PLM参数，只训练降维矩阵A和升维矩阵B，模型输入与输出维度不变。输出时将BA与PLM的参数叠加

在微调sd的情况下，lora可以应用于将图像表示于描述他们的提示词相关联的交叉注意层。

也就是说lora在不干涉神经网络运算层的情况下，插入一些低阶运算层，通过训练这些人低阶运算层来影响出图

顺便看看和超网络的区别，下面是超网络，看出来两者区别没有：

研究发现，LoRA的微调质量与全模型微调相当，我愿称之为神器。

要做个比喻的话，就好比是大模型的一个小模型，或者说是一个插件。

LoRA本来是给大语言模型准备的，但把它用在cross-attention layers（交叉关注层）也能影响用文字生成图片的效果。

1.2 通过分层控制lora

（TB）

二、作用：

2.1 复刻人物特征

对人物和物品的复刻，只要挂载lora，就可以九成九的复刻指定人物的特征。
比如，你下载了光头强的Lora，只要你加载了它，那么你生成的任何一张图片都是他的样子。

2.2 固定人物的动作特征

2.3 确定某种画风

总结：Lora可以确定某种特征。

Lora一定要和底模配合使用。

三、lora和embedding 区别：

3.1 文件大小不一样

文件小携带的信息就比较少。

3.2 lora比embedding要好的多

embedding在多视图和动漫人物上要好一点。

3.3 embedding是文字描述，lora是真实的物品描述。

四 、lora使用

4.1 用lora配套的大模型

因为lora训练是需要一个大模型的，是通过大模型来训练参数的人，所以它的一切和大模型都绑定了。
所以使用的时候，要和训练的原始模型匹配。
下面是hanfu lora和不同大模型的对比：

4.2 .有触发词一定要使用触发词

<lora:baihuaniang_1.0:0.8>,baihuaniang，flower

一般来说，不用把全部的触发词加上去，只要加其中一个就可以了，比如下面的有汉服，有唐服等等，这个要根据lora模型来确定，有写lora是没有提示词的。

4.3 正确设置lora的权重

权重越大，lora对结果影响就很大。但是也不能过大，过小，一般在0.6-0.9之间。

4.4 新手不要混用lora

多了会打架。

五、 收集到的lora模型

１、baihuaniang_1.0

下载链接：https://www.liblibai.com/modelinfo/1d2839ecaaf829e0161beb8c70294dbf
基础模型：SD1.5（AWPainting_v1.2_liblib_sd15.safetensors）
触发词语：baihuaniang，flower

案例：
正向提示词：

Best quality,masterpiece,ultra high res,(photorealistic:1.4),
solo,1girl,.<lora:baihuaniang_1.0:0.8>,baihuaniang,Flowers,
Flowers,Soft sunlight,Wind,Flying long hair,Film Light,
Facing the screen,.copper,,upper body,fatigues,
outstretched arms,.

反向提示词：

ng_deepnegative_v1_75t,badhandv4 (worst quality:2),
(low quality:2),(normal quality:2),lowres,bad anatomy,
bad hands,normal quality,((monochrome)),((grayscale)),, 
ng_deepnegative_v1_75t, badhandv4 (worst quality:2), 
(low quality:2), (normal quality:2), lowres, bad anatomy, 
bad hands, normal quality, ((monochrome)), ((grayscale)):.

采样：
DPM++2MKarras
随机种子：2514900086
CFGscale:11
步数：28

生成结果：

2、long.safetensors　中国龙

下载链接：https://www.liblibai.com/modelinfo/870bee7ee8377e037097ae2d16839709
触发词：no humans， dragon， long
基础模型SD1.5 （AWPainting_v1.2_liblib_sd15.safetensors）
正向提示词：

Best quality,masterpiece,ultra high res,nu no humans, (long:1.2),
<lora:long:0.8>,no humans， dragon， long eastern dragon, 
east asian architecture, sky, architecture, outdoors, cloud, day, 
mountain, open mouth, blue sky, teeth, scales, sharp teeth, 
horns, glowing, fangs, tree, glowing eyes, pagoda, scenery.

反向提示词：

ng_deepnegative_v1_75t,badhandv4 (worst quality:2),(low quality:2),
(normal quality:2),lowres,bad anatomy,bad hands,normal quality,
((monochrome)),((grayscale)),nsfw,..

采样：
DPM++2MKarras
随机种子：2330800025
CFGscale:11
步数：28

生成结果：

3、xiaorenshu_v2.0.safetensors

下载链接：https://www.liblibai.com/modelinfo/7755bdd92e3baeb7bd8ac79a455070ad
基础模型：SD1.5 （AWPainting_v1.2_liblib_sd15.safetensors）
触发词语：没有

正向提示词：
shz,((bestquality)),((masterpiece)),(detailed),realistic,multipleboys,(ridinghorse),mustache,lookingatviewer,outdoors,house,amy,horse

反向提示词：

采样：DPM++2MKarras
随机种子：3836055402
CFGscale: 5
步数：28

生成结果：

4、Lora_GuoFeng3_Lora_v3.2_Lora_big_light.safetensors

下载链接：https://www.liblibai.com/modelinfo/7b3ce08ce87271586f0fd00de32604be
基础模型：SD1.5 （AWPainting_v1.2_liblib_sd15.safetensors）
触发词语：没有

正向提示词：

bestquality,masterpiece,highres,1girl,chinadress,Beautifulface,
hairornament,solo,lookingatviewer,smile,closedmouth,lips,
chinadress,dress,hairornament,necklace,jewelry,longhair,earrings,chineseclothes,
architecture,eastasianarchitecture,building,outdoors,rooftop,city,cityscape,
darkstudio,rimlighting,twotonelighting,dimlylit,lowkey
<lora:GuoFeng3.2_Lora_light:1.5>

采样：Eulera
随机种子：3985684671
CFGscale: 7
步数：30

生成结果：

5、Lora Real dragons _dragon_real_base_V1.safetensors

下载链接：https://www.liblibai.com/modelinfo/6d1bbd25afd9c54f442606d1774dc425
基础模型：SD1.5 （AWPainting_v1.2_liblib_sd15.safetensors）
触发词语：dragon_real，flame

正向提示词：

dragon_real,(extremely detailed CG unity 8k wallpaper,masterpiece, best quality, 
ultra-detailed, beautiful detailed eyes:1.2),best illumination, (best shadow, an extremely delicate and beautiful, bloom),
the girl was surrounded by dragons,.(1girl:1.4),breasts, solo,full body,Back tattoo,white dress,(color dragon:1.4),
Bare leg,long dress, 

.
反向提示词：

nude, (badhandv4:1.2), (easynegative), verybadimagenegative_v1.3, deformation), blurry, poorly dissected, disfigured,
 poorly drawn face, mutation, mutation, (extra_limb), (ugly), (poorly drawn hand), finger fusion, messy drawing, 
broken leg examination, examination, examination stick, multiple breasts, (mutated hand and finger: 1.5), (long body: 1.3), 
(mutated, poorly drawn: 1.2), black and white, poorly dissected, liquid body, disfigured Deformed, mutated, anatomical nonsense, 
text font ui, errors, malformed hands, long necks, blurry, lowered, anatomically poor, proportion not good, shadows not good, 
body not coordinated, body not natural, fused breasts, poor breasts, huge breasts, poorly drawn breasts, excess breasts, 
liquid breasts, heavy breasts, missing breasts, large buttocks, huge thighs, Huge calves, bad hands, fused hands, missing hands, 
missing arms, missing thighs, missing calves, missing legs, fused arms, bad ears, poorly drawn ears, excess ears, liquid ears, 
heavy ears, missing ears, text, ui, errors, missing fingers, missing limbs, fused fingers, one hand with more than 5 fingers, 
one hand with less than 5 fingers, The other hand has more than 5 digits, the other hand is missing less than 5 digits, 
excess digits, fewer digits, fused digits, missing digits, bad digits, liquid digits, cropped, blurry, JPEG artifacts, signatures, 
deformed feet, excess feet, bad feet, bad drawings, fused feet, lost feet, excess shoes, bad shoes, fused shoes,
 two or more pairs of shoes, poorly drawn shoes Bad gloves, poorly drawn gloves, fused gloves, bad hair, poorly drawn hair, 
fused hair, big muscles, ugly, bad face, fused face, poorly drawn face, cloned face, big face, long face, bad eyes, 
fused eyes well drawn eyes, excess eyes, deformed limbs, two or more nipples, rough ratio. Short arm, (((missing arm)), 
thigh missing, calf missing, leg missing, mutation, duplication, deformity, poorly drawn hand, more than one left hand, 
less than one right hand, deformation, (blurring), disfigurement, missing leg, excess arm, excess thigh, two or more thighs, 
excess calf, fused calf, excess leg, poor knee, excess knee, more than two legs, Bad mouth, fused mouth, bad mouth, 
bad tongue, tongue in mouth, too long tongue, black tongue, big mouth, cracked mouth, bad mouth, dirty face, 
dirty teeth, bad underpants, fused cloth, bad cloth, bad underpants, yellow teeth, thick lips, bad camel toe, 
colorful camel toe, bad crotch seam, missing clip, bad clip, fused clip, color clip, black clip, liquid clip, QR code,
 barcode, reviewed, secure underwear, extra numbers, fewer numbers, cropped, worst quality, low quality, 
normal quality, jpeg artifacts, signature, watermark, username, blurry, out of focus, reviewed, ugly, old, deformed, 
amateur, strange, fat, deformed, black and white, realistic eyes,.

采样：DPM++2MKarras
随机种子：2656310671
CFGscale: 7
步数：55

生成结果：

6、korean doll likeness

下载链接：https://www.aigc.wtf/korean-doll-likeness-free-download/
成图参考：https://zhuanlan.zhihu.com/p/606825889
基础模型：chilloutmix_NiPrunedFp16Fix.safetensors
触发词语：

正向提示词：

(RAW photo:1.2), (photorealistic:1.4),(masterpiece:1.3),(best quality:1.4),
nikon d750 f/1.4 55mm, detailed shadow, (realistic, photo-realistic:1.37),(sharp focus:1.2),(ISO high), (masterpiece,best quality), professional lighting, ultra high res,

dreamlike,
(detailed eyes),(detailed facial features), (detailed clothes features),
skin tight, (shiny skin),(slender girl),(slim girl),
1girl, ((full body)),solo,
cute,smlie,
, (medium breasts), bangs, straight hair, long hair, black hair, black eyes, crystal pendant,
Long blue scarf,White long wool coat,Black thick pantyhose,Brown snow boots,
<lora:koreanDollLikeness_v15:1>

.
反向提示词：

nsfw,EasyNegative,bad_prompt,ng_deepnegative_v1_75t,(worst quality:2), (low quality:2), (normal quality:2), lowres, normal quality, ((monochrome)), ((grayscale)), skin spots, acnes, skin blemishes, age spot, (outdoor:1.6), manboobs, backlight,(ugly:1.331), (duplicate:1.331), (morbid:1.21), (mutilated:1.21), (tranny:1.331), mutated hands, (poorly drawn hands:1.331), blurry, (bad anatomy:1.21), (bad proportions:1.331), extra limbs, (disfigured:1.331), (more than 2 nipples:1.331), (missing arms:1.331), (extra legs:1.331), (fused fingers:1.61051), (too many fingers:1.61051), (unclear eyes:1.331), bad hands, missing fingers, extra digit, (futa:1.1), bad body, pubic hair, glans,
Steps: 20, Sampler: Euler a, CFG scale: 7, Seed: 546686100, Size: 720x960, Model hash: fc2511737a, Model: chilloutmix_NiPrunedFp32Fix (2), Clip skip: 2, ENSD: 31337

采样：DPM++2MKarras
随机种子：2514900086
CFGscale: 7
步数：22
生成结果：

7、Beautiful_Dress连衣裙

下载链接：https://www.liblibai.com/modelinfo/3aa33b228cda4a0980754ae921eb51d4
成图参考：
基础模型：chilloutmix_NiPrunedFp16Fix.safetensors
触发词语：dress
说明：这个只会生成裙子，一般没有人头，最好和其它的配合使用

正向提示词：

<lora:Beautiful_Dress:0.6>,(tightlacedressblackdeepvnecklayeredskirtankletdecorateholdflowersultryversatile:1.1),Girl,lightblue,(fullbody:1.2),(wholebody:1.2),(Lightweight:1.1),(Smile:1.1),Legs,Pure,Feet,(Lightmakeup:1.1),(45°LookUp:1.1),blackhair,Longhair,(whiteground:1.2),(Lightbackground:1.2),LongDress,(Stand.:1.2),Silkytexture,Cool,(Summer:1.1),(masterpiece:1,2),bestquality,masterpiece,highres,original,extremelydetailedwallpaper,perfectlighting,(extremelydetailedCG:1.2),drawing,paintbrush,Freedom,Romantic,mediumbreasts,

.
反向提示词：

NSFW,(worst quality:2),(low quality:2),(normal quality:2),lowres,normal quality,((monochrome)),((grayscale)),skin spots,acnes,skin blemishes,age spot,(ugly:1.331),(duplicate:1.331),(morbid:1.21),(mutilated:1.21),(tranny:1.331),mutated hands,(poorly drawn hands:1.5),blurry,(bad anatomy:1.21),(bad proportions:1.331),extra limbs,(disfigured:1.331),(missing arms:1.331),(extra legs:1.331),(fused fingers:1.61051),(too many fingers:1.61051),(unclear eyes:1.331),lowers,bad hands,missing fingers,extra digit,bad hands,missing fingers,(((extra arms and legs))),

采样：DPM++2MKarras
随机种子：1172117708

CFGscale: 7
步数：22

生成结果：

8、qianlijiangshanXXMix-V1.0

下载链接：https://www.liblibai.com/modelinfo/2581675514bd453984cc0f544b8f6cfe
成图参考：
基础模型：SD 1.5
触发词语：1.用于风景：no humans，mountain，outdoors，scenery，traditional media，landscape，river，birds，water，boat，moon 2.用于人物：1girl，solo，realistic，sky，hanfu，moon
说明：

正向提示词：

guofeng,illustration,1girl,full body,best quality,highly detailed,best quality,masterpiece,an extremely delicate and beautiful,extremely detailed,8k wallpaper,Amazing,finely detail,best quality,official art,extremely detailed CG unity 8k wallpaper,(style of Victo Ngai:0.5) <lora:qianlijiangshanXXMix-V1.0:0.65>, (moon:1.0),(river:1.2),(forest:1.0)(mountain :1.3)<lora:smokeanywhere:0.45>colorful smoke,

.
反向提示词：

lowres,bad anatomy,bad hands,text,error,extra digit,fewer digits,cropped,worst quality,low quality,normal quality,jpeg artifacts,signature,watermark,username,two moons,six fingers

采样：Euler a
随机种子：329501653

CFGscale: 7
步数：25

总结

`以上就是今天要讲的内容。