在task2中，已经借助AI工具对AIGC生图的代码进行精读。在本章中则更加关注于了解Lora微调的基本原理以及文生图的工作流平台工具ComfyUI的使用。
task2链接：Datawhale X 魔搭 AI夏令营 第四期魔搭-AIGC文生图方向Task2笔记
Lora微调
Lora简介
在深度学习领域，Lora是Low-Rank Adaptation的缩写，是一种在预训练模型上进行高效微调的技术。它可以通过高效且灵活的方式实现模型的个性化调整，使其能够适应特定的任务或领域，同时保持良好的泛化能力和较低的资源消耗，在大规模预训练模型的实际应用中发挥着重要的作用。
Lora基本原理及其优势
Lora的详细论文可以阅读LORA: LOW-RANK ADAPTATION OF LARGE LANGUAGE MODELS，同时作者开源了代码，可以查看https://github.com/microsoft/LoRA。其基本原理如下图所示：
针对一个已经预训练好的模型，作者冻结模型原有的权值，而将需要更新的部分分解成两个低秩矩阵即和。其中，，，。由于秩，因此实际训练中需要优化的参数为，只占很小的一部分，因此能够达到更少的可训练参数的目的。在训练过程中，矩阵A被随机初始化为高斯分布矩阵，而矩阵B则初始化为0矩阵。由上述公式可知，在训练开始时， 的初始值也为0。
相比于其他的微调方式，LoRA有以下的优势：
（1）快速适应新任务：不同的任务之间可以共享一个相同的预训练模型，而任务的差异部分则可通过构建许多小型的LoRA模块来实现。即通过替换图中的矩阵来有效的切换任务，从而显著减少存储需求和任务切换开销。
（2）更高效的存储和资源效率：当使用自适应优化器时，LoRA可以提供更高效的训练，只需要优化低秩矩阵，避免了大部分参数的存储、求导和优化。
（3）不增加额外的推理延迟：使用简单的线性方式设计能够在部署时将可训练矩阵与冻结权值合并，通过构造的方式，与完全微调的模型相比能够不引入推理延迟。
task中的Lora微调代码
import os
cmd = """
python DiffSynth-Studio/examples/train/kolors/train_kolors_lora.py \ # 选择使用可图的Lora训练脚本DiffSynth-Studio/examples/train/kolors/train_kolors_lora.py
--pretrained_unet_path models/kolors/Kolors/unet/diffusion_pytorch_model.safetensors \ # 选择unet模型
--pretrained_text_encoder_path models/kolors/Kolors/text_encoder \ # 选择text_encoder
--pretrained_fp16_vae_path models/sdxl-vae-fp16-fix/diffusion_pytorch_model.safetensors \ # 选择vae模型
--lora_rank 16 \ # lora_rank 16 表示在权衡模型表达能力和训练效率时，选择了使用 16 作为秩，适合在不显著降低模型性能的前提下，通过 LoRA 减少计算和内存的需求
--lora_alpha 4.0 \ # 设置 LoRA 的 alpha 值，影响调整的强度
--dataset_path data/lora_dataset_processed \ # 指定数据集路径，用于训练模型
--output_path ./models \ # 指定输出路径，用于保存模型
--max_epochs 1 \ # 设置最大训练轮数为 1
--center_crop \ # 启用中心裁剪，这通常用于图像预处理
--use_gradient_checkpointing \ # 启用梯度检查点技术，以节省内存
--precision "16-mixed" # 指定训练时的精度为混合 16 位精度（half precision），这可以加速训练并减少显存使用
""".strip()
os.system(cmd) # 执行可图Lora训练 其中，关乎LoRA对模型的影响的参数有两个，分别是lora_rank和lora_alpha。lora_rank也就是r的大小，在取值时需要在模型性能、计算和内存两者之间进行权衡。lora_alpha则是设置LoRA的alpha 值，在训练过程中会影响梯度的更新，使用适当的lora_alpha可以影响调整的强度。
ComfyUI工具的探索和使用
ComfyUI简介
ComfyUI是一个基于节点的图形用户界面（GUI），其特点在于将图像生成的过程分解成多个步骤，每个步骤都作为一个节点，因此通过链接不同的节点能够构建复杂的图像生成工作流程。一般而言这些节点可以包括各种任务，如加载检查点模型、输入提示、指定采样器等，有利于实现了工作流的精准定制和可靠复现。
ComfyUI的特点和其他UI工具的对比
目前，AIGC的三大主流GUI分别是窗口式function（代表是WebUI）、prompt only（代表是fooocus）和节点式pipeline（代表是ComfyUI）。在这三大GUI中，各有各的特点，其优缺点总结如下：
GUI WebUI fooocus ComfyUI 优点操作界面固定且非常清晰、直观，操作简单，使用方便，易于学习和快速上手
只需关注提示词而无需关注各种复杂参数就可以获得高质量图片；对硬件要求较低，但是界面友好，易于快速上手
对硬件要求较低；具有极高的自由度和灵活性，支持高度的定制化和工作流复用；在出图速度和性能表现更佳
缺点对硬件要求较高，且比较占资源；工作流程复制方面存在局限性，用户需在每次操作时手动进行设置。
自由度并不是很高，不适合定制化的工作使用拥有众多的插件节点以及较为复杂的操作流程，上手相对困难
ComfyUI的图片生成流程
ComfyUI的安装和使用
ComfyUI的安装
首先，下载安装ComfyUI的执行文件和task1中微调完成Lora文件
git lfs install
git clone https://www.modelscope.cn/datasets/maochase/kolors_test_comfyui.git
mv kolors_test_comfyui/* ./
rm -rf kolors_test_comfyui/
mkdir -p /mnt/workspace/models/lightning_logs/version_0/checkpoints/
mv epoch=0-step=500.ckpt /mnt/workspace/models/lightning_logs/version_0/checkpoints/ 其次，进入到ComfyUI的安装文件中，一键执行安装程序。程序的代码步骤如下：
1、克隆ComfyUI的代码，并安装相关依赖
# #@title Environment Setup
from pathlib import Path
OPTIONS = {}
UPDATE_COMFY_UI = True #@param {type:"boolean"}
INSTALL_COMFYUI_MANAGER = True #@param {type:"boolean"}
INSTALL_KOLORS = True #@param {type:"boolean"}
INSTALL_CUSTOM_NODES_DEPENDENCIES = True #@param {type:"boolean"}
OPTIONS['UPDATE_COMFY_UI'] = UPDATE_COMFY_UI
OPTIONS['INSTALL_COMFYUI_MANAGER'] = INSTALL_COMFYUI_MANAGER
OPTIONS['INSTALL_KOLORS'] = INSTALL_KOLORS
OPTIONS['INSTALL_CUSTOM_NODES_DEPENDENCIES'] = INSTALL_CUSTOM_NODES_DEPENDENCIES
current_dir = !pwd
WORKSPACE = f"{current_dir[0]}/ComfyUI"
%cd /mnt/workspace/
![ ! -d $WORKSPACE ] && echo -= Initial setup ComfyUI =- && git clone https://github.com/comfyanonymous/ComfyUI
%cd $WORKSPACE
if OPTIONS['UPDATE_COMFY_UI']:
!echo "-= Updating ComfyUI =-"
!git pull
if OPTIONS['INSTALL_COMFYUI_MANAGER']:
%cd custom_nodes
![ ! -d ComfyUI-Manager ] && echo -= Initial setup ComfyUI-Manager =- && git clone https://github.com/ltdrdata/ComfyUI-Manager
%cd ComfyUI-Manager
!git pull
if OPTIONS['INSTALL_KOLORS']:
%cd ../
![ ! -d ComfyUI-KwaiKolorsWrapper ] && echo -= Initial setup KOLORS =- && git clone https://github.com/kijai/ComfyUI-KwaiKolorsWrapper.git
%cd ComfyUI-KwaiKolorsWrapper
!git pull
%cd $WORKSPACE
if OPTIONS['INSTALL_CUSTOM_NODES_DEPENDENCIES']:
!pwd
!echo "-= Install custom nodes dependencies =-"
![ -f "custom_nodes/ComfyUI-Manager/scripts/colab-dependencies.py" ] && python "custom_nodes/ComfyUI-Manager/scripts/colab-dependencies.py"
!wget "https://modelscope.oss-cn-beijing.aliyuncs.com/resource/cloudflared-linux-amd64.deb"
!dpkg -i cloudflared-linux-amd64.deb2、下载一些经典的文生图模型（可以包含SD基础模型，Lora，Controlnet等），并存放到models目录的相关子目录下
#@markdown ###Download standard resources
OPTIONS = {}
#@markdown **unet**
!wget -c "https://modelscope.cn/models/Kwai-Kolors/Kolors/resolve/master/unet/diffusion_pytorch_model.fp16.safetensors" -P ./models/diffusers/Kolors/unet/
!wget -c "https://modelscope.cn/models/Kwai-Kolors/Kolors/resolve/master/unet/config.json" -P ./models/diffusers/Kolors/unet/
#@markdown **encoder**
!modelscope download --model=ZhipuAI/chatglm3-6b-base --local_dir ./models/diffusers/Kolors/text_encoder/
#@markdown **vae**
!wget -c "https://modelscope.cn/models/AI-ModelScope/sdxl-vae-fp16-fix/resolve/master/sdxl.vae.safetensors" -P ./models/vae/ #sdxl-vae-fp16-fix.safetensors
#@markdown **scheduler**
!wget -c "https://modelscope.cn/models/Kwai-Kolors/Kolors/resolve/master/scheduler/scheduler_config.json" -P ./models/diffusers/Kolors/scheduler/
#@markdown **modelindex**
!wget -c "https://modelscope.cn/models/Kwai-Kolors/Kolors/resolve/master/model_index.json" -P ./models/diffusers/Kolors/
3、安装LoRA节点（用于后面加载LoRA模型使用）
lora_node = """
import torch
from peft import LoraConfig, inject_adapter_in_model
class LoadKolorsLoRA:
@classmethod
def INPUT_TYPES(s):
return {
"required": {
"kolors_model": ("KOLORSMODEL", ),
"lora_path": ("STRING", {"multiline": False, "default": "",}),
"lora_alpha": ("FLOAT", {"default": 2.0, "min": 0.0, "max": 4.0, "step": 0.01}),
},
}
RETURN_TYPES = ("KOLORSMODEL",)
RETURN_NAMES = ("kolors_model",)
FUNCTION = "add_lora"
CATEGORY = "KwaiKolorsWrapper"
def convert_state_dict(self, state_dict):
prefix_rename_dict = {
"blocks.7.transformer_blocks": "down_blocks.1.attentions.0.transformer_blocks",
"blocks.10.transformer_blocks": "down_blocks.1.attentions.1.transformer_blocks",
"blocks.15.transformer_blocks": "down_blocks.2.attentions.0.transformer_blocks",
"blocks.18.transformer_blocks": "down_blocks.2.attentions.1.transformer_blocks",
"blocks.21.transformer_blocks": "mid_block.attentions.0.transformer_blocks",
"blocks.25.transformer_blocks": "up_blocks.0.attentions.0.transformer_blocks",
"blocks.28.transformer_blocks": "up_blocks.0.attentions.1.transformer_blocks",
"blocks.31.transformer_blocks": "up_blocks.0.attentions.2.transformer_blocks",
"blocks.35.transformer_blocks": "up_blocks.1.attentions.0.transformer_blocks",
"blocks.38.transformer_blocks": "up_blocks.1.attentions.1.transformer_blocks",
"blocks.41.transformer_blocks": "up_blocks.1.attentions.2.transformer_blocks",
}
suffix_rename_dict = {
".to_out.lora_A.default.weight": ".to_out.0.lora_A.default.weight",
".to_out.lora_B.default.weight": ".to_out.0.lora_B.default.weight",
}
state_dict_ = {}
for name, param in state_dict.items():
for prefix in prefix_rename_dict:
if name.startswith(prefix):
name = name.replace(prefix, prefix_rename_dict[prefix])
for suffix in suffix_rename_dict:
if name.endswith(suffix):
name = name.replace(suffix, suffix_rename_dict[suffix])
state_dict_[name] = param
lora_rank = state_dict_["up_blocks.1.attentions.2.transformer_blocks.1.attn2.to_q.lora_A.default.weight"].shape[0]
return state_dict_, lora_rank
def load_lora(self, model, lora_rank, lora_alpha, state_dict):
lora_config = LoraConfig(
r=lora_rank,
lora_alpha=lora_alpha,
init_lora_weights="gaussian",
target_modules=["to_q", "to_k", "to_v", "to_out.0"],
)
model = inject_adapter_in_model(lora_config, model)
model.load_state_dict(state_dict, strict=False)
return model
def add_lora(self, kolors_model, lora_path, lora_alpha):
state_dict = torch.load(lora_path, map_location="cpu")
state_dict, lora_rank = self.convert_state_dict(state_dict)
kolors_model["pipeline"].unet = self.load_lora(kolors_model["pipeline"].unet, lora_rank, lora_alpha, state_dict)
return (kolors_model,)
NODE_CLASS_MAPPINGS = {
"LoadKolorsLoRA": LoadKolorsLoRA,
}
NODE_DISPLAY_NAME_MAPPINGS = {
"LoadKolorsLoRA": "Load Kolors LoRA",
}
__all__ = ["NODE_CLASS_MAPPINGS", "NODE_DISPLAY_NAME_MAPPINGS"]
""".strip()
import os
os.makedirs("/mnt/workspace/ComfyUI/custom_nodes/ComfyUI-LoRA", exist_ok=True)
with open("/mnt/workspace/ComfyUI/custom_nodes/ComfyUI-LoRA/__init__.py", "w", encoding="utf-8") as f:
f.write(lora_node)4、运行ComfyUI获取访问链接
%cd /mnt/workspace/ComfyUI
import subprocess
import threading
import time
import socket
import urllib.request
def iframe_thread(port):
while True:
time.sleep(0.5)
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
result = sock.connect_ex(('127.0.0.1', port))
if result == 0:
break
sock.close()
print("\nComfyUI finished loading, trying to launch cloudflared (if it gets stuck here cloudflared is having issues)\n")
p = subprocess.Popen(["cloudflared", "tunnel", "--url", "http://127.0.0.1:{}".format(port)], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
for line in p.stderr:
l = line.decode()
if "trycloudflare.com " in l:
print("This is the URL to access ComfyUI:", l[l.find("http"):], end='')
#print(l, end='')
threading.Thread(target=iframe_thread, daemon=True, args=(8188,)).start()
!python main.py --dont-print-server上述代码执行之后可以看到ComfyUI的访问链接，如下图所示，将链接复制到浏览器之后就可以打开ComfyUI的界面。
ComfyUI的使用
进入ComfyUI界面之后，可以构建自己的工作流，可以通过点击Load按键上传搭建工作流的json文件，然后点击Queue Prompt可稳定运行生图，最后生成的图片可以鼠标右键保存到本地。
这里分别提供了不带LoRA的工作流案例和带LoRA的工作流案例的json文件。
# 不带LoRA
{
"last_node_id": 15,
"last_link_id": 18,
"nodes": [
{
"id": 11,
"type": "VAELoader",
"pos": [
1323,
240
],
"size": {
"0": 315,
"1": 58
},
"flags": {},
"order": 0,
"mode": 0,
"outputs": [
{
"name": "VAE",
"type": "VAE",
"links": [
12
],
"shape": 3
}
],
"properties": {
"Node name for S&R": "VAELoader"
},
"widgets_values": [
"sdxl.vae.safetensors"
]
},
{
"id": 10,
"type": "VAEDecode",
"pos": [
1368,
369
],
"size": {
"0": 210,
"1": 46
},
"flags": {},
"order": 6,
"mode": 0,
"inputs": [
{
"name": "samples",
"type": "LATENT",
"link": 18
},
{
"name": "vae",
"type": "VAE",
"link": 12,
"slot_index": 1
}
],
"outputs": [
{
"name": "IMAGE",
"type": "IMAGE",
"links": [
13
],
"shape": 3,
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "VAEDecode"
}
},
{
"id": 14,
"type": "KolorsSampler",
"pos": [
1011,
371
],
"size": {
"0": 315,
"1": 222
},
"flags": {},
"order": 5,
"mode": 0,
"inputs": [
{
"name": "kolors_model",
"type": "KOLORSMODEL",
"link": 16
},
{
"name": "kolors_embeds",
"type": "KOLORS_EMBEDS",
"link": 17
}
],
"outputs": [
{
"name": "latent",
"type": "LATENT",
"links": [
18
],
"shape": 3,
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "KolorsSampler"
},
"widgets_values": [
1024,
1024,
1000102404233412,
"fixed",
25,
5,
"EulerDiscreteScheduler"
]
},
{
"id": 6,
"type": "DownloadAndLoadKolorsModel",
"pos": [
201,
368
],
"size": {
"0": 315,
"1": 82
},
"flags": {},
"order": 1,
"mode": 0,
"outputs": [
{
"name": "kolors_model",
"type": "KOLORSMODEL",
"links": [
16
],
"shape": 3,
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "DownloadAndLoadKolorsModel"
},
"widgets_values": [
"Kwai-Kolors/Kolors",
"fp16"
]
},
{
"id": 3,
"type": "PreviewImage",
"pos": [
1366,
468
],
"size": [
535.4001724243165,
562.2001106262207
],
"flags": {},
"order": 7,
"mode": 0,
"inputs": [
{
"name": "images",
"type": "IMAGE",
"link": 13
}
],
"properties": {
"Node name for S&R": "PreviewImage"
}
},
{
"id": 12,
"type": "KolorsTextEncode",
"pos": [
519,
529
],
"size": [
457.2893696934723,
225.28656056301645
],
"flags": {},
"order": 4,
"mode": 0,
"inputs": [
{
"name": "chatglm3_model",
"type": "CHATGLM3MODEL",
"link": 14,
"slot_index": 0
}
],
"outputs": [
{
"name": "kolors_embeds",
"type": "KOLORS_EMBEDS",
"links": [
17
],
"shape": 3,
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "KolorsTextEncode"
},
"widgets_values": [
"cinematic photograph of an astronaut riding a horse in space |\nillustration of a cat wearing a top hat and a scarf |\nphotograph of a goldfish in a bowl |\nanime screencap of a red haired girl",
"",
1
]
},
{
"id": 15,
"type": "Note",
"pos": [
200,
636
],
"size": [
273.5273818969726,
149.55464588512064
],
"flags": {},
"order": 2,
"mode": 0,
"properties": {
"text": ""
},
"widgets_values": [
"Text encoding takes the most VRAM, quantization can reduce that a lot.\n\nApproximate values I have observed:\nfp16 - 12 GB\nquant8 - 8-9 GB\nquant4 - 4-5 GB\n\nquant4 reduces the quality quite a bit, 8 seems fine"
],
"color": "#432",
"bgcolor": "#653"
},
{
"id": 13,
"type": "DownloadAndLoadChatGLM3",
"pos": [
206,
522
],
"size": [
274.5334274291992,
58
],
"flags": {},
"order": 3,
"mode": 0,
"outputs": [
{
"name": "chatglm3_model",
"type": "CHATGLM3MODEL",
"links": [
14
],
"shape": 3
}
],
"properties": {
"Node name for S&R": "DownloadAndLoadChatGLM3"
},
"widgets_values": [
"fp16"
]
}
],
"links": [
[
12,
11,
0,
10,
1,
"VAE"
],
[
13,
10,
0,
3,
0,
"IMAGE"
],
[
14,
13,
0,
12,
0,
"CHATGLM3MODEL"
],
[
16,
6,
0,
14,
0,
"KOLORSMODEL"
],
[
17,
12,
0,
14,
1,
"KOLORS_EMBEDS"
],
[
18,
14,
0,
10,
0,
"LATENT"
]
],
"groups": [],
"config": {},
"extra": {
"ds": {
"scale": 1.1,
"offset": {
"0": -114.73954010009766,
"1": -139.79705810546875
}
}
},
"version": 0.4
}示例图：
# 带LoRA
{
"last_node_id": 16,
"last_link_id": 20,
"nodes": [
{
"id": 11,
"type": "VAELoader",
"pos": [
1323,
240
],
"size": {
"0": 315,
"1": 58
},
"flags": {},
"order": 0,
"mode": 0,
"outputs": [
{
"name": "VAE",
"type": "VAE",
"links": [
12
],
"shape": 3
}
],
"properties": {
"Node name for S&R": "VAELoader"
},
"widgets_values": [
"sdxl.vae.safetensors"
]
},
{
"id": 10,
"type": "VAEDecode",
"pos": [
1368,
369
],
"size": {
"0": 210,
"1": 46
},
"flags": {},
"order": 7,
"mode": 0,
"inputs": [
{
"name": "samples",
"type": "LATENT",
"link": 18
},
{
"name": "vae",
"type": "VAE",
"link": 12,
"slot_index": 1
}
],
"outputs": [
{
"name": "IMAGE",
"type": "IMAGE",
"links": [
13
],
"shape": 3,
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "VAEDecode"
}
},
{
"id": 15,
"type": "Note",
"pos": [
200,
636
],
"size": {
"0": 273.5273742675781,
"1": 149.5546417236328
},
"flags": {},
"order": 1,
"mode": 0,
"properties": {
"text": ""
},
"widgets_values": [
"Text encoding takes the most VRAM, quantization can reduce that a lot.\n\nApproximate values I have observed:\nfp16 - 12 GB\nquant8 - 8-9 GB\nquant4 - 4-5 GB\n\nquant4 reduces the quality quite a bit, 8 seems fine"
],
"color": "#432",
"bgcolor": "#653"
},
{
"id": 13,
"type": "DownloadAndLoadChatGLM3",
"pos": [
206,
522
],
"size": {
"0": 274.5334167480469,
"1": 58
},
"flags": {},
"order": 2,
"mode": 0,
"outputs": [
{
"name": "chatglm3_model",
"type": "CHATGLM3MODEL",
"links": [
14
],
"shape": 3
}
],
"properties": {
"Node name for S&R": "DownloadAndLoadChatGLM3"
},
"widgets_values": [
"fp16"
]
},
{
"id": 6,
"type": "DownloadAndLoadKolorsModel",
"pos": [
201,
368
],
"size": {
"0": 315,
"1": 82
},
"flags": {},
"order": 3,
"mode": 0,
"outputs": [
{
"name": "kolors_model",
"type": "KOLORSMODEL",
"links": [
19
],
"shape": 3,
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "DownloadAndLoadKolorsModel"
},
"widgets_values": [
"Kwai-Kolors/Kolors",
"fp16"
]
},
{
"id": 12,
"type": "KolorsTextEncode",
"pos": [
519,
529
],
"size": {
"0": 457.28936767578125,
"1": 225.28656005859375
},
"flags": {},
"order": 4,
"mode": 0,
"inputs": [
{
"name": "chatglm3_model",
"type": "CHATGLM3MODEL",
"link": 14,
"slot_index": 0
}
],
"outputs": [
{
"name": "kolors_embeds",
"type": "KOLORS_EMBEDS",
"links": [
17
],
"shape": 3,
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "KolorsTextEncode"
},
"widgets_values": [
"二次元，长发，少女，白色背景",
"",
1
]
},
{
"id": 3,
"type": "PreviewImage",
"pos": [
1366,
469
],
"size": {
"0": 535.400146484375,
"1": 562.2001342773438
},
"flags": {},
"order": 8,
"mode": 0,
"inputs": [
{
"name": "images",
"type": "IMAGE",
"link": 13
}
],
"properties": {
"Node name for S&R": "PreviewImage"
}
},
{
"id": 16,
"type": "LoadKolorsLoRA",
"pos": [
606,
368
],
"size": {
"0": 317.4000244140625,
"1": 82
},
"flags": {},
"order": 5,
"mode": 0,
"inputs": [
{
"name": "kolors_model",
"type": "KOLORSMODEL",
"link": 19
}
],
"outputs": [
{
"name": "kolors_model",
"type": "KOLORSMODEL",
"links": [
20
],
"shape": 3,
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "LoadKolorsLoRA"
},
"widgets_values": [
"/mnt/workspace/models/lightning_logs/version_0/checkpoints/epoch=0-step=500.ckpt",
2
]
},
{
"id": 14,
"type": "KolorsSampler",
"pos": [
1011,
371
],
"size": {
"0": 315,
"1": 266
},
"flags": {},
"order": 6,
"mode": 0,
"inputs": [
{
"name": "kolors_model",
"type": "KOLORSMODEL",
"link": 20
},
{
"name": "kolors_embeds",
"type": "KOLORS_EMBEDS",
"link": 17
},
{
"name": "latent",
"type": "LATENT",
"link": null
}
],
"outputs": [
{
"name": "latent",
"type": "LATENT",
"links": [
18
],
"shape": 3,
"slot_index": 0
}
],
"properties": {
"Node name for S&R": "KolorsSampler"
},
"widgets_values": [
1024,
1024,
0,
"fixed",
25,
5,
"EulerDiscreteScheduler",
1
]
}
],
"links": [
[
12,
11,
0,
10,
1,
"VAE"
],
[
13,
10,
0,
3,
0,
"IMAGE"
],
[
14,
13,
0,
12,
0,
"CHATGLM3MODEL"
],
[
17,
12,
0,
14,
1,
"KOLORS_EMBEDS"
],
[
18,
14,
0,
10,
0,
"LATENT"
],
[
19,
6,
0,
16,
0,
"KOLORSMODEL"
],
[
20,
16,
0,
14,
0,
"KOLORSMODEL"
]
],
"groups": [],
"config": {},
"extra": {
"ds": {
"scale": 1.2100000000000002,
"offset": {
"0": -183.91309381910426,
"1": -202.11110769225016
}
}
},
"version": 0.4
}加载 LoRA 时，需要在“lora_path”处填入 LoRA 模型的路径。
示例图：
在github官网中可以查看到更多关于ComfyUI的信息：https://github.com/comfyanonymous/ComfyUI
参考：
https://arxiv.org/pdf/2106.09685
https://www.uisdc.com/comfyui-3
https://mp.weixin.qq.com/s/W5ZOQOJsSs7grpbirbfwqg
https://www.bilibili.com/video/BV1ch4y1B7vp/?spm_id_from=333.1350.jump_directly
https://github.com/comfyanonymous/ComfyUI
https://modelscope.cn/headlines/article/429
总结