Khiqiza Izithombe Ezinombhalo Omningi Nezakhiwo Ezilawulwayo nge-MOSS-Image

Yini i-MOSS-Image?

I-MOSS-Image iphrojekthi evulekile yokukhiqiza izithombe evela ku-OpenMOSS Team. Ihlinzeka ngemodeli eyisisekelo yokuguqula umbhalo ube isithombe kanye nemodeli eqeqeshwe kahle ngokugadwa, kanye nepayiphi ye-Diffusers yangokwezifiso, isixhumi esibonakalayo se-Gradio, i-plugin yomdabu ye-ComfyUI, namathuluzi okulungiselela iziqondiso ezinemininingwane.

Le phrojekthi yakhelwe ukwakheka okuhlanganisa izici ezibonakalayo nombhalo ofundekayo. Isekela iziqondiso ezingafika kumathokheni angu-4K futhi ingakhiqiza izitayela ezisuka ku-anime ziye ezithombeni ezibukeka njengezangempela. I-backbone yayo engu-5B ihloselwe ukuhlinzeka ngokukhiqizwa kwezithombe okuhlangene futhi okusebenza kahle.

Izici eziyinhloko

• Ukulawula ukwakheka: Beka umbhalo nezici ezibonakalayo usebenzisa okokufaka kwe-JSON okuhlelekile namabhokisi abophayo.
• Ukuboniswa kombhalo: Khiqiza umbhalo ocacile ngaphakathi kwamaphosta, amanothi, amabhodi amhlophe, nokunye ukwakheka kwezithombe.
• Izindida ezinde: Thumela iziqondiso zombhalo eziqukethe amathokheni afinyelela ku-4K.
• Izitayela eziningi: Dala izithombe zesitayela se-anime nezibukeka njengezangempela.
• Ukwesekwa kokulungisa kahle: Guqula imodeli eyisisekelo ukuze ivumelane nemisebenzi nezizinda ezithile.
• Isakhiwo esisebenza kahle: Khiqiza izithombe nge-backbone engu-5B.
• Izixhumi ezibonakalayo eziningi: Sebenzisa i-Python, i-Gradio, i-ComfyUI, noma ikhono elihlukile lokuthuthukisa iziqondiso.

Khetha imodeli

I-MOSS-Image itholakala ngezinhlobo ezimbili:

• MOSS-Image-Base: Imodeli eyisisekelo yokuguqula umbhalo ube isithombe. Isisindo sayo sitholakala ku-Hugging Face naku-ModelScope.
• MOSS-Image-SFT: Inguqulo elungiswe kahle isuselwa kumodeli eyisisekelo kusetshenziswa idatha ekhethiwe. Isisindo sayo sitholakala ku-Hugging Face naku-ModelScope.

Ekuqhathaniseni okuhlinzekwe yiphrojekthi, imodeli ye-SFT ikhiqiza ukuhlanganiswa kombhalo okucace kakhudlwana nokwakheka kwebhodi elimhlophe okuhlelekile kakhulu. Qala ngemodeli ye-Base uma udinga i-checkpoint yokuqala esetshenziswa kabanzi noma uhlela ukuyilungisa kahle. Cabangela i-SFT lapho ukwethulwa kombhalo okuhlelekile kubaluleke kakhulu.

Faka i-MOSS-Image

Clone-a i-repository, ngena kufolda yayo, dala i-virtual environment, yenze isebenze, bese ufaka izidingo:

git clone https://github.com/OpenMOSS/MOSS-Image.git
cd MOSS-Image
python -m venv .venv
source .venv/bin/activate

pip install -r requirements.txt

Izibonelo zisebenzisa i-CUDA futhi zihambisa ipayiphi ku-GPU nge-pipe.to("cuda"). Qinisekisa ukuthi indawo yakho ye-PyTorch ilungiselelwe inkundla ye-GPU ohlela ukusebenzisa kuyo i-inference.

Khiqiza isithombe nge-Python

I-MOSS-Image ilayishwa nge-Diffusers kusetshenziswa ipayiphi yayo yangokwezifiso. Isibonelo esilandelayo sicela imodeli ukuthi yakhe isigcawu sasekhishini esiqukethe isikhumbuzi sesiShayina esibhalwe ngesandla:

import torch
from diffusers import DiffusionPipeline

pipe = DiffusionPipeline.from_pretrained(
    "OpenMOSS-Team/MOSS-Image-Base",
    custom_pipeline="pipeline_moss_image",
    trust_remote_code=True,
    torch_dtype={"default": torch.bfloat16, "vae": torch.float32},
)
pipe.scheduler = type(pipe.scheduler).from_config(pipe.scheduler.config)
pipe.to("cuda")

prompt = (
    "A wide view of a refrigerator beside a kitchen counter with a fruit bowl. "
    "A small sticky note on the refrigerator door is handwritten in Chinese:“\n"
    "记得买牛奶和鸡蛋！\n牛奶两盒，鸡蛋一打。\n顺便带一袋面包。”"
)

image = pipe(
    prompt,
    height=1456,
    width=1456,
    num_inference_steps=50,
    guidance_scale=3.0,
    uncond_skip_layers=[22],
    generator=torch.Generator("cpu").manual_seed(42),
).images[0]

image.save("refrigerator_note.png")

Qonda izimpikiswano ezibalulekile

• custom_pipeline="pipeline_moss_image" ikhetha ipayiphi ye-MOSS-Image.
• trust_remote_code=True ivumela i-Diffusers ukuthi ilayishe ukuqaliswa kwepayiphi yangokwezifiso yemodeli.
• torch_dtype isebenzisa i-bfloat16 ngokuzenzakalelayo, kuyilapho igcina i-float32 ye-VAE.
• height ne-width zinquma ubukhulu bomphumela.
• num_inference_steps=50 ilandela isincomo esizenzakalelayo sephrojekthi sokwenza amasampula.
• guidance_scale=3.0 isebenzisa inani elinconyiwe le-classifier-free guidance.
• uncond_skip_layers=[22] inika amandla ukweqa izendlalelo egatsheni eline-negative kusendlalelo 22.
• manual_seed(42) yenza ukuphinda ukusebenza ngokucushwa okufanayo kuthembeke kakhulu.

Qalisa isixhumi esibonakalayo se-Gradio

Faka okuncikile kokuzikhethela kwesixhumi esibonakalayo bese uqala uhlelo olufakiwe:

pip install -e '.[ui]'
python gradio_app.py --model OpenMOSS-Team/MOSS-Image-Base

Vula ikheli lendawo elibikwe yi-Gradio, faka isiqondiso esihle, khetha ubukhulu besithombe, bese uqala ukukhiqiza. Ukuze uhlole i-checkpoint elungiswe kahle, shintsha impikiswano yemodeli ibe ngu-OpenMOSS-Team/MOSS-Image-SFT.

Izilungiselelo ezinconyiwe zokwenza amasampula

• I-sampler: Flow-matching Euler
• Izinyathelo: 50
• CFG: 3
• Shift: 5
• Ukweqa izendlalelo: Ukweqa egatsheni eline-negative kusendlalelo 22
• Isiqondiso esine-negative se-Python API: Akukho

Lezi yizilungiselelo ezizenzakalelayo ezibhalwe phansi zephrojekthi futhi zinikeza isiqalo esisebenzisekayo. Bheka umhlahlandlela womsebenzisi wesiNgisi wekhosombe ukuze uthole ezinye izincomo mayelana nama-prompt namasu okusampula.

Lawula ukwakheka usebenzisa izakhiwo ze-JSON

Ama-prompt ajwayelekile achaza ukuthi isithombe kufanele siqukathe ini, kodwa awachazi ngokunembile ukuthi ingxenye ngayinye kufanele ibe kuphi. Esikhundleni salokho, i-MOSS-Image ingamukela isakhiwo se-JSON esichaza indawo yonke, izakhi zombhalo noma zezinto ngazinye, kanye namabhokisi azo okubopha.

Isakhiwo esilandelayo sichaza iphosta yokuhamba elula enobukhulu obungu-1024 x 1024. Yigcine njenge-layout.json uma ufuna ukuyilayisha usebenzisa i-Python, noma uyinamathisele ngqo enkambini ye-Prompt ye-Gradio.

{
  "summary": {
    "meta": ["poster"],
    "canvas": "single",
    "caption": "A minimalist travel poster with a cream background, a blue mountain illustration, and the title EXPLORE at the top."
  },
  "items": [
    {
      "label": "text",
      "caption": "Large dark-blue uppercase title centered at the top.",
      "text": "EXPLORE",
      "bbox": [-416, -384, -288, 384]
    },
    {
      "label": "object",
      "caption": "A flat blue mountain illustration in the lower center.",
      "bbox": [-192, -288, 384, 288]
    }
  ],
  "bbox_coord_space": "abs_centralized",
  "bbox_format": "yxyx"
}

Indlela ifomethi yesakhiwo esebenza ngayo

• summary.caption ichaza indawo yonke nesitayela sayo esibonakalayo.
• items iqukethe izakhi zombhalo nezinto ezibekwe ngokuzimela.
• label ikhomba into njengombhalo noma into.
• caption ichaza ukubukeka nendima yento.
• text iqukethe amagama aqondile okufanele aboniswe into yombhalo.
• bbox ichaza indawo nobubanzi bento.
• bbox_format: "yxyx" kusho ukuthi ibhokisi ngalinye lihlelwe njenge-[y_min, x_min, y_max, x_max].
• bbox_coord_space: "abs_centralized" kusho ukuthi izixhumanisi zikalwa kusukela maphakathi kwe-canvas.

Kwi-canvas engu-1024 x 1024, izimbazo zombili zisuka ku--512 ziye ku-+512. Isixhumanisi se-x siyanda uma siya kwesokudla, kanti esika-y siyanda uma siya phansi. Ukuze uguqule izixhumanisi ezijwayelekile ze-canvas, susa uhhafu wobubanzi be-canvas kumanani e-x kanye nohhafu bobude be-canvas kumanani e-y.

Njalo sebenzisa ubukhulu bokukhiqiza obufana nobukhulu isakhiwo esilungiselelwe bona. Ukungafani kushintsha uhlelo lwezixhumanisi futhi kungasusa izakhi ezindaweni ebezihlosiwe.

Thumela isakhiwo nge-Python

Layisha ifayela le-JSON, liguqule libe uchungechunge lombhalo, bese udlulisa lolo chungechunge njenge-prompt yepayipi:

import json
import torch

with open("layout.json", "r", encoding="utf-8") as file:
    layout = json.load(file)

layout_prompt = json.dumps(layout, ensure_ascii=False)

image = pipe(
    layout_prompt,
    height=1024,
    width=1024,
    num_inference_steps=50,
    guidance_scale=3.0,
    uncond_skip_layers=[22],
    generator=torch.Generator("cpu").manual_seed(42),
).images[0]

image.save("travel_poster.png")

Ungabuyekeza ukwakheka ngaphandle kokubhala kabusha yonke i-prompt. Hlela umbhalo, i-caption, noma ibhokisi lokubopha lento bese ukhiqiza isithombe futhi. I-README iphinde iphawule ukuthi le ndlela ehlelekile ingasekela ukuhlela okulawulwayo namathuluzi okwakha izakhiwo asebenziseka kalula.

Lawula inkumbulo ye-GPU

Isixhumi esibonakalayo se-Gradio sinikeza izindlela ezintathu zenkumbulo ngaphansi kwezilungiselelo ezithuthukile:

• GPU resident: Igcina imodeli ephelele ku-GPU ukuze ifinyelelwe ngokuqondile.
• Component: Ihambisa izingxenye zemodeli phakathi kwe-CPU ne-GPU njengoba kudingeka.
• Sequential fallback: Yehlisa ukusetshenziswa kwenkumbulo kodwa yenza inference ihambe kancane.

Uma imodeli ingalingani kwinkumbulo etholakalayo ye-GPU, khetha imodi ye-Sequential noma wehlise usayizi wesithombe esiphumayo. Ukuhlanganiswa kwe-DiT kukhutshaziwe ngokuzenzakalelayo futhi akutholakali kwimodi ye-Sequential.

Sebenzisa i-MOSS-Image ne-ComfyUI

Ikhosombe lifaka i-plugin yomdabu ye-MOSS-Image ye-ComfyUI. Sesha i-MOSS-Image ku-ComfyUI Manager ukuze uyifake. Iphrojekthi iphinde inikeze ukugeleza komsebenzi okuyisibonelo okungangeniswa njengesiqalo.

Iphrojekthi isebenzela ukuhlanganiswa nekhosombe elisemthethweni le-ComfyUI. Okwamanje, landela imiyalelo yokufaka nokusebenzisa i-ComfyUI yekhosombe.

Ukwenza ama-prompt athuthukile nge-Prompt Enhancer

I-MOSS-Image Prompt Enhancer efakiwe iguqula umbono oyisisekelo wesithombe ube yi-caption enemininingwane noma i-prompt yesakhiwo sendawo. Yakhelwe ama-agent omndeni we-GPT-6 anamathuluzi endawo wamafayela ne-Python.

1. Faka ifolda ethi skills/moss-image-prompt-enhancer ohlwini lwamakhono lwe-agent yakho.
2. Shayela i-$moss-image-prompt-enhancer nombono wesithombe.
3. Chaza ukuthi ufuna i-caption enemininingwane noma okukhiphayo kwe-Layout.
4. Faka ubukhulu be-canvas obuhlosiwe lapho udala isakhiwo.
5. Namathisela okukhiphayo okuhlanganisiwe ku-positive prompt ye-MOSS-Image.
6. Ngokukhiphayo kwe-Layout, khiqiza isithombe usebenzisa ubukhulu obufanayo be-canvas obusetshenziswe ngesikhathi sokuhlanganisa.

Iskripthi esifakiwe siguqula okusalungiselelwe okusekelwe ezixhumanisini zamaphikseli kube yi-JSON emfushane sisebenzisa izixhumanisi ezigxile ezipheleleyo. Silungiselela kuphela imiyalelo futhi asibizi isevisi yokukhiqiza izithombe.

Amathiphu asebenzayo

• Qala ngezilungiselelo ezizenzakalelayo ezibhaliwe: Sebenzisa izinyathelo ezingu-50, i-CFG engu-3, i-shift engu-5, nokweqa igatsha elinegethivu kusendlalelo esingu-22 ngaphambi kokuzama ezinye izilungiselelo.
• Gcina umbhalo oceliwe ucacile: Faka amagama aqondile emyalweni noma endaweni ethi text yento.
• Sebenzisa izakhiwo ezihlelekile kumiklamo eyinkimbinkimbi: Amaphosta, imidwebo, nezigcawu ezigcwele umbhalo kusizakala ngokuchaza izinto namabhokisi emingcele ngokwehlukana.
• Qondanisa ubukhulu ngokucophelela: Izixhumanisi zesakhiwo zixhumene nobubanzi nobude bekhanvasi ekhethiwe.
• Gcina i-seed ingashintshi ngesikhathi sokuphindaphinda: Ukugcina i-seed efanayo kukusiza uqhathanise izinguquko kumyalo, kumodeli, nasesakhiweni.
• Yehlisa usayizi wesithombe uma inkumbulo ilinganiselwe: Leli elinye lamakhambi anconywe ngokusobala yiphrojekthi.
• Qhathanisa i-Base ne-SFT: I-checkpoint ye-SFT ingase ifaneleke kakhulu lapho kubaluleke khona ukuhlanganiswa okucacile nesakhiwo sombhalo esihleleke kangcono.

Isiphetho

I-MOSS-Image ihlanganisa ukukhiqizwa kwezithombe kusuka embhalweni okuvamile nokunikezwa kombhalo okunamandla, kanye nokulawula ukwakheka okuhlelekile. Ungaqala ngepayipi le-Python noma ngesixhumi esibonakalayo se-Gradio, bese uya ezakhiweni ze-JSON lapho ukubekwa okuqondile kubalulekile. Ukuhlanganiswa kwe-ComfyUI, izindlela zokusebenzisa inkumbulo, nekhono lokuthuthukisa imiyalo kunikeza ezinye izindlela zokufaka imodeli ezinqubweni zokusebenza ezibonakalayo. Ikhodi yepayipi ezimele itholakala ngaphansi kwe-Apache License 2.0.