Khiqiza Ama-Embedding Aezinhlobo Eziningi nge-WeMM-Embedding

Yini i-WeMM-Embedding?

I-WeMM-Embedding iwumndeni wamamodeli e-universal multimodal embedding athuthukiswe ithimba le-WeChat Vision. Iguqula umbhalo, izithombe, amavidiyo, amadokhumenti abonakalayo, kanye nokufakwayo kwe-multimodal okuhlanganisiwe kube izethulo ze-vector ezihlangene. Lawa ma-vector angasekela izinqubo ezidinga ukumela nokuqhathanisa okuqukethwe phakathi kwama-modality ahlukahlukene.

Imodeli ngayinye ithola i-embedding yayo ku-hidden state yesendlalelo sokugcina endaweni ezinikele yethokheni ethi <embedding>, bese isebenzisa i-L2 normalization. Okufakwayo komsindo akukasekelwa okwamanje.

Izici Ezisemqoka

• Izethulo ezihlangene ze-multimodal: Umndeni ofanayo wamamodeli uphatha umbhalo, izithombe, amavidiyo, amadokhumenti abonakalayo, kanye nokufakwayo okuhlanganisiwe.
• Osayizi bamamodeli abaningi: Khetha phakathi kwezinhlobo ezinamapharamitha angu-2B, 4B, no-9B.
• Matryoshka Representation Learning: Sebenzisa ubukhulu be-vector obufushane obusekelwayo lapho ukusebenza kahle kwesitoreji noma kokusesha kubalulekile.
• Izixhumanisi ezimbili ze-inference: Kunikezwe izibonelo ze-Transformers ne-Sentence Transformers.
• Ukusebenzela ukukhiqiza: I-repository ihlanganisa ama-configuration nama-wrapper script e-vLLM ne-SGLang.
• Amathuluzi okuhlola: Iphayiphi ye-MMEB-v3 efakiwe isekela i-inference ku-node eyodwa noma kuma-node amaningi.

Khetha Imodeli Nobukhulu

I-model zoo iqukethe izinhlobo ezintathu ku-Hugging Face. Khetha imodeli ngokwamandla akho e-compute kanye nobukhulu obugcwele be-embedding obudingayo.

• WeMM-Embedding-2B: Isekela ubukhulu obungu-64, 128, 256, 512, 1024, no-2048.
• WeMM-Embedding-4B: Isekela ubukhulu obungu-64, 128, 256, 512, 1024, no-2560.
• WeMM-Embedding-9B: Isekela ubukhulu obungu-64, 128, 256, 512, 1024, 2048, no-4096.

Ubukhulu obuncane obusekelwayo bunciphisa isitoreji sama-vector nokubalwa okwenziwa kamuva. Isibonelo, i-README ibika ukuthi imodeli engu-2B enobukhulu obungu-256 igcina u-98.7% wokusebenza kwesithombe nevidiyo kobukhulu bayo obugcwele ku-MMEB-v2.

Ukufaka Nokulungiselela

1. Thola i-repository nemodeli

Sebenza ngekhophi yendawo ye-repository, bese ulanda imodeli ekhasini layo le-Hugging Face noma ulungiselele isibonelo esifanele sisebenzise isikhombi semodeli ye-Hugging Face lapho kusekelwa khona.

2. Faka ama-dependency

Faka izinguqulo ezishiwo yiphrojekthi:

pip install -r requirements.txt

3. Khiya i-Transformers ukuze kube nokuphindaphindeka

Iphrojekthi incoma i-transformers==5.2.0 nge-inference nokuphindaphindeka, ngoba ukukhishwa okusha kungase kuhluke ekuziphatheni kwe-preprocessing.

pip install transformers==5.2.0

Lungisa izindlela eziya kumodeli yakho, esithombeni sokuhlola, nakwividiyo yokuhlola ngaphambi kokusebenzisa izibonelo.

Ukusetshenziswa Okuyisisekelo nge-Transformers

Isibonelo se-Transformers sikhiqiza ama-embedding azimele okufakwayo kombhalo, kwesithombe, nokwevidiyo. Sisebenzise ngendlela enendlela yendawo yemodeli nobukhulu obusekelwayo bokuphumayo:

python examples/transformers_inference.py \
  --model /path/to/WeMM-Embedding-2B \
  --image /path/to/image.jpg \
  --video /path/to/video.mp4 \
  --dimension 2048

Shintsha izindlela eziyizibambeli ngamafayela akusistimu yakho. Kumodeli engu-2B, i-2048 iwubukhulu obugcwele be-vector. Ukuze ucele ubukhulu obugcwele be-embedding yemodeli ngaphandle kokubucacisa, susa u---dimension:

python examples/transformers_inference.py \
  --model /path/to/WeMM-Embedding-2B \
  --image /path/to/image.jpg \
  --video /path/to/video.mp4

Ukusetshenziswa Okuyisisekelo nge-Sentence Transformers

Isibonelo se-Sentence Transformers sidlulisela okufakwayo kombhalo, kwesithombe, nokwevidiyo ku-SentenceTransformer.encode(). Siphinde sisekele ukukhethwa kobukhulu be-Matryoshka nge---dimension.

python examples/sentence_transformers_inference.py \
  --model /path/to/WeMM-Embedding-2B \
  --image /path/to/image.jpg \
  --video /path/to/video.mp4 \
  --dimension 2048

Ngenxa yokuthi i-SentenceTransformer ingalayisha imodeli ngokuqondile, ungashintsha indlela yendawo ufake isikhombi semodeli ye-Hugging Face:

python examples/sentence_transformers_inference.py \
  --model tencent/WeMM-Embedding-2B \
  --image /path/to/image.jpg \
  --video /path/to/video.mp4 \
  --dimension 2048

Sebenzisa Ama-Embedding e-Matryoshka Ngokufanele

Ama-embedding e-Matryoshka akuvumela ukuba ufinyeze i-vector egcwele ibe kolunye lobukhulu obusekelwa imodeli ekhethiwe. Sika i-axis yokugcina bese wenza i-normalization futhi ku-vector efinyeziwe:

embedding = torch.nn.functional.normalize(
    embedding[..., :d],
    dim=-1
)

Setha u-d kuphela kubukhulu obusohlwini bemodeli yakho. Isinyathelo sesibili se-normalization sibalulekile ngoba ukusika kushintsha i-vector norm. Uma usebenzisa izibonelo zomugqa womyalo ezinikeziwe, khetha ubukhulu obufunwayo be-Matryoshka ngo---dimension.

Sebenzisa Imodeli nge-vLLM

I-repository ibika ukuhlolwa nge-vLLM 0.27.0. Qalisa iseva ye-pooling bese unikeza i-embedding chat template yemodeli:

MODEL_PATH=/path/to/WeMM-Embedding-2B
vllm serve "$MODEL_PATH" \
  --runner pooling \
  --chat-template "$MODEL_PATH/embedding_chat_template.jinja"

I-wrapper elinganayo iyatholakala ku-scripts/serve_vllm.sh.

Sebenzisa Imodeli nge-SGLang

I-SGLang 0.5.9 iyinguqulo ehloliwe ebhalwe yiphrojekthi. Sebenzisa i-video patch enikeziwe ngaphambi kokuqalisa iseva ye-embedding:

MODEL_PATH=/path/to/WeMM-Embedding-2B
python scripts/patch_sglang_video.py
python -m sglang.launch_server \
  --model-path "$MODEL_PATH" \
  --is-embedding \
  --enable-precise-embedding-interpolation

Indawo yokugcina iphinde inikeze scripts/serve_sglang.sh njengesisongeli esisebenza ngomyalo owodwa.

Qalisa Ukuhlolwa kwe-MMEB-v3

Uhla lwemibhalo mmeb_v3_eval/ luqukethe ikhodi yokuhlola esetshenziselwe imiphumela ebikiwe ye-MMEB-v3. Lusekelwe ku-VLM2Vec pipeline esemthethweni futhi lunezela i-WeMM-Embedding backbone, imiyalelo yedathasethi ehambisanayo, i-batched inference, ukusebenza kwama-node amaningi nama-GPU amaningi, kanye nokusampula ividiyo ngamafreyimu angu-64.

Qala ngokungena kuhla lwemibhalo lokuhlola bese ulanda idatha:

cd mmeb_v3_eval
DATA_ROOT=/path/to/MMEB-V3 bash scripts/download_data.sh

Bese uqhuba ukuhlola ngemodeli, idathasethi, kanye nezindlela zokuphuma ozikhethile:

MODEL_PATH=/path/to/WeMM-Embedding-2B \
DATA_BASEDIR=/path/to/MMEB-V3 \
OUTPUT_DIR=exps/wemm_embedding \
bash scripts/run_eval.sh

Bheka mmeb_v3_eval/README.md ukuze uthole ukusethwa kwedatha okuchaziwe kanye nemiyalo ye-node eyodwa noma ama-node amaningi.

Amathiphu Athuthukile

1. Qondanisa ubukhulu nemodeli: Ungaceli ubukhulu obungu-2560 kumodeli engu-2B noma obungu-4096 kumodeli engu-4B. Sebenzisa kuphela ubukhulu obubhalwe be-checkpoint ekhethiwe.
2. Yenza kabusha i-normalization yamavekhtha ancishisiwe: Uma unciphisa mathupha i-embedding ephelele, sebenzisa i-L2 normalization njalo ngemva kwalokho.
3. Gcina i-preprocessing ikhiqiza imiphumela efanayo: Sebenzisa inguqulo enconyiwe ye-Transformers lapho uphinda ukhiqiza imiphumela, ngoba i-preprocessing ingashintsha phakathi kwezinguqulo.
4. Sebenzisa indlela efanele yokulayisha: Umyalo obhaliwe we-Transformers usebenzisa indlela yemodeli, kanti isibonelo se-Sentence Transformers singalayisha ngokuqondile ne-tencent/WeMM-Embedding-2B.
5. Qaphela umsindo ongasekelwa: Amamodeli okwamanje awamukeli umsindo, ngakho isikolo somsindo esibikiwe se-MMEB-v3 singu-zero.
6. Sebenzisa izisongeli zokusevisa ukuze kube lula: Izikripthi ze-shell eziku-scripts/ zihlanganisa izilungiselelo zokuqalisa ezibhaliwe ze-vLLM ne-SGLang.

Umongo Wokusebenza

I-README ibika imiphumela kuwo wonke amadathasethi angu-78 e-MMEB-v2 kanye nayo yonke imisebenzi engu-190 ye-MMEB-v3. Ku-MMEB-v2, izinhlobo ze-2B, 4B, ne-9B zibika izikolo ezimaphakathi ezingu-77.9, 79.2, ne-80.6 ngokulandelana. Ku-MMEB-v3, izikolo zazo ezibikiwe sezizonke zingu-56.0, 58.2, ne-59.5. Amamethrikhi ayahluka ngokweqembu lomsebenzi: imisebenzi yesithombe nevidiyo isebenzisa i-Hit@1, imisebenzi yamadokhumenti abonakalayo nombhalo isebenzisa i-NDCG@5, kanti imisebenzi ye-agent, MCMR, nomsindo isebenzisa i-Hit@1.

Izikolo ze-benchmark kufanele zihunyushwe kucatshangelwa uhla lwemisebenzi olubhaliwe. I-MMEB-v3 inikeza isikolo esingu-zero emisebenzini engasekelwa, okuhlanganisa umsindo ku-WeMM-Embedding.

Isiphetho

I-WeMM-Embedding inikeza umndeni wemodeli owodwa wokukhiqiza izethulo ezenziwe zalinganiswa ezivela ezinhlotsheni eziningana zokufaka ezibonakalayo nezombhalo. Qala nge-checkpoint engu-2B kanye nesibonelo se-inference esinikeziwe, khetha ubukhulu obusekelwayo be-Matryoshka lapho amavekhtha ahlangene ewusizo, bese udlulela ku-vLLM, SGLang, noma ku-pipeline ye-MMEB-v3 lapho udinga uku-sevisa noma ukuhlola. Ikhodi yendawo yokugcina ebhalwe ngu-Tencent ikhishwa ngaphansi kwe-Apache License 2.0 ngaphandle kwalapho kuchazwe ngenye indlela.