Dala iziqeshana ze-MiniMax H3 ze-Text-to-Video ne-Image-to-Video ku-Apple Silicon usebenzisa i-TensorFold

Lokho okwenziwa yile phrojekthi

Le repository ipakisha imodeli yevidiyo i-MiniMax H3 ukuze isetshenziswe ku-TensorFold ku-Apple Silicon. Ihlanganisa umndeni we-H3 we-TensorFold nama-kernel e-MLX int8 Metal kanye ne-adapter ye-LightX2V Turbo ukuze ikhiqize ividiyo enomsindo we-stereo ovumelanisiwe.

Iphakheji isekela kokubili i-text-to-video futhi, kusukela kunguqulo 1.1, i-image-to-video kusetshenziswa ifreyimu yokuqala enikeziwe. Ku-Mac Studio M5 Ultra elinganisiwe enememori engu-256 GB, ikliphu yamasekhondi angu-5 engu-864x480 ithatha cishe imizuzwana engu-50 ne-adapter ye-Turbo. Ikhiphu engu-768x448 ithatha cishe imizuzwana engu-40 ngaphansi kwezimo zebhenchimakhi ezibhaliwe.

Le repository iyindlela yokwenza ama-render entuthukweni, hhayi iseva noma umkhiqizo. Awukho umyalo othi tensorfold generate; imodeli ilayishwa kuyo yonke i-render kusetshenziswa ama-script afakiwe.

Izici ezibalulekile

• I-Text-to-video: chaza isigcawu bese ukhiqiza i-MP4 enevidiyo nomsindo we-stereo.
• I-Image-to-video: nikeza ifreyimu yokuqala bese uchaza okwenzeka ngokulandelayo.
• Ukukhiqiza nge-Turbo: i-adapter ye-LightX2V yehlisa i-render ibe ama-forward amathathu endleleni esheshayo ebhaliwe.
• Ama-kernel e-MLX ne-int8: imisebenzi ekhethiwe ye-transformer ne-decoder isebenzisa ama-kernel e-int8 Metal ku-hardware ye-M5 tensor ehambisanayo.
• Ukusetha okuphindaphindekayo: isifaki sibopha i-TensorFold, i-MLX, imitapo yolwazi esekelayo, izinguqulo zamamodeli, namachecksum e-adapter.
• Ukukhiqiza ngemodeli eyisisekelo ngokuzikhethela: khubaza i-adapter bese usebenzisa cishe izinyathelo zokusampula ezingu-20 uma ufuna i-render esondelene nemodeli eyisisekelo.

Izidingo ze-hardware ne-software

Izinombolo zokusebenza ezishicilelwe zikalwe ku-Mac Studio M5 Ultra enememori ehlanganisiwe engu-256 GB, esebenzisa i-macOS 27.0.1. Iphrojekthi ithi kudingeka okungenani imemori engu-128 GB, kanti ukuhlanganiswa kwe-adapter kungafinyelela cishe ku-103 GB yememori ephezulu. Ngaphandle kwe-adapter, imemori ephezulu ibhalwe cishe ku-65 GB.

Ama-kernel e-int8 adinga imisebenzi ye-Metal 4 tensor, ngakho ukusheshiswa okubikiwe kuhloselwe i-hardware ye-M5 ehambisanayo. Ezinhlelweni ezingenayo le misebenzi, umndeni we-H3 usebenza nge-bfloat16 futhi ungase uhambe kancane kunezinye izindlela ezisekelwe ku-C ezichazwe ku-README.

Udinga futhi i-Homebrew, i-Python 3.11, i-uv, ne-FFmpeg. Ukulanda imodeli ye-MiniMax H3 cishe kungu-144 GB, kanti i-adapter ye-Turbo cishe ingu-1.96 GB.

Faka iphrojekthi

Clone-a i-repository bese ufaka amathuluzi adingekayo e-host:

git clone https://github.com/drowzeys/keys-Mac-TensorFold-MiniMax-H3-MLX.git
cd keys-Mac-TensorFold-MiniMax-H3-MLX
brew install python@3.11 uv ffmpeg

Qalisa i-script yokusetha konke ngomyalo owodwa:

bash oneshot-setup.sh

Isifaki sakha indawo ehlukanisiwe ku-~/.local/opt/tensorfold-h3 bese sifaka i-TensorFold 0.6.5 nomndeni we-H3 kanye ne-dependency lock eboshiwe. Uma i-Docker itholakala, ingathola i-payload ye-TensorFold eqinisekiswe nge-checksum ku-prebuilt carrier ye-GHCR yephrojekthi; uma ingatholakali, ithola i-source commit ehambisanayo ku-Git.

Ukusetha kuphinde ku-clone inguqulo edingekayo ye-minimax-h3-mlx, kulande i-MiniMax H3 FL2VA partition iye ku-~/h3-models/MiniMax-H3, kulande i-adapter ye-Turbo, bese kwenza i-test clip ibe yi-render ku-outputs/test.mp4.

Ukuhlola ukufakwa okukhona ngaphandle kokukufaka kabusha, sebenzisa:

bash oneshot-setup.sh --verify

Ukweqa i-test render yokuqala, sebenzisa:

bash oneshot-setup.sh --no-render

Ungashintsha izindawo ezizenzakalelayo zokufakwa nemodeli ngokusebenzisa iziguquguquko zemvelo ze-PREFIX ne-H3_MODEL_DIR.

Khiqiza ikliphu ye-text-to-video

Sebenzisa i-scripts/generate.sh nge-prompt negama lefayela lokuphuma:

bash scripts/generate.sh "A hummingbird hovering over red flowers, soft wing hum" out.mp4

Indlela ezenzakalelayo isebenzisa i-adapter ye-Turbo emisiwe kanye nezilungiselelo ezizenzakalelayo zokukhiqiza ze-repository. I-prompt ichaza kokubili isigcawu esibonakalayo nomsindo ohlosiwe lapho kudingeka.

Ukuze uthole okuhlukile okunqunyelwe, nikeza i-seed bese ukhetha ikhanvasi esekelwayo:

WIDTH=768 HEIGHT=448 SEED=7 bash scripts/generate.sh "A quiet train crossing a snowy valley at dawn" out.mp4

Ungagcina ama-prompt efayeleni lombhalo. Kuleso simo, dlulisa i-prompt engenalutho bese usetha i-PROMPT_FILE:

PROMPT_FILE=prompts/black-mirror-scene.txt SEED=2077 bash scripts/generate.sh "" out.mp4

Khiqiza ikliphu ye-image-to-video

Setha i-FIRST_FRAME ibe yifayela lesithombe. Isithombe siba yifreyimu yokuqala, kanti i-prompt ichaza ukunyakaza noma isigameko esilandelayo:

FIRST_FRAME=photo.jpg WIDTH=1344 HEIGHT=768 bash scripts/generate.sh "She turns to the camera and smiles" out.mp4

Isithombe selulwa ukuze silingane nekhanvasi ekhethiwe, ngakho khetha ubukhulu obunesilinganiso sobude nobubanzi esifanayo noma usike isithombe ngaphambi kokwenza i-render. I-README incoma u-1344x768 ku-16:9, u-768x1344 ku-9:16, no-768x768 kokuqukethwe okuyisikwele.

Ama-prompt e-image-to-video esitayela se-MiniMax angakhomba ngokusobala ifreyimu yokuqala, isibonelo:

For the target video, at 0.00 seconds into the target video, <Picture 1> (from [Shot 1]) is fully referenced. The subject walks through the doorway and looks toward the window.

Ukwenziwa kwefreyimu yokuqala kuyatholakala nge-adaptha ye-Turbo kanye nendlela eyisisekelo yezinyathelo ezingu-20. Izindlela zefreyimu yokugcina nezithombe eziyisithenjwa azikaxhunywa kule phakheji.

Lawula amafreyimu, ukulungiswa, kanye nesampula

Izibalo zamafreyimu kufanele zilandele uhlobo lwe-17n + 5. I-README ibala izibonelo ezisekelwayo ezihlanganisa amafreyimu angu-56, 73, 90, 124, 192, 243, kanye no-362. Ububanzi nobude kufanele kube iziphindaphindo zika-32, futhi umkhawulo wemodeli ekhishiwe uyisayizi yekhanvasi ephelele engadluli ku-768x1344 pixels.

Indlela esheshayo ye-Turbo isebenzisa ukudlulisa okuya phambili okuthathu. Ukuvala i-adaptha nokusebenzisa ukunikezwa kwezinyathelo ezingu-20, setha ADAPTER=none kanye ne-POINTS=21:

ADAPTER=none POINTS=21 bash scripts/generate.sh "A lantern glowing beside a forest path" base-model.mp4

I-README ichaza lesi silungiselelo njengesisondelene nemodeli eyisisekelo, kuyilapho i-Turbo ingakhiqiza amafreyimu acijile ngokushesha kodwa ngezinye izikhathi ingaphinda izibalo zabantu. Lezi zindlela ezimbili zingakhiqiza ukwakheka okuhlukile noma zisebenzisa imbewu efanayo.

Qonda ukuhwebelana kwesivinini

Ukuthuthuka okukhulu kwesivinini kuvela ekudluliseni okuthathu kwe-adaptha ye-Turbo esikhundleni kwezingu-20. I-int8 SwiGLU MLP, ukusebenza kwe-int8 QKV, okukhiphayo kokunaka, kanye nama-kernel edikhoda yevidiyo kunikeza izinzuzo ezengeziwe ku-hardware esekelwayo ye-M5 tensor.

Kubenchmark ebhaliwe engu-864x480, indlela ye-Turbo ithathe cishe imizuzwana engu-27.6 yokususa umsindo kanye nemizuzwana ecishe ibe ngu-50 isiyonke, okuhlanganisa ukufakwa kwekhodi yombhalo, ukulayisha nokwenza i-quantization, kanye nokudikhoda. Ukunikezwa kwe-TensorFold int8 kwezinyathelo ezingu-20 kuthathe cishe imizuzwana engu-210 isiyonke ekuqhathaniseni okufanayo.

Lezi zibalo zincike ku-hardware ethile futhi akufanele zithathwe njengezisebenza kuyo yonke indawo. Indawo yokugcina nayo iphawula ukuthi ukunaka uqobo kusasebenzisa i-MLX scaled dot-product attention, ngoba enye indlela ye-int8 yohlobo lwe-flash ayizange inikeze inzuzo ewusizo yesivinini ezilinganisweni.

Ukusetha okuthuthukile namathiphu okusebenza

Sebenzisa i-carrier eyakhiwe kusengaphambili

Isifaki singasebenzisa i-GHCR carrier, kodwa i-carrier ayiyona i-runtime ye-Mac. I-Metal ayisebenzi ngaphakathi kwe-container; i-carrier inikeza kuphela i-TensorFold wheel, i-dependency lock, iskripthi sokukhiqiza, kanye namachecksum okufakwa ngokwendabuko.

docker pull ghcr.io/drowzeys/keys-mac-tensorfold-minimax-h3-mlx:1.1
docker run --rm -v "$PWD":/out ghcr.io/drowzeys/keys-mac-tensorfold-minimax-h3-mlx:1.1 cp -a /payload/. /out/payload/

Ukuze wakhe kabusha futhi uthumele i-carrier, indawo yokugcina inikeza:

PUSH=1 bash scripts/build-carrier.sh

Qaphela ukusetshenziswa kwememori

Ukulayishwa kwemodeli nokuhlanganiswa kwe-adaptha kudla imemori eningi. Vala izinhlelo ezingadingekile ngaphambi kokuqala ukunikezwa, ikakhulukazi uma usebenzisa i-adaptha ye-Turbo. Indawo yokugcina ibika ukuthi ukuveza amatafula e-AdaLN kanye kuphela ekusebenzeni ngakunye nokukhulula izisindo konga cishe u-24 GiB endleleni elinganisiwe.

Khetha ikhanvasi efanele yesithombe ukuya kuvidiyo

Ngenxa yokuthi isithombe sokuqala sinwetshwa esikhundleni sokunqunywa ngokuzenzakalelayo ukuze kugcinwe ukwakheka kwaso, lungiselela isithombe somthombo ngokwesilinganiso sobukhulu esiqondiswe kuso. Lokhu kugwema ukuhlanekezelwa okungafuneki lapho ukhiqiza iziqeshana eziqondile, ezivundlile, noma eziyisikwele.

Sebenzisa ama-adaptha ahlukile ngokucophelela

I-loader ingafunda i-FastH3 isebenzisa indlela ye-adaptha kanye ne-POINTS=5, isibonelo nge-ADAPTER=.../adapter_model.safetensors. I-README ibika ukuthi le ndlela yahlolwa ku-864x480, yathatha imizuzwana engu-76 kuphrompti yokuhlola, futhi yabonakala imatasa kakhulu kune-Turbo. Yashicilelwa ku-1344x768, okungazange kuhlolwe kuphrojekthi.

Humusha ukuqhathaniswa kwekhwalithi ngendlela efanele

I-README igcizelela ukuthi ikhwalithi ayizange ihlolwe ngokuphelele. Indlela ye-int8 ingagcina ukwakheka kwe-bfloat16 kodwa iguqule imininingwane, futhi ukunikezwa kwe-Turbo kokudlulisela kathathu kungahluka ekunikezweni kwe-bfloat16 Turbo. Ngakho amathebula ebhentshimakhi aqhathanisa isivinini esilinganisiwe nezithombe ezimile ezikhethiwe, kunokuba aqinisekise ukunyakaza noma ikhwalithi yomsindo efanayo.

Ilayisensi nesimo sephrojekthi

Le phakheji ayithumeli izisindo zemodeli. Ngaphambi kokulanda i-MiniMax H3, funda i-MiniMax H3 Community License, okuhlanganisa nokungafakwa kwezinye izindawo. Phinda ubuyekeze amakhredithi endawo yokugcina futhi ucaphune ababhali bokuqala be-MiniMax, TensorFold, MLX, H3, i-adaptha, kanye namaphrojekthi ahlobene lapho kufanele khona.

Umndeni we-H3 osetshenziswe lapha uchazwa njengedrafti ehlongozwayo ye-pull request evela phezulu, kunokuba ube yingxenye yenguqulo ekhishiwe ye-TensorFold. Ngakho-ke ukubophela izinguqulo ezibhalwe phansi kubalulekile ukuze kusetshenziswe ukusetha okufanayo.

Isiphetho

Le ndawo yokugcina inikeza indlela egxile ekusebenziseni ukukhiqizwa kwevidiyo kwe-MiniMax H3 ngokwendabuko kuma-Mac ane-Apple Silicon ahambisanayo. Ngemva kokufaka indawo eboshiwe nokulanda imodeli, ungakhiqiza iziqeshana zombhalo ukuya kuvidiyo, uphilise ifreyimu yokuqala, ushintshe phakathi kwe-adaptha ye-Turbo esheshayo nendlela eyisisekelo yezinyathelo ezingu-20, futhi ulungise ikhanvasi, inani lamafreyimu, imbewu, kanye nezilungiselelo zeprompti. Lindela izidingo ezinkulu zememori nesitoreji, futhi thatha izikhathi ezishicilelwe njengezilinganiso zesistimu ye-M5 eshiwo, hhayi njengeziqinisekiso ezijwayelekile.