Yakha Ifilimu Yetheminali Ye-AI Ekhiqizwe Ngekhodi nge-ff-tracking

ff-tracking yifilimu emfushane ekhiqizwe ngekhodi, lapho ikhamera ebonakalayo ilandela khona i-ejenti ye-AI ebhala kutheminali. Isendlalelo esifana nombono wekhompyutha sigxila kuma-glyph ngamunye, kuyilapho umbono, ukujula kwenkundla, ukucutshungulwa kwemibala, ukukhanya okuchichimayo, ukuphazamiseka kwemigqa yokuskena, umsindo wokusanhlamvu, nomsindo kukhiqizwa ngekhodi.

Le phrojekthi yakhiwe nge-Rust, fframes, Skia Metal, SkSL, FFmpeg, ne-Python. Ikhiqiza ividiyo yemizuzwana eyisithupha ngezigaba ezimbili ezihambisanayo, lapho imiphumela ebonakalayo nomsindo kuqhutshwa khona idatha efanayo yokulandelela.

Okukhiqizwa yiphrojekthi

Ifilimu iveza uchungechunge lwemilayezo yetheminali efana nokuthi thinking…, reading 14 files, tool_call: render_frame(), kanye no-✓ 0 problems. Umlandeli ulandela izinhlamvu ezintsha futhi ubonisa izixhumanisi zazo zangempela ngamaphikseli, abakaki abaqondisiwe, amalebula, nezixhumanisi phakathi kwamabhokisi.

• Izixhumanisi zangempela: amalebula omlandeleli asebenzisa indawo yangempela yebhokisi ngalinye kuhlaka lokuphumayo.
• Ukugxila okuhambayo: okuqondiswe kukho ukujula kwenkundla kulandela ibhokisi elilandelwayo njengoba i-ejenti ibhala.
• Ukulandelela okupopayiwe: izixhumanisi zixhumanisa amabhokisi ngokulandelana kokufundwa, nechashazi elihamba kuxhumano ngalunye.
• Imiphumela yokukhiya: igama elisha kakhulu liba bomvu amafreyimu amabili, lithole ukuphazamiseka komugqa wokuskena, futhi liqalise umsindo okhishwa ngokusabalalisa kwendawo.
• Ukubukeka okuningi: izithombe eziyishumi nambili zisebenzisa amaphalethi e-Paper, Thermal, Mono, Phosphor, Amber, Navy, ne-Neon.
• Ukugoqa kokugcina: amabhokisi ayishumi nane ahlanganiswa abe ibhokisi elilodwa eliba iphilisi le-Done ngaphambi kokufiphala kube mnyama.

Qonda inqubo enezigaba ezimbili

Iphrojekthi ihlukanisa ukwakheka kwetheminali nemiphumela yekhamera nelensi. Lokhu kwenza indawo ye-glyph ngayinye inqunywe ngokungaguquki, kuyilapho isigaba sesibili siphatha itheminali ekhiqiziwe njengokufakwayo kwe-shader.

1. Idatha ye-HUD: i-crate ye-hud ichaza izigcawu, amabhokisi omlandeleli, ukunyakaza kwekhamera, nedatha yokuqagela.
2. Isigaba esiyisicaba: ibhinari ye-flat ikhiqiza itheminali njenge-SVG ku-3840×2160. Ibeka into eyodwa ye-<text> ku-glyph ngayinye ukuze izindawo zaziwe ngokunembile.
3. Isigaba selensi: ibhinari ye-lens isampula ividiyo eyisicaba njenge-iChannel0 futhi isebenzisa ikhamera ebonakalayo nemiphumela ye-SkSL.
4. Isigaba somsindo: i-tools/sfx.py ifunda i-out/tracks.json futhi ikhiqiza imicimbi yomsindo ehambisana nokuthayipha, ukusikeka, ukukhiya, nokufika kokugcina.
5. Ukufaka ikhodi: i-FFmpeg ihlanganisa ividiyo ecubunguliwe nomsindo ibe yi-out/tracking.mp4.

I-lens shader iphatha umbono, ukujula kwenkundla, ukumepha amaphalethi, ukuphambuka kwe-chromatic, ukukhanya okuchichimayo, igridi ye-LED, ukuphazamiseka, umsindo wokusanhlamvu, kanye nesendlalelo somlandeleli esikhaleni sekhamera. Umphumela wokujula kwenkundla usebenzisa amasampula angu-36 e-golden-angle, kuyilapho izindawo ezikhanyayo zithola ukukalwa okufana ne-bokeh.

Faka izimfuneko

Ukukhiqiza kudinga i-Mac ene-Apple silicon ngoba iphrojekthi isebenzisa i-Skia ku-Metal. Faka amathuluzi alandelayo ngaphambi kokwakha:

• i-Rust esekela uhlelo luka-2024
• i-FFmpeg
• i-Python 3
• amaphakheji e-Python i-numpy ne-scipy
• i-Pillow ne-img2webp uma ufuna ukwakha kabusha imidiya ye-README

Yenza i-clone yendawo yokugcina bese uya kuhla lwemibhalo yayo:

git clone https://github.com/mrsarac/ff-tracking
cd ff-tracking

Khiqiza ifilimu ezenzakalelayo

Isikripthi esikhulu sakha i-workspace ye-Rust, sikhiphe idatha yokulandelela, sikhiqize isigaba esiyisicaba, sikhiqize umsindo, bese sikhiqiza isigaba selensi:

tools/render.sh

Ividiyo ephelele ibhalwa ku-out/tracking.mp4. Ku-M3 Pro, ukukhiqiza kuthatha cishe imizuzwana engu-40 ngemva kokwakhiwa kokuqala.

Idatha ekhiqiziwe ihlanganisa i-out/tracks.json, echaza imicimbi yokulandelela futhi eqhuba ukwakheka komsindo okuzenzakalelayo. Ibhinary yekhamera ongayisebenzisa nayo ibhala i-out/camera.json, equkethe ukusondeza, ukutshekisa, ukuzungezisa, indawo yebhokisi elilandelwayo, nolwazi lwendawo emaphakathi yesikrini kuhlaka ngalunye.

Hlola amafreyimu, umsindo, nezigaba ze-shader

Njengoba isigaba ngasinye siyi-fframes CLI, ungahlola okukhiphayo ngemiyalo ejwayelekile:

target/release/lens strip all -n 12
target/release/lens frame 81 -o frames
target/release/lens preview
target/release/lens audio analyze

Sebenzisa i-strip ukuze wakhe ishidi lokubuka elihlanganisiwe, i-frame ukuze ukhiphe uhlaka olugcwele, kanye ne-preview ukuze uvule ukubuka kwangempela okunomsindo. Umyalo wokuhlaziya umsindo ubika ulwazi lokuphakama komsindo ne-true peak.

Ukuze umise i-shader ngemva kwesigaba esithile, setha i-LENS_STAGE. Izigaba ezisekelwayo zibalwa kusukela ku-0 kuya ku-3:

LENS_STAGE=1 target/release/lens frame 81

Lokhu kuyasiza lapho ulungisa uguquko olusuka esithombeni somthombo luye ekwelashweni kokugcina kwelensi.

Yenza izigcawu ngendlela oyifunayo

Isithombe ngasinye siyinto eku-SCENE_LIST ngaphakathi kwe-hud/src/lib.rs. Isigcawu sichaza uhlobo lwaso, umbhalo, iphalethi, ikhamera, indawo yokuqala, nosayizi wefonti:

SceneDef {
    kind: Kind::Typing,
    text: "thinking…",
    palette: Palette::Thermal,
    cam: cam(0.38, -0.22, 0.04, 2.5, 2.8, 50.0, 0.0, 1.2),
    hero: (560.0, 540.0),
    size: 104.0,
},

Shintsha umbhalo, khetha elinye iphalethi, noma lungisa amanani ekhamera, bese unikeza futhi:

tools/render.sh

I-tracker, ithagethi yokugxila, amalebula, namabhiphu okukhiya kulandela ngokuzenzakalelayo idatha yesigcawu ebuyekeziwe. Amagama amaphalethi atholakalayo ahlanganisa Paper, Thermal, Mono, Phosphor, Amber, Navy, kanye ne-Neon.

Qalisa izivivinyo ze-HUD ngemva kokuhlela izigcawu:

cargo test -p hud

Lezi zivivinyo zihlola ukuthi umugqa ngamunye uyangena yini esikrinini nokuthi ikhamera igcina wonke amatargethi engaphakathi kwefreyimu.

Sebenzisa ingoma ezenzakalelayo noma ozikhethela yona

Ingoma ezenzakalelayo yenziwa yi-tools/sfx.py. Ihlanganisa umsindo ophansi wesikrini, ama-glitch ancishiswe ngamabhithi lapho kusikwa, ukuchofoza kokucindezelwa kokhiye, namabhiphu okukhiya e-pentatonic aqondiswe ebhokisini elilandelelwayo. Iphinde ingeze i-riser lapho kuqoqwa khona isithombe, kanye ne-chime enomsindo ophansi lapho kuvela i-pill ye-Done. Ukuphakama komsindo kukalwa nge-ebur128 ye-FFmpeg bese kusethwa ku--14 LUFS.

Indawo yokugcina nayo ihlanganisa ingoma emnyama ozikhethela yona ye-Slab ku-120 BPM. Ilandela idatha efanayo yokulandelela nekhamera, ngakho ama-lock blip, ukucindezelwa kokhiye, amaphedi, ama-riser, ama-glitch, ukuhlungwa komsindo ophansi, nokudabuka komsindo ophansi kusabela ku-animation.

Sebenzisa ukuhumusha kwengoma okugcinwe nge:

tools/score.sh

Ukuze uphinde uhumushe ingoma, nikeza i-Slab checkout eyakhiwe nge-zig build -Doptimize=ReleaseFast:

SLAB=/path/to/slab tools/score.sh

I-script yengoma ibhala ku-score/ff_tracking.slab, kanti umsindo ohunyushiwe utholakala njenge-score/score.wav. I-pipeline yengoma idlulela ngale kwesithombe ngenxa yomsila we-reverb, bese isika umphumela ube imizuzwana eyisithupha futhi iwufade kanye nevidiyo.

Qaphela ukuthi i-score/score.py ingenisa i-slabkit, esatshalaliswa ne-Slab ngaphansi kwe-GPL-3.0. I-Slab idingeka kuphela ukuze uphinde uhumushe ingoma; amafayela engoma agciniwe anikezwa ngaphansi kwelayisensi ye-MIT yendawo yokugcina ngokwe-README.

Yakha kabusha imidiya yamadokhumenti

Indawo yokugcina ihlanganisa i-script yokukhiqiza kabusha izithombe ze-README. Yiqalise nge-Python ngemva kokufaka okuncikile kwezithombe:

python3 -I tools/readme_media.py

Isakhiwo sephrojekthi

• hud/: izigcawu, amabhokisi e-tracker, idatha yekhamera, kanye nomqondo wokuphrojektha.
• flat/: ukudluliswa kokuqala kwetheminali ku-3840×2160.
• lens/: ukudluliswa kwesibili, okuhlanganisa lens/shaders/lens.sksl ne-tracker esikhaleni sekhamera.
• tools/: i-script eyinhloko yokuhumusha, umkhiqizi womsindo, umkhiqizi wemidiya ye-README, kanye ne-pipeline ye-Slab.
• score/: ingoma ye-Slab ozikhethela yona, iphrojekthi, kanye nefayela le-WAV elihunyushiwe.
• docs/: izincazelo zomklamo, ama-prompt, izinqumo, kanye neminikelo ye-fframes.

Amanothi okusetshenziswa athuthukile

Umklamo wokudlulisela kabili ubalulekile ngoba i-shader okwamanje ayikwazi ukusampula i-subtree ye-SVG kusukela kufreyimu efanayo. Kunalokho, ifreyimu eyisicaba ephelele iyavumelaniswa bese idluliselwa ku-shader njengokufakwa kwesithombe:

.image("iChannel0", &frame.get_synced_video_frame(..)?.into_image())

I-README iphinde ibhale imikhawulo eminingana esebenzayo:

• flat igodliwe ku-SkSL, ngakho gwema ukuyisebenzisa njengegama lomsizi.
• Ezingeni le-Video, buyisa i-root <svg> esikhundleni se-Svgr::empty(); i-root engenalutho ingavela njengephutha lokuhumusha elivela sekwephuzile.
• Umsebenzi we-CPU wokuphrojektha i-hud::project ugcina i-tracker overlay iqondaniswe nekhamera ye-shader.
• Kokubili ukudluliswa okubonakalayo nomkhiqizi womsindo kufunda ulwazi olufanayo lokulandelela lwe-HUD, okuvimbela ukugxila nemicimbi yomsindo ukuthi kuphambuke kuma-glyph.

Isiphetho

i-ff-tracking ikhombisa ukuthi ukulandelana okufushane kwesinema kungakhiqizwa ngokuphelele ngedatha yesigcawu ehlelekile, i-SVG, i-Rust, i-SkSL, ukuhumusha kwe-GPU, kanye nomsindo owenziwe nge-synthesis. Qala nge-tools/render.sh, hlola amafreyimu aphakathi, bese uhlela i-SCENE_LIST ukuze udale eyakho imiyalezo yetheminali, amaphalethi, ukunyakaza kwekhamera, kanye nezinhlanganisela ezilandelelwayo. Sebenzisa i-pipeline ye-Slab ozikhethela yona lapho ufuna ingoma elandela isikhathi esifanayo nokunyakaza kwekhamera.