Yakha I-ejenti ye-iPhone Esekelwe ku-AI nge-ESP32-S3

Okungenziwa yi-ESP32 iPhone Agent

I-ESP32 iPhone Agent ivumela imodeli ye-AI ukuthi ibone futhi isebenzise i-iPhone yangempela. Isebenzisa ukuboniswa kwesikrini nge-AirPlay ukuze ithole okubonakalayo, kanye nebhodi le-ESP32-S3 elingabizi kakhulu elisebenza njengegundane nekhibhodi ye-USB. Ngenxa yokuthi isebenzisana nesikrini nesikhombi, ayidingi i-Xcode, i-akhawunti yonjiniyela yakwa-Apple, i-jailbreak, i-Developer Mode, noma uhlelo olusizayo olufakwe ocingweni.

Ikhompyutha ingasebenzisa i-macOS noma i-Linux. Isebenzisa i-UxPlay, iseva ye-Python, isixhumi esibonakalayo sewebhu, kanye nokuxhumana nemodeli yokubona ehambisana ne-OpenAI. I-ESP32-S3 ixhuma kukhompyutha nge-Wi-Fi, kuyilapho i-iPhone iyibona njengedivayisi evamile ye-USB Human Interface Device.

Indlela Uhlelo Olusebenza Ngayo

I-AirPlay inikeza amehlo

I-iPhone ibonisa isikrini sayo ku-UxPlay ekhompyutheni. Ipayipi le-GStreamer liguqula lokho kusakazwa kube amafreyimu e-JPEG asetshenziswa yi-ejenti ye-AI kanye nombono obukhoma osesipheqululini.

I-ESP32-S3 inikeza izandla

Uma ixhunywe ku-iPhone, leli bhodi liziphatha njengegundane nekhibhodi ye-USB. Uma i-iOS AssistiveTouch ivuliwe, ukuchofoza kwegundane kuba ukuthinta, ukudonsa isikhombi kuba ukuswayipha, isondo lokuskrola liskrola isixhumi esibonakalayo, kanti imiyalo yekhibhodi ingafaka umbhalo noma icindezele izinqamuleli.

I-firmware inenani elincane ngamabomu lomqondo wokusebenza. Iseva ithumela imiyalo esuselwe emigqeni efana ne-move, click, scroll, type, kanye ne-key nge-TCP. Ibhodi liphendula ngo-ok noma ngomlayezo wephutha.

Umhleli nomenzi wesenzo banikeza ubuchopho

Imodeli yomhleli yenza uhlelo olufushane ngokususelwa emgomweni oceliwe nasesikrinini samanje. Imodeli yomenzi wesenzo ihlola isithombe-skrini esisha ngasinye bese ikhetha isenzo esilandelayo, njengokuthinta, ukuswayipha, ukudweba indlela, ukuthayipha, ukucindezela ukhiye, ukubuyela ekhaya, noma ukuqeda. Imodeli efanayo yokubona ingasebenza kuzo zombili lezi zindima.

Ukuhlola okuyindilinga kuthuthukisa ukwethembeka

Ukunyakaza kwesikhombi kuhlobene nendawo yaso yangaphambilini, ngakho iseva ilandelela indawo yayo elinganisiwe. Ngaphambi kokuthinta, inganyakazisa kancane isikhombi, isithole esithombeni esibukiswayo, ilungise iphutha, futhi ithuthukise ukulinganisa. Ngemva kwesenzo, i-ejenti ihlola ukuthi isikrini sishintshile yini. Ngakho izenzo ezingaphumelelanga zingabonwa ziphindwe, esikhundleni sokuba yisisekelo sezinyathelo ezilandelayo.

Uhlelo lungaphinde lwenze uhlelo olusha ngemva kwezenzo eziphindaphindiwe futhi lululame ekunqamukeni kwezixhumanisi noma ezimpendulweni zemodeli ezingavumelekile. Uma uxhumano lwe-Wi-Fi lunqamuka, ibhodi likhulula izinkinobho zegundane bese lixhuma kabusha ngokuzenzakalelayo.

Izici Ezibalulekile

• Akukho ukusethwa kokuthuthukisa kwe-Apple: akudingeki i-Xcode, i-akhawunti yonjiniyela, ukusayinwa kwekhodi, i-Developer Mode, noma uhlelo lokuhlola ukusebenza.
• Ukuhambisana okubanzi nezinhlelo: i-ejenti isebenza ngamaphikseli esikhundleni sokuthembela esihlahleni sokufinyeleleka, ngakho ingasebenza ezinhlelweni nasemidlalweni evamile.
• Ikhompyutha esebenzisa i-Mac noma i-Linux: iseva ayigcini ku-macOS kuphela.
• Ukusekelwa kwamamodeli okuguquguqukayo: kungasetshenziswa noma iyiphi i-Chat API ehambisana ne-OpenAI futhi eyamukela izithombe.
• Ukulawula ngesandla nangokuzenzakalelayo: isixhumi esibonakalayo sewebhu sisekela ukuthinta okuqondile, ukuswayipha, ukuskrola, ukuthayipha, u-Home, u-Back, kanye nemisebenzi eqhutshwa yi-AI.
• Ukwenza kubonakale: isixhumi esibonakalayo sibonisa isikrini sefoni bukhoma, izinhlelo, amalogi, izithombe-skrini, izenzo, ukucabanga, kanye nolwazi lwesikhathi.
• Izilawuli zokuphepha: ukumisa okuphuthumayo kumisa i-ejenti futhi kukhulule isikhombi.

Okudingayo

• Ibhodi le-ESP32-S3: iphrojekthi ihlolwe nge-Seeed Studio XIAO ESP32S3 engaphethe lutho olwengeziwe. Elinye ibhodi le-ESP32-S3 lingasebenza uma imbobo yalo ye-USB-C ixhunywe kusixhumi esibonakalayo se-USB somdabu se-chip.
• iPhone: amamodeli ane-USB-C angaxhuma ngokuqondile. Amamodeli ane-Lightning adinga i-Lightning to USB Camera Adapter.
• Ikhompyutha: umshini osebenzisa i-macOS noma i-Linux kunethiwekhi efanayo ne-iPhone.
• Wi-Fi: ukumiswa kwebhodi kudinga igama nephasiwedi yenethiwekhi engu-2.4 GHz.
• Imodeli yokubona: indawo yokugcina esebenza efwini noma endaweni ehambisana ne-OpenAI eyamukela okokufaka kwezithombe.

I-iPhone inika ibhodi amandla. Uma ifoni kufanele ishaje ngesikhathi esifanayo, sebenzisa i-USB-C hub enokudlulisa amandla.

Faka Futhi Lungisa I-ejenti

1. Klona indawo yokugcina

git clone https://github.com/onehumanbeing/esp32-iphone-agent.git
cd esp32-iphone-agent

2. Lungisa futhi ufake i-ESP32-S3

Faka i-arduino-cli kanye ne-ESP32 core engu-3.x. Xhuma i-ESP32-S3 kukhompyutha nge-USB, kopisha ifayela lezimfihlo lesibonelo, bese uhlela ifayela elisha.

cp hardware/hid_bridge/secrets.example.h hardware/hid_bridge/secrets.h

Ku-hardware/hid_bridge/secrets.h, faka igama lenethiwekhi ye-Wi-Fi engu-2.4 GHz, iphasiwedi ye-Wi-Fi, kanye nekheli le-IP lekhompyutha ezosebenzisa iseva. Bese ufaka i-firmware ebhodini:

hardware/flash.sh

Iskripthi sokufaka i-firmware singamukela ngokuzikhethela imbobo ye-serial kanye nefulegi elithi --debug. I-firmware yokulungisa amaphutha iphinda ibonise imiyalo nobude bayo nge-USB serial.

3. Faka i-UxPlay

Ku-macOS, faka i-UxPlay nge-Homebrew:

brew install uxplay

Ku-Debian noma ku-Ubuntu, faka i-UxPlay namaphakheji adingekayo e-GStreamer:

sudo apt install uxplay gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-libav

4. Dala indawo ye-Python

Ngena kufolda yeseva, dala indawo ebonakalayo, bese ufaka iphrojekthi:

cd server
python3 -m venv .venv
.venv/bin/pip install -e .

Kudingeka i-Python 3.10 noma entsha.

5. Lungisa imodeli

Kopisha ithempulethi yemvelo:

cp .env.example .env

Hlela server/.env bese usetha okungenani i-endpoint ehambisana ne-OpenAI, igama lemodeli, kanye nokhiye we-API:

LLM_BASE_URL=https://your-endpoint.example/v1
LLM_MODEL=your-vision-model
LLM_API_KEY=your-api-key

Imodeli ekhethiwe kufanele isekele izithombe. I-API yefu iyanele; ukusebenzisa imodeli yasendaweni kuyazikhethela.

6. Qalisa iseva

set -a && . ./.env && set +a && .venv/bin/python -m hidagent

Vula http://<computer-ip>:8890 esipheqululini. I-IP yekhompyutha kufanele ifinyeleleke kunethiwekhi esetshenziswa ibhodi.

Lungisa i-iPhone

Lokhu kulungiselelwa kwefoni kuvame ukudingeka kanye kuphela:

1. Vula Settings, bese uye ku-Accessibility, Touch, naku-AssistiveTouch.
2. Vula i-AssistiveTouch.
3. Nqamula ibhodi eline-firmware kwikhompyutha bese ulixhuma ku-iPhone.
4. Qinisekisa ukuthi kuvela isikhombisi esiyindilinga nokuthi ikhasi lewebhu libonisa ukuthi isilawuli sixhunyiwe.
5. Vula i-Control Center ku-iPhone bese ukhetha i-Screen Mirroring.
6. Khetha i-iPhone Agent. Isibonisi esibukhoma sefoni kufanele sivele kusixhumi esibonakalayo sewebhu.
7. Esixhumi esibonakalayo sewebhu, vula Manual controls bese ukhetha Calibrate.
8. Chofoza ngqo esikhombisini esibonakalayo esikrinini esibukhoma.

Phinda ukulinganisa uma kamuva ushintsha isivinini sokulandelela se-AssistiveTouch.

Qalisa Umsebenzi Wakho Wokuqala

Faka umgomo esixhumi esibonakalayo sewebhu bese ucindezela Start. Isibonelo:

Open Settings and go to Wi-Fi

Umhleli wenza uhlelo olufushane, bese umlingisi ehlola kaningi isithombe-skrini samanje akhethe isenzo. Buka ifoni ebukhoma kanye nerekhodi ngesikhathi umsebenzi uqhubeka.

Udonga lwezinyathelo lugcina isithombe-skrini esisetshenziswe esinqumweni ngasinye. Luphinde ludwebe ukuthepha noma ukuswayipha okukhethiwe phezu kwesithombe, lubonise ukucabanga kwemodeli nesikhathi. Ukukhetha ikhadi lesinyathelo kuvula ukubuka okukhudlwana.

Sebenzisa izilawuli ezenziwa ngesandla

• Chofoza isithombe sefoni ebukhoma ukuze uthephe kuleyo ndawo.
• Donsa unqamule isithombe ukuze uswayiphe.
• Sebenzisa okokufaka kokuskrola ukuze uskrole ifoni.
• Sebenzisa izilawuli ezinikeziwe ze-Home, Back, ukufaka umbhalo, nokhiye.

Uma i-agent iziphatha ngendlela engalindelekile, vula Emergency stop noma ucindezele u-Esc. Lokhu kumisa ukusebenza futhi kukhulule isikhombisi. Isixhumi esibonakalayo siphinde sinikeze isilawuli sokusula umlando wokusebenza.

Izinketho Ezibalulekile Zokumisa

• LLM_BASE_URL: i-endpoint ehambisana ne-OpenAI. Okuzenzakalelayo ngu-http://localhost:8000/v1.
• LLM_MODEL kanye ne-LLM_API_KEY: igama lemodeli kanye nemininingwane ye-API.
• LLM_EXTRA_BODY: i-JSON ehlanganiswa kuzo zonke izicelo zemodeli, njengezilungiselelo ezivala imodi yokucabanga yemodeli.
• ACTOR_* kanye ne-PLANNER_*: izilungiselelo ezithile zendima ezishintsha izilungiselelo ezihambisanayo ze-LLM_*.
• AIRPLAY_NAME: igama lomamukeli eliboniswa ohlwini lwe-iPhone lwe-Screen Mirroring. Okuzenzakalelayo ngu-iPhone Agent.
• PORT: imbobo yesixhumi esibonakalayo sewebhu, okuzenzakalelayo kungu-8890.
• DEVICE_PORT: imbobo esetshenziswa yi-ESP32-S3, okuzenzakalelayo kungu-7300.
• HIDAGENT_DATA: ifolda yokulinganisa nokulandelela, okuzenzakalelayo kungu-~/.hidagent.
• HIDAGENT_DEBUG: ivula ukuloga kokulungisa iphutha ohlangothini lweseva uma isethelwe ku-1.

Ukusetshenziswa Okuthuthukile Nokuxazulula Izinkinga

Yabela amamodeli ahlukene umhleli nomlingisi

Umhleli nomlingisi bangasebenzisa imodeli efanayo, kodwa okuguquguqukayo kwemvelo okuqondene nendima kukuvumela ukuthi ushintshe izilungiselelo zemodeli ngayinye ngokuzimela. Sebenzisa izinhlobo ze-PLANNER_* ne-ACTOR_* zeziguquguquko ezifanele ze-LLM_* lapho kudingeka ama-endpoint noma amamodeli ahlukene.

Xhuma iseva yemodeli yasendaweni

Noma iyiphi iseva ehambisana ne-OpenAI enemodeli yokubona ingalungiswa ngendlela efanayo, okuhlanganisa i-vLLM, i-SGLang, i-llama.cpp, ne-Ollama. Iphrojekthi yathuthukiswa futhi yalinganiswa kusetshenziswa i-NVIDIA Qwen3.8-Flash-Next-NVFP4 ehlinzekwa ama-NVIDIA DGX Sparks amabili kusetshenziswa i-tensor parallelism.

Isibonelo sokumisa lokho yilesi:

LLM_BASE_URL=http://<dgx-spark-ip>:8888/v1
LLM_MODEL=Qwen3.8-Flash-Next
LLM_EXTRA_BODY='{"chat_template_kwargs": {"enable_thinking": false}}'

Ukumiswa kokuthuthukiswa okubikiwe kwafinyelela cishe kumasekhondi angu-1.3 ngesenzo ngasinye namasekhondi angu-2 ngohlelo ngalunye, imodi yokucabanga ivaliwe. Lezi zibalo zichaza leyo hardware nokumiswa kwemodeli okuthile, hhayi isivinini esiqinisekisiwe kokunye ukuthunyelwa.

Vula ukuqoqwa kokulandelela

Setha leli nani ku-server/.env:

HIDAGENT_DEBUG=1

Imodi yokulungisa iphutha iloga yonke imiyalo yebhodi kanye nokubambezeleka kwayo. Iphinde idale ifolda eyodwa yokusebenza ngakunye kwe-agent ngaphansi kwe-~/.hidagent/traces/. Ifolda iqukethe i-steps.jsonl, ehlanganisa imiyalelo yemodeli, izimpendulo ezingacutshungulwanga, izilinganiso zokubambezeleka, nezenzo, kanye nezithombe-skrini ezibonwe izingcingo zemodeli.

Qonda iphrothokholi yebhodi

• ping yenza ukuhlolwa kokusebenza kahle.
• home iphushela isikhombisi ekhoneni eliphezulu kwesokunxele ukuze kusungulwe indawo eyaziwayo.
• move <dx> <dy> ihambisa isikhombisi ngokuhlobene ngesivinini esingaguquki.
• press, release, kanye ne-click kulawula inkinobho yegundane yesokunxele.
• scroll <v> <h> ithumela okokufaka kokuskrola okuqondile nokuvundlile.
• type <text> ifaka umbhalo we-ASCII.
• key <combo> ithumela okhiye noma izinhlanganisela ezifana ne-enter, cmd+h, noma cmd+space.

Lungisa izinkinga zokutholwa kwe-AirPlay ku-Linux

Uma ikhompuyutha inezixhumi eziningi zenethiwekhi futhi i-iPhone ingamfaki ohlwini umamukeli, khawulela ukukhangisa kwe-Avahi mDNS kusixhumi esixhunywe kunethiwekhi yocingo.

Imikhawulo Eyaziwayo

• I-iOS ayibonisi ukufakwa kwekhodi eyimfihlo, i-Apple Pay, amavidiyo avikelwe nge-DRM, nezinye izikrini ezivikelwe, ngakho i-ejenti ayikwazi ukuzibona.
• Ukuboniswa kwesikrini kuyama uma ucingo lukhiya. Gcina ukukhiya okuzenzakalelayo kuvaliwe ngenkathi i-ejenti isebenza.
• Ukufakwa kombhalo ngekhibhodi kukhawulelwe ku-ASCII.
• Ukulinganisa isikhombisi kufanele kuphindwe ngemva kokushintsha isivinini sokulandelela se-AssistiveTouch.
• I-ejenti incike endleleni yenethiwekhi esebenzayo phakathi kwebhodi neseva kanye nokusakazwa kwe-AirPlay okutholakalayo phakathi kocingo nekhompuyutha.

Ukusebenza Ngendlela Enesibopho

Yenza ngokuzenzakalela kuphela amadivayisi nama-akhawunti okungewakho noma ogunyazwe ukuwasebenzisa, futhi ulandele imigomo yezinhlelo zokusebenza ezihilelekile.

Ngenxa yokuthi i-ejenti ingasebenzisa izinhlobo ezifanayo zokufaka njengomuntu, gada ukusebenza okuqhubekayo futhi ugcine inkinobho yokumisa eziphuthumayo ifinyeleleka.

Isiphetho

I-ESP32 iPhone Agent ihlanganisa ukuboniswa kwesikrini nge-AirPlay, ibhuloho le-USB HID elingabizi elisebenzisa i-ESP32-S3, kanye nemodeli yombono ehambisana ne-OpenAI ukuze isebenze ngokuzenzakalela ku-iPhone ephathekayo ngaphandle kohlelo olujwayelekile lokuthuthukisa lwe-Apple. Uma i-firmware, iseva, indawo yokugcina yemodeli, i-AssistiveTouch, ukuboniswa kwesikrini, nokulinganiswa kwesikhombisi sekulungisiwe, ungakhipha imigomo ngolimi lwemvelo, uhlole zonke izinqumo, ungenelele ngesandla, futhi uqoqe iminonjana enemininingwane yokulungisa amaphutha.