Indlela Yokusebenzisa Umcibisholo Omkhulu Ukuqondisa Abantu Ezinhlelweni Zokusebenza Ze-Mac Eziqhutshwa yi-AI

Okwenziwa i-big-arrow-on-the-screen

big-arrow-on-the-screen, ebizwa nangokuthi bigarrow, iyithuluzi lomugqa womyalo le-macOS elidweba umcibisholo nophawu olubonakala kakhulu phezu kwawo wonke amawindi. Lakhelwe ama-agent e-AI angenza imisebenzi eminingi yekhompyutha kodwa okufanele abuyisele izinqumo ezithile kumuntu.

Isibonelo, i-agent ingakhomba inkinobho ethi Allow ku-System Settings, ikhombe iwindi lesiphequluli elilungile, noma iqondise umuntu enqubweni enezinyathelo eziningi. Umsebenzisi uhlala elawula: i-bigarrow ayilokothi ichofoze, ibhale, noma ithwebule isikrini.

Isendlalelo siboniswa phezu kwamawindi ajwayelekile, izingxoxo, izinhlelo ezisebenza kusikrini esigcwele, nama-Spaces, kuyilapho sigcina ukugxila kwekhibhodi kuhlelo lokusebenza oselunakho. Ukuchofoza kulokho okuqondisiwe kudlulela kuhlelo lokusebenza olungaphansi. Umcibisholo uyazisusa ngokuzenzakalelayo lapho isikhathi sawo siphela noma inqubo engumnikazi wawo iphuma.

I-agent yakho ingachaza okufanele kwenziwe, bese ibeka umunwe obonakalayo phezu kwesilawuli esiqondile umuntu okufanele asisebenzise.

Isikhathi sokuyisebenzisa

bigarrow iwusizo noma nini lapho i-agent ye-AI idinga ukuthi umuntu enze isinqumo noma aqedele isenzo. Izibonelo ezivamile zihlanganisa izicelo zemvume, izikrini zemvume ye-OAuth, ukuqinisekiswa kwezinyathelo ezimbili, ama-CAPTCHA, ama-passkey, izinkokhelo, amasignesha, nokuqinisekiswa kwezomthetho.

Ingakwazi futhi ukwenza imiyalelo icace uma kubonakala amawindi noma izilawuli eziningana ezifanayo. I-agent ingakhetha uhlelo lokusebenza, iwindi, noma ithebhu yesiphequluli ethile esikhundleni sokucela umsebenzisi ukuthi afune phakathi kwamawindi amaningi avuliwe.

Okunye ukusetshenziswa kufaka ukufundisa umuntu ukusebenzisa uhlelo lokusebenza, ukunikeza usizo olukude, ukugqamisa izilawuli ngesikhathi sokurekhoda isikrini, nokuhlola ukuthi izixhumanisi ezibuyiswe elinye ithuluzi le-automation zilungile yini.

Faka ku-macOS

Le phrojekthi isekela i-macOS 14 noma entsha. Indlela elula yokuyifaka ukusebenzisa i-Homebrew:

brew install franzenzenhofer/tap/bigarrow

Uma ufuna i-Claude Code ne-Codex zazi ukuthi leli thuluzi lisetshenziswa nini futhi kanjani, faka ne-agent skill yalo:

bigarrow install-skill

Lokhu kufaka i-skill ye-Claude Code ngaphansi kwe-~/.claude/skills neye-Codex ngaphansi kwe-~/.agents/skills. Ungaphinda ubize i-bigarrow ngokuqondile kushell ngaphandle kokufaka i-skill.

Yakha kusukela emthonjeni

Ukuhlanganisa le phrojekthi ngokwakho, sebenzisa i-Xcode 16 noma entsha:

swift build -c release

I-binary ephumayo itholakala ku-.build/release/bigarrow. Uhlelo lokusebenza ngokwalo lubhalwe nge-Swift. Amafayela e-shell ne-Python aku-scripts/ asetshenziselwa izithombe-skrini nokuhlola.

Imiyalo yakho yokuqala yokukhomba

Amaphethini amathathu ayisisekelo ukukhomba into yokufinyeleleka, ukukhomba izixhumanisi, nokugcina umcibisholo ubonakala uze uwumise.

Khomba isilawuli esinelebula

bigarrow point --element "Allow" --app "System Settings" --text "Franz, click Allow: Ghostty may control your Mac"

--element ifanisa isilawuli ngelebula yaso, kuyilapho --app ikhomba uhlelo lokusebenza oluqukethe sona. Uphawu luchaza lokho umsebenzisi acelwa ukuba akwenze.

Khomba izixhumanisi

bigarrow point --at 760,500 --text "Franz, click HERE"

Izixhumanisi zisebenzisa amaphoyinti anengqondo omhlaba aqala phezulu kwesokunxele, indawo efanayo ebikwa yi-macOS Accessibility, i-CGWindowList, ne-Peekaboo. Ungakhomba nerectangle:

bigarrow point --rect 600,400,240,80 --text "This is the area to review"

Gcina umcibisholo ubonakala kuze kufike isinyathelo esilandelayo

bigarrow start --window "Safari:Inbox" --text "This window"
bigarrow stop

start ibuya ngokushesha futhi ishiye umcibisholo ubonakala. Sebenzisa stop ukususa umcibisholo wamanje noma stop --all ukususa yonke imicibisholo.

Umyalo we-point osebenza kanye uhlala imizuzwana eyisishiyagalombili ngokuzenzakalelayo. Shintsha lokho nge---duration; i-start izenzakalela ibe yimizuzwana engu-300. Sebenzisa --duration 0 ukuze kungabi nomkhawulo wesikhathi.

Khetha okuqondiswe kukho okufanele

bigarrow isekela imithombo eminingi yalokho okuqondiswe kukho:

• --at X,Y ikhomba isixhumanisi.
• --rect X,Y,W,H igqamisa indawo eyi-rectangular.
• --mouse ikhomba indawo yegundane yamanje.
• --window App[:title] ikhomba iwindi.
• --element Label --app App ithola into yokufinyeleleka.
• --peekaboo ID --snapshot see.json isebenzisa i-ID yento evela kumphumela we-Peekaboo othi see --json.

Ukuhlola izilawuli ezivezwa uhlelo lokusebenza, sebenzisa:

bigarrow elements --app "System Settings"

Kumadispleyi amaningi, --display N yenza amanani ezixhumanisi nawerectangle abe maqondana nedispleyi eyodwa. Kusekelwa nezixhumanisi zomhlaba ezingezinhle kumadispleyi abekwe kwesokunxele noma ngaphezu kwedispleyi enkulu.

Khomba iwindi noma ithebhu yesiphequluli elungile

Uma unikeza --app App[:window or tab title] noma --window, i-bigarrow ingaletha uhlelo lokusebenza, iwindi, noma ithebhu ye-Chrome/Safari efanele ngaphambili ngaphambi kokudweba. Lokhu kuvimbela umcibisholo onembe ngenye indlela ukuthi uvele phezu kwewindi elibonakalayo elingalungile.

bigarrow point --app "Google Chrome:Pull request" --text "Buyekeza lesi sicelo sokuhlanganisa"

Sebenzisa --no-raise uma ufuna ukushiya amawindi omsebenzisi esesimweni sawo. Uma olunye uhlelo luvala okuqondiwe kamuva, umcibisholo uyacasha kuze kubonakale okuqondiwe futhi.

Yenza imicibisholo icace futhi ibe wusizo

Ungenza ngendlela oyifisayo umumo, isitayela, usayizi, amakhona, imibala, nokuziphatha komcibisholo:

• --shape bend|straight|zigzag|spiral ikhetha indlela yomcibisholo.
• --style arrow|ring|box ikhetha umcibisholo ojwayelekile, iringi engenamaphakathi enomngcele kuphela, noma ibhokisi elinomngcele kuphela.
• --size S|M|L ishintsha usayizi.
• --corners round|sharp ilawula amakhona ophawu.
• --color red|orange|yellow|green|teal|blue|purple|pink|black|white|#RRGGBB isetha umbala oyinhloko.
• --border shadow|white-black|black ishintsha indlela yomngcele.
• --border-color, --text-color, kanye ne---edge-color kwenza imibala ngayinye ibe ngokwezifiso.
• --close-button yengeza u-X ochofozekayo ophawini.

Isibonelo, lo myalo udweba ibhokisi eliluhlaza eligqamile elizungeze okuqondiwe:

bigarrow point --at 760,500 --style box --color green --text "Buyekeza lesi silawuli"

Inkinobho yokuvala kufanele uyikhethe ngokuzithandela futhi inika umuntu indlela eqondile yokususa isendlalelo:

bigarrow start --element "Save" --app "TextEdit" --close-button --text "Chofoza u-Save uma usulungele"

Izinketho zokusebenzisana eziwusizo

Izinketho eziningana zisiza ukuxhumanisa isendlalelo nokugeleza komsebenzi okuqondisiwe:

• --follow ihambisa umcibisholo kanye newindi noma into.
• --until-click iphetha umcibisholo ngemva kokuchofozwa kokuqondiwe.
• --say ifunda uphawu ngokuzwakalayo, okuwusizo uma umsebenzisi engase angabheki isikrini.
• --json inikeza okukhiphayo okufundeka yimishini kuma-agent namaskripthi.

Ngakho-ke ukulandelana okuqondisiwe kungakhomba isilawuli esisodwa, kulinde umsebenzisi enze okuthile, kumise umcibisholo, bese kukhomba isilawuli esilandelayo. Ithuluzi alizenzi ngokwalo ukuchofoza.

Qonda izimvume

Ukudweba isendlalelo akudingi imvume ye-macOS ye-Screen Recording noma ye-Accessibility. Nokho, ezinye izindlela zokuthola noma ukwenza okuqondiwe kusebenze zidinga izimvume.

• --at, --rect, --mouse, --window App, --peekaboo, kanye ne---app App azidingi imvume.
• --element, elements, --until-click, kanye ne---app App:title kudinga i-Accessibility.
• --window App:title idinga i-Screen Recording ukuze ifunde izihloko zamawindi kanye ne-Accessibility ukuze iphakamise iwindi, ngaphandle kokuthi usebenzise --no-raise.

i-macOS inika lezi zimvume uhlelo olwethule i-bigarrow, njenge-Terminal, iTerm2, Ghostty, VS Code, noma i-Claude, esikhundleni se-bigarrow binary uqobo.

Qalisa umyalo wokuxilonga ukuze ubone izimvume ezifanele, uhlelo olunazo, kanye nezibonisi ezixhunyiwe:

bigarrow doctor

Uma imvume edingekayo ingekho, umyalo uphuma ngekhodi yesimo 4 futhi ukhomba uhlelo kanye nephaneli yezilungiselelo okuthintekayo.

Sebenzisa ukuhlanzwa okuzenzakalelayo ngokuphepha

Imicibisholo yakhelwe ukuthi ingahlali ngemva kokuqeda kwe-agent. Umyalo we-point onesikhathi uyazisusa ngokuzenzakalelayo. Umcibisholo nawo uyaphela lapho inqubo ye-agent ewudalile iphuma, kusetshenziswa i-CLAUDE_PID noma i-BIGARROW_OWNER_PID uma ikhona.

Kokugeleza komsebenzi kwe-Claude Code, i-bigarrow stop --hook ingasetshenziswa njenge-hook ye-UserPromptSubmit ukuze isuse imicibisholo yaleso seshini lapho umuntu ephendula. Imiyalo ecacile yokuhlanza yile:

bigarrow stop
bigarrow stop --all

Lezi zivikelo ziwusizo kuma-agent angase aphazamiseke noma athathelwe indawo ngaphambi kokufinyelela esinyathelweni sawo esivamile sokuhlanza.

Sebenzisa i-JSON namakhodi okuphuma ekugelezeni komsebenzi kwama-agent

Wonke umyalo wamukela --json, okwenza kube lula ukuhlola imiphumela ngohlelo. Amakhodi okuphuma abhaliwe yilawa:

• 0: umyalo uqedwe ngempumelelo.
• 2: okokufaka okungavumelekile.
• 3: okuqondiwe akutholakalanga.
• 4: imvume edingekayo ayikho.

I-agent ingasebenzisa le miphumela ukunquma ukuthi iphinde izame ngenye indlela yokuqondisa, icele umsebenzisi imvume, noma ichaze ukuthi isilawuli esiceliwe asitholakalanga.

Ukulungisa okuqondiwe okuthuthukile

Uma ungaqiniseki ukuthi izixhumanisi ezivela ku-Accessibility, Peekaboo, noma kwelinye ithuluzi zilungile yini, sebenzisa ukuhlola ngaphandle kokwenza ushintsho:

bigarrow point --at 760,500 --dry-run --json

Lokhu kubika lapho ithuluzi belizokhomba khona ngaphandle kokudweba isendlalelo. Kuwusizo ekuqinisekiseni ukuguqulwa kwezixhumanisi, ukukhethwa kwesibonisi, kanye nemiphumela yokuzenzakalela ngaphambi kokubonisa noma yini kumsebenzisi.

I-Peekaboo inganikeza omunye umthombo wokuqondiwe. Thatha isithombe-skrini se-JSON se-Peekaboo, bese udlulisa i-ID yento ehambisanayo:

bigarrow point --peekaboo ELEMENT_ID --snapshot see.json --text "Buyekeza le nto"

Ku-Chrome, izici zekhasi zingase zidinge ifulegi le-Chrome elithi --force-renderer-accessibility noma i-VoiceOver. Ibha yamathuluzi ye-Chrome uqobo isatholakala ngendlela evamile yokufinyeleleka. Uma okuqukethwe kwekhasi kungavezwa, sebenzisa izixhumanisi zendawo esikrinini esikhundleni salokho.

Indlela ama-agent okufanele asebenzise ngayo le skill

I-skill efakiwe isekela kokubili i-Claude Code ne-Codex nge-SKILL.md eyodwa ehambisana ne-Agent Skills, kanye nemethadatha ye-Codex eku-agents/openai.yaml. Ifundisa i-agent ukuthi ikhombe nini, ikhethe kanjani okuqondiwe, ibhale kanjani umusho ophelele ophawulweni, isebenzise nini i---say, nokuthi ime nini ngemva kokuba umsebenzisi esesenzile isenzo.

Uchungechunge olusebenzayo lwezenzo ze-agent lungabukeka kanje:

1. Thola isenzo esidinga ukuqinisekiswa noma usizo lomuntu.
2. Khetha ilebula, iwindi, ithebhu, indawo, unxande, noma okuqondiwe kwe-Peekaboo.
3. Bhala uphawu olusho ngokuqondile lokho umsebenzisi okufanele akwenze nokuthi kungani.
4. Veza uhlelo lokusebenza okuqondiwe kulo, ngaphandle kwalapho ukugcina ukwakheka kwamanje kubalulekile.
5. Lindela umuntu ukuthi enze isenzo noma aphendule.
6. Misa umcibisholo ngaphambi kokuqhubekela esinyathelweni esilandelayo.

Lokho leli thuluzi elingakwenzi

bigarrow inemikhawulo ngenhloso. Ayilona ithuluzi lokubeka izichasiselo esikrinini kubantu, i-bot yokuchofoza, noma ithuluzi lokuthatha izithombe-skrini. Ayilokothi ichofoze, ibhale, noma ithwebule isikrini. Inhloso yayo ukwenza umyalelo wohlelo ubonakale, kuyilapho isinqumo nesenzo kushiyelwa kumuntu.

Izitayela zendandatho nebhokisi zisebenzisa imigqa yangaphandle ukuze okuqondiwe kuhlale kubonakala. Uphawu lubekwa kude nalokho okuqondiwe uma kunendawo, futhi inkinobho yokuvala, uma inikwe amandla, ivumela umuntu ukuba avale umcibisholo. Lezi zinqumo zokuklama zenza leli thuluzi lifanele izenzo ezibucayi njengezimvume, ukuqinisekiswa kobunikazi, izinkokhelo, nokukhishwa kwezinhlelo.

Isiphetho

bigarrow inikeza ama-agent e-AI asekelwe ku-macOS indlela elula yokubuyisela ukulawula kumuntu ngaphandle kokusebenzisa imilayezo yetheminali engacacile. Yifake nge-Homebrew, engeza i-Claude Code noma i-Codex skill uma kudingeka, bese usebenzisa i-point emiyalweni emifushane noma i-start ne-stop ekuhambeni komsebenzi okuqondisiwe.

Ngezinto eziqondiwe ezifana nelebula, iwindi, indawo, unxande, igundane, ne-Peekaboo, kanye nezitayela ezenziwa ngokwezifiso, inkulumo, okukhiphayo kwe-JSON, ukuhlanzwa okuzenzakalelayo, nokusekelwa kwezikrini eziningi, leli thuluzi liguqula ukuthi “sicela uchofoze lokhu” kube isiqondiso esinembile esisesikrinini, kuyilapho isenzo sokugcina silawulwa ngumuntu.