Isifundo Somhloli Wombhalo we-AI wesiShayina: Ukuqagela Kwasendaweni, Ukuqeqeshwa Kwemodeli Nokuqinisekisa Imiphumela

Isingeniso Sephrojekthi

I-Chinese Guba AI Detector iyithuluzi lokuhlukanisa imibhalo yesiShayina ye-AI ngezigaba ezimbili. Iphrojekthi isekelwe ku-hfl/chinese-roberta-wwm-ext, futhi yaqeqeshwa ngokwengeziwe kusetshenziswa idatha ye-C-ReD. Yayike yasetshenziselwa ukukala imibhalo yasezinkundleni ze-Guba ngezinjongo zocwaningo. Indawo yokugcina ikhodi manje iveza imibhalo nemibhalo esebenzayo yokuqeqesha, ukuqagela nokuhlola.

Leli thuluzi lilayisha imodeli endaweni futhi alishayi ama-API angaphandle. Qaphela ngokukhethekile: indawo yokugcina okwamanje ayinawo ama-weight namafayela e-tokenizer aqeqeshiwe. Umsebenzisi kufanele athole imodeli ngokusemthethweni ayibeke kufolda ebekiwe.

Imikhawulo ebalulekile: Lena akuyona imodeli esemthethweni ye-C-ReD. Lokhu kuqeqeshwa ngokwengeziwe akuzange kusebenzise okuthunyelwe kwe-Guba, futhi akuzange kuhlolwe ngezimpawu ezenziwe abantu emkhakheni wezinkundla zezezimali. Amaphuzu emodeli awakwazi ukufakazela ubunikazi bangempela bombhali wombhalo, futhi akufanele asetshenziswe njengesisekelo esisodwa sokunquma ubuwena bomuntu noma ukujezisa okuqukethwe.

Imisebenzi Eyinhloko

• Ukuhlukanisa imibhalo yesiShayina ye-AI neyabantu ku-CSV ene-UTF-8 ngezigaba ezimbili.
• Ukukhipha amaphuzu e-softmax esigaba se-AI kanye nomphumela wokuhlukanisa osuselwe emngceleni.
• Ukufunda ngokuzenzakalelayo 帖子内容文本, futhi kusekela amakholomu wombhalo nekhodi yesitoko ezenziwe ngokwezifiso.
• Ukumisa 股吧代码 ibe umbhalo onezinhlamvu eziyisithupha nokugcina o-zero bokuqala.
• Ukugcina amanye amakholomu efayela lokufaka ngaphandle kokuguqula okuqukethwe kwawo kombhalo.
• Ukunciphisa ingozi yokonakala kwefayela lomphumela ngokusebenzisa ifayela lesikhashana nokushintsha kwe-athomu.
• Ukweseka ukuhlolwa kobuqotho, ukubala kabusha okubhalayo phezu kokudala, nokuvikela ukubhala kwenqubo eyodwa.
• Ukuhlinzeka ngemibhalo yokuqeqesha, ukuhlola kweyunithi, kanye nokuhlola okuqukethwe okushicilelwe.

Ukufaka Indawo Yokusebenza

1. Lungisa i-Python ne-PyTorch

Indawo yokuhlola yephrojekthi yi-Python 3.10. Kunconywa ukusebenzisa indawo ebonakalayo ezimele, bese ufaka i-PyTorch 2.5.1 efanele idivayisi yakho ye-CPU noma ye-CUDA ngokulandela imiyalelo esemthethweni ye-PyTorch.

Kwi-Windows PowerShell, ngena kufolda eyimpande yendawo yokugcina, bese udala futhi uvule indawo ebonakalayo:

python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install -r requirements.txt

I-CPU ingaqhuba ukuqagela, kodwa i-CUDA GPU ngokuvamile iyashesha. Uma kuvela ukuthi inkumbulo ye-GPU ayanele, yehlisa kuqala --batch-size kusuka kunani elizenzakalelayo noma elikhulu uye ku-8 noma ku-4.

2. Lungisa Imodeli Yasendaweni

Beka imodeli eqeqeshiwe oyithole ngokusemthethweni ku-models/detector. Le folda kufanele ibe namafayela alandelayo:

• model.safetensors
• config.json
• tokenizer.json
• tokenizer_config.json
• special_tokens_map.json
• vocab.txt

Lawa ma-weight namafayela e-tokenizer awafakiwe endaweni yokugcina yomphakathi. Uma ifolda ingaphelele, umbhalo wokuqagela ngeke ukwazi ukulayisha imodeli kahle.

Ukuqhuba Ukuhlola Kokuqala

1. Lungisa i-CSV Yokufaka

Ifayela lokufaka kufanele libe yi-CSV ene-UTF-8. Igama lekholomu yombhalo elizenzakalelayo ngu-帖子内容文本, kanti lekholomu yekhodi yesitoko ngu-股吧代码. Izibonelo ezinikezwe yindawo yokugcina ziyimibhalo eqanjiwe, azinabo abasebenzisi bangempela, okuthunyelwe kwangempela noma amalebula eqiniso enziwe abantu ne-AI.

2. Yenza Ukuqagela Kwenqwaba

Ngemva kokulungisa imodeli, sebenzisa lokhu kufolda eyimpande yendawo yokugcina:

python scripts/predict.py --model-dir models/detector --input examples/synthetic_posts.csv --output outputs/demo.csv --batch-size 32

Umyalo uzolayisha imodeli yasendaweni ku-models/detector, ufunde i-CSV yesibonelo, bese ubhala imiphumela yokuhlola ku-outputs/demo.csv. Yonke inqubo yokuhlola ayishayi ama-API angaphandle.

3. Qonda Izinkambu Zomphumela

Ifayela lomphumela lizogcina amakholomu okuqala futhi lengeze lezi zinkambu:

• 是否AI生成: Iba ngu-1 uma iphuzu lemodeli elingazange lisondelwe lilingana noma likhulu kunomngcele, kungenjalo libe ngu-0; umbhalo ongenalutho uhlala ungenalutho.
• AI生成概率: Iphuzu le-softmax lesigaba se-AI. Leli phuzu alikalwanga ngokuvumelana namathuba, ngakho alikwazi ukubhekwa ngokuqondile njengamathuba angempela.
• AI检测模型: Isihlonzi semodeli yomphakathi; indlela eya kumodeli yasendaweni ayibhalwa.
• AI检测阈值: Inani elizenzakalelayo ngu-0.7. Lokhu ukumiswa kwephrojekthi, hhayi umngcele obonwe yiphepha njengongcono kakhulu.
• AI检测状态: ok kusho ukuthi ukuhlola kuqediwe, kanti empty_text kusho ukuthi umbhalo wokufaka awunalutho.

Ukusebenzisa Amakholomu E-CSV Enziwe Ngokwezifiso

Uma ikholomu yombhalo ibizwa nge-text futhi ikholomu yekhodi yesitoko ibizwa nge-id, ungayichaza ngokusobala:

python scripts/predict.py --model-dir models/detector --input input.csv --output outputs/result.csv --text-column text --stock-column id

Uma ifayela lokufaka lingenayo ikholomu yekhodi yesitoko, dlulisa umbhalo ongenalutho:

python scripts/predict.py --model-dir models/detector --input input.csv --output outputs/result.csv --text-column text --stock-column ""

Uma ikholomu yekhodi yesitoko ikhona, uhlelo luzoyimisa ibe umbhalo onezinhlamvu eziyisithupha futhi lugcine o-zero bokuqala. Amanye amakholomu okufaka azogcinwa njengombhalo wokuqala.

Ukubhala Phezu Kokudala, Ukuhlola Nokucubungula Ngokuhambisana

Ukukhiqiza Kabusha Umphumela Ophelele

Umphumela wokuqagela uqala ngokubhalwa kufayela lesikhashana, bese ifayela eliqondiwe lishintshwa nge-athomu ngemva kokuphumelela. Lapho usebenzisa umyalo ofanayo futhi, uhlelo luzokweqa kuphela umphumela okhona uma amanani okuhlola okufaka, imodeli, amapharamitha kanye nomphumela ophelele ehambisana.

Ukuze ungayinaki imiphumela ekhona futhi ukhiqize ifayela eliphelele kabusha, faka --overwrite:

python scripts/predict.py --model-dir models/detector --input examples/synthetic_posts.csv --output outputs/demo.csv --batch-size 32 --overwrite

Ukuphazamiseka Nokusebenza Okunemisebenzi Eningi

Ngemva kokuphazamiseka, uhlelo luzobala kabusha kusukela ekuqaleni kwefayela, hhayi ukuqhubeka ngokungacabangi ngokusekelwe enanini lemigqa ekhona. Indlela efanayo yomphumela ingabhalwa inqubo eyodwa kuphela ngesikhathi. Ukuze usebenze ngokuhambisana, nikeza inqubo ngayinye ifayela lokufaka nelomphumela elehlukile.

Imikhawulo Yobude Nokuchazwa Kwemiphumela

Ngokuzenzakalelayo, umbhalo ngamunye ungafaka ama-token afinyelela ku-256; okweqa kulokho kuyanqunywa. Inani lama-token alilingani nenani lezinhlamvu zesiShayina, ngakho lo mngcele akufanele uqondwe njengezinhlamvu zesiShayina ezingu-256.

Eminye imibhalo yokuqala yase-Guba ocwaningweni yayinezingcezu eziqoqiwe ezingaba nezinhlamvu ezingu-203 kuphela, ngakho into ehlolwayo ingaba yingxenye yombhalo wokuqala. Lapho uchaza imiphumela, cabangela ukunqunywa kombhalo, umehluko womkhakha kanye nokuntuleka komongo.

AI生成概率 yiphuzu le-softmax elingakalwanga, kanti umngcele ozenzakalelayo othi 0.7 uwuk設定wa kwephrojekthi kuphela. Ukushintsha umngcele kuzoshintsha umphumela wokuhlukanisa ngezigaba ezimbili, kodwa i-README ayisho ukuthi yimuphi umngcele oqinisekiswe yiphepha njengongcono kakhulu.

Ukuqeqesha Imodeli Kusetshenziswa Idatha Ye-C-ReD

Imfuneko Yemvume

Ngaphambi kokuqeqesha, kufanele uqinisekise ngokwakho futhi uthole imvume esemthethweni yokusebenzisa idatha ye-C-ReD. Indawo yokugcina yomthombo ayinayo ilayisense ecacile, futhi ukucaphuna iphepha akusho ukuthi uthole imvume yokusebenzisa noma ukusabalalisa kabusha idatha. Le phrojekthi ayilingisi idatha ye-C-ReD, kanti ilayisense ye-MIT yendawo yokugcina ayifaki idatha ye-C-ReD noma ama-weight athathwe kuyo.

Faka Okuncike Kuko Kokuqeqesha

python -m pip install -r requirements-train.txt

Qala Ukuqeqesha

Uma idatha etholwe ngokusemthethweni ibekwe ku-data/C-ReD, ungasebenzisa:

python scripts/train.py --data-root data/C-ReD --output-dir models/detector --epochs 3 --batch-size 8 --eval-batch-size 16 --no-fp16

Uhlelo lokuqeqesha lufunda ama-CSV amagama awo aqukethe CReD_, bese lusebenzisa amakholomu text kanye label. Amalebula edatha yomthombo achazwa ngokuthi 0=AI, 1=umuntu; ngesikhathi sokuqeqesha aguqulwa abe ngu-0=umuntu, 1=AI.

Imithetho yokuqeqesha nezinkomba zomlando kubhalwe ku-MODEL_CARD.md. Ukuqeqesha kabusha kusebenzisa ukuqaliswa okuphelele ngembewu engahleliwe, kodwa ukuqeqeshwa komlando akuzange kusethe ngokusobala imbewu yomhlaba wonke ngaphambi kokuqalisa imodeli, ngakho ukuphindaphinda ama-weight amadala ngokunembile akuqinisekisiwe.

Ukuhlola Nokucwaninga Ngaphambi Kokushicilela

Qalisa konke ukuhlola kweyunithi:

python -m unittest discover -s tests -v

Ukuhlola kweyunithi kusebenzisa idatha eqanjiwe nokuqagela okulingisiwe, ngakho akudingi ukulanda imodeli. Ukuhlola okusheshayo kwemodeli yangempela kungabonisa kuphela ukuthi imodeli iyalayisheka futhi iyakwazi ukuqagela; akufakazeli ukuthi imiphumela yokuqagela ilungile.

Qalisa ukuhlolwa kokuqukethwe komphakathi:

python scripts/audit_release.py

Indawo yokugcina yomphakathi ihlolwa ngohlu lwamafayela avunyelwe namaphethini olwazi olubucayi. Ama-CSV okuqala e-Guba, ama-ID abashicileli, amagama eziteketiso, ama-IP, ama-URL okuthunyelwe, amaphaneli amaphepha, imininingwane yezohwebo, amakheji, izingodo, imiphumela yama-Notebook, izindawo zokuhlola eziphakathi zokuqeqesha kanye nomlando we-Git akufanele kungene endaweni yokugcina. Ububanzi obuqondile bokususa imininingwane buyatholakala ku-PRIVACY.md, kanti inqubo yokushicilela iku-docs/PUBLISHING.md.

Izincomo Zokusebenzisa Okuthuthukile

1. Qala ngedatha eqanjiwe ukuqinisekisa inqubo: Qinisekisa ukuthi indawo, amagama amakholomu, ifolda yemodeli kanye nendlela yomphumela kulungile ngaphambi kokucubungula idatha yakho esemthethweni.
2. Yehlisa inqwaba uma i-GPU inganele: Zama kuqala --batch-size 8 noma --batch-size 4.
3. Gwema ukwabelana ngendlela yomphumela: Imisebenzi ehambisanayo kufanele isebenzise amafayela emiphumela ahlukene ukuze ingabangeli umngcele wokubhala wenqubo eyodwa.
4. Gcina amapharamitha okusebenza: Imodeli, okufakwayo, amapharamitha namanani okuhlola omphumela kuthinta ukusetshenziswa kabusha kwemiphumela; ngemva kokulungisa noma iyiphi yazo, hlola kabusha umphumela.
5. Humusha imibhalo emide ngokuqapha: Okuqukethwe okudlula ama-token angu-256 kuyanqunywa, ngakho imodeli ingase ingayiboni ingxenye yokugcina.
6. Ungaphathi amaphuzu njengobufakazi bobuwena: Amaphuzu e-softmax awakalwanga, futhi iphrojekthi ayinakho ukuhlola okwenziwe ngabantu emkhakheni wezinkundla zezezimali.
7. Qinisekisa imvume ngaphambi kokushicilela ama-weight: Ikhodi nemibhalo kunelayisense ye-MIT, imodeli eyisisekelo yona ine-Apache-2.0, kodwa idatha ye-C-ReD nama-weight athathwe kuyo kudinga ukuqinisekiswa kwemvume ngokwehlukile.

Isiphetho

I-Chinese Guba AI Detector inikeza inqubo yokuhlukanisa imibhalo yesiShayina ye-AI engasebenza endaweni, ehlanganisa ukuqagela kwe-CSV, ukuqeqesha imodeli, ukuhlola kweyunithi kanye nokucwaninga ngaphambi kokushicilela. Ekusetshenzisweni kwangempela, kufanele uqale uthole imodeli noma idatha yokuqeqesha ngokusemthethweni, futhi uphathe umphumela njengephuzu lemodeli locwaningo, hhayi njengobufakazi obuqinisekile bobuwena bombhali wombhalo.