Sebenzisa futhi Ufake i-OpenJev ukuze Uhlukanise Ngethokheni Eyodwa nge-SGLang

Yini i-openjev-sglang?

openjev-sglang yiseva esebenzisa i-TypeSafe/Jev HTTP API isebenzisa i-Qwen3.6-35B-A3B ku-SGLang. Iguqula imibuzo yokuhlukanisa ibe ukufundwa okusebenzayo kwamathuba ethokheni yokuqala, kunokukhiqiza izimpendulo ezinde.

Ukusetshenziswa okuzenzakalelayo kuqhuba isitsha esisodwa se-SGLang ku-GPU ngayinye ye-NVIDIA B200. I-SGLang 0.5.19 inikeza i-Rust frontend yayo, i-radix caching, nama-CUDA graphs e-prefill anqamukayo. Inqubo ehlukile ye-Python iphatha i-API yomphakathi nge-FastAPI, uvloop, i-Hugging Face tokenizer esekelwe ku-Rust, kanye nokuxhumana kwe-asynchronous okuhlanganisiwe okuya ku-SGLang ku-localhost.

Azikho izicelo ezithunyelwa ku-TypeSafe. I-endpoint yomphakathi iveza i-evaluation API, kanti izindlela zokukhiqiza nezokuphatha ze-SGLang zihlala ziyimfihlo ku-localhost.

Izici ezibalulekile

• Izinhlobo ezintathu zokuhlola: i-Noul yamathuba ka-yebo/cha, i-Choice yokukhetha ngokwezigaba, kanye ne-Score yamazinga ahleliwe.
• Inference yethokheni eyodwa: umbuzo ngamunye uhlolwa kusetshenziswa ama-log probabilities amalebula ezimpendulo, ngaphandle kwe-chain of thought ekhiqiziwe.
• Ukugcina ama-prefix abiwe: i-prompt prefix efanayo iyafudunyezwa ngaphambi kokuba amagatsha emibuzo asebenze ngesikhathi esisodwa.
• Kufika emibuzweni engu-64: imibuzo ye-Choice ne-Score isekela izimpendulo eziphakathi kuka-2 no-64 ngokuzenzakalelayo.
• Isimo esihlelekile: isimo nemiyalelo kungaba ama-string, ama-JSON object, noma ama-array.
• Endpoints zokusebenza: i-health, i-liveness, ikhathalogi yamamodeli, imikhawulo, i-OpenAPI schema, i-Swagger UI, kanye ne-Scalar reference esebenzisanayo kufakiwe.
• Ukusetshenziswa ku-Modal: uhlelo olunikeziwe lusekela i-autoscaling, ama-model cache aqhubekayo, ukubuyiselwa kokuqalisa, kanye nokuziphatha kwe-scale-to-zero.
• Ukuvikelwa okulungisekayo: imikhawulo yezicelo, izilawuli ze-concurrency, ama-timeout, kanye ne-Bearer authentication ongayikhetha kuyatholakala.

Okudingekayo ngaphambi kokuqala

Faka i-uv bese uthola i-akhawunti ye-Modal uma uhlela ukusebenzisa i-cloud deployment enikeziwe. I-build yokuqala ingenisa i-SGLang image enkulu, kanti ukuqala kokuqala kwe-GPU kulanda ama-model weights futhi kuhlanganise noma kuthwebule ama-kernel.

Izidingo ze-CUDA zihlala ngaphakathi kwesitsha se-SGLang. Ukuqhuba i-uv sync ku-workstation yakho kufaka kuphela i-API, izinsiza ze-deployment, kanye nama-test.

Faka iphrojekthi

Clone-a i-repository, ngena ku-directory yayo, bese uvumelanisa imvelo ye-Python:

uv sync

Uma i-Modal ingakaqinisekiswa emshinini wamanje, qedela ukusethwa kwayo:

uv run modal setup

Qalisa i-Modal deployment yesikhashana

Ukuze wenze ukuhlola okuphelele, qala iseva yesikhashana, sebenzisa ukuhlolwa kwe-inference okufakiwe, bese ucisha iseva ngemva kwalokho:

uv run modal run modal_app.py

Lo myalo ulondoloza umbiko we-smoke test njenge-smoke-result.json. Umbiko uhlanganisa isikhathi sokulinda kokuqalisa, i-inference latency, kanye nolwazi olutholakalayo lokusetshenziswa kwe-cache.

Deploy-a i-endpoint ezinzile

Deploy-a uhlelo njengesevisi eqhubekayo ye-Modal:

uv run modal deploy modal_app.py

I-deployment iphrinta i-URL efana ne-https://YOUR-SERVER.us-west.modal.direct. Yigcine ku-environment variable ukuze uyifinyelele kalula:

export OPENJEV_URL="https://YOUR-SERVER.us-west.modal.direct"

I-Modal endpoint ezenzakalelayo ayidingi i-authentication ngamabomu. Isebenzisa isifunda somzila sase-US West futhi ingahlela i-compute ezifundeni zase-US ezilungiselelwe. Awukho umkhawulo ocacile wezi-container, futhi ama-instance ehla abe ngu-zero ngemva kwemizuzu emihlanu engasebenzi.

Iseva eyehle yaba ngu-zero ibuyisa i-HTTP 503 ngenkathi iqala. I-smoke command efakiwe iphinda izame lezi zimpendulo zokuqalisa ngokuzenzakalelayo.

Qalisa i-smoke test usebenzisa i-URL edeployed:

uv run openjev smoke "$OPENJEV_URL"

I-smoke test ihlola zonke izinhlobo ezintathu zezimpendulo, ihlole umbuzo onezimpendulo ezingu-64, yenze ukuhlolwa okuyisisekelo kwencazelo, futhi iqinisekise ukuthi izimpendulo ezingu-65 ziyenqatshwa.

Thumela ukuhlola kwakho kokuqala

Indlela eyinhloko yi-POST /v1/systemone. Isicelo esilandelayo sihlukanisa umlayezo wekhasimende ngemibuzo ye-Noul, Choice, ne-Score:

curl "$OPENJEV_URL/v1/systemone" \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "jev-latest",
    "state": [
      {"role": "system", "content": "You are a support assistant."},
      {"role": "user", "content": "I was charged twice. Please refund the duplicate."}
    ],
    "questions": {
      "refund": {
        "type": "noul",
        "instructions": "Does the user request a refund?"
      },
      "department": {
        "type": "choice",
        "instructions": "Which department should handle this?",
        "criteria": {
          "billing": "Payments and refunds",
          "technical": "Software bugs"
        }
      },
      "urgency": {
        "type": "score",
        "instructions": "How urgent is the request?",
        "criteria": ["Routine", "Urgent", "Emergency"]
      }
    }
  }'

Ungagcina futhi i-JSON efayeleni bese uyithumela ngaphandle kokucaphuna kwe-shell:

curl "$OPENJEV_URL/v1/systemone" \
  -H 'Content-Type: application/json' \
  --data-binary @examples/request.json

Ukuqonda izinhlobo zemibuzo

• Noul: ibuyisa amathuba ajwayelekile ka-yes.
• Choice: ibuyisa inketho enamathuba aphezulu kanye nokusatshalaliswa okuphelele kwamathuba. Amagama ezinketho okuqala agcinwa empendulweni.
• Score: ibuyisa izinga elilindelekile eliqala ku-zero, elibalwa kusetshenziswa inkomba yezinga ngalinye namathuba alo, kanye ne-legend.

Izinketho zokukhetha zethulwa kumodeli njengamalebula ezinhlamvu alandelwa izincazelo. Okhiye bezimpendulo ngokuvamile bafihlwa kumodeli. Uma incazelo yenketho ingu-null, ukhiye wayo usinika incazelo esikhundleni salokho.

Ukunikeza isimo nemiyalelo

Kokubili state kanye instructions kwamukela izintambo, izinto ze-JSON, noma uhlu. Uhlu lwemilayezo yengxoxo, noma into equkethe ngqo {"messages": [...]}, lunikezwa kusetshenziswa ithempulethi yengxoxo yomdabu yemodeli.

Izindima nezinto zomlayezo wokuqala ziyagcinwa, futhi umbuzo ngamunye wokuhlukanisa wengezwa njengomunye umzuliswano womsebenzisi. Amanye amanani ahlelekile aguqulwa ngokuphelele abe umlayezo womsebenzisi. Into equkethe messages kanye nezinye izinkambu nayo iyagcinwa ukuze imethadatha ingalahleki buthule.

Isimo sengxoxo sisekela okuqukethwe kombhalo kuphela. Okuqukethwe kwesithombe, komsindo, nevidiyo akusekelwa.

Hlola iziphetho zesevisi

• POST /v1/systemone yenza ukuhlola kwe-Noul, Choice, ne-Score.
• GET /v1/models ibuyisa ikhathalogi yamamodeli enezinkambu ze-TypeSafe nezohlobo lwe-OpenAI.
• GET /v1/limits ibika imikhawulo yokwamukela izicelo.
• GET /health ibika ukulungela, impilo ye-SGLang, nesikhathi sokuqalisa.
• GET /health/live ihlola ukuthi inqubo ye-API isaphila yini.
• GET / ivula ireferensi ye-Scalar API enesibonelo sesicelo esihlelekayo.
• GET /docs ivula i-Swagger UI eyakhelwe ngaphakathi.
• GET /openapi.json ibuyisa i-schema ye-OpenAPI ekhiqiziwe.

Igama lemodeli yesicelo elithi jev-latest liyisiteketiso sokuhambisana. I-ID yemodeli yomphakathi ezenzakalelayo ngu-Qwen/Qwen3.6-35B-A3B, kanti inqolobane ye-NVIDIA isetshenziswa ngaphakathi njengomthombo wama-weight.

Indlela i-inference esebenza ngayo

1. I-API iqinisekisa i-schema, inani lezimpendulo, usayizi womzimba, ubude bomongo, kanye nesabelomali samathokheni esihlanganisiwe.
2. I-rendera ithempulethi yengxoxo yomdabu kanye kuphela lapho ukucabanga kukhutshaziwe, bese ihlukanisa isiqalo esabiwe kwizijobelelo zemibuzo ezithokenezwe ngokuzimela.
3. Ithumela isiqalo esivamile ku-SGLang nge-max_new_tokens=1. Ithokheni esampuliwe iyalahlwa, kushiywe isiqalo sitholakale ukuze kusetshenziswe kabusha i-radix cache uma kudingeka.
4. Ithumela ngokuhambisana igatsha elilodwa lombuzo ngamunye. Igatsha ngalinye liqukethe isiqalo, isijobelelo sombuzo, unhlokweni womsizi, kanye nomaka othi Answer:.
5. Icela amathuba e-log kuphela kumalebula ezimpendulo avumelekile bese iwenza ajwayeleke kabusha nge-softmax ezinzile.

Emibuzweni engu-N, iseva yenza izingcingo ezingu-N + 1 zethokheni eyodwa: ucingo olulodwa lokufudumeza i-cache kanye nocingo olulodwa lombuzo ngamunye. Ithokheni ekhiqiziwe iyazitshwa, ngakho akukho kuqhubeka kwe-autoregressive noma uchungechunge lokucabanga olukhiqiziwe. I-speculative decoding ayivunyelwe.

Amalebula aqala ku-A aze afike ku-Z, bese esebenzisa izinhlanganisela eziqinisekisiwe zethokheni eyodwa ezifana no-AA no-AB. Wonke amalebula angu-64 ahlolwa nge-tokenizer esebenzayo ngesikhathi sokuqalisa. Lokhu kugwema ukusebenzisa amalebula ezinombolo ezingamathokheni amaningi afana no-10 noma 64.

Qonda ukusetshenziswa namamethrikhi e-cache

Unhlokweni wempendulo othi x-openjev-prefix-tokens ubika usayizi wesiqalo esivamile esiceliwe. Uma i-SGLang inikeza izibalo ze-cache, x-openjev-cached-tokens iqukethe isamba sokushaywa kwe-cache kwamagatsha.

I-frontend ye-Rust ye-SGLang 0.5.19 ayizivezi lezo zibalo ze-cache. Kulokho kulungiselelwa unhlokweni awukho futhi umbiko we-smoke ubhala null, hhayi uziro. Amalogi e-scheduler asengabonisa ukushaywa kwe-cache kwangempela. Unhlokweni we-Server-Timing uhlukanisa ukulungiselela i-prompt, i-prefill eyabiwe, kanye ne-inference yamagatsha.

usage.input_tokens uyisamba samanani aphelele e-prompt abikwe kuzo zonke izingcingo zokufudumeza namagatsha, kuhlanganise namathokheni agcinwe ku-cache. usage.output_tokens ngu-N + 1. Lezi zibalo zichaza umsebenzi wangemuva kunamathokheni abaliwe ahlukile noma izilinganiso zokukhokhisa ze-TypeSafe.

Hlela imikhawulo nokuqinisekisa ubunikazi

Imikhawulo ezenzakalelayo yokwamukela izicelo ihlanganisa imibuzo engu-64, izimpendulo ezingu-2 kuya kwezi-64 ngombuzo we-Choice noma we-Score, umzimba we-JSON ongu-2 MiB, amathokheni angu-32,768 ngegatsha kuhlanganise nokuphumayo, amathokheni okufaka athunyelwe angu-262,144 esewonke, ukuhlola okungu-16 okwenziwa ngesikhathi esisodwa, kanye nezingcingo zangemuva ezingu-64 ezenziwa ngesikhathi esisodwa.

Izicelo ezingavumelekile zibuyisa i-HTTP 422 ngaphambi kwe-inference. Imizimba edlula umkhawulo wosayizi ibuyisa u-413, ukugcwala ngokweqile kubuyisa u-529 one-Retry-After, kanti ukuphela kwesikhathi kwe-backend kubuyisa u-504. Ukuhlola okuhlulekile noma okukhanseliwe kukhansela amagatsha ahambisana nakho futhi kuzama ukuwunqamula ku-SGLang.

Ukucushwa kuyatholakala ngama-environment variable e-OPENJEV_*. Izilungiselelo ezivamile zihlanganisa:

• OPENJEV_MODEL: okuzenzakalelayo ngu-nvidia/Qwen3.6-35B-A3B-NVFP4.
• OPENJEV_SERVED_MODEL_NAME: kudlula i-ID yemodeli yomphakathi futhi idluliselwe ku-SGLang.
• OPENJEV_REVISION: ikhetha i-revision ye-checkpoint ephiniwe.
• OPENJEV_FRONTEND: okuzenzakalelayo ngu-rust; python iyindlela yokubuyela emuva ecaciswe ngokusobala.
• OPENJEV_MAX_INPUT_TOKENS: okuzenzakalelayo ngu-32768.
• OPENJEV_MAX_TOTAL_INPUT_TOKENS: okuzenzakalelayo ngu-262144.
• OPENJEV_MAX_CONCURRENT_REQUESTS: okuzenzakalelayo ngu-16.
• OPENJEV_MAX_CONCURRENT_BRANCHES: okuzenzakalelayo ngu-64.
• OPENJEV_REQUEST_TIMEOUT: okuzenzakalelayo imizuzwana engu-120.
• OPENJEV_TEMPERATURE: okuzenzakalelayo ngu-1.0 futhi kuthinta ukwenziwa kwelebula kube ngokujwayelekile.
• OPENJEV_API_KEY: kwenza ukuqinisekiswa kobunikazi kwe-Bearer okukhethwayo kusebenze ku-API yomphakathi.
• OPENJEV_BACKEND_API_KEY: kulungiselela ukhiye we-Bearer ohlukile nowokuzikhethela we-SGLang.

Isiqalisi se-Modal sidlulisela ngokuzenzakalelayo i-OPENJEV_PROFILE, i-OPENJEV_FRONTEND, kanye ne-OPENJEV_SERVED_MODEL_NAME kusuka endaweni yendawo. Engeza ezinye izilungiselelo zesilawuli kude ku-image.env(...), noma usebenzise i-Modal Secret ukuze ugcine imininingwane yokufakazela ubuwena.

Sebenzisa endaweni nge-backend ye-SGLang ekhona

Ukuze usebenzise i-backend esivele isebenza, qondisa i-API ku-URL yayo yendawo:

uv run openjev serve --connect http://127.0.0.1:30000

I-backend kufanele isebenzise inguqulo ehambisanayo yemodeli ne-tokenizer. Kufanele futhi isekele amathuba e-log ama-token akhethiwe, i-radix caching, nobude obanele bomongo.

Kumsingathi noma esitsheni se-B200 lapho i-SGLang 0.5.19 ifakwe khona endaweni ehlukile ye-Python, nikeza ngokusobala i-interpreter yayo:

uv run openjev serve --sglang-python /path/to/sglang/bin/python

Ukuthuthukisa nokuqinisekisa

Sebenzisa imiyalo efakiwe ukuze usebenzise ukuhlolwa okungaxhunyiwe ku-inthanethi, ukuhlola ukuhlanganiswa kwe-tokenizer, i-linting, kanye nokukhiqizwa kwe-schema:

uv run pytest
uv run pytest -m integration
uv run ruff check .
uv run openjev schema

Ukuhlola okuvamile kuhlanganisa ukuziphatha kwe-unit ne-API kungaxhunyiwe ku-inthanethi. Ukuhlola kokuhlanganiswa kusebenzisa i-tokenizer yangempela nokulanda okuncane kwa-Hugging Face, kodwa akudingi i-GPU. Ukukhiqizwa kwe-schema nakho akudingi i-GPU noma ukulanda imodeli.

Amathiphu okuphakela athuthukile

Gcina isitsha se-GPU sifudumele

I-Modal yehlisa isevisi ibe nguziro ngemva kwemizuzu emihlanu ingasebenzi. Setha i-min_containers=1 ku-modal_app.py lapho ukubambezeleka okuphansi njalo kokuqalisa kubaluleke kakhulu kunokugwema ukusetshenziswa kwe-GPU engasebenzi.

Lindela ukuqaliswa kokuqala

Ukuqalisa kwe-GPU kokuqala kulanda ama-weights futhi kuhlanganise noma kuthwebule ama-kernel. Ama-weights emodeli, idatha yokulungisa i-SGLang, nedatha yokuhlanganisa i-Triton kugcinwa ku-Modal Volume ethi openjev-huggingface. Ukuqalisa okulandelayo kuphinde kusebenzise lawo mafayela, nakuba ukuthwebula i-CUDA graph kusenzeka ekuqaleni.

Thatha ukuphinda usebenzise i-cache njengethuba

Ukuphinda kusetshenziswe i-Radix akuyona iseshini ye-KV eboshiwe ngesicelo ngasinye. Isimo esiphindaphindwayo kumodeli eyi-hybrid Qwen, imingcele yamakhasi e-cache, ingcindezi ye-cache, kanye nethrafikhi ehambisanayo kunganciphisa ama-hit. I-backend isebenzisa i---mamba-radix-cache-strategy extra_buffer.

Humusha ukuzethemba ngokucophelela

I-OpenJev ichaza ukuzethemba njenge-1 - H(probabilities) / log(number_of_options), okubekwe phakathi kuka-zero no-one. Amathuba alinganayo akhiqiza ukuzethemba okuyiqanda, kanti i-point mass ikhiqiza ukuzethemba okukodwa.

Lawa mathuba ancike ezinkethweni ezinikeziwe futhi angathinteka amagama e-prompt kanye nokuhleleka kwamalebula. Awazona izilinganiso ezilinganisiwe zokunemba.

Hlela ukwehluleka kwe-backend

Uma i-SGLang iphuma ngokungalindelekile, i-API nayo iyaphuma. Isiqalisi se-Modal bese siqeda isitsha ukuze i-Modal ikwazi ukusifaka esikhundleni, kunokuba ishiye inqubo yomphakathi ye-HTTP ixhumeke ku-backend yokwenza inference efile. Ukuvalwa okuvamile kucisha bobabili ababukeli ngokuhlanzekile.

Isiphetho

i-openjev-sglang inikeza indlela egxile yokusebenzisa ukuhlukanisa okuhambisana ne-Jev ku-Qwen3.6 ngokufundwa kahle kwamathuba e-token eyodwa. Ukumiswa kwayo kwe-Modal kunikeza ukuphakelwa kwe-B200 okukhulisekayo, kuyilapho imiyalo yayo yendawo isekela ukuhlola nokuxhuma kusevisi ye-SGLang ekhona. Qala ngokuhlolwa okuyisisekelo, qinisekisa imikhawulo nezilungiselelo zakho zokuqinisekisa, futhi uqaphe isikhathi kanye nama-header e-cache njengoba ithrafikhi ikhula.