Sheshisa Imisebenzi ye-PyTorch TopK nge-DeepSelect

Yini i-DeepSelect?

I-DeepSelect iwukusetshenziswa okusebenza kahle kakhulu kwe-TopK kernel esetshenziswa ku-DeepSeek Sparse Attention nasekuthatheni amasampula. Ngokwe-README yephrojekthi, inganikeza ukusheshisa okuphindwe ka-2 kuya ka-20 uma iqhathaniswa ne-torch.topk evamile emithwalweni yomsebenzi esekelwa yile labhulali.

Esikhundleni sokuzama ukwenza kahle konke ukumiswa kwe-TopK okungenzeka, i-DeepSelect igxile ezimweni ezimbili: umthwalo we-Lightning Indexer osebenzisa okokufaka kwe-torch.bfloat16, kanye nomthwalo wokuthatha amasampula osebenzisa okokufaka kwe-torch.float32.

Imithwalo Yomsebenzi Esekelwayo

Isimo se-Lightning Indexer

• Uhlobo lokokufaka: torch.bfloat16
• Usayizi we-batch: kusuka ku-1 kuya phezulu, ngokuthuthukiswa kwama-batch amancane namakhulu
• Usayizi we-vocabulary: kusuka ku-1 kuya phezulu, ngokuthuthukiswa kwama-vocabulary amancane namakhulu
• Usayizi we-TopK: mncane futhi awudluli ku-4096

Ukuze uthole ukusebenza okungcono, shiya i-sorted_index ivaliwe ngaphandle kokuthi imiphumela kufanele ihlelwe. Uma amanani engadingeki, setha i-return_value=False ukuze weqe ukukhishwa kwawo.

Isimo sokuthatha amasampula

• Uhlobo lokokufaka: torch.float32
• Usayizi we-batch: kusuka ku-1 kuya phezulu
• Usayizi we-vocabulary: cishe u-128K
• Usayizi we-TopK: mncane futhi awudluli ku-4096

Indawo yokugcina ikala ukusebenza isebenzisa umkhawulokudlulisa wememori osebenzayo, hhayi umthamo wokucubungula izinombolo zamaphuzu antantayo, ngoba i-TopK ayenzi izibalo zamaphuzu antantayo. Ibhentshimakhi yayo ibika ukusebenza uma kuqhathaniswa ne-torch.topk kokokufaka okufanayo.

Izici Ezibalulekile

• Ama-kernel e-TopK akhethekile emithwalweni yomsebenzi ye-DeepSeek Sparse Attention neyokuthatha amasampula
• Ukusekelwa kokokufaka kwe-torch.bfloat16 kanye ne-torch.float32 ezimweni zazo ezihlosiwe
• Ukukhetha ukukhishwa kwama-index e-torch.int32 noma e-torch.int64
• Ukukhishwa kwamanani ongakukhetha ukuze kuncishiswe umthwalo lapho kudingeka ama-index kuphela
• Imigqa enobude obuhlukahlukene kusetshenziswa umkhawulo ophezulu okhethekile, ongafakiwe, womugqa ngamunye
• Ama-buffer okukhipha ama-index aphethwe ngumsebenzisi
• Ukuhlolwa kwe-NaN okwakhelwe ngaphakathi

Ukufakwa

Yenza i-clone yendawo yokugcina, qalisa ama-submodule ayo, bese ufaka iphakheji usuka kufolda yephrojekthi:

git clone https://github.com/deepseek-ai/DeepSelect.git
cd DeepSelect
git submodule update --init --recursive
pip install -v .

Izibonelo zisebenzisa ama-tensor e-CUDA, ngakho dala okokufaka ku-device="cuda".

Ukusebenzisa i-TopK Okuyisisekelo

Ngenisa i-PyTorch ne-DeepSelect, dala i-tensor yokokufaka enezinhlangothi ezimbili, bese ubiza i-deep_select.topk:

import torch
import deep_select

batch_size = 4
vocab_size = 204800
topk = 1024

x = torch.randn(
    batch_size,
    vocab_size,
    dtype=torch.bfloat16,
    device="cuda",
)

values, indices = deep_select.topk(
    x,
    topk,
    sorted_index=True,
    indices_type=torch.int32,
    return_value=True,
)

print(values.shape)   # (4, 1024)
print(indices.shape)  # (4, 1024)

Okokufaka kufanele kube nesimo esithi (batch_size, vocab_size). Ubukhulu bokugcina kufanele bube contiguous, futhi i-row stride yaso kufanele ibe ukuphindaphinda kwamabhayithi e-deep_select.get_stride_requirement()[0].

Amanani abuyiswayo anohlobo lwedatha olufanayo no-x. Ama-index asebenzisa uhlobo olukhethwe nge-indices_type, olungaba u-torch.int32 noma u-torch.int64. Uma i-sorted_index=True, ama-index abuyiswayo emugqeni ngamunye ahlelwa ngokukhuphukayo.

Ukubuyisa ama-index kuphela

Uma ikhodi elandelayo ingawadingi amanani akhethiwe, vala ukukhishwa kwamanani:

values, indices = deep_select.topk(
    x,
    topk,
    sorted_index=False,
    indices_type=torch.int32,
    return_value=False,
)

I-README iphawula ukuthi ukweqa ukukhishwa kwamanani kushesha cishe ngo-10%. Gwema ukucela okukhiphayo okuhleliwe ngaphandle kokuthi ukuhleleka kudingeke, ngoba ukuhlela kwengeza umthwalo.

Ukuphatha Ukuqondaniswa Kwe-Stride

I-DeepSelect ibeka izimfuneko zokuqondanisa emigqeni yokokufaka nasemigqeni yokukhipha. Zihlole ngesikhathi sokusebenza:

input_alignment, output_alignment = deep_select.get_stride_requirement()

print("Input row-stride alignment:", input_alignment, "bytes")
print("Output row-stride alignment:", output_alignment, "bytes")

Uma i-row stride yokokufaka ingeyona ukuphindaphinda kokuqondaniswa kokokufaka okudingekayo, faka i-padding ku-tensor ngaphambi kokuyidlulisela ku-DeepSelect. Ubukhulu bokugcina kufanele buqhubeke bube contiguous.

I-DeepSelect yabela kokubili okukhiphayo ngokuzenzakalelayo. Ama-stride ako aqondaniswa namabhayithi e-deep_select.get_stride_requirement()[1], okusho ukuthi ama-tensor abuyiswayo angase angabi contiguous. Ikhodi esebenzisa okukhiphayo akufanele icabange ukuthi ama-stride ajwayelekile e-contiguous.

Ukusebenzisa i-index buffer ekhona

I-API yamukela i-output_idx= lapho ama-index kufanele abhalwe ku-buffer ephethwe ngumshayi-khodi. Leyo buffer kufanele ihlangabezane nemfuneko efanayo ye-output stride. Bheka deep_select/interface.py ukuze uthole isiginesha ephelele yomsebenzi ngaphambi kokumisa i-buffer yangokwezifiso.

Ukusebenza Ngemigqa Enobude Obuhlukahlukene

Sebenzisa i-end ukuze unikeze umugqa ngamunye umkhawulo ophezulu ongafakiwe. Lokhu kuyasiza lapho imigqa eku-batch efanayo inobude obuvumelekile obuhlukahlukene:

import torch
import deep_select

batch_size = 2
vocab_size = 129280

x = torch.randn(
    batch_size,
    vocab_size,
    dtype=torch.float32,
    device="cuda",
)

end = torch.tensor(
    [129280, 100000],
    dtype=torch.int32,
    device="cuda",
)

values, indices = deep_select.topk(
    x,
    1000,
    end=end,
    sorted=True,
    indices_type=torch.int64,
)

Lapha, umugqa wokuqala ungasebenzisa zonke izakhi ezingu-129280, kanti umugqa wesibili ulinganiselwe ezakhiweni zokuqala ezingu-100000. Uma umugqa umfushane kunosayizi we-TopK oceliwe, i-DeepSelect igcwalisa imiphumela yawo nge-value_oob_fill_value kanye ne-idx_oob_fill_value.

Ukuphathwa kwe-NaN

Ukuhlolwa kwe-NaN kuhlala kuvuliwe. Ngokuzenzakalela, i-abort_when_nan_found=True, ngakho i-kernel ibiza i-trap() bese iyama lapho ithola i-NaN. Imigqa ubude bayo obuvumelekile bungaphansi noma bulingana nosayizi we-TopK oceliwe ayihlolwa i-NaN.

Qinisekisa noma uhlanze idatha yokufaka ngaphambi kokuqalisa i-kernel uma ama-NaN engenzeka, ikakhulukazi uma ugcina indlela yokumisa ezenzakalelayo.

Ukusebenza namathiphu athuthukile

1. Hlala ngaphakathi kobubanzi be-TopK obusekelwayo. I-DeepSelect isekela amanani amancane e-TopK afika futhi afake phakathi u-4096. Amanani amakhulu awasekelwa.
2. Qondanisa nomthwalo ohleliwe. Sebenzisa i-torch.bfloat16 esimweni se-Lightning Indexer kanye ne-torch.float32 esimweni sokusampula esichazwe iphrojekthi.
3. Yeqa ukuhlela lapho kungenzeka. Ukuhleleka ngenkomba noma ngenani kudla ukusebenza, ngakho kucela kuphela lapho ukucutshungulwa okulandelayo kukudinga.
4. Yeqa amanani lapho kungenzeka. Setha i-return_value=False lapho kusetshenziswa izinkomba kuphela.
5. Hlola ukuqondanisa kusenesikhathi. Hlela izimo zama-tensor nokupheda ngokusekelwe ku-get_stride_requirement() esikhundleni sokulungisa ama-tensor angahambisani phakathi kweluphu ebucayi ekusebenzeni.
6. Ungacabangi ukuthi imiphumela iyaqhubeka ngokulandelana kwenkumbulo. I-DeepSelect iqondanisa ama-stride emigqa yemiphumela, ngakho ama-tensor aphumayo angase angabi contiguous.
7. Yenza i-benchmark ngesimo sangempela. Ukusebenza kwe-TopK kuncike ohlotsheni lokufaka, kusayizi we-batch, kusayizi we-vocabulary, nakusayizi we-TopK. Hlola ukucushwa okuqondile okusetshenziswa uhlelo.

Ukuqalisa i-benchmark enikeziwe

Indawo yokugcina ikhodi iqukethe imodi yokusebenza kuskripthi sayo sokuhlola:

python3 tests/test.py --perf-only

Le benchmark iqhathanisa i-DeepSelect ne-torch.topk kokokufaka okufanayo futhi ibike isilinganiso isebenzisa i-effective memory bandwidth.

Isiphetho

I-DeepSelect inikeza ama-kernel e-TopK akhethekile emithwalweni ehambisanayo ye-DeepSeek Sparse Attention kanye neyokusampula. Qala nge-CUDA tensor enezinhlangothi ezimbili eqondanisiwe, gcina i-topk iku-4096 noma ngaphansi, bese ukhubaza ukuhlela noma ukukhishwa kwamanani lapho kungadingeki. Ekuhlanganisweni kokukhiqiza, qinisekisa izidingo ze-stride, cabangela imiphumela engase ingabi contiguous, futhi wenze i-benchmark ngezimo eziqondile zama-tensor ezisetshenziswa uhlelo lwakho.