Reinforcement learning has become a mainstream paradigm for training large models, driving model technologies toward engineering deployment in core industry scenarios. Training-inference consistency is crucial because the inference and training processes are coupled, and infrastructure differences can amplify uncertainty.
Huawei Computing officially announced on August 5 that Ascend provides a complete set of training-inference consistency operators based on the Ascend C programming language. Testing on the Qwen3-30B MoE model achieved a Logdiff of 0. With Ascend-optimized FA operator improvements, Eager mode delivered performance gains of 20%-60%, providing developers with an efficient and reliable reinforcement learning training solution.
The fundamental reason for training-inference inconsistency during RL post-training is that accumulation in low-level computations is not performed in a fixed order. Ensuring that the training and inference engines use the same accumulation order every time, from top to bottom, requires simultaneous efforts at both the framework and operator levels. As model parameters grow, the problem becomes more pronounced: tensor parallelism, expert parallelism, and other partitioning strategies, as well as collective communication paths, can prevent true on-policy behavior if any link is misaligned. Operators must also cover multiple invocation scenarios, apply constraints to key computation steps, and keep performance losses within an acceptable range. This is a systemic engineering challenge.
Training-inference inconsistency refers to numerical discrepancies between the Rollout (inference) engine and the training engine. Even when both use exactly the same model weights, the log probabilities (logprob) calculated for the same Token sequence may differ slightly; this difference is typically measured using logdiff.
Using the Ascend C programming language, Ascend has systematically constrained and aligned key operators throughout the training-inference pipeline. The core approach is to make training and inference follow the same numerical path wherever accumulation is required. The main changes include the following four items:
Unified attention semantics: Align the computation range at valid positions between the training mask Softmax and step-by-step inference attention, eliminating numerical differences caused by different sets of visible Tokens.
Fixed reduce accumulation order: Constrain the partitioning and reduction order along the reduction dimensions for training FA and inference PFA / PagedAttention, avoiding accumulation order differences caused by using different partitioning with the same template.
Aligned online Softmax blocking strategy: Unify block sizes and reduction cadence, avoiding inconsistencies between multi-block merged Softmax on the decoder side and block-based Softmax on the training side.
Aligned precision on critical paths: Unify precision conversion strategies at sensitive steps such as Softmax accumulation, reducing mantissa errors introduced by differences in mixed-precision implementations.
Building on this, FA dispatch and scheduling optimizations ensure that training-inference consistency no longer comes at the cost of significant performance degradation, achieving both logdiff=0 and end-to-end acceleration in testing.
Huawei announced that the related infrastructure has been open-sourced and that it will launch a “training-inference consistency issue identification Skill” to help investigate residual logdiff and determine whether the issue lies in the operator path or in differences between framework implementations.
The open-source address is:
https://github.com/verl-project/verl-ascend-recipe/tree/main/true_on_policy