[Vulkan] Fuse SwiGLU elementwise operations

[mergennachin]

Summary

SwiGLU currently runs a sigmoid and two multiplies as separate Vulkan dispatches. Fuse gate * sigmoid(gate) * up into one binary kernel. For Qwen3-0.6B, this removes 56 dispatches per token (737 → 681). Models need to be re-exported to use the fusion.

The matcher handles either multiply operand order and skips shared sigmoid/SiLU intermediates. The fused op reuses the existing binary-op broadcasting, packing, and dynamic-resize logic for FP16/FP32 buffers and 3D textures. FP16 inputs are evaluated in FP32 with one output conversion, so results can differ slightly from the unfused intermediate rounding.

On an Apple M1 Pro with MoltenVK, Qwen3-0.6B W4 (group size 32, FP16 storage, context limit 1024):

Prompt tokens	Median decode ms/token, before → after	Median paired decode TPS gain	Median prefill ms, before → after
41	20.47 → 19.52	+4.1%	106.5 → 106.0
768	21.87 → 20.59	+6.5%	662.5 → 656.5

Four alternating pairs per prompt, full warmup, and 64 measured decode tokens per run using the same unprofiled Release runner. All eight pairs favored fusion, with identical generated text within each prompt. Prefill showed no regression in these runs. Both benchmark arms include the W4 specialization from #22973 and the merged view removal from #22898; this PR contains only SwiGLU fusion. Both runners used the same 255-token prefill chunk cap to respect the export's shape bound. Adreno performance has not been measured.

Test plan

All 21 tests in backends/vulkan/test/test_vulkan_passes.py passed. The five new SwiGLU tests cover operand order, broadcasting, dynamic projection ordering, shared intermediates, and mixed precision; they were rerun after test cleanup.

All 56 cases selected by vulkan_compute_api_test --gtest_filter='*VulkanSwiGLUTest*' passed on MoltenVK. Each exercises row counts 1 → 7 → 63 → 1, covering FP16/FP32, buffers/textures, packed layouts, odd widths, broadcasting, and the 3072-wide LLM case. Results are checked against an independent float reference and the unfused GPU path with half-precision rounding bounds. Full model runs also exercised prefill chunks up to 255 followed by decode.

The new shaders and C++ dispatch code built in Release. Python/C++ lint and formatting checks passed. GPU traces confirmed that the only dispatch-count changes were 28 fewer sigmoid kernels, 56 fewer multiply kernels, and 28 new SwiGLU kernels.

Authored with assistance from OpenAI Codex.

cc @SS-JIA @manuelcandales @digantdesai @cbilgin

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/22979

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

• server certificate verification failed on github.com

❌ 2 New Failures

As of commit 5721f54 with merge base dd8297a ():

NEW FAILURES - The following jobs have failed:

• pull / nxp-mcuxpresso-test / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / unittest-nxp-neutron / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[meta-codesync]

@mergennachin has imported this pull request. If you are a Meta employee, you can view this in D121198447.

[Copilot]

Copilot review overview

🟢 Approval recommended

No unresolved blocking issues were identified, and the reported test coverage passed.

Review effort: Lite
Findings: None

What changed in this PR

Fuses Vulkan SwiGLU sigmoid/multiply chains into a single binary kernel, reducing dispatch overhead while retaining dtype, layout, broadcasting, and resize support.

Changes:

• Adds SwiGLU pattern detection and replacement.
• Implements buffer and texture shader variants.
• Registers the fused operator and adds pass/runtime coverage.

File	Summary
backends/​vulkan/​test/​vulkan_compute_api_test.cpp	Native fused-kernel correctness and resize tests
backends/​vulkan/​test/​test_vulkan_passes.py	Pattern fusion tests
backends/​vulkan/​runtime/​graph/​ops/​impl/​BinaryOp.cpp	Runtime operator registration
backends/​vulkan/​runtime/​graph/​ops/​glsl/​binary_op_texture.yaml	Texture shader variants
backends/​vulkan/​runtime/​graph/​ops/​glsl/​binary_op_texture.glsl	Texture implementation
backends/​vulkan/​runtime/​graph/​ops/​glsl/​binary_op_buffer.yaml	Buffer shader variants
backends/​vulkan/​runtime/​graph/​ops/​glsl/​binary_op_buffer.glsl	Buffer implementation
backends/​vulkan/​patterns/​swiglu.py	Pattern detection and replacement
backends/​vulkan/​patterns/​BUCK	Build inclusion
backends/​vulkan/​patterns/​__init__.py	Pattern registration
backends/​vulkan/​op_registry.py	Operator capability registration
backends/​vulkan/​custom_ops_lib.py	Custom operator definition

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.