ACL: INT8 3x3 slower than F32 on small-channel conv (Neoverse-V2, SVE2)

[allnes]

Hi, @milpuz01

I catch perf degradation

Environment

• HW: ARM Neoverse-V2 (flags: sve, sve2, i8mm, bf16, dotprod)
• ACL build (scons):
  scons -j8 Werror=0 debug=0 neon=1 opencl=0 embed_kernels=1
  benchmark_tests=0 validation_tests=0 asserts=0
  arch=arm64-v8.2-a-sve2
  (compiles with -march=armv8.2-a+sve2+fp16+dotprod; SVE2 enabled)

Minimal reproducer (standalone)

Single NEConvolutionLayer benchmark, parameters via CLI.

Build (save code below as repro_conv.cpp):

g++ -std=c++14 -O3 -march=armv8.2-a+sve2+fp16+dotprod
-I<ACL_ROOT> -I<ACL_ROOT>/include
repro_conv.cpp -L<ACL_ROOT>/build -larm_compute -pthread
-Wl,-rpath,<ACL_ROOT>/build -o conv_repro

Runs (NCHW, N=1, threads=4):

1. Problem case: 3x3, H=W=75, C=32->32, stride=1, pad=1

./conv_repro --prec=i8 --h=75 --w=75 --in_c=32 --out_c=32 --k=3 --stride=1 --pad=1 --iters=50 --warmup=5 --threads=4

avg_ms ≈ 0.471

./conv_repro --prec=f32 --h=75 --w=75 --in_c=32 --out_c=32 --k=3 --stride=1 --pad=1 --iters=50 --warmup=5 --threads=4

avg_ms ≈ 0.276

INT8 ~1.7x slower than F32

2. Reference where INT8 is faster (first 7x7 layer)

./conv_repro --prec=i8 --iters=50 --warmup=5 --threads=4 # avg_ms ≈ 1.192 (H=W=300, C:3->32, k=7, s=2, p=3)
./conv_repro --prec=f32 --iters=50 --warmup=5 --threads=4 # avg_ms ≈ 1.882

Source (repro_conv.cpp)

#include "arm_compute/runtime/NEON/NEFunctions.h"
#include "arm_compute/runtime/Tensor.h"
#include "arm_compute/runtime/Scheduler.h"
#include "arm_compute/core/Helpers.h"
#include "arm_compute/core/TensorShape.h"
#include "arm_compute/core/Types.h"
#include <chrono>
#include <iostream>
#include <random>
#include <string>

using namespace arm_compute;

struct Options {
    std::string prec = "i8"; // i8 | f32
    unsigned iters = 50, warmup = 5, threads = 4;
    unsigned h = 300, w = 300, in_c = 3, out_c = 32;
    unsigned k = 7, stride = 2, pad = 3;
};

Options parse(int argc, char** argv) {
    Options o;
    for (int i = 1; i < argc; ++i) {
        std::string a = argv[i];
        auto val = [&](const std::string&) { auto p = a.find('='); return p == std::string::npos ? std::string() : a.substr(p + 1); };
        if      (a.rfind("--prec=",0)==0) o.prec    = val(a);
        else if (a.rfind("--iters=",0)==0)o.iters   = std::stoul(val(a));
        else if (a.rfind("--warmup=",0)==0)o.warmup = std::stoul(val(a));
        else if (a.rfind("--threads=",0)==0)o.threads=std::stoul(val(a));
        else if (a.rfind("--h=",0)==0)    o.h      = std::stoul(val(a));
        else if (a.rfind("--w=",0)==0)    o.w      = std::stoul(val(a));
        else if (a.rfind("--in_c=",0)==0) o.in_c   = std::stoul(val(a));
        else if (a.rfind("--out_c=",0)==0)o.out_c  = std::stoul(val(a));
        else if (a.rfind("--k=",0)==0)    o.k      = std::stoul(val(a));
        else if (a.rfind("--stride=",0)==0)o.stride= std::stoul(val(a));
        else if (a.rfind("--pad=",0)==0)  o.pad    = std::stoul(val(a));
    }
    return o;
}

void fill_random(Tensor& t) {
    std::mt19937 gen(0);
    std::uniform_int_distribution<int> dist_i(-128, 127);
    std::uniform_real_distribution<float> dist_f(-1.f, 1.f);
    auto dt = t.info()->data_type();
    Window win; win.use_tensor_dimensions(t.info()->tensor_shape());
    Iterator it(&t, win);
    execute_window_loop(win, [&](const Coordinates&) {
        if (dt == DataType::QASYMM8 || dt == DataType::QASYMM8_SIGNED)
            *reinterpret_cast<int8_t*>(it.ptr()) = static_cast<int8_t>(dist_i(gen));
        else
            *reinterpret_cast<float*>(it.ptr()) = dist_f(gen);
    }, it);
}

int main(int argc, char** argv) {
    Options opt = parse(argc, argv);
    Scheduler::get().set_num_threads(opt.threads);

    TensorShape src_shape(opt.w, opt.h, opt.in_c, 1U);
    TensorShape wei_shape(opt.k, opt.k, opt.in_c, opt.out_c);
    unsigned out_w = (opt.w + 2 * opt.pad - opt.k) / opt.stride + 1;
    unsigned out_h = (opt.h + 2 * opt.pad - opt.k) / opt.stride + 1;
    TensorShape dst_shape(out_w, out_h, opt.out_c, 1U);
    PadStrideInfo conv_info(opt.stride, opt.stride, opt.pad, opt.pad);

    DataType dt = (opt.prec == "f32") ? DataType::F32 : DataType::QASYMM8_SIGNED;
    QuantizationInfo qi(1.f, 0);
    Tensor src, wei, dst;
    src.allocator()->init(TensorInfo(src_shape, 1, dt, qi));
    wei.allocator()->init(TensorInfo(wei_shape, 1, dt, qi));
    dst.allocator()->init(TensorInfo(dst_shape, 1, dt, qi));

    NEConvolutionLayer conv;
    conv.configure(&src, &wei, nullptr, &dst,
                   conv_info, WeightsInfo(), Size2D(1U, 1U),
                   ActivationLayerInfo(), false, 1);

    src.allocator()->allocate();
    wei.allocator()->allocate();
    dst.allocator()->allocate();
    fill_random(src);
    fill_random(wei);

    for (unsigned i = 0; i < opt.warmup; ++i) conv.run();
    auto t0 = std::chrono::high_resolution_clock::now();
    for (unsigned i = 0; i < opt.iters; ++i) conv.run();
    auto t1 = std::chrono::high_resolution_clock::now();

    double ms = std::chrono::duration<double, std::milli>(t1 - t0).count();
    std::cout << "prec=" << opt.prec
              << " h=" << opt.h << " w=" << opt.w
              << " in_c=" << opt.in_c << " out_c=" << opt.out_c
              << " k=" << opt.k << " s=" << opt.stride << " p=" << opt.pad
              << " threads=" << opt.threads
              << " iters=" << opt.iters
              << " avg_ms=" << ms / opt.iters
              << std::endl;
    return 0;
}

Observation

• On Neoverse-V2 with SVE2 build, INT8 3×3 (75×75, C=32) is still ~1.7× slower than F32. Large 7×7 remains faster in INT8. Please investigate INT8 kernel selection/perf for this pattern.

[milpuz01]

cc @morgolock

[morgolock]

Hi @allnes

This is expected to be so because for F32 we use ConvolutionMethod::WINOGRAD and for INT8 we use ConvolutionMethod::GEMM for this specific workload configuration.

See below, if I disable Winograd for F32 the INT8 conv is faster.

[build]$ ./conv_repro --prec=f32 --h=75 --w=75 --in_c=64 --out_c=64 --k=3 --stride=1 --pad=1 --iters=50 --warmup=5 --threads=4
prec=f32 h=75 w=75 in_c=64 out_c=64 k=3 s=1 p=1 threads=4 iters=50 avg_ms=1.74266
[build]$ ./conv_repro --prec=i8 --h=75 --w=75 --in_c=64 --out_c=64 --k=3 --stride=1 --pad=1 --iters=50 --warmup=5 --threads=4
prec=i8 h=75 w=75 in_c=64 out_c=64 k=3 s=1 p=1 threads=4 iters=50 avg_ms=0.846248

Hope this helps

[allnes]

@morgolock

I reproduced this on Neoverse‑V2 (SVE2) using ACL master, and the fastest time is still F32 Winograd, not INT8.

• Repro: repro_conv.cpp (same as in the issue)

Important: how I forced GEMM/WINOGRAD
ACL does not expose a public runtime switch to force the convolution method.
So I made a local change in the library:

• File: src/cpu/operators/CpuConv2d.cpp

• Function: CpuConv2d::get_convolution_method(...)

• Change: added an env var override:

• If ACL_FORCE_CONV_METHOD=GEMM, return ConvolutionMethod::GEMM

• If ACL_FORCE_CONV_METHOD=WINOGRAD, return ConvolutionMethod::WINOGRAD

• (also supported: GEMM_CONV2D, DIRECT, FFT)

This is the only way I could force the method without changing the public API.

How I measured

• Default:

./conv_repro --prec=i8 --h=75 --w=75 --in_c=32 --out_c=32 --k=3 --stride=1 --pad=1 --iters=50 --warmup=5 --threads=4
./conv_repro --prec=f32 --h=75 --w=75 --in_c=32 --out_c=32 --k=3 --stride=1 --pad=1 --iters=50 --warmup=5 --threads=4

• Forced methods (requires the local change above):

ACL_FORCE_CONV_METHOD=GEMM ./conv_repro ...
ACL_FORCE_CONV_METHOD=WINOGRAD ./conv_repro ...

Table (3×3, 75×75, C=32→32, s=1, p=1, threads=4)

Precision	GEMM avg_ms	WINOGRAD avg_ms
INT8	0.461086	N/A (not supported)
F32	0.764315	0.273943

Conclusion

• In the default configuration INT8 is slower than F32 (~1.72×).
• Even when forcing methods, the fastest time is still F32 Winograd (0.2739 ms).
• INT8 cannot beat F32 here because INT8 Winograd on CPU is not supported.

We’re hitting a performance dead‑end for the INT8 networks we’re analyzing at Phoronix: even with method forcing, the fastest results still come from F32 Winograd, so INT8 can’t win on these cases without an INT8 Winograd (or a faster INT8 path).

[alvoron]

@morgolock please correct me if I'm missing something: I assume we need to use fp32/fp16 winograd in int8 models when it's possible since it's faster than int8 GEMM, right?
Do you have plans to support int8 Winograd?

[allnes]

@morgolock could you please answer for @alvoron questions

[morgolock]

Hi @allnes

You’re slightly mixing two different things.

ACL Winograd is currently FP-only (F16/F32), so an INT8 model cannot directly “use Winograd” unless that layer is run in mixed precision, i.e. dequantize to FP16/FP32, run Winograd, then requantize. That is not a general recommendation, because it adds conversion overhead and changes the numerics. For the reported case, the main issue is not “INT8 vs Winograd” in isolation, but that INT8/NCHW falls back to GEMM, while F32/NCHW can use Winograd. In fact, on this shape, INT8/NHWC is already much faster because it can use the faster INT8-native GEMM_CONV2D path.

So the practical recommendation for INT8 models is: prefer the INT8-native fast path (NHWC in this case), rather than trying to route INT8 layers through FP Winograd.

ACL does not currently support INT8 Winograd. I’m not aware of a committed plan to add INT8 Winograd, so I would not want to promise that. On CPU, the faster INT8 GEMM_CONV2D path is NHWC-only. For INT8/NCHW, ACL may fall back to the separate GEMM (im2col + GEMM) path, which is slower for this shape.

For 75x75x32, 3x3 s1 p1 case on the Neoverse-V2 test host, the numbers I measured were:

• F32 / NCHW / WINOGRAD: about 0.239 ms
• QASYMM8_SIGNED / NCHW / GEMM: about 0.387 ms
• QASYMM8_SIGNED / NHWC / GEMM_CONV2D: about 0.080 ms

So INT8/NHWC was faster than F32/NCHW/Winograd in that setup.

Hope this helps

[morgolock]

Hi @alvoron

I used neon_issue1225_conv_probe.cpp on the Neoverse-V2 machine.

Build:

scons -j$(nproc) build=native Werror=0 debug=0 neon=1 opencl=0 embed_kernels=1 \
      benchmark_tests=0 validation_tests=0 asserts=0 examples=1 gemm_tuner=0 \
      arch=arm64-v8.2-a-sve2 build_dir=issue1225_v82 neon_issue1225_conv_probe

Run the reported shape:

LD_LIBRARY_PATH=$PWD/build/issue1225_v82 \
   ./build/issue1225_v82/examples/neon_issue1225_conv_probe \
   --preset=problem --dtype=f32 --layout=nchw --threads=4 --iterations=30

Then rerun with

• --dtype=qasymm8_signed --layout=nchw
• --dtype=qasymm8_signed --layout=nhwc
  to compare the INT8 paths.

Read method.selected and timing.steady_state_avg_ms from the output.

On our Neoverse-V2 rerun for this shape:

• F32/NCHW selected WINOGRAD and measured about 0.245 ms
• QASYMM8_SIGNED/NCHW selected GEMM and measured about 0.408 ms
• QASYMM8_SIGNED/NHWC selected GEMM_CONV2D and measured about 0.080 ms.

For INT8 CPU workloads in ACL, prefer an end-to-end NHWC layout where feasible. In this case, INT8/NCHW falls back to a slower GEMM path, while INT8/NHWC can use the faster INT8-native GEMM_CONV2D path.

Hope this helps

[alvoron]

@morgolock today we discussed #1273 on the call
I was interested in pooling results reproducer, not this one :)

[morgolock]

@morgolock today we discussed #1273 on the call I was interested in pooling results reproducer, not this one :)

I've also shared the pooling test here #1273 (comment)

Hope this helps

[morgolock]

Hi @alvoron

Do you still need support with this?