2 GB at the start

[tim-lebedkov]

Just entering the main() in a program linking to "openblas" under MSYS2 consumes 2 GB of RAM.
VMMap shows it as 16 blocks 128 MB each. Are these allocations necessary? Can they be postponed to when I really need something from OpenBLAS?

Setting these variables reduces the memory to 600 MB. It looks like 128 MB is allocated per thread in DLLMain

set OPENBLAS_NUM_THREADS=4
set GOTO_NUM_THREADS=4
set OMP_NUM_THREADS=4

This does not help either:
openblas_set_num_threads(1);

The original issue was against libllama: ggml-org/llama.cpp#18024

[martin-frbg]

That is due to the "GEMM buffer" and associated management structures needed for data exchange in multithreaded operation - to improve performance, this is created as a static array on startup, instead of calling malloc later.
The size of this buffer directly relates to the maximum matrix dimensions the library will be able to handle - if you are sure that you will not work with matrix sizes above 30000, you can reduce it at compile time by passing BUFFERSIZE=20 (or lower) to make or CMake (see Makefile.rule, the 20 translates to 32<<20 i.e. 32MB instead of the default 128MB corresponding to 32<<22)

[tim-lebedkov]

Do you agree that this behavior is bad or should I elaborate further?

[martin-frbg]

I consider it a feature, and it has been that way since the early days of GotoBLAS, 20+ years.
What has changed is the diversity of applications, and the expectation to find a pre-installed library instead of tailoring it to one's particular needs when building from source.

[tim-lebedkov]

I agree that this is a feature, but a bad one, the one that should be removed. A well behaving library should not initialize itself via DLLMain. It should have a function init() and the corresponding done().

As an example, my application consumes about 100 MB. It is a UI. If and when the user decides to talk to AI, I would like to initialize libllama and openblas and compute the answers. And if the users closes the "Chat with AI" window, I would want to free the used memory.

Your argument about a static array and performance is not valid as OpenBLAS calls VirtualAlloc 16 times and this could easily be done in a init(). Obviously, everything what is done in DLLMain should be done in init().

[tim-lebedkov]

While you are still thinking, here is an idea how to do this and keep the backwards compatibility: check GetProcAddress("LIBOPENBLAS_AUTOINIT") in DLLMain and skip the initialization if the function returns a non-null value. The main program could then export a function named "LIBOPENBLAS_AUTOINIT" and prevent the auto-initialization. I have not tested this though.

[jschueller]

hello, what about this patch ?

threads.patch.txt

[martin-frbg]

Sorry, your attachment is unreadable - all I see is an xml'ified error message about range being invalid for the resource.
Also note that this problem may be related to #4665 - there was a somewhat optimistic attempt at reorganizing the GEMM buffer code, but it (probably) led to spurious allocation of a per-thread buffer that goes unused in subsequent operations. Reverting that part of the original PR may already be sufficient to solve this