Improving glibc malloc for high reliability large data multi threaded applications

Описание: As memory and performance capabilities of computer have increased, virtualization has been the preferred approach to exploit all those resources.
To efficiently use such a system it is required to maximize the number of concurrent applications.
This makes it extremely important to fully optimize any running application not only for time (cpu) but also space (memory).
Moreover, high reliability and large data multi-threaded applications require not only an efficient allocation strategy but also a reliable and fast concurrency mechanism.

The solution for parallelism in glibc malloc (arenas) is based on minimizing concurrency through isolation, splitting the memory space through different threads using virtual memory as an abstraction layer.
This solution defers memory management to the kernel rather than keeping the responsibility to explicitly perform minimum virtual allocation.
Glibc malloc has been identified by projects such as MySQL and JVM as using more memory than required, and this has justified the adoption of other more multi-threaded friendly allocators, such as jemalloc and tcmalloc.

This talk will present MySQL performance and data size analysis using both glibc malloc and other competitive allocators, and show how recent improvements in the glibc malloc implementation greatly reduce virtual memory consumption in real-world MySQL usage, making it unnecessary to resort to other specialized allocators. Further possible improvements will also be discussed, which can make glibc the best allocator for high availability large data multi-threaded applications.