mariadb

Commit Graph

Author	SHA1	Message	Date
Vladislav Vaintroub	b005b6097f	Cleanup CMake code (Windows-specific) Prepare for a more modern CMake version than the current minimum. - Use CMAKE_MSVC_RUNTIME_LIBRARY instead of the custom MSVC_CRT_TYPE. - Replace CMAKE_{C,CXX}_FLAGS modifications with add_compile_definitions/options and add_link_options. The older method already broke with new pcre2. - Fix clang-cl compilation and ASAN build. - Avoid modifying CMAKE_C_STANDARD_LIBRARIES/CMAKE_CXX_STANDARD_LIBRARIES, as this is discouraged by CMake. - Reduce system checks.	7 months ago
Marko Mäkelä	b6923420f3	MDEV-29445: Reimplement SET GLOBAL innodb_buffer_pool_size We deprecate and ignore the parameter innodb_buffer_pool_chunk_size and let the buffer pool size to be changed in arbitrary 1-megabyte increments. innodb_buffer_pool_size_max: A new read-only startup parameter that specifies the maximum innodb_buffer_pool_size. If 0 or unspecified, it will default to the specified innodb_buffer_pool_size rounded up to the allocation unit (2 MiB or 8 MiB). The maximum value is 4GiB-2MiB on 32-bit systems and 16EiB-8MiB on 64-bit systems. This maximum is very likely to be limited further by the operating system. The status variable Innodb_buffer_pool_resize_status will reflect the status of shrinking the buffer pool. When no shrinking is in progress, the string will be empty. Unlike before, the execution of SET GLOBAL innodb_buffer_pool_size will block until the requested buffer pool size change has been implemented, or the execution is interrupted by a KILL statement a client disconnect, or server shutdown. If the buf_flush_page_cleaner() thread notices that we are running out of memory, the operation may fail with ER_WRONG_USAGE. SET GLOBAL innodb_buffer_pool_size will be refused if the server was started with --large-pages (even if no HugeTLB pages were successfully allocated). This functionality is somewhat exercised by the test main.large_pages, which now runs also on Microsoft Windows. On Linux, explicit HugeTLB mappings are apparently excluded from the reported Redident Set Size (RSS), and apparently unshrinkable between mmap(2) and munmap(2). The buffer pool will be mapped to a contiguous virtual memory area that will be aligned and partitioned into extents of 8 MiB on 64-bit systems and 2 MiB on 32-bit systems. Within an extent, the first few innodb_page_size blocks contain buf_block_t objects that will cover the page frames in the rest of the extent. The number of such frames is precomputed in the array first_page_in_extent[] for each innodb_page_size. In this way, there is a trivial mapping between page frames and block descriptors and we do not need any lookup tables like buf_pool.zip_hash or buf_pool_t::chunk_t::map. We will always allocate the same number of block descriptors for an extent, even if we do not need all the buf_block_t in the last extent in case the innodb_buffer_pool_size is not an integer multiple of the of extents size. The minimum innodb_buffer_pool_size is 256*5/4 pages. At the default innodb_page_size=16k this corresponds to 5 MiB. However, now that the innodb_buffer_pool_size includes the memory allocated for the block descriptors, the minimum would be innodb_buffer_pool_size=6m. my_large_virtual_alloc(): A new function, similar to my_large_malloc(). my_virtual_mem_reserve(), my_virtual_mem_commit(), my_virtual_mem_decommit(), my_virtual_mem_release(): New interface mostly by Vladislav Vaintroub, to separately reserve and release virtual address space, as well as to commit and decommit memory within it. After my_virtual_mem_decommit(), the virtual memory range will be read-only or unaccessible, depending on whether the build option cmake -DHAVE_UNACCESSIBLE_AFTER_MEM_DECOMMIT=1 has been specified. This option is hard-coded on Microsoft Windows, where VirtualMemory(MEM_DECOMMIT) will make the memory unaccessible. On IBM AIX, Linux, Illumos and possibly Apple macOS, the virtual memory will be zeroed out immediately. On other POSIX-like systems, madvise(MADV_FREE) will be used if available, to give the operating system kernel a permission to zero out the virtual memory range. We prefer immediate freeing so that the reported resident set size (RSS) of the process will reflect the current innodb_buffer_pool_size. Shrinking the buffer pool is a rarely executed resource intensive operation, and the immediate configuration of the MMU mappings should not incur significant additional penalty. opt_super_large_pages: Declare only on Solaris. Actually, this is specific to the SPARC implementation of Solaris, but because we lack access to a Solaris development environment, we will not revise this for other MMU and ISA. buf_pool_t::chunk_t::create(): Remove. buf_pool_t::create(): Initialize all n_blocks of the buf_pool.free list. buf_pool_t::allocate(): Renamed from buf_LRU_get_free_only(). buf_pool_t::LRU_warned: Changed to Atomic_relaxed<bool>, only to be modified by the buf_flush_page_cleaner() thread. buf_pool_t::shrink(): Attempt to shrink the buffer pool. There are 3 possible outcomes: SHRINK_DONE (success), SHRINK_IN_PROGRESS (the caller may keep trying), and SHRINK_ABORT (we seem to be running out of buffer pool). While traversing buf_pool.LRU, release the contended buf_pool.mutex once in every 32 iterations in order to reduce starvation. Use lru_scan_itr for efficient traversal, similar to buf_LRU_free_from_common_LRU_list(). buf_pool_t::shrunk(): Update the reduced size of the buffer pool in a way that is compatible with buf_pool_t::page_guess(), and invoke my_virtual_mem_decommit(). buf_pool_t::resize(): Before invoking shrink(), run one batch of buf_flush_page_cleaner() in order to prevent LRU_warn(). Abort if shrink() recommends it, or no blocks were withdrawn in the past 15 seconds, or the execution of the statement SET GLOBAL innodb_buffer_pool_size was interrupted. buf_pool_t::first_to_withdraw: The first block descriptor that is out of the bounds of the shrunk buffer pool. buf_pool_t::withdrawn: The list of withdrawn blocks. If buf_pool_t::resize() is aborted before shrink() completes, we must be able to resurrect the withdrawn blocks in the free list. buf_pool_t::contains_zip(): Added a parameter for the number of least significant pointer bits to disregard, so that we can find any pointers to within a block that is supposed to be free. buf_pool_t::is_shrinking(): Return the total number or blocks that were withdrawn or are to be withdrawn. buf_pool_t::to_withdraw(): Return the number of blocks that will need to be withdrawn. buf_pool_t::usable_size(): Number of usable pages, considering possible in-progress attempt at shrinking the buffer pool. buf_pool_t::page_guess(): Try to buffer-fix a guessed block pointer. If HAVE_UNACCESSIBLE_AFTER_MEM_DECOMMIT is set, the pointer will be validated before being dereferenced. buf_pool_t::get_info(): Replaces buf_stats_get_pool_info(). innodb_init_param(): Refactored. We must first compute srv_page_size_shift and then determine the valid bounds of innodb_buffer_pool_size. buf_buddy_shrink(): Replaces buf_buddy_realloc(). Part of the work is deferred to buf_buddy_condense_free(), which is being executed when we are not holding any buf_pool.page_hash latch. buf_buddy_condense_free(): Do not relocate blocks. buf_buddy_free_low(): Do not care about buffer pool shrinking. This will be handled by buf_buddy_shrink() and buf_buddy_condense_free(). buf_buddy_alloc_zip(): Assert !buf_pool.contains_zip() when we are allocating from the binary buddy system. Previously we were asserting this on multiple recursion levels. buf_buddy_block_free(), buf_buddy_free_low(): Assert !buf_pool.contains_zip(). buf_buddy_alloc_from(): Remove the redundant parameter j. buf_flush_LRU_list_batch(): Add the parameter to_withdraw to keep track of buf_pool.n_blocks_to_withdraw. buf_do_LRU_batch(): Skip buf_free_from_unzip_LRU_list_batch() if we are shrinking the buffer pool. In that case, we want to minimize the page relocations and just finish as quickly as possible. trx_purge_attach_undo_recs(): Limit purge_sys.n_pages_handled() in every iteration, in case the buffer pool is being shrunk in the middle of a purge batch. Reviewed by: Debarun Banerjee	7 months ago
Sergei Golubchik	049d839350	mhnsw: inter-statement shared cache * preserve the graph in memory between statements * keep it in a TABLE_SHARE, available for concurrent searches * nodes are generally read-only, walking the graph doesn't change them * distance to target is cached, calculated only once * SIMD-optimized bloom filter detects visited nodes * nodes are stored in an array, not List, to better utilize bloom filter * auto-adjusting heuristic to estimate the number of visited nodes (to configure the bloom filter) * many threads can concurrently walk the graph. MEM_ROOT and Hash_set are protected with a mutex, but walking doesn't need them * up to 8 threads can concurrently load nodes into the cache, nodes are partitioned into 8 mutexes (8 is chosen arbitrarily, might need tuning) * concurrent editing is not supported though * this is fine for MyISAM, TL_WRITE protects the TABLE_SHARE and the graph (note that TL_WRITE_CONCURRENT_INSERT is not allowed, because an INSERT into the main table means multiple UPDATEs in the graph) * InnoDB uses secondary transaction-level caches linked in a list in in thd->ha_data via a fake handlerton * on rollback the secondary cache is discarded, on commit nodes from the secondary cache are invalidated in the shared cache while it is exclusively locked * on savepoint rollback both caches are flushed. this can be improved in the future with a row visibility callback * graph size is controlled by @@mhnsw_cache_size, the cache is flushed when it reaches the threshold	1 year ago
Sergei Golubchik	32e6f8ff2e	cleanup: remove unconditional #ifdef's	2 years ago
Monty	bddbef3573	MDEV-34533 asan error about stack overflow when writing record in Aria The problem was that when using clang + asan, we do not get a correct value for the thread stack as some local variables are not allocated at the normal stack. It looks like that for example clang 18.1.3, when compiling with -O2 -fsanitize=addressan it puts local variables and things allocated by alloca() in other areas than on the stack. The following code shows the issue Thread 6 "mariadbd" hit Breakpoint 3, do_handle_one_connection (connect=0x5080000027b8, put_in_cache=<optimized out>) at sql/sql_connect.cc:1399 THD thd; 1399 thd->thread_stack= (char) &thd; (gdb) p &thd (THD *) 0x7fffedee7060 (gdb) p $sp (void ) 0x7fffef4e7bc0 The address of thd is 24M away from the stack pointer (gdb) info reg ... rsp 0x7fffef4e7bc0 0x7fffef4e7bc0 ... r13 0x7fffedee7060 140737185214560 r13 is pointing to the address of the thd. Probably some kind of "local stack" used by the sanitizer I have verified this with gdb on a recursive call that calls alloca() in a loop. In this case all objects was stored in a local heap, not on the stack. To solve this issue in a portable way, I have added two functions: my_get_stack_pointer() returns the address of the current stack pointer. The code is using asm instructions for intel 32/64 bit, powerpc, arm 32/64 bit and sparc 32/64 bit. Supported compilers are gcc, clang and MSVC. For MSVC 64 bit we are using _AddressOfReturnAddress() As a fallback for other compilers/arch we use the address of a local variable. my_get_stack_bounds() that will return the address of the base stack and stack size using pthread_attr_getstack() or NtCurrentTed() with fallback to using the address of a local variable and user provided stack size. Server changes are: - Moving setting of thread_stack to THD::store_globals() using my_get_stack_bounds(). - Removing setting of thd->thread_stack, except in functions that allocates a lot on the stack before calling store_globals(). When using estimates for stack start, we reduce stack_size with MY_STACK_SAFE_MARGIN (8192) to take into account the stack used before calling store_globals(). I also added a unittest, stack_allocation-t, to verify the new code. Reviewed-by: Sergei Golubchik <serg@mariadb.org>	1 year ago
Marko Mäkelä	6acada713a	MDEV-34062: Implement innodb_log_file_mmap on 64-bit systems When using the default innodb_log_buffer_size=2m, mariadb-backup --backup would spend a lot of time re-reading and re-parsing the log. For reads, it would be beneficial to memory-map the entire ib_logfile0 to the address space (typically 48 bits or 256 TiB) and read it from there, both during --backup and --prepare. We will introduce the Boolean read-only parameter innodb_log_file_mmap that will be OFF by default on most platforms, to avoid aggressive read-ahead of the entire ib_logfile0 in when only a tiny portion would be accessed. On Linux and FreeBSD the default is innodb_log_file_mmap=ON, because those platforms define a specific mmap(2) option for enabling such read-ahead and therefore it can be assumed that the default would be on-demand paging. This parameter will only have impact on the initial InnoDB startup and recovery. Any writes to the log will use regular I/O, except when the ib_logfile0 is stored in a specially configured file system that is backed by persistent memory (Linux "mount -o dax"). We also experimented with allowing writes of the ib_logfile0 via a memory mapping and decided against it. A fundamental problem would be unnecessary read-before-write in case of a major page fault, that is, when a new, not yet cached, virtual memory page in the circular ib_logfile0 is being written to. There appears to be no way to tell the operating system that we do not care about the previous contents of the page, or that the page fault handler should just zero it out. Many references to HAVE_PMEM have been replaced with references to HAVE_INNODB_MMAP. The predicate log_sys.is_pmem() has been replaced with log_sys.is_mmap() && !log_sys.is_opened(). Memory-mapped regular files differ from MAP_SYNC (PMEM) mappings in the way that an open file handle to ib_logfile0 will be retained. In both code paths, log_sys.is_mmap() will hold. Holding a file handle open will allow log_t::clear_mmap() to disable the interface with fewer operations. It should be noted that ever since commit `685d958e38` (MDEV-14425) most 64-bit Linux platforms on our CI platforms (s390x a.k.a. IBM System Z being a notable exception) read and write /dev/shm/*/ib_logfile0 via a memory mapping, pretending that it is persistent memory (mount -o dax). So, the memory mapping based log parsing that this change is enabling by default on Linux and FreeBSD has already been extensively tested on Linux. ::log_mmap(): If a log cannot be opened as PMEM and the desired access is read-only, try to open a read-only memory mapping. xtrabackup_copy_mmap_snippet(), xtrabackup_copy_mmap_logfile(): Copy the InnoDB log in mariadb-backup --backup from a memory mapped file.	1 year ago
Dimitri John Ledkov	bf77f9793d	openssl: add a more specific DES support detection Improve detection for DES support in OpenSSL, to allow compilation against system OpenSSL without DES. Note that MariaDB needs to be compiled against OpenSSL-like library that itself has DES support which cmake detected. Positive detection is indicated with CMake variable HAVE_des 1. Signed-off-by: Dimitri John Ledkov <dimitri.ledkov@surgut.co.uk>	2 years ago
Marko Mäkelä	f4d75452fc	MDEV-33379 innodb_log_file_buffering=OFF causes corruption on bcachefs Apparently, invoking fcntl(fd, F_SETFL, O_DIRECT) will lead to unexpected behaviour on Linux bcachefs and possibly other file systems, depending on the operating system version. So, let us avoid doing that, and instead just attempt to pass the O_DIRECT flag to open(). This should make us compatible with NetBSD, IBM AIX, as well as Solaris and its derivatives. We will only implement innodb_log_file_buffering=OFF on systems where we can determine the physical block size (typically 512 or 4096 bytes). Currently, those operating systems are Linux and Microsoft Windows. HAVE_FCNTL_DIRECT, os_file_set_nocache(): Remove. OS_FILE_OVERWRITE, OS_FILE_CREATE_PATH: Remove (never used parameters). os_file_log_buffered(), os_file_log_maybe_unbuffered(): Helper functions. os_file_create_func(): When applicable, initially attempt to open files in O_DIRECT mode. For type==OS_LOG_FILE && create_mode != OS_FILE_CREATE we will first invoke stat(2) on the file name to find out if the size is compatible with O_DIRECT. If create_mode == OS_FILE_CREATE, we will invoke fstat(2) on the created log file afterwards, and may close and reopen the file in O_DIRECT mode if applicable. create_temp_file(): Support O_DIRECT. This is only used if O_TMPFILE is available and innodb_disable_sort_file_cache=ON (non-default value). Notably, that setting never worked on Microsoft Windows. row_merge_file_create_mode(): Split from row_merge_file_create_low(). Create a temporary file in the specified mode. Reviewed by: Vladislav Vaintroub	2 years ago
Marko Mäkelä	7f7329f092	MDEV-33379 innodb_log_file_buffering=OFF causes corruption on bcachefs Apparently, invoking fcntl(fd, F_SETFL, O_DIRECT) will lead to unexpected behaviour on Linux bcachefs and possibly other file systems, depending on the operating system version. So, let us avoid doing that, and instead just attempt to pass the O_DIRECT flag to open(). This should make us compatible with NetBSD, IBM AIX, as well as Solaris and its derivatives. This fix does not change the fact that we had only implemented innodb_log_file_buffering=OFF on systems where we can determine the physical block size (typically 512 or 4096 bytes). Currently, those operating systems are Linux and Microsoft Windows. HAVE_FCNTL_DIRECT, os_file_set_nocache(): Remove. OS_FILE_OVERWRITE, OS_FILE_CREATE_PATH: Remove (never used parameters). os_file_log_buffered(), os_file_log_maybe_unbuffered(): Helper functions. os_file_create_simple_func(): When applicable, initially attempt to open files in O_DIRECT mode. os_file_create_func(): When applicable, initially attempt to open files in O_DIRECT mode. For type==OS_LOG_FILE && create_mode != OS_FILE_CREATE we will first invoke stat(2) on the file name to find out if the size is compatible with O_DIRECT. If create_mode == OS_FILE_CREATE, we will invoke fstat(2) on the created log file afterwards, and may close and reopen the file in O_DIRECT mode if applicable. create_temp_file(): Support O_DIRECT. This is only used if O_TMPFILE is available and innodb_disable_sort_file_cache=ON (non-default value). Notably, that setting never worked on Microsoft Windows. row_merge_file_create_mode(): Split from row_merge_file_create_low(). Create a temporary file in the specified mode. Reviewed by: Vladislav Vaintroub	2 years ago
Marko Mäkelä	a6290a5bc5	MDEV-33095 innodb_flush_method=O_DIRECT creates excessive errors on Solaris The directio(3C) function on Solaris is supported on NFS and UFS while the majority of users should be on ZFS, which is a copy-on-write file system that implements transparent compression and therefore cannot support unbuffered I/O. Let us remove the call to directio() and simply treat innodb_flush_method=O_DIRECT in the same way as the previous default value innodb_flush_method=fsync on Solaris. Also, let us remove some dead code around calls to os_file_set_nocache() on platforms where fcntl(2) is not usable with O_DIRECT. On IBM AIX, O_DIRECT is not documented for fcntl(2), only for open(2).	2 years ago
Sergei Golubchik	761d5c8987	MDEV-33092 Undefined reference to concurrency on Solaris remove thr_setconcurrency() followup for `8bbcaab160` Fix by Rainer Orth	2 years ago
Sergei Golubchik	8b0fb154f7	MDEV-33093 plugin/disks/information_schema_disks.cc doesn't compile on Solaris second part of the fix by Rainer Orth	2 years ago
Vladislav Vaintroub	3fad2b1155	MDEV-33096 mysys/my_timezone.cc does not compile on AIX AIX compilation failed, because glibc's non-standard extension to `struct tm` were used - additional members tm_gmtoff and tm_zone. The patch fixes it by adding corresponding compile-time check. Additionally, for the calculation of GMT offset on AIX, a portable variant of timegm() was required.Implementation here is inspired by SergeyD's answer on Stackoverflow : https://stackoverflow.com/questions/16647819/timegm-cross-platform	2 years ago
Vladislav Vaintroub	f8600b1755	MDEV-32567 Remove thr_alarm from server codebase Remove alarm() remnants - Replace thread-unsafe use of alarm() inside my_lock.c with a timed loop. - Remove configure time checks - Remove mysys my_alarm.c/my_alarm.h	2 years ago
Vladislav Vaintroub	013fc02a23	MDEV-32567 Remove thr_alarm from server codebase This allows to simplify net_real_read() and net_real_write() a bit. Removed some superfluous #ifdef/ifndef MYSQL_SERVER from net_serv.cc The code always runs in server, either normal or embedded. Dead code for switching socket between blocking and non-blocking modes, is also removed. Removed pthread_kill() with alarm signal that woke up main thread on server shutdown. Used shutdown(2) on polling sockets instead, to the same effect. Removed yet another superstitious pthread_kill(), that ran on non-Windows in terminate_slave_thread().	2 years ago
Sergei Golubchik	4f9396b9f8	MDEV-31474 KDF() function KDF(key_str, salt [, {info \| iterations} [, kdf_name [, width ]]]) kdf_name is "hkdf" or "pbkdf2_hmac" (default). width (in bits) can be any number divisible by 8, by default it's taken from @@block_encryption_mode iterations must be positive, and is 1000 by default OpenSSL 1.0 doesn't support HKDF, so it'll return NULL. This OpenSSL version is still used in SLES 12 and CentOS 7	2 years ago
Anel Husakovic	602124bb3b	Remove redudant defines USE_MB and USE_MB_IDENT Reviewer: <wlad@mariadb.com>	3 years ago
Marko Mäkelä	df97eb1432	Remove HAVE_SNPRINTF This fixes up commit `77c184df7c` which explicitly specifies that we use ISO/IEC 9899:1999 (C99), which includes the snprintf() function.	3 years ago
Daniel Black	129616c70a	MDEV-28592 disks plugin - getmntinfo (BSD) & getmntent (AIX) Thanks to references from Brad Smith, BSDs use getmntinfo as a system call for mounted filesystems. Most BSDs return statfs structures, (and we use OSX's statfs64), but NetBSD uses a statvfs structure. Simplify Linux getmntent_r to just use getmntent. AIX uses getmntent. An attempt at writing Solaris compatibility with a small bit of HPUX compatibility was made based on man page entries only. Fixes welcome. statvfs structures now use f_bsize for consistency with statfs Test case adjusted as PATH_MAX is OS defined (e.g. 1023 on AIX) Fixes: `0ee5cf837e` also fixes: MDEV-27818: Disk plugin does not show zpool mounted devices This is because zpool mounted point don't begin with /. Due to the proliferation of multiple filesystem types since this was written, we restrict the entries listed in the disks plugin to excude: * read only mount points (no point monitoring, and includes squash, snaps, sysfs, procfs, cgroups...) * mount points that aren't directories (excludes /etc/hostname and similar mounts in containers). (getmntent (Linux/AIX) only) * exclude systems where there is no capacity listed (excludes various virtual filesystem types). Reviewer: Sergei Golubchik	3 years ago
Brad Smith	f02ca429f7	Revert aligned_alloc() addition from MDEV-28836 As pointed out with MDEV-29308 there are issues with the code as is. MariaDB is built as C++11 / C99. aligned_alloc() is not guarenteed to be exposed when building with any mode other than C++17 / C11. The other *BSD's have their stdlib.h header to expose the function with C+11 anyway, but the issue exists in the C99 code too, the build just does not use -Werror. Linux globally defines _GNU_SOURCE hiding the issue as well.	3 years ago
Marko Mäkelä	3794673111	MDEV-28836: Memory alignment cleanup Table_cache_instance: Define the structure aligned at the CPU cache line, and remove a pad[] data member. Krunal Bauskar reported this to improve performance on ARMv8. aligned_malloc(): Wrapper for the Microsoft _aligned_malloc() and the ISO/IEC 9899:2011 <stdlib.h> aligned_alloc(). Note: The parameters are in the Microsoft order (size, alignment), opposite of aligned_alloc(alignment, size). Note: The standard defines that size must be an integer multiple of alignment. It is enforced by AddressSanitizer but not by GNU libc on Linux. aligned_free(): Wrapper for the Microsoft _aligned_free() and the standard free(). HAVE_ALIGNED_ALLOC: A new test. Unfortunately, support for aligned_alloc() may still be missing on some platforms. We will fall back to posix_memalign() for those cases. HAVE_MEMALIGN: Remove, along with any use of the nonstandard memalign(). PFS_ALIGNEMENT (sic): Removed; we will use CPU_LEVEL1_DCACHE_LINESIZE. PFS_ALIGNED: Defined using the C++11 keyword alignas. buf_pool_t::page_hash_table::create(), lock_sys_t::hash_table::create(): lock_sys_t::hash_table::resize(): Pad the allocation size to an integer multiple of the alignment. Reviewed by: Vladislav Vaintroub	3 years ago
Dmitry Shulga	bc7ba7afee	MDEV-27758: Errors when building Connect engine on os x 11.6.2 Added checking for support of vfork by a platform where building being done. Set HAVE_VFORK macros in case vfork() system call is supported. Use vfork() system call if the macros HAVE_VFORK is set, else use fork().	4 years ago
Monty	b1d81974b2	Added support to MEM_ROOT for write protected memory This is useful for thing like Item_true and Item_false that we allocated and initalize once and want to ensure that nothing can change them Main changes: - Memory protection is achived by allocating memory with mmap() and protect it from write with mprotect() - init_alloc_root(...,MY_ROOT_USE_MPROTECT) will create a memroot that one can later use with protect_root() to turn it read only or turn it back to read-write. All allocations to this memroot is done with mmap() to ensure page alligned allocations. - alloc_root() code was rearranged to combine normal and valgrind code. - init_alloc_root() now changes block size to be power of 2's, to get less memory fragmentation. - Changed MEM_ROOT structure to make it smaller. Also renamed MEM_ROOT m_psi_key to psi_key. - Moved MY_THREAD_SPECIFIC marker in MEM_ROOT from block size (old hack) to flags. - Added global variable my_system_page_size. This is initialized at startup.	4 years ago
Vladislav Vaintroub	3d6eb7afcf	MDEV-25602 get rid of __WIN__ in favor of standard _WIN32 This fixed the MySQL bug# 20338 about misuse of double underscore prefix __WIN__, which was old MySQL's idea of identifying Windows Replace it by _WIN32 standard symbol for targeting Windows OS (both 32 and 64 bit) Not that connect storage engine is not fixed in this patch (must be fixed in "upstream" branch)	4 years ago
David CARLIER	9a3cbc0541	mysqld: print status display subset of memory usage. leveling up to some degree with linux's mallinfo* api for the memory usage display with debug build with malloc_zone Darwin api. Closes: #1803	5 years ago
Daniel Black	460d480c74	MDEV-5536: add systemd socket activation Systemd has a socket activation feature where a mariadb.socket definition defines the sockets to listen to, and passes those file descriptors directly to mariadbd to use when a connection occurs. The new functionality is utilized when starting as follows: systemctl start mariadb.socket The mariadb.socket definition only needs to contain the network information, ListenStream= directives, the mariadb.service definition is still used for service instigation. When mariadbd is started in this way, the socket, port, bind-address backlog are all assumed to be self contained in the mariadb.socket definition and as such the mariadb settings and command line arguments of these network settings are ignored. See man systemd.socket for how to limit this to specific ports. Extra ports, those specified with extra_port in socket activation mode, are those with a FileDescriptorName=extra. These need to be in a separate service name like mariadb-extra.socket and these require a Service={mariadb.service} directive to map to the original service. Extra ports need systemd v227 or greater (not RHEL/Centos7 - v219) when FileDescriptorName= was added, otherwise the extra ports are treated like ordinary ports. The number of sockets isn't limited when using systemd socket activation (except by operating system limits on file descriptors and a minimal amount of memory used per file descriptor). The systemd sockets passed can include any ownership or permissions, including those the mariadbd process wouldn't normally have the permission to create. This implementation is compatible with mariadb.service definitions. Those services started with: systemctl start mariadb.service does actually start the mariadb.service and used all the my.cnf settings of sockets and ports like it previously did.	11 years ago
Michael Okoko	48141f3c17	Replace mallinfo with mallinfo2 on supported systems `mallinfo` is deprecated since glibc 2.33 and has been replaced by mallinfo2. The deprecation causes building the server to fail if glibc version is > 2.33. Check if mallinfo2 exist on the system and use it instead.	5 years ago
Daniel Black	3b486c28f7	MDEV-24125: linux large pages, linux/mman.h needed Centos/RHEL7 have the MAP_HUGE_SHIFT constant defined in linux/mman.h which needed to get included.	5 years ago
Vladislav Vaintroub	ccbe6bb6fc	MDEV-19935 Create unified CRC-32 interface Add CRC32C code to mysys. The x86-64 implementation uses PCMULQDQ in addition to CRC32 instruction after Intel whitepaper, and is ported from rocksdb code. Optimized ARM and POWER CRC32 were already present in mysys.	5 years ago
Marko Mäkelä	9ef36faa61	MDEV-23618 InnoDB lacks IA-32 CRC-32C acceleration on GCC 4 When MDEV-22669 introduced CRC-32C acceleration to IA-32, it worked around a compiler bug by disabling the acceleration on GCC 4 for IA-32 altogether, even though the compiler bug only affects -fPIC builds that are targeting IA-32. Let us extend the solution `fe5dbfe723` and define HAVE_CPUID_INSTRUCTION that allows us to implement a necessary and sufficient work-around of the compiler bug.	5 years ago
Marko Mäkelä	b47d61d04f	MDEV-23585: Fix HAVE_CLMUL_INSTRUCTION MDEV-22641 in commit `dec3f8ca69` refactored a SIMD implementation of CRC-32 for the ISO 3309 polynomial that uses the IA-32/AMD64 carry-less multiplication (pclmul) instructions. The code was previously only available in Mariabackup; it was changed to be a general replacement of the zlib crc32(). There exist AMD64 systems where CMAKE_SYSTEM_PROCESSOR matches the pattern i[36]86 but not x86_64 or amd64. This would cause a link failure, because mysys/checksum.c would basically assume that the compiler support for instruction is always available on GCC-compatible compilers on AMD64. Furthermore, we were unnecessarily disabling the SIMD acceleration for 32-bit executables. Note: Until MDEV-22749 has been implemented, the PCLMUL instruction will not be used on Microsoft Windows. Closes: #1660	5 years ago
Karthik Kamath	e6cb263ef3	MDEV-15961: Fix stacktraces under FreeBSD (aarch64) Largely based on MySQL commit `75271e51d6` MySQL Ref: BUG#24566529: BACKPORT BUG#23575445 TO 5.6 (cut) Also, the PTR_SANE macro which tries to check if a pointer is invalid (used when printing pointer values in stack traces) gave false negatives on OSX/FreeBSD. On these platforms we now simply check if the pointer is non-null. This also removes a sbrk() deprecation warning when building on OS X. (It was before only disabled with building using XCode). Removed execinfo path of MySQL patch that was already included. sbrk doesn't exist on FreeBSD aarch64. Removed HAVE_BSS_START based detection and replaced with __linux__ as it doesn't exist on OSX, Solaris or Windows. __bss_start exists on mutiple Linux architectures. Tested on FreeBSD and Linux x86_64. Being in FreeBSD ports for 2 years implies a good testing there on all FreeBSD architectures there too. MySQL-8.0.21 code is functionally identical to original commit.	5 years ago
Vladislav Vaintroub	9c55f83eae	Cleanup - remove HAVE_AIOWAIT and associated code from mysys HAVE_AIOWAIT had not been disabled and unused for at least 10 years.	5 years ago
Daniel Black	9bf3a3a47a	HAVE_LARGE_PAGES no longer global	6 years ago
Sergey Vojtovich	30379b487f	HAVE_LARGE_PAGE_OPTION to HAVE_LARGE_PAGES It isn't an "option" anymore.	6 years ago
Sergey Vojtovich	fc6e8b4b1b	Cleanup HAVE_MMAP_ALIGNED: one cmake check less	6 years ago
Daniel Black	e8351934b6	Merge pull request #1221 from grooverdan/10.4-MDEV-18851-multiple-sized-large-page-support MDEV-18851: multiple sized large page support (linux)	6 years ago
Jon Olav Hauglid	52d7980753	Bug#18913935: REMOVE SUPPORT FOR LINUXTHREADS This patch removes support for LinuxThreads. It was superseded by NPTL in Linux 2.6 (2003).	12 years ago
Anel Husakovic	0d1dd2e79d	Clean wrong cherry-pick from previous commit - Delete variable HAVE_PTHREAD_CONDATTR_SETCLOCK and check - Delete second HAVE_PTHREAD_KEY_DELETE	6 years ago
Daniel Black	fb01cc3766	my_getncpus based on threads available Detecting the cpus based on sysconf of the online CPUs can significantly over estimate the number of cpus available. Wheither via numactl, cgroups, taskset, systemd constraints, docker containers and probably other mechanisms, the number of threads mysqld can be run on can be quite less. As such we use the pthread_getaffinity_np function on Linux and FreeBSD (identical API) to get the number of CPUs. The number of CPUs is the default for the thread_pool_size and a too high default will resulting in large memory usage and high context switching overhead. Closes PR #922	7 years ago
Alexey Botchkov	9dadfdcde5	MDEV-14024 PCRE2. Related changes in the server code.	6 years ago
Vlad Lesin	6718d3bc32	MDEV-21082: isnan/isinf compilation errors, isfinite warnings on MacOS The fix consists of three commits backported from 10.3: 1) Cleanup isnan() portability checks (cherry picked from commit `7ffd7fe962`) 2) Cleanup isinf() portability checks Original problem reported by Wlad: re-compilation of 10.3 on top of 10.2 build would cache undefined HAVE_ISINF from 10.2, whereas it is expected to be 1 in 10.3. std::isinf() seem to be available on all supported platforms. (cherry picked from commit `bc469a0bdf`) 3) Use std::isfinite in C++ code This is addition to parent revision fixing build failures. (cherry picked from commit `54999f4e75`)	8 years ago
Robert Bindar	e8392e58b2	MDEV-19696 - Cleanup gcc sync builtins Since 10.4 requires C++11 capable compiler, gcc sync builtins became dead code. Remove relevant cmake checks and cleanup include files.	6 years ago
Marko Mäkelä	5e929ee8a0	MDEV-19845: Define my_timer_cycles() inline On clang, use __builtin_readcyclecounter() when available. Hinted by Sergey Vojtovich. (This may lead to runtime failure on ARM systems. The hardware should be available on ARMv8 (AArch64), but access to it may require special privileges.) We remove support for the proprietary Sun Microsystems compiler, and rely on clang or the __GNUC__ assembler syntax instead. For now, we retain support for IA-64 (Itanium) and 32-bit SPARC, even though those platforms are likely no longer widely used. We remove support for clock_gettime(CLOCK_SGI_CYCLE), because Silicon Graphics ceased supporting IRIX in December 2013. This was the only cycle timer interface available for MIPS. On PowerPC, we rely on the GCC 4.8 __builtin_ppc_get_timebase() (or clang __builtin_readcyclecounter()), which should be equivalent to the old assembler code on both 64-bit and 32-bit targets.	6 years ago
Alexander Barkov	3e7e87ddcc	MDEV-19897 Rename source code variable names from utf8 to utf8mb3	6 years ago
Marko Mäkelä	042fc29597	MDEV-19845: Adaptive spin loops Starting with the Intel Skylake microarchitecture, the PAUSE instruction latency is about 140 clock cycles instead of earlier 10. On AMD processors, the latency could be 10 or 50 clock cycles, depending on microarchitecture. Because of this big range of latency, let us scale the loops around the PAUSE instruction based on timing results at server startup. my_cpu_relax_multiplier: New variable: How many times to invoke PAUSE in a loop. Only defined for IA-32 and AMD64. my_cpu_init(): Determine with RDTSC the time to run 16 PAUSE instructions in two unrolled loops according, and based on the quicker of the two runs, initialize my_cpu_relax_multiplier. This form of calibration was suggested by Mikhail Sinyavin from Intel. LF_BACKOFF(), ut_delay(): Use my_cpu_relax_multiplier when available. ut_delay(): Define inline in my_cpu.h. UT_COMPILER_BARRIER(): Remove. This does not seem to have any effect, because in our ut_delay() implementation, no computations are being performed inside the loop. The purpose of UT_COMPILER_BARRIER() was to prohibit the compiler from reordering computations. It was not emitting any code.	6 years ago
Yuqi	0928596a8b	Armv8 CRC32 optimization (#772 ) ARMv8 (AArch64) CPUs implement the CRC32 extension which is implemented by inline assembly , so they can also benefit from hardware acceleration in IO-intensive workloads. The patch optimizes crc32c calculate with the armv8 crypto instruction(Intrinsics) when available rather than original linear crc instructions. Change-Id: I05d36a64c726d910c47befad93390108f4e6567f Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>	7 years ago
Vladislav Vaintroub	5e4b657dd4	MDEV-18531 : Use WolfSSL instead of YaSSL as "bundled" SSL/encryption library - Add new submodule for WolfSSL - Build and use wolfssl and wolfcrypt instead of yassl/taocrypt - Use HAVE_WOLFSSL instead of HAVE_YASSL - Increase MY_AES_CTX_SIZE, to avoid compile time asserts in my_crypt.cc (sizeof(EVP_CIPHER_CTX) is larger on WolfSSL)	7 years ago
Vicențiu Ciorbaru	5543b75550	Update FSF Address * Update wrong zip-code	7 years ago
Marko Mäkelä	10dd290b4b	MDEV-17380 innodb_flush_neighbors=ON should be ignored on SSD For tablespaces that do not reside on spinning storage, it does not make sense to attempt to write nearby pages when writing out dirty pages from the InnoDB buffer pool. It is actually detrimental to performance and to the life span of flash ROM storage. With this change, MariaDB will detect whether an InnoDB file resides on solid-state storage. The detection has been implemented for Linux and Microsoft Windows. For other systems, we will err on the safe side and assume that files reside on SSD. As part of this change, we will reduce the number of fstat() calls when opening data files on POSIX systems and slightly clean up some file I/O code. FIXME: os_is_sparse_file_supported() on POSIX works in a destructive manner. Thus, we can only invoke it when creating files, not when opening them. For diagnostics, we introduce the column ON_SSD to the table INFORMATION_SCHEMA.INNODB_TABLESPACES_SCRUBBING. The table INNODB_SYS_TABLESPACES might seem more appropriate, but its purpose is to reflect the contents of the InnoDB system table SYS_TABLESPACES, which we would like to remove at some point. On Microsoft Windows, querying StorageDeviceSeekPenaltyProperty sometimes returns ERROR_GEN_FAILURE instead of ERROR_INVALID_FUNCTION or ERROR_NOT_SUPPORTED. We will silently ignore also this error, and assume that the file does not reside on SSD. On Linux, the detection will be based on the files /sys/block//queue/rotational and /sys/block//dev. Especially for USB storage, it is possible that /sys/block//queue/rotational will wrongly report 1 instead of 0. fil_node_t::on_ssd: Whether the InnoDB data file resides on solid-state storage. fil_system_t::ssd: Collection of Linux block devices that reside on non-rotational storage. fil_system_t::create(): Detect ssd on Linux based on the contents of /sys/block//queue/rotational and /sys/block/*/dev. fil_system_t::is_ssd(dev_t): Determine if a Linux block device is non-rotational. Partitions will be identified with the containing block device by assuming that the least significant 4 bits of the minor number identify a partition, and that the "partition number" of the entire device is 0.	7 years ago

1 2 3 4 5 ...

266 Commits (mariadb-12.0.1)