mariadb

Commit Graph

Author	SHA1	Message	Date
Vladislav Vaintroub	3bd23b76c5	MDEV-34740 mariadb-import: optimize index and constraint creation For InnoDB tables, parse the CREATE TABLE statement to defer index and constraint creation until after data loading. For other storage engines, the DISABLE/ENABLE KEYS commands achieve similar optimization. This behavior is controlled by a new option, innodb-optimize-keys (default: ON), compatible with mydumper. Additionally, this commit separates the table creation phase from data loading. Running DDL statements (such as DROP IF EXISTS) in a single thread avoids the "table not locked" issue from MDEV-34741. As a bonus, view creation no longer requires a separate step.	9 months ago
Vladislav Vaintroub	9255206b86	appveyor - run builds in "main" branch	9 months ago
Vicențiu Ciorbaru	b8fa8b9b3d	MDEV-35921: s3.mysqldump fails in buildbot This is a fixup for MDEV-32250, introduced via `b24ecd7ca6`, the test case was not recorded given the new mariadb-dump format.	9 months ago
Alexander Barkov	89f5d28191	MDEV-22217 Make OS character sets "utf8" and "utf-8" map to MariaDB character set "utf8mb4" Map Unix utf8 locales to utf8mb4 instead of utf8mb3.	1 year ago
Sergei Golubchik	e11592aed3	MDEV-35450 VEC_DISTANCE() function to autouse the available index type	11 months ago
Sergei Golubchik	528249a20a	cleanup: one Item_func_vec_distance class, not three prepare for MDEV-35450 VEC_DISTANCE auto-detection	11 months ago
Sergei Golubchik	d2ec5ec9c2	cleanup: move test w/ character_set_results=utf16 into separate file	11 months ago
Sergei Golubchik	e36ca5b1a6	cleanup: extraneous quotes in errmsg.txt	11 months ago
Vladislav Vaintroub	5482155df6	MDEV-34703 followup - reenable Innodb bulk load in mariadb-import Since bulk load is fixed, in some newer versions of the server, reenable it, after checking whether MDEV-34703 was really fixed in server.	10 months ago
Sergey Vojtovich	11a6c1b30a	MDEV-34699 - mhnsw: support aarch64 SIMD instructions SIMD implementations of bloom filters and dot product calculation. A microbenchmark shows 1.7x dot product performance improvement compared to regular -O2/-O3 builds and 2.4x compared to builds with auto-vectorization disabled. Performance improvement (microbenchmark) for bloom filters is less exciting, within 10-30% ballpark depending on compiler options and load. Misc implementation notes: CalcHash: no _mm256_shuffle_epi8(), use explicit XOR/shift. CalcHash: no 64bit multiplication, do scalar multiplication. ConstructMask/Query: no _mm256_i64gather_epi64, access array elements explicitly. Query: no _mm256_movemask_epi8, accumulate bits manually. Closes #3671	11 months ago
Dave Gosselin	cacaaebf01	MDEV-35837 Move to c++17 Move from c++11 to c++17.	10 months ago
Vladislav Vaintroub	2563839853	MDEV-34979 generate SBOM from server builds This commit adds the capability to generate a Software Bill of Materials (SBOM) from server builds. It introduces a new WITH_SBOM variable, which defaults to ON for package builds (i.e if BUILD_CONFIG is used) and to OFF otherwise. When enabled, the build process will produce an sbom.json document in CycloneDX format, capturing information about various dependencies, which is gathered from various sources. We use git submodule information and CMake external projects properties to gather version information for 3rd party code, but also handle dependencies if external code is part of our repository (zlib, or Connect storage engine's minizip) The SBOM document is stored in the root build directory in sbom.json file, but is not currently installed.	10 months ago
Marko Mäkelä	18dbeae1b8	MDEV-35849: index records in a wrong order page_cur_dtuple_cmp(): Take DESC index fields into account also when comparing a NULL value in a ROW_FORMAT≠REDUNDANT record. Thanks to Elena Stepanova for finding this bug.	10 months ago
Marko Mäkelä	3761a7fec8	MDEV-35312 page_cur_search_with_match() could avoid rec_get_offsets() page_cur_search_with_match(): Remove rec_get_offsets(), and instead determine the start and end of each field while comparing. page_dir_slot_get_rec(), page_dir_slot_get_rec_validate(): Add a parameter to avoid invoking page_align(). page_cur_dtuple_cmp(): Replaces cmp_dtuple_rec_leaf() for both leaf and non-leaf pages. In SPATIAL INDEX, non-leaf records are special, because the child page number may be part of the comparison. Reviewed by: Vladislav Lesin	10 months ago
Marko Mäkelä	793a2fc8ba	MDEV-35049: Always enable page_cur_search_with_match_bytes() For some reason, page_cur_search_with_match_bytes(), which can speed up append operations (PAGE_CUR_LE used by INSERT), was only enabled if innodb_adaptive_hash_index=ON even though it has nothing to do with the adaptive hash index. Furthermore, mysql/mysql-server@c9bbc83d11b56831658b03475b23306e3d48acca a.k.a. commit `c9bbc83d11` reduced a limit from 3 to 2 but forgot to adjust the PAGE_N_DIRECTION limit accordingly. We are adjusting that as well. Reviewed by: Vladislav Lesin	10 months ago
Marko Mäkelä	4221ed1d7d	MDEV-35049: Avoid building AHI beyond unique field prefix During a workload, an adaptive hash index had been built on UNIQUE INDEX(ID) on SYS_TABLES, and during a DROP TABLE operation the adaptive hash index would be widened to cover also the PRIMARY KEY(NAME) field that the index includes: (ID,NAME). Such an adaptive hash index is unlikely to satisfy (m)any queries. Let us limit the AHI prefix to the unique fields. Reviewed by: Vladislav Lesin	10 months ago
Marko Mäkelä	5f7b2a3ced	MDEV-35049: Improve btr_search_drop_page_hash_index() btr_search_drop_page_hash_index(): Replace the Boolean parameter with const dict_index_t *not_garbage. If buf_block_t::index points to that, there is no need to acquire btr_sea::partition::latch. The old parameter bool garbage_collect=false is equivalent to the parameter not_garbage=nullptr. The parameter garbage_collect=true will be replaced either with the actual index that is associated with the buffer page, or with a bogus pointer not_garbage=-1 to indicate that any lazily entries for a freed index need to be removed. buf_page_get_low(), buf_page_get_gen(), mtr_t::page_lock(), mtr_t::upgrade_buffer_fix(): Do not invoke btr_search_drop_page_hash_index(). Our caller will have to do it when appropriate. buf_page_create_low(): Keep invoking btr_search_drop_page_hash_index(). This is the normal way of lazily dropping the adaptive hash index after a DDL operation such as DROP INDEX operation. btr_block_get(), btr_root_block_get(), btr_root_adjust_on_import(), btr_read_autoinc_with_fallback(), btr_cur_instant_init_low(), btr_cur_t::search_leaf(), btr_cur_t::pessimistic_search_leaf(), btr_pcur_optimistic_latch_leaves(), dict_stats_analyze_index_below_cur(): Invoke btr_search_drop_page_hash_index(block, index) for pages that may be leaf pages. No adaptive hash index may have been created on anything else than a B-tree leaf page. btr_cur_search_to_nth_level(): Do not invoke btr_search_drop_page_hash_index(), because we are only accessing non-leaf pages and the adaptive hash index may only have been created on leaf pages. btr_page_alloc_for_ibuf() and many other callers of buf_page_get_gen() or similar functions do not invoke btr_search_drop_page_hash_index(), because the adaptive hash index is never created on such pages. If a page in the tablespace was freed as part of a DDL operation and reused for something else, then buf_page_create_low() will take care of dropping the adaptive hash index before the freed page will be modified. It is notable that while the flst_ functions may access pages that are related to allocating B-tree index pages (the BTR_SEG_TOP and BTR_SEG_LEAF linked from the index root page), those pages themselves can never be stored in the adaptive hash index. Therefore, it is not necessary to invoke btr_search_drop_page_hash_index() on them. Reviewed by: Vladislav Lesin	10 months ago
Marko Mäkelä	c942b31340	MDEV-35049: Fix bogus rebuild on BTR_CUR_HASH_FAIL btr_search_info_update_hash(): Do nothing if the record is positioned on the page supremum or infimum pseudo-record. The adaptive hash index can only include user records. This deficiency would cause the adaptive hash index parameters to change between hashing a prefix of 1 field or a prefix of 1 byte. Reviewed by: Vladislav Lesin	10 months ago
Marko Mäkelä	6b58ee769f	MDEV-35049: Fix bogus BTR_CUR_HASH_FAIL on contention btr_search_guess_on_hash(): Only set BTR_CUR_HASH_FAIL on actual mismatch. If the page latch cannot be acquired, the hash search might very well have succeeded. Do not count that as a failure, that is, do not unnecessarily invoke btr_search_update_hash_ref() after a normal search. Set cursor->flag=BTR_CUR_HASH_ABORT if the current parameters of the adaptive hash index are not suitable for the search and a call to btr_cur_t::search_info_update() might help. btr_cur_t::search_leaf(): Do not invoke search_info_update() if btr_search_guess_on_hash() failed due to contention. btr_cur_t::pessimistic_search_leaf(): Do not invoke search_info_update() on the change buffer tree. Preivously, this condition was being checked inside search_info_update().	10 months ago
Marko Mäkelä	68cac26108	MDEV-35049: Fix bogus BTR_CUR_HASH_FAIL on PAGE_CUR_LE btr_cur_t::search_leaf(): Do not attempt to use the adaptive hash index for PAGE_CUR_G or PAGE_CUR_L, because those modes expect an inequal result, and the adaptive hash index can only deliver equal results. btr_cur_t::check_mismatch(): Only handle PAGE_CUR_LE and PAGE_CUR_GE. For PAGE_CUR_LE (bool ge=false), qualify a full match for the last record of a page that is not at the end of the index. Previously, an adaptive hash index lookup would fail when the record is at the end of an index page but not at the end of the index. This would lead to unnecessary rebuild of the adaptive hash index in read-only workloads. Reviewed by: Vladislav Lesin	10 months ago
Marko Mäkelä	4dd6131711	MDEV-35049: Privatize ut_fold_ulint_pair() Now that ut_fold_ulint_pair() and ut_fold_binary() are no longer needed for anything else than compatibility with old InnoDB data files that may use innodb_checksum_algorithm=innodb, let us move the code to a single compilation unit. Reviewed by: Vladislav Lesin	10 months ago
Marko Mäkelä	4dcb1b575b	MDEV-35049: Use CRC-32C and avoid allocating heap For the adaptive hash index, dtuple_fold() and rec_fold() were employing a slow rolling hash algorithm, computing hash values ("fold") for one field and one byte at a time, while depending on calls to rec_get_offsets(). We already have optimized implementations of CRC-32C and have been successfully using that function in some other InnoDB tables, but not yet in the adaptive hash index. Any linear function such as any CRC will fail the avalanche test that any cryptographically secure hash function is expected to pass: any single-bit change in the input key should affect on average half the bits in the output. But we always were happy with less than cryptographically secure: in fact, ut_fold_ulint_pair() or ut_fold_binary() are just about as linear as any CRC, using a combination of multiplication and addition, partly carry-less. It is worth noting that exclusive-or corresponds to carry-less subtraction or addition in a binary Galois field, or GF(2). We only need some way of reducing key prefixes into hash values. The CRC-32C should be better than a Rabin–Karp rolling hash algorithm. Compared to the old hash algorithm, it has the drawback that there will be only 32 bits of entropy before we choose the hash table cell by a modulus operation. The size of each adaptive hash index array is (innodb_buffer_pool_size / 512) / innodb_adaptive_hash_index_parts. With the maximum number of partitions (512), we would not exceed 1<<32 elements per array until the buffer pool size exceeds 1<<50 bytes (1 PiB). We would hit other limits before that: the virtual address space on many contemporary 64-bit processor implementations is only 48 bits (256 TiB). So, we can simply go for the SIMD accelerated CRC-32C. rec_fold(): Take a combined parameter n_bytes_fields. Determine the length of each field on the fly, and compute CRC-32C over a single contiguous range of bytes, from the start of the record payload area to the end of the last full or partial field. For secondary index records in ROW_FORMAT=REDUNDANT, also the data area that is reserved for NULL values (to facilitate in-place updates between NULL and NOT NULL values) will be included in the count. Luckily, InnoDB always zero-initialized such unused area; refer to data_write_sql_null() in rec_convert_dtuple_to_rec_old(). For other than ROW_FORMAT=REDUNDANT, no space is allocated for NULL values, and therefore the CRC-32C will only cover the actual payload of the key prefix. dtuple_fold(): For ROW_FORMAT=REDUNDANT, include the dummy NULL values in the CRC-32C, so that the values will be comparable with rec_fold(). innodb_ahi-t: A unit test for rec_fold() and dtuple_fold(). btr_search_build_page_hash_index(), btr_search_drop_page_hash_index(): Use a fixed-size stack buffer for computing the fold values, to avoid dynamic memory allocation. btr_search_drop_page_hash_index(): Do not release part.latch if we need to invoke multiple batches of rec_fold(). dtuple_t: Allocate fewer bits for the fields. The maximum number of data fields is about 1023, so uint16_t will be fine for them. The info_bits is stored in less than 1 byte. ut_pair_min(), ut_pair_cmp(): Remove. We can actually combine and compare int(n_fields << 16 \| n_bytes). PAGE_CUR_LE_OR_EXTENDS, PAGE_CUR_DBG: Remove. These were never defined, because they would only work with latin1_swedish_ci if at all. btr_cur_t::check_mismatch(): Replaces !btr_search_check_guess(). cmp_dtuple_rec_bytes(): Replaces cmp_dtuple_rec_with_match_bytes(). Determine the offsets of fields on the fly. page_cur_try_search_shortcut_bytes(): This caller of cmp_dtuple_rec_bytes() will not be invoked on the change buffer tree. cmp_dtuple_rec_leaf(): Replaces cmp_dtuple_rec_with_match() for comparing leaf-page records. buf_block_t::ahi_left_bytes_fields: Consolidated Atomic_relaxed<uint32_t> of curr_left_side << 31 \| curr_n_bytes << 16 \| curr_n_fields. The other set of parameters (n_fields, n_bytes, left_side) was removed as redundant. btr_search_update_hash_node_on_insert(): Merged to btr_search_update_hash_on_insert(). btr_search_build_page_hash_index(): Take combined left_bytes_fields instead of n_fields, n_bytes, left_side. btr_search_update_block_hash_info(), btr_search_update_hash_ref(): Merged to btr_search_info_update_hash(). btr_cur_t::n_bytes_fields: Replaces n_bytes << 16 \| n_fields. We also remove many redundant checks of btr_search.enabled. If we are holding any btr_sea::partition::latch, then a nonnull pointer in buf_block_t::index must imply that the adaptive hash index is enabled. Reviewed by: Vladislav Lesin	10 months ago
Marko Mäkelä	9c8bdc6c15	MDEV-35049: btr_search_check_free_space_in_heap() is a bottleneck Let us use implement a simple fixed-size allocator for the adaptive hash index, insted of complicating mem_heap_t or mem_block_info_t. MEM_HEAP_BTR_SEARCH: Remove. mem_block_info_t::free_block(), mem_heap_free_block_free(): Remove. mem_heap_free_top(), mem_heap_get_top(): Remove. btr_sea::partition::spare: Replaces mem_block_info_t::free_block. This keeps one spare block per adaptive hash index partition, to process an insert. We must not wait for buf_pool.mutex while holding any btr_sea::partition::latch. That is why we cache one block for future allocations. This is protected by a new btr_sea::partition::blocks_mutex in order to relieve pressure on btr_sea::partition::latch. btr_sea::partition::prepare_insert(): Replaces btr_search_check_free_space_in_heap(). btr_sea::partition::erase(): Replaces ha_search_and_delete_if_found(). btr_sea::partition::cleanup_after_erase(): Replaces the most part of ha_delete_hash_node(). Unlike the previous implementation, we will retain a spare block for prepare_insert(). This should reduce some contention on buf_pool.mutex. btr_search.n_parts: Replaces btr_ahi_parts. btr_search.enabled: Replaces btr_search_enabled. This must hold whenever buf_block_t::index is set while a thread is holding a btr_sea::partition::latch. dict_index_t::search_info: Remove pointer indirection, and use Atomic_relaxed or Atomic_counter for most fields. btr_search_guess_on_hash(): Let the caller ensure that latch_mode is BTR_MODIFY_LEAF or BTR_SEARCH_LEAF. Release btr_sea::partition::latch before buffer-fixing the block. The page latch that we already acquired is preventing buffer pool eviction. We must validate both block->index and block->page.state while holding part.latch in order to avoid race conditions with buffer page relocation or buf_pool_t::resize(). btr_search_check_guess(): Remove the constant parameter can_only_compare_to_cursor_rec=false. ahi_node: Replaces ha_node_t. This has been tested by running the regression test suite with the adaptive hash index enabled: ./mtr --mysqld=--loose-innodb-adaptive-hash-index=ON Reviewed by: Vladislav Lesin	10 months ago
Marko Mäkelä	5e8714b7b2	Merge 11.7 into main	10 months ago
Marko Mäkelä	3e9da8c923	Disable atomic.alter_table on MSAN because of slowness	10 months ago
Marko Mäkelä	15700f54c2	Merge 11.4 into 11.7	10 months ago
Marko Mäkelä	17f01186f5	Merge 10.11 into 11.4	10 months ago
Tony Chen	28b2958082	Add MTR to verify behavior on incompatible TLS configuration Add a simple test to verify the server behaves in a safe manner if configured with ciphers that aren't compatible with the server certificate. All new code of the whole pull request, including one or several files that are either new files or modified ones, are contributed under the BSD-new license. I am contributing on behalf of my employer Amazon Web Services, Inc.	1 year ago
Tony Chen	2294ecdf0e	Add MTR to ensure startup fails with invalid ssl-cipher Add a simple test to verify that the server will fail to start up when no valid cipher suites are passed to `ssl-cipher`. As different TLS libraries and versions have differing cipher suite support, it would be a good idea to ensure the server behaves in a safe manner if it is configured with invalid cipher suites. All new code of the whole pull request, including one or several files that are either new files or modified ones, are contributed under the BSD-new license. I am contributing on behalf of my employer Amazon Web Services, Inc.	1 year ago
Marko Mäkelä	420d9eb27f	Merge 10.6 into 10.11	10 months ago
Marko Mäkelä	b251cb6a4f	Merge 10.5 into 10.6	10 months ago
Sergei Golubchik	3bbbeae792	fix a memory leak	10 months ago
Sergei Golubchik	1d6f857534	MDEV-35607 Compile error with gcc-15 (signal returns) set policy CMP0067 to NEW, this will make try_compile to use project-wide CMAKE_C_STANDARD and CMAKE_CXX_STANDARD settings	10 months ago
Sergei Golubchik	3bf8b60caf	clarify the message when filesort is aborted by LIMIT ROWS EXAMINED	10 months ago
Sergei Golubchik	9508a44c37	enforce no trailing \n in Diagnostic_area messages that is in my_error(), push_warning(), etc	10 months ago
Sergei Golubchik	0031f4a74f	MDEV-35663 Sporadic connection failures during FLUSH PRIVILEGES during FLUSH PRIVILEGES, allow_all_hosts temporarily goes out of sync with acl_check_hosts and acl_wild_hosts. As it's tested in acl_check_host() without a mutex, let's re-test it under a mutex to make sure the value is correct. Note that it's just an optimization and it's ok to see outdated allow_all_hosts value here.	10 months ago
Sergei Golubchik	9f9072c344	MDEV-34733 main.mysqld--help-aria test failure: feedback plugin: failed to retrieve the MAC address * replace the message away in the test result * remove "feedback plugin:" prefix, it's a server message, not plugin's * downgrade to the warning, because 1) it's not a failure, no operation was aborted, server still works 2) it's something actionable, so not a [Note] either	10 months ago
Sergei Golubchik	b059f60510	MDEV-35704 Error message mispelled when altering table engine to MEMORY reset progress report output before printing the error message. need to flush(stdout), because error message goes to stderr	10 months ago
Sergei Golubchik	828b928fce	MDEV-35651 NO_UNSIGNED_SUBTRACTION does not work for multiple unsigned integers restore correct unsigned-unsigned logic followup for `031f11717d`	11 months ago
Sergei Golubchik	680d461b5d	MDEV-35239 Mariabackup incorrectly thinks we are on a multithreaded slave if slave_parallel_workers > 0 don't require GTID enabled for multi-threaded slave in MariaDB it's only needed for MySQL with it out-of-order commits.	1 year ago
Vladislav Vaintroub	4e9c7031a5	MDEV-35575 Fix memory leak, when installing auth_gssapi plugin fails. Make sure to release memory, which was allocated by gss_import_name(). Also when plugin_init() fails.	10 months ago
Vladislav Vaintroub	a2f510fccf	MDEV-33978 P_S.THREADS is not showing all server threads This patch only makes sure Linux getevents thread is shown in PS	10 months ago
Dave Gosselin	437550b7cf	MDEV-35773 ER_PSEUDO_THREAD_ID_OVERWRITE in 11.4 shifts error messages Copy error messages from 11.7 down to 11.4 (they are unused) to preserve their order	10 months ago
Thirunarayanan Balathandayuthapani	f8cf493290	MDEV-34898 Doublewrite recovery of innodb_checksum_algorithm=full_crc32 encrypted pages does not work - InnoDB fails to recover the full crc32 encrypted page from doublewrite buffer. The reason is that buf_dblwr_t::recover() fails to identify the space id from the page because the page has been encrypted from FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION bytes. Fix: === buf_dblwr_t::recover(): preserve any pages whose space_id does not match a known tablespace. These could be encrypted pages of tablespaces that had been created with innodb_checksum_algorithm=full_crc32. buf_page_t::read_complete(): If the page looks corrupted and the tablespace is encrypted and in full_crc32 format, try to restore the page from doublewrite buffer. recv_dblwr_t::recover_encrypted_page(): Find the page which has the same page number and try to decrypt the page using space->crypt_data. After decryption, compare the space id. Write the recovered page back to the file.	10 months ago
Monty	cc5d738999	Disable mmap usage in Aria and MyISAM when compiling with valgrind This removes a valgrind warning "cannot read program header" while it tries to search for memory leaks.	10 months ago
Julius Goryavsky	fd9a11d8a5	MDEV-35749: Add support for --use-memory option for SST with mariabackup Mariabackup (mariadb-backup) supports the --use-memory option that sets the buffer pool size for innodb. However, current SST scripts do not use this option. This commit adds support for this option, the value for which can be specified via the "use_memory" parameter in the configuration file in the [sst], [mariabackup] or [xtrabackup] sections (supported only for compatibility with old configurations). In addition, if the innodb_buffer_pool_size option is specified in the user configuration (in the main server configuration sections) or passed to the SST scripts or the server via arguments, its value is also passed to mariadb-backup as the value for the --use-memory option. A new section name [mariabackup] has also been added, which can be used instead of the deprecated [xtrabackup] (the section name "mariabackup" was specified in the documentation, but was not actually supported by SST scripts before this commit).	10 months ago
Sergei Golubchik	6abbfdef7a	sporadic failures of binlog_encryption.rpl_parallel_gco_wait_kill CURRENT_TEST: binlog_encryption.rpl_parallel_gco_wait_kill mysqltest: In included file "./suite/rpl/t/rpl_parallel_gco_wait_kill.test": included from /home/buildbot/amd64-ubuntu-2004-debug/build/mysql-test/suite/binlog_encryption/rpl_parallel_gco_wait_kill.test at line 2: At line 334: Can't initialize replace from 'replace_result $thd_id THD_ID' An sql thread can reach the "Slave has read all relay log" state and then start reading relay log again. Let's use a more generic pattern to retrieve the sql thread ID even if it's not in the "read all relay log" state.	2 years ago
Monty	a2d37705ca	Only print "InnoDB: Transaction was aborted..." if log_warnings >= 4 This is a minor fixup for MDEV-24035 Failing assertion UT_LIST_GET_LEN(lock.trx_locks) == 0 causing disruption and replication failure	10 months ago
Monty	130d6f9c4b	Fixed memory leak in get_window_functions_required_cursors() Found by buildbot with test main.gis	10 months ago
Monty	2085f36c6c	Removed not used and not visible send_metdata_skip variable. Reviewed-by: Sergei Golubchik <serg@mariadb.org>	10 months ago

1 2 3 4 5 ...

202473 Commits (3bd23b76c5b867396a148e3603d8172aec58d259) All Branches Search

202473 Commits (3bd23b76c5b867396a148e3603d8172aec58d259)

All Branches