mariadb

Commit Graph

Author	SHA1	Message	Date
Marko Mäkelä	b9f177f66a	MDEV-11254 cleanup: Remove buf_page_t::write_size commit `6495806e59` removed all reads of buf_page_t::write_size. Let us remove the field altogether.	6 years ago
Thirunarayanan Balathandayuthapani	6697135c6d	MDEV-21572 buf_page_get_gen() should apply buffered page initialized redo log during recovery - InnoDB unnecessarily reads the page even though it has fully initialized buffered redo log records. Allow the page initialization redo log to apply for the page in buf_page_get_gen() during recovery. - Renamed buf_page_get_gen() to buf_page_get_low() - Newly added buf_page_get_gen() will check for buffered redo log for the particular page id during recovery - Added new function buf_page_mtr_lock() which basically latches the page for the given latch type. - recv_recovery_create_page() is inline function which creates a page if it has page initialization redo log records.	6 years ago
Eugene Kosov	884d22f288	remove fishy reinterpret_cast from buf_page_is_zeroes() In my micro-benchmarks memcmp(4196) 3 times faster than old implementation. Also, it's generally better to use as less reinterpret_casts<> as possible. buf_is_zeroes(): renamed from buf_page_is_zeroes() and argument changed to span<> for convenience. st_::span<T>::const_iterator: fixed page_zip-verify_checksum(): make argument byte* instead of void*	6 years ago
Eugene Kosov	df88e7cefa	fix typedef-related warning and cleanup using namespace std	6 years ago
Thirunarayanan Balathandayuthapani	c65cb244b3	MDEV-19335 Remove buf_page_t::encrypted The field buf_page_t::encrypted was added in MDEV-8588. It was made mostly redundant in MDEV-12699. Remove the field.	6 years ago
Thirunarayanan Balathandayuthapani	723a4b1d78	MDEV-17228 Encrypted temporary tables are not encrypted - Introduce a new variable called innodb_encrypt_temporary_tables which is a boolean variable. It decides whether to encrypt the temporary tablespace. - Encrypts the temporary tablespace based on full checksum format. - Introduced a new counter to track encrypted and decrypted temporary tablespace pages. - Warnings issued if temporary table creation has conflict value with innodb_encrypt_temporary_tables - Added a new test case which reads and writes the pages from/to temporary tablespace.	6 years ago
Thirunarayanan Balathandayuthapani	e9145aab44	MDEV-19435 buf_fix_count > 0 for corrupted page when it exits the LRU list Problem: ========= One of the purge thread access the corrupted page and tries to remove from LRU list. In the mean time, other purge threads are waiting for same page in buf_wait_for_read(). Assertion(buf_fix_count == 0) fails for the purge thread which tries to remove the page from LRU list. Solution: ======== - Set the page id as FIL_NULL to indicate the page is corrupted before removing the block from LRU list. Acquire hash lock for the particular page id and wait for the other threads to release buf_fix_count for the block. - Added the error check for btr_cur_open() in row_search_on_row_ref().	7 years ago
Marko Mäkelä	9d142a895c	Define page_id_t in buf0types.h	7 years ago
Vicențiu Ciorbaru	15f1e03d46	Follow-up to changing FSF address Some places didn't match the previous rules, making the Floor address wrong. Additional sed rules: sed -i -e 's/Place.Suite ., Boston/Street, Fifth Floor, Boston/g' sed -i -e 's/Suite .*, Boston/Fifth Floor, Boston/g'	7 years ago
Marko Mäkelä	5fb4c0a8a8	Clean up ut_list ut_list_validate(), ut_list_map(): Add variants with const Functor& so that these functions can be called with an rvalue. Remove wrapper macros, and add #ifdef UNIV_DEBUG around debug-only code.	7 years ago
Marko Mäkelä	169c00994b	MDEV-12699 Improve crash recovery of corrupted data pages InnoDB crash recovery used to read every data page for which redo log exists. This is unnecessary for those pages that are initialized by the redo log. If a newly created page is corrupted, recovery could unnecessarily fail. It would suffice to reinitialize the page based on the redo log records. To add insult to injury, InnoDB crash recovery could hang if it encountered a corrupted page. We will fix also that problem. InnoDB would normally refuse to start up if it encounters a corrupted page on recovery, but that can be overridden by setting innodb_force_recovery=1. Data pages are completely initialized by the records MLOG_INIT_FILE_PAGE2 and MLOG_ZIP_PAGE_COMPRESS. MariaDB 10.4 additionally recognizes MLOG_INIT_FREE_PAGE, which notifies that a page has been freed and its contents can be discarded (filled with zeroes). The record MLOG_INDEX_LOAD notifies that redo logging has been re-enabled after being disabled. We can avoid loading the page if all buffered redo log records predate the MLOG_INDEX_LOAD record. For the internal tables of FULLTEXT INDEX, no MLOG_INDEX_LOAD records were written before commit `aa3f7a107c`. Hence, we will skip these optimizations for tables whose name starts with FTS_. This is joint work with Thirunarayanan Balathandayuthapani. fil_space_t::enable_lsn, file_name_t::enable_lsn: The LSN of the latest recovered MLOG_INDEX_LOAD record for a tablespace. mlog_init: Page initialization operations discovered during redo log scanning. FIXME: This really belongs in recv_sys->addr_hash, and should be removed in MDEV-19176. recv_addr_state: Add the new state RECV_WILL_NOT_READ to indicate that according to mlog_init, the page will be initialized based on redo log record contents. recv_add_to_hash_table(): Set the RECV_WILL_NOT_READ state if appropriate. For now, we do not treat MLOG_ZIP_PAGE_COMPRESS as page initialization. This works around bugs in the crash recovery of ROW_FORMAT=COMPRESSED tables. recv_mark_log_index_load(): Process a MLOG_INDEX_LOAD record by resetting the state to RECV_NOT_PROCESSED and by updating the fil_name_t::enable_lsn. recv_init_crash_recovery_spaces(): Copy fil_name_t::enable_lsn to fil_space_t::enable_lsn. recv_recover_page(): Add the parameter init_lsn, to ignore any log records that precede the page initialization. Add DBUG output about skipped operations. buf_page_create(): Initialize FIL_PAGE_LSN, so that recv_recover_page() will not wrongly skip applying the page-initialization record due to the field containing some newer LSN as a leftover from a different page. Do not invoke ibuf_merge_or_delete_for_page() during crash recovery. recv_apply_hashed_log_recs(): Remove some unnecessary lookups. Note if a corrupted page was found during recovery. After invoking buf_page_create(), do invoke ibuf_merge_or_delete_for_page() via mlog_init.ibuf_merge() in the last recovery batch. ibuf_merge_or_delete_for_page(): Relax a debug assertion. innobase_start_or_create_for_mysql(): Abort startup if a corrupted page was found during recovery. Corrupted pages will not be flagged if innodb_force_recovery is set. However, the recv_sys->found_corrupt_fs flag can be set regardless of innodb_force_recovery if file names are found to be incorrect (for example, multiple files with the same tablespace ID).	7 years ago
Eugene Kosov	a06a3e4670	MDEV-18233 Moving the hash_node_t to improve locality of reference When performing a hash search via HASH_SEARCH we first look at a key of a node and then at its pointer to the next node in chain. If we have those in one cache line instead of a two we reduce memory reads. I found dict_table_t, fil_space_t and buf_page_t suitable for such improvement.	7 years ago
Marko Mäkelä	8c43f96388	Follow-up to MDEV-12112: corruption in encrypted table may be overlooked The initial fix only covered a part of Mariabackup. This fix hardens InnoDB and XtraDB in a similar way, in order to reduce the probability of mistaking a corrupted encrypted page for a valid unencrypted one. This is based on work by Thirunarayanan Balathandayuthapani. fil_space_verify_crypt_checksum(): Assert that key_version!=0. Let the callers guarantee that. Now that we have this assertion, we also know that buf_page_is_zeroes() cannot hold. Also, remove all diagnostic output and related parameters, and let the relevant callers emit such messages. Last but not least, validate the post-encryption checksum according to the innodb_checksum_algorithm (only accepting one checksum for the strict variants), and no longer try to validate the page as if it was unencrypted. buf_page_is_zeroes(): Move to the compilation unit of the only callers, and declare static. xb_fil_cur_read(), buf_page_check_corrupt(): Add a condition before calling fil_space_verify_crypt_checksum(). This is a non-functional change. buf_dblwr_process(): Validate the page only as encrypted or unencrypted, but not both.	7 years ago
Marko Mäkelä	1a780eefc9	MDEV-17958 Make bug-endian innodb_checksum_algorithm=crc32 optional In MySQL 5.7, it was noticed that files are not portable between big-endian and little-endian processor architectures (such as SPARC and x86), because the original implementation of innodb_checksum_algorithm=crc32 was not byte order agnostic. A byte order agnostic implementation of innodb_checksum_algorithm=crc32 was only added to MySQL 5.7, not backported to 5.6. Consequently, MariaDB Server versions 10.0 and 10.1 only contain the CRC-32C implementation that works incorrectly on big-endian architectures, and MariaDB Server 10.2.2 got the byte-order agnostic CRC-32C implementation from MySQL 5.7. MySQL 5.7 introduced a "legacy crc32" variant that is functionally equivalent to the big-endian version of the original crc32 implementation. Thanks to this variant, old data files can be transferred from big-endian systems to newer versions. Introducing new variants of checksum algorithms (without introducing new names for them, or something on the pages themselves to identify the algorithm) generally is a bad idea, because each checksum algorithm is like a lottery ticket. The more algorithms you try, the more likely it will be for the checksum to match on a corrupted page. So, essentially MySQL 5.7 weakened innodb_checksum_algorithm=crc32, and MariaDB 10.2.2 inherited this weakening. We introduce a build option that together with MDEV-17957 makes innodb_checksum_algorithm=strict_crc32 strict again by only allowing one variant of the checksum to match. WITH_INNODB_BUG_ENDIAN_CRC32: A new cmake option for enabling the bug-compatible "legacy crc32" checksum. This is only enabled on big-endian systems by default, to facilitate an upgrade from MariaDB 10.0 or 10.1. Checked by #ifdef INNODB_BUG_ENDIAN_CRC32. ut_crc32_byte_by_byte: Remove (unused function). legacy_big_endian_checksum: Remove. This variable seems to have unnecessarily complicated the logic. When the weakening is enabled, we must always fall back to the buggy checksum. buf_page_check_crc32(): A helper function to compute one or two CRC-32C variants.	7 years ago
Marko Mäkelä	b3009059d0	Minor cleanup	7 years ago
Eugene Kosov	14be814380	MDEV-17491 micro optimize page_id_t page_id_t: remove m_fold member various places: pass page_id_t by value instead of by reference	7 years ago
Marko Mäkelä	2ca904f0ca	MDEV-13103 Deal with page_compressed page corruption fil_page_decompress(): Replaces fil_decompress_page(). Allow the caller detect errors. Remove duplicated code. Use the "safe" instead of "fast" variants of decompression routines. fil_page_compress(): Replaces fil_compress_page(). The length of the input buffer always was srv_page_size (innodb_page_size). Remove printouts, and remove the fil_space_t* parameter. buf_tmp_buffer_t::reserved: Make private; the accessors acquire() and release() will use atomic memory access. buf_pool_reserve_tmp_slot(): Make static. Remove the second parameter. Do not acquire any mutex. Remove the allocation of the buffers. buf_tmp_reserve_crypt_buf(), buf_tmp_reserve_compression_buf(): Refactored away from buf_pool_reserve_tmp_slot(). buf_page_decrypt_after_read(): Make static, and simplify the logic. Use the encryption buffer also for decompressing. buf_page_io_complete(), buf_dblwr_process(): Check more failures. fil_space_encrypt(): Simplify the debug checks. fil_space_t::printed_compression_failure: Remove. fil_get_compression_alg_name(): Remove. fil_iterate(): Allocate a buffer for compression and decompression only once, instead of allocating and freeing it for every page that uses compression, during IMPORT TABLESPACE. Also, validate the page checksum before decryption, and reduce the scope of some variables. fil_page_is_index_page(), fil_page_is_lzo_compressed(): Remove (unused). AbstractCallback::operator()(): Remove the parameter 'offset'. The check for it in FetchIndexRootPages::operator() was basically redundant and dead code since the previous refactoring.	8 years ago
Marko Mäkelä	f5eb37129f	MDEV-13103 Deal with page_compressed page corruption fil_page_decompress(): Replaces fil_decompress_page(). Allow the caller detect errors. Remove duplicated code. Use the "safe" instead of "fast" variants of decompression routines. fil_page_compress(): Replaces fil_compress_page(). The length of the input buffer always was srv_page_size (innodb_page_size). Remove printouts, and remove the fil_space_t* parameter. buf_tmp_buffer_t::reserved: Make private; the accessors acquire() and release() will use atomic memory access. buf_pool_reserve_tmp_slot(): Make static. Remove the second parameter. Do not acquire any mutex. Remove the allocation of the buffers. buf_tmp_reserve_crypt_buf(), buf_tmp_reserve_compression_buf(): Refactored away from buf_pool_reserve_tmp_slot(). buf_page_decrypt_after_read(): Make static, and simplify the logic. Use the encryption buffer also for decompressing. buf_page_io_complete(), buf_dblwr_process(): Check more failures. fil_space_encrypt(): Simplify the debug checks. fil_space_t::printed_compression_failure: Remove. fil_get_compression_alg_name(): Remove. fil_iterate(): Allocate a buffer for compression and decompression only once, instead of allocating and freeing it for every page that uses compression, during IMPORT TABLESPACE. fil_node_get_space_id(), fil_page_is_index_page(), fil_page_is_lzo_compressed(): Remove (unused code).	8 years ago
Marko Mäkelä	9ed2b2b2b8	Do not divide or multiply by srv_page_size Instead, shift by srv_page_size_shift.	8 years ago
Marko Mäkelä	a90100d756	Replace univ_page_size and UNIV_PAGE_SIZE Try to use one variable (srv_page_size) for innodb_page_size. Also, replace UNIV_PAGE_SIZE_SHIFT with srv_page_size_shift.	8 years ago
Marko Mäkelä	4cad42392a	MDEV-12266: Change dict_table_t::space to fil_space_t* InnoDB always keeps all tablespaces in the fil_system cache. The fil_system.LRU is only for closing file handles; the fil_space_t and fil_node_t for all data files will remain in main memory. Between startup to shutdown, they can only be created and removed by DDL statements. Therefore, we can let dict_table_t::space point directly to the fil_space_t. dict_table_t::space_id: A numeric tablespace ID for the corner cases where we do not have a tablespace. The most prominent examples are ALTER TABLE...DISCARD TABLESPACE or a missing or corrupted file. There are a few functional differences; most notably: (1) DROP TABLE will delete matching .ibd and .cfg files, even if they were not attached to the data dictionary. (2) Some error messages will report file names instead of numeric IDs. There still are many functions that use numeric tablespace IDs instead of fil_space_t, and many functions could be converted to fil_space_t member functions. Also, Tablespace and Datafile should be merged with fil_space_t and fil_node_t. page_id_t and buf_page_get_gen() could use fil_space_t& instead of a numeric ID, and after moving to a single buffer pool (MDEV-15058), buf_pool_t::page_hash could be moved to fil_space_t::page_hash. FilSpace: Remove. Only few calls to fil_space_acquire() will remain, and gradually they should be removed. mtr_t::set_named_space_id(ulint): Renamed from set_named_space(), to prevent accidental calls to this slower function. Very few callers remain. fseg_create(), fsp_reserve_free_extents(): Take fil_space_t as a parameter instead of a space_id. fil_space_t::rename(): Wrapper for fil_rename_tablespace_check(), fil_name_write_rename(), fil_rename_tablespace(). Mariabackup passes the parameter log=false; InnoDB passes log=true. dict_mem_table_create(): Take fil_space_t* instead of space_id as parameter. dict_process_sys_tables_rec_and_mtr_commit(): Replace the parameter 'status' with 'bool cached'. dict_get_and_save_data_dir_path(): Avoid copying the fil_node_t::name. fil_ibd_open(): Return the tablespace. fil_space_t::set_imported(): Replaces fil_space_set_imported(). truncate_t: Change many member function parameters to fil_space_t*, and remove page_size parameters. row_truncate_prepare(): Merge to its only caller. row_drop_table_from_cache(): Assert that the table is persistent. dict_create_sys_indexes_tuple(): Write SYS_INDEXES.SPACE=FIL_NULL if the tablespace has been discarded. row_import_update_discarded_flag(): Remove a constant parameter.	8 years ago
Marko Mäkelä	112df06996	MDEV-15529 IMPORT TABLESPACE unnecessarily uses the doublewrite buffer fil_space_t::atomic_write_supported: Always set this flag for TEMPORARY TABLESPACE and during IMPORT TABLESPACE. The page writes during these operations are by definition not crash-safe because they are not written to the redo log. fil_space_t::use_doublewrite(): Determine if doublewrite should be used. buf_dblwr_update(): Add assertions, and let the caller check whether doublewrite buffering is desired. buf_flush_write_block_low(): Disable the doublewrite buffer for the temporary tablespace and for IMPORT TABLESPACE. fil_space_set_imported(), fil_node_open_file(), fil_space_create(): Initialize or revise the space->atomic_write_supported flag. buf_page_io_complete(), buf_flush_write_complete(): Add the parameter dblwr, to indicate whether doublewrite was used for writes. buf_dblwr_sync_datafiles(): Remove an unnecessary flush of persistent tablespaces when flushing temporary tablespaces. (Move the call to buf_dblwr_flush_buffered_writes().)	8 years ago
Eugene Kosov	f02f1eda7e	review fixes	8 years ago
Eugene Kosov	e14790b89d	let buf_page_hash_lock_get() be function, not macro	8 years ago
Eugene Kosov	365f478240	make buf_block_t::lock_hash_val uint32_t	8 years ago
Vladislav Vaintroub	ac8e3c85a4	MDEV-15295 Type mismatch for srv_fatal_semaphore_wait_threshold	8 years ago
Marko Mäkelä	51b4366bfb	MDEV-13328 ALTER TABLE…DISCARD TABLESPACE takes a lot of time With a big buffer pool that contains many data pages, DISCARD TABLESPACE took a long time, because it would scan the entire buffer pool to remove any pages that belong to the tablespace. With a large buffer pool, this would take a lot of time, especially when the table-to-discard is empty. The minimum amount of work that DISCARD TABLESPACE must do is to remove the pages of the to-be-discarded table from the buf_pool->flush_list because any writes to the data file must be prevented before the file is deleted. If DISCARD TABLESPACE does not evict the pages from the buffer pool, then IMPORT TABLESPACE must do it, because we must prevent pre-DISCARD, not-yet-evicted pages from being mistaken for pages of the imported tablespace. It would not be a useful fix to simply move the buffer pool scan to the IMPORT TABLESPACE step. What we can do is to actively evict those pages that could be mistaken for imported pages. In this way, when importing a small table into a big buffer pool, the import should still run relatively fast. Import is bypassing the buffer pool when reading pages for the adjustment phase. In the adjustment phase, if a page exists in the buffer pool, we could replace it with the page from the imported file. Unfortunately I did not get this to work properly, so instead we will simply evict any matching page from the buffer pool. buf_page_get_gen(): Implement BUF_EVICT_IF_IN_POOL, a new mode where the requested page will be evicted if it is found. There must be no unwritten changes for the page. buf_remove_t: Remove. Instead, use trx!=NULL to signify that a write to file is desired, and use a separate parameter bool drop_ahi. buf_LRU_flush_or_remove_pages(), fil_delete_tablespace(): Replace buf_remove_t. buf_LRU_remove_pages(), buf_LRU_remove_all_pages(): Remove. PageConverter::m_mtr: A dummy mini-transaction buffer PageConverter::PageConverter(): Complete the member initialization list. PageConverter::operator()(): Evict any 'shadow' pages from the buffer pool so that pre-existing (garbage) pages cannot be mistaken for pages that exist in the being-imported file. row_discard_tablespace(): Remove a bogus comment that seems to refer to IMPORT TABLESPACE, not DISCARD TABLESPACE.	8 years ago
Marko Mäkelä	6b45355e6b	MDEV-13103 Assertion `flags & BUF_PAGE_PRINT_NO_CRASH' failed in buf_page_print buf_page_print(): Remove the parameter 'flags', and when a server abort is intended, perform that in the caller. In this way, page corruption reports due to different reasons can be distinguished better. This is non-functional code refactoring that does not fix any page corruption issues. The change is only made to avoid falsely grouping together unrelated causes of page corruption.	8 years ago
Jan Lindström	34eef269eb	MDEV-11939: innochecksum mistakes a file for an encrypted one (page 0 invalid) Always read full page 0 to determine does tablespace contain encryption metadata. Tablespaces that are page compressed or page compressed and encrypted do not compare checksum as it does not exists. For encrypted tables use checksum verification written for encrypted tables and normal tables use normal method. buf_page_is_checksum_valid_crc32 buf_page_is_checksum_valid_innodb buf_page_is_checksum_valid_none Modify Innochecksum logging to file to avoid compilation warnings. fil0crypt.cc fil0crypt.h Modify to be able to use in innochecksum compilation and move fil_space_verify_crypt_checksum to end of the file. Add innochecksum logging to file. univ.i Add innochecksum strict_verify, log_file and cur_page_num variables as extern. page_zip_verify_checksum Add innochecksum logging to file and remove unnecessary code. innochecksum.cc Lot of changes most notable able to read encryption metadata from page 0 of the tablespace. Added test case where we corrupt intentionally FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION (encryption key version) FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION+4 (post encryption checksum) FIL_DATA+10 (data)	8 years ago
Jan Lindström	8b019f87dd	MDEV-11939: innochecksum mistakes a file for an encrypted one (page 0 invalid) Always read full page 0 to determine does tablespace contain encryption metadata. Tablespaces that are page compressed or page compressed and encrypted do not compare checksum as it does not exists. For encrypted tables use checksum verification written for encrypted tables and normal tables use normal method. buf_page_is_checksum_valid_crc32 buf_page_is_checksum_valid_innodb buf_page_is_checksum_valid_none Add Innochecksum logging to file buf_page_is_corrupted Remove ib_logf and page_warn_strict_checksum calls in innochecksum compilation. Add innochecksum logging to file. fil0crypt.cc fil0crypt.h Modify to be able to use in innochecksum compilation and move fil_space_verify_crypt_checksum to end of the file. Add innochecksum logging to file. univ.i Add innochecksum strict_verify, log_file and cur_page_num variables as extern. page_zip_verify_checksum Add innochecksum logging to file. innochecksum.cc Lot of changes most notable able to read encryption metadata from page 0 of the tablespace. Added test case where we corrupt intentionally FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION (encryption key version) FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION+4 (post encryption checksum) FIL_DATA+10 (data)	8 years ago
Jan Lindström	90c52e5291	MDEV-12615: InnoDB page compression method snappy mostly does not compress pages Snappy compression method require that output buffer used for compression is bigger than input buffer. Similarly lzo require additional work memory buffer. Increase the allocated buffer accordingly. buf_tmp_buffer_t: removed unnecessary lzo_mem, crypt_buf_free and comp_buf_free. buf_pool_reserve_tmp_slot: use alligned_alloc and if snappy available allocate size based on snappy_max_compressed_length and if lzo is available increase buffer by LZO1X_1_15_MEM_COMPRESS. fil_compress_page: Remove unneeded lzo mem (we use same buffer) and if output buffer is not yet allocated allocate based similarly as above. Decompression does not require additional work area. Modify test to use same test as other compression method tests.	9 years ago
Sergei Golubchik	0072d2e9a1	InnoDB cleanup: remove a bunch of #ifdef UNIV_INNOCHECKSUM innochecksum uses global variables. great, let's use them all the way down, instead of passing them as arguments to innodb internals, conditionally modifying function prototypes with #ifdefs	9 years ago
Marko Mäkelä	b82c602db5	MDEV-12602 InnoDB: Failing assertion: space->n_pending_ops == 0 This fixes a regression caused by MDEV-12428. When we introduced a variant of fil_space_acquire() that could increment space->n_pending_ops after space->stop_new_ops was set, the logic of fil_check_pending_operations() was broken. fil_space_t::n_pending_ios: A new field to track read or write access from the buffer pool routines immediately before a block write or after a block read in the file system. fil_space_acquire_for_io(), fil_space_release_for_io(): Similar to fil_space_acquire_silent() and fil_space_release(), but modify fil_space_t::n_pending_ios instead of fil_space_t::n_pending_ops. Adjust a number of places accordingly, and remove some redundant tablespace lookups. The following parts of this fix differ from the 10.2 version of this fix: buf_page_get_corrupt(): Add a tablespace parameter. In 10.2, we already had a two-phase process of freeing fil_space objects (first, fil_space_detach(), then release fil_system->mutex, and finally free the fil_space and fil_node objects). fil_space_free_and_mutex_exit(): Renamed from fil_space_free(). Detach the tablespace from the fil_system cache, release the fil_system->mutex, and then wait for space->n_pending_ios to reach 0, to avoid accessing freed data in a concurrent thread. During the wait, future calls to fil_space_acquire_for_io() will not find this tablespace, and the count can only be decremented to 0, at which point it is safe to free the objects. fil_node_free_part1(), fil_node_free_part2(): Refactored from fil_node_free().	9 years ago
Marko Mäkelä	4b24467ff3	MDEV-12602 InnoDB: Failing assertion: space->n_pending_ops == 0 This fixes a regression caused by MDEV-12428. When we introduced a variant of fil_space_acquire() that could increment space->n_pending_ops after space->stop_new_ops was set, the logic of fil_check_pending_operations() was broken. fil_space_t::n_pending_ios: A new field to track read or write access from the buffer pool routines immediately before a block write or after a block read in the file system. fil_space_acquire_for_io(), fil_space_release_for_io(): Similar to fil_space_acquire_silent() and fil_space_release(), but modify fil_space_t::n_pending_ios instead of fil_space_t::n_pending_ops. fil_space_free_low(): Wait for space->n_pending_ios to reach 0, to avoid accessing freed data in a concurrent thread. Future calls to fil_space_acquire_for_io() will not find this tablespace, because it will already have been detached from fil_system. Adjust a number of places accordingly, and remove some redundant tablespace lookups. FIXME: buf_page_check_corrupt() should take a tablespace from fil_space_acquire_for_io() as a parameter. This will be done in the 10.1 version of this patch and merged from there. That depends on MDEV-12253, which has not been merged from 10.1 yet.	9 years ago
Marko Mäkelä	472b5f0d1f	Follow-up to Bug#24346574 PAGE CLEANER THREAD, ASSERT BLOCK->N_POINTERS == 0 Silence the Valgrind warnings on instrumented builds (-DWITH_VALGRIND). assert_block_ahi_empty_on_init(): A variant of assert_block_ahi_empty() that declares n_pointers initialized and then asserts that n_pointers==0. In Valgrind-instrumented builds, InnoDB declares allocated memory uninitialized.	9 years ago
Marko Mäkelä	e63ead68bf	Bug#24346574 PAGE CLEANER THREAD, ASSERT BLOCK->N_POINTERS == 0 btr_search_drop_page_hash_index(): Do not return before ensuring that block->index=NULL, even if !btr_search_enabled. We would typically still skip acquiring the AHI latch when the AHI is disabled, because block->index would already be NULL. Only if the AHI is in the process of being disabled, we would wait for the AHI latch and then notice that block->index=NULL and return. The above bug was a regression caused in MySQL 5.7.9 by the fix of Bug#21407023: DISABLING AHI SHOULD AVOID TAKING AHI LATCH The rest of this patch improves diagnostics by adding assertions. assert_block_ahi_valid(): A debug predicate for checking that block->n_pointers!=0 implies block->index!=NULL. assert_block_ahi_empty(): A debug predicate for checking that block->n_pointers==0. buf_block_init(): Instead of assigning block->n_pointers=0, assert_block_ahi_empty(block). buf_pool_clear_hash_index(): Clarify comments, and assign block->n_pointers=0 before assigning block->index=NULL. The wrong ordering could make block->n_pointers appear incorrect in debug assertions. This bug was introduced in MySQL 5.1.52 by Bug#13006367 62487: INNODB TAKES 3 MINUTES TO CLEAN UP THE ADAPTIVE HASH INDEX AT SHUTDOWN i_s_innodb_buffer_page_get_info(): Add a comment that the IS_HASHED column in the INFORMATION_SCHEMA views INNODB_BUFFER_POOL_PAGE and INNODB_BUFFER_PAGE_LRU may show false positives (there may be no pointers after all.) ha_insert_for_fold_func(), ha_delete_hash_node(), ha_search_and_update_if_found_func(): Use atomics for updating buf_block_t::n_pointers. While buf_block_t::index is always protected by btr_search_x_lock(index), in ha_insert_for_fold_func() the n_pointers-- may belong to another dict_index_t whose btr_search_latches[] we are not holding. RB: 13879 Reviewed-by: Jimmy Yang <jimmy.yang@oracle.com>	9 years ago
Jan Lindström	765a43605a	MDEV-12253: Buffer pool blocks are accessed after they have been freed Problem was that bpage was referenced after it was already freed from LRU. Fixed by adding a new variable encrypted that is passed down to buf_page_check_corrupt() and used in buf_page_get_gen() to stop processing page read. This patch should also address following test failures and bugs: MDEV-12419: IMPORT should not look up tablespace in PageConverter::validate(). This is now removed. MDEV-10099: encryption.innodb_onlinealter_encryption fails sporadically in buildbot MDEV-11420: encryption.innodb_encryption-page-compression failed in buildbot MDEV-11222: encryption.encrypt_and_grep failed in buildbot on P8 Removed dict_table_t::is_encrypted and dict_table_t::ibd_file_missing and replaced these with dict_table_t::file_unreadable. Table ibd file is missing if fil_get_space(space_id) returns NULL and encrypted if not. Removed dict_table_t::is_corrupted field. Ported FilSpace class from 10.2 and using that on buf_page_check_corrupt(), buf_page_decrypt_after_read(), buf_page_encrypt_before_write(), buf_dblwr_process(), buf_read_page(), dict_stats_save_defrag_stats(). Added test cases when enrypted page could be read while doing redo log crash recovery. Also added test case for row compressed blobs. btr_cur_open_at_index_side_func(), btr_cur_open_at_rnd_pos_func(): Avoid referencing block that is NULL. buf_page_get_zip(): Issue error if page read fails. buf_page_get_gen(): Use dberr_t for error detection and do not reference bpage after we hare freed it. buf_mark_space_corrupt(): remove bpage from LRU also when it is encrypted. buf_page_check_corrupt(): @return DB_SUCCESS if page has been read and is not corrupted, DB_PAGE_CORRUPTED if page based on checksum check is corrupted, DB_DECRYPTION_FAILED if page post encryption checksum matches but after decryption normal page checksum does not match. In read case only DB_SUCCESS is possible. buf_page_io_complete(): use dberr_t for error handling. buf_flush_write_block_low(), buf_read_ahead_random(), buf_read_page_async(), buf_read_ahead_linear(), buf_read_ibuf_merge_pages(), buf_read_recv_pages(), fil_aio_wait(): Issue error if page read fails. btr_pcur_move_to_next_page(): Do not reference page if it is NULL. Introduced dict_table_t::is_readable() and dict_index_t::is_readable() that will return true if tablespace exists and pages read from tablespace are not corrupted or page decryption failed. Removed buf_page_t::key_version. After page decryption the key version is not removed from page frame. For unencrypted pages, old key_version is removed at buf_page_encrypt_before_write() dict_stats_update_transient_for_index(), dict_stats_update_transient() Do not continue if table decryption failed or table is corrupted. dict0stats.cc: Introduced a dict_stats_report_error function to avoid code duplication. fil_parse_write_crypt_data(): Check that key read from redo log entry is found from encryption plugin and if it is not, refuse to start. PageConverter::validate(): Removed access to fil_space_t as tablespace is not available during import. Fixed error code on innodb.innodb test. Merged test cased innodb-bad-key-change5 and innodb-bad-key-shutdown to innodb-bad-key-change2. Removed innodb-bad-key-change5 test. Decreased unnecessary complexity on some long lasting tests. Removed fil_inc_pending_ops(), fil_decr_pending_ops(), fil_get_first_space(), fil_get_next_space(), fil_get_first_space_safe(), fil_get_next_space_safe() functions. fil_space_verify_crypt_checksum(): Fixed bug found using ASAN where FIL_PAGE_END_LSN_OLD_CHECKSUM field was incorrectly accessed from row compressed tables. Fixed out of page frame bug for row compressed tables in fil_space_verify_crypt_checksum() found using ASAN. Incorrect function was called for compressed table. Added new tests for discard, rename table and drop (we should allow them even when page decryption fails). Alter table rename is not allowed. Added test for restart with innodb-force-recovery=1 when page read on redo-recovery cant be decrypted. Added test for corrupted table where both page data and FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION is corrupted. Adjusted the test case innodb_bug14147491 so that it does not anymore expect crash. Instead table is just mostly not usable. fil0fil.h: fil_space_acquire_low is not visible function and fil_space_acquire and fil_space_acquire_silent are inline functions. FilSpace class uses fil_space_acquire_low directly. recv_apply_hashed_log_recs() does not return anything.	9 years ago
Marko Mäkelä	0871a00a62	MDEV-12545 Reduce the amount of fil_space_t lookups buf_flush_write_block_low(): Acquire the tablespace reference once, and pass it to lower-level functions. This is only a start; further calls may be removed. fil_decompress_page(): Remove unsafe use of fil_space_get_by_id().	9 years ago
Marko Mäkelä	996c7d5cb5	MDEV-12545 Reduce the amount of fil_space_t lookups buf_flush_write_block_low(): Acquire the tablespace reference once, and pass it to lower-level functions. This is only a start; further calls may be removed later.	9 years ago
Marko Mäkelä	97acc4a1c3	MDEV-12270 Port MySQL 8.0 Bug#21141390 REMOVE UNUSED FUNCTIONS AND CONVERT GLOBAL SYMBOLS TO STATIC InnoDB defines some functions that are not called at all. Other functions are called, but only from the same compilation unit. Remove some function declarations and definitions, and add 'static' keywords. Some symbols must be kept for separately compiled tools, such as innochecksum.	11 years ago
Marko Mäkelä	4e1116b2c6	MDEV-12271 Port MySQL 8.0 Bug#23150562 REMOVE UNIV_MUST_NOT_INLINE AND UNIV_NONINL Also, remove empty .ic files that were not removed by my MySQL commit. Problem: InnoDB used to support a compilation mode that allowed to choose whether the function definitions in .ic files are to be inlined or not. This stopped making sense when InnoDB moved to C++ in MySQL 5.6 (and ha_innodb.cc started to #include .ic files), and more so in MySQL 5.7 when inline methods and functions were introduced in .h files. Solution: Remove all references to UNIV_NONINL and UNIV_MUST_NOT_INLINE from all files, assuming that the symbols are never defined. Remove the files fut0fut.cc and ut0byte.cc which only mattered when UNIV_NONINL was defined.	10 years ago
Marko Mäkelä	89d80c1b0b	Fix many -Wconversion warnings. Define my_thread_id as an unsigned type, to avoid mismatch with ulonglong. Change some parameters to this type. Use size_t in a few more places. Declare many flag constants as unsigned to avoid sign mismatch when shifting bits or applying the unary ~ operator. When applying the unary ~ operator to enum constants, explictly cast the result to an unsigned type, because enum constants can be treated as signed. In InnoDB, change the source code line number parameters from ulint to unsigned type. Also, make some InnoDB functions return a narrower type (unsigned or uint32_t instead of ulint; bool instead of ibool).	9 years ago
Marko Mäkelä	27b9989d31	MDEV-12121 Introduce build option WITH_INNODB_AHI to disable innodb_adaptive_hash_index The InnoDB adaptive hash index is sometimes degrading the performance of InnoDB, and it is sometimes disabled to get more consistent performance. We should have a compile-time option to disable the adaptive hash index. Let us introduce two options: OPTION(WITH_INNODB_AHI "Include innodb_adaptive_hash_index" ON) OPTION(WITH_INNODB_ROOT_GUESS "Cache index root block descriptors" ON) where WITH_INNODB_AHI always implies WITH_INNODB_ROOT_GUESS. As part of this change, the misleadingly named function trx_search_latch_release_if_reserved(trx) will be replaced with the macro trx_assert_no_search_latch(trx) that will be empty unless BTR_CUR_HASH_ADAPT is defined (cmake -DWITH_INNODB_AHI=ON). We will also remove the unused column INFORMATION_SCHEMA.INNODB_TRX.TRX_ADAPTIVE_HASH_TIMEOUT. In MariaDB Server 10.1, it used to reflect the value of trx_t::search_latch_timeout which could be adjusted during row_search_for_mysql(). In 10.2, there is no such field. Other than the removal of the unused column TRX_ADAPTIVE_HASH_TIMEOUT, this is an almost non-functional change to the server when using the default build options. Some tests are adjusted so that they will work with both -DWITH_INNODB_AHI=ON and -DWITH_INNODB_AHI=OFF. The test innodb.innodb_monitor has been renamed to innodb.monitor in order to track MySQL 5.7, and the duplicate tests sys_vars.innodb_monitor_* are removed.	9 years ago
Marko Mäkelä	a13a636c74	MDEV-11802 innodb.innodb_bug14676111 fails The function trx_purge_stop() was calling os_event_reset(purge_sys->event) before calling rw_lock_x_lock(&purge_sys->latch). The os_event_set() call in srv_purge_coordinator_suspend() is protected by that X-latch. It would seem a good idea to consistently protect both os_event_set() and os_event_reset() calls with a common mutex or rw-lock in those cases where os_event_set() and os_event_reset() are used like condition variables, tied to changes of shared state. For each os_event_t, we try to document the mutex or rw-lock that is being used. For some events, frequent calls to os_event_set() seem to try to avoid hangs. Some events are never waited for infinitely, only timed waits, and os_event_set() is used for early termination of these waits. os_aio_simulated_put_read_threads_to_sleep(): Define as a null macro on other systems than Windows. TODO: remove this altogether and disable innodb_use_native_aio on Windows. os_aio_segment_wait_events[]: Initialize only if innodb_use_native_aio=0. log_write_flush_to_disk_low(): Invoke log_mutex_enter() at the end, to avoid race conditions when changing the system state. (No potential race condition existed before MySQL 5.7.)	9 years ago
Marko Mäkelä	13493078e9	MDEV-11802 innodb.innodb_bug14676111 fails The function trx_purge_stop() was calling os_event_reset(purge_sys->event) before calling rw_lock_x_lock(&purge_sys->latch). The os_event_set() call in srv_purge_coordinator_suspend() is protected by that X-latch. It would seem a good idea to consistently protect both os_event_set() and os_event_reset() calls with a common mutex or rw-lock in those cases where os_event_set() and os_event_reset() are used like condition variables, tied to changes of shared state. For each os_event_t, we try to document the mutex or rw-lock that is being used. For some events, frequent calls to os_event_set() seem to try to avoid hangs. Some events are never waited for infinitely, only timed waits, and os_event_set() is used for early termination of these waits. os_aio_simulated_put_read_threads_to_sleep(): Define as a null macro on other systems than Windows. TODO: remove this altogether and disable innodb_use_native_aio on Windows. os_aio_segment_wait_events[]: Initialize only if innodb_use_native_aio=0.	9 years ago
Jan Lindström	41cd80fe06	After review fixes.	9 years ago
Jan Lindström	0340067608	After review fixes for MDEV-11759. buf_page_is_checksum_valid_crc32() buf_page_is_checksum_valid_innodb() buf_page_is_checksum_valid_none(): Use ULINTPF instead of %lu and %u for ib_uint32_t fil_space_verify_crypt_checksum(): Check that page is really empty if checksum and LSN are zero. fil_space_verify_crypt_checksum(): Correct the comment to be more agurate. buf0buf.h: Remove unnecessary is_corrupt variable from buf_page_t structure.	9 years ago
Jan Lindström	ddf2fac733	MDEV-11759: Encryption code in MariaDB 10.1/10.2 causes compatibility problems Pages that are encrypted contain post encryption checksum on different location that normal checksum fields. Therefore, we should before decryption check this checksum to avoid unencrypting corrupted pages. After decryption we can use traditional checksum check to detect if page is corrupted or unencryption was done using incorrect key. Pages that are page compressed do not contain any checksum, here we need to fist unencrypt, decompress and finally use tradional checksum check to detect page corruption or that we used incorrect key in unencryption. buf0buf.cc: buf_page_is_corrupted() mofified so that compressed pages are skipped. buf0buf.h, buf_block_init(), buf_page_init_low(): removed unnecessary page_encrypted, page_compressed, stored_checksum, valculated_checksum fields from buf_page_t buf_page_get_gen(): use new buf_page_check_corrupt() function to detect corrupted pages. buf_page_check_corrupt(): If page was not yet decrypted check if post encryption checksum still matches. If page is not anymore encrypted, use buf_page_is_corrupted() traditional checksum method. If page is detected as corrupted and it is not encrypted we print corruption message to error log. If page is still encrypted or it was encrypted and now corrupted, we will print message that page is encrypted to error log. buf_page_io_complete(): use new buf_page_check_corrupt() function to detect corrupted pages. buf_page_decrypt_after_read(): Verify post encryption checksum before tring to decrypt. fil0crypt.cc: fil_encrypt_buf() verify post encryption checksum and ind fil_space_decrypt() return true if we really decrypted the page. fil_space_verify_crypt_checksum(): rewrite to use the method used when calculating post encryption checksum. We also check if post encryption checksum matches that traditional checksum check does not match. fil0fil.ic: Add missed page type encrypted and page compressed to fil_get_page_type_name() Note that this change does not yet fix innochecksum tool, that will be done in separate MDEV. Fix test failures caused by buf page corruption injection.	9 years ago
Marko Mäkelä	1eabad5dbe	Remove MYSQL_COMPRESSION. The MariaDB 10.1 page_compression is incompatible with the Oracle implementation that was introduced in MySQL 5.7 later. Remove the Oracle implementation. Also remove the remaining traces of MYSQL_ENCRYPTION. This will also remove traces of PUNCH_HOLE until it is implemented better. The only effective call to os_file_punch_hole() was in fil_node_create_low() to test if the operation is supported for the file. In other words, it looks like page_compression is not working in MariaDB 10.2, because no code equivalent to the 10.1 os_file_trim() is enabled.	9 years ago
Marko Mäkelä	63574f1275	MDEV-11690 Remove UNIV_HOTBACKUP The InnoDB source code contains quite a few references to a closed-source hot backup tool which was originally called InnoDB Hot Backup (ibbackup) and later incorporated in MySQL Enterprise Backup. The open source backup tool XtraBackup uses the full database for recovery. So, the references to UNIV_HOTBACKUP are only cluttering the source code.	9 years ago

1 2 3 4

166 Commits (15fa70b8408a2146b300a4e1b19819addafa5929)