mariadb

Commit Graph

Author	SHA1	Message	Date
Marko Mäkelä	3dd7b0a80c	Cleanup: Remove OS_FILE_ON_ERROR_NO_EXIT Ever since commit `412ee0330c` or commit `a440d6ed3a` InnoDB should generally not abort when failing to open or create files. In Datafile::open_or_create() we had failed to set the flag to avoid abort() on failure, but everywhere else we were setting it. We may still call abort() via os_file_handle_error(). Reviewed by: Vladislav Vaintroub	2 years ago
Marko Mäkelä	7f7329f092	MDEV-33379 innodb_log_file_buffering=OFF causes corruption on bcachefs Apparently, invoking fcntl(fd, F_SETFL, O_DIRECT) will lead to unexpected behaviour on Linux bcachefs and possibly other file systems, depending on the operating system version. So, let us avoid doing that, and instead just attempt to pass the O_DIRECT flag to open(). This should make us compatible with NetBSD, IBM AIX, as well as Solaris and its derivatives. This fix does not change the fact that we had only implemented innodb_log_file_buffering=OFF on systems where we can determine the physical block size (typically 512 or 4096 bytes). Currently, those operating systems are Linux and Microsoft Windows. HAVE_FCNTL_DIRECT, os_file_set_nocache(): Remove. OS_FILE_OVERWRITE, OS_FILE_CREATE_PATH: Remove (never used parameters). os_file_log_buffered(), os_file_log_maybe_unbuffered(): Helper functions. os_file_create_simple_func(): When applicable, initially attempt to open files in O_DIRECT mode. os_file_create_func(): When applicable, initially attempt to open files in O_DIRECT mode. For type==OS_LOG_FILE && create_mode != OS_FILE_CREATE we will first invoke stat(2) on the file name to find out if the size is compatible with O_DIRECT. If create_mode == OS_FILE_CREATE, we will invoke fstat(2) on the created log file afterwards, and may close and reopen the file in O_DIRECT mode if applicable. create_temp_file(): Support O_DIRECT. This is only used if O_TMPFILE is available and innodb_disable_sort_file_cache=ON (non-default value). Notably, that setting never worked on Microsoft Windows. row_merge_file_create_mode(): Split from row_merge_file_create_low(). Create a temporary file in the specified mode. Reviewed by: Vladislav Vaintroub	2 years ago
Marko Mäkelä	a6290a5bc5	MDEV-33095 innodb_flush_method=O_DIRECT creates excessive errors on Solaris The directio(3C) function on Solaris is supported on NFS and UFS while the majority of users should be on ZFS, which is a copy-on-write file system that implements transparent compression and therefore cannot support unbuffered I/O. Let us remove the call to directio() and simply treat innodb_flush_method=O_DIRECT in the same way as the previous default value innodb_flush_method=fsync on Solaris. Also, let us remove some dead code around calls to os_file_set_nocache() on platforms where fcntl(2) is not usable with O_DIRECT. On IBM AIX, O_DIRECT is not documented for fcntl(2), only for open(2).	2 years ago
Marko Mäkelä	3613fb2aa8	MDEV-33112 innodb_undo_log_truncate=ON is blocking page write When innodb_undo_log_truncate=ON causes an InnoDB undo tablespace to be truncated, we must guarantee that the undo tablespace will be rebuilt atomically: After mtr_t::commit_shrink() has durably written the mini-transaction that rebuilds the undo tablespace, we must not write any old pages to the tablespace. To guarantee this, in trx_purge_truncate_history() we used to traverse the entire buf_pool.flush_list in order to acquire exclusive latches on all pages for the undo tablespace that reside in the buffer pool, so that those pages cannot be written and will be evicted during mtr_t::commit_shrink(). But, this traversal may interfere with the page writing activity of buf_flush_page_cleaner(). It would be better to lazily discard the old pages of the truncated undo tablespace. fil_space_t::is_being_truncated, fil_space_t::clear_stopping(): Remove. fil_space_t::create_lsn: A new field, identifying the LSN of the latest rebuild of a tablespace. buf_page_t::flush(), buf_flush_try_neighbors(): Evict pages whose FIL_PAGE_LSN is below fil_space_t::create_lsn. mtr_t::commit_shrink(): Update fil_space_t::create_lsn and fil_space_t::size right before the log is durably written and the tablespace file is being truncated. fsp_page_create(), trx_purge_truncate_history(): Simplify the logic. Reviewed by: Thirunarayanan Balathandayuthapani, Vladislav Lesin Performance tested by: Axel Schwenke Correctness tested by: Matthias Leich	2 years ago
Marko Mäkelä	f21a6cbf6e	MDEV-32939 If tables are frequently created, renamed, dropped, a backup cannot be restored During mariadb-backup --backup, a table could be renamed, created and dropped. We could have both oldname.ibd and oldname.new, and one of the files would be deleted before the InnoDB recovery starts. The desired end result would be that we will recover both oldname.ibd and newname.ibd. During normal crash recovery, at most one file operation (create, rename, delete) may require to be replayed from the write-ahead log before the DDL recovery starts. deferred_spaces.create(): In mariadb-backup --prepare, try to create the file in case it does not exist. fil_name_process(): Display a message about not found files not only if innodb_force_recovery is set, but also in mariadb-backup --prepare. If we are processing a FILE_RENAME for a tablespace whose recovery is deferred, suppress the message and adjust the file name in case fil_ibd_load() returns FIL_LOAD_NOT_FOUND or FIL_LOAD_DEFER. fil_ibd_load(): Remove a redundant file name comparison. The caller already compared that the file names are different. We used to wrongly return FIL_LOAD_OK instead of FIL_LOAD_ID_CHANGED if only the schema name differed, such as a/t1.ibd and b/t1.ibd. Tested by: Matthias Leich Reviewed by: Thirunarayanan Balathandayuthapani	2 years ago
Marko Mäkelä	7443ad1c8a	MDEV-32374 log_sys.lsn_lock is a performance hog The log_sys.lsn_lock that was introduced in commit `a635c40648` had better be located in the same cache line with log_sys.latch so that log_t::append_prepare() needs to modify only two first cache lines where log_sys is stored. log_t::lsn_lock: On Linux, change the type from pthread_mutex_t to something that may be as small as 32 bits, to pack more data members in the same cache line. On Microsoft Windows, CRITICAL_SECTION works better. log_t::check_flush_or_checkpoint_: Renamed to need_checkpoint. There is no need to pause all writer threads in log_free_check() when we only need to write log_sys.buf to ib_logfile0. That will be done in mtr_t::commit(). log_t::append_prepare_wait(): Make the member function non-static to simplify the call interface, and add a parameter for the LSN. log_t::append_prepare(): Invoke append_prepare_wait() at most once. Only set_check_for_checkpoint() if a log checkpoint needs to be written. If the log buffer needs to be written, we will take care of it ourselves later in our caller. This will reduce interference with log_free_check() in other threads. mtr_t::commit(): Call log_write_up_to() if needed. log_t::get_write_target(): Return a log_write_up_to() target to mtr_t::commit(). buf_flush_ahead(): If we are in furious flushing, call log_sys.set_check_for_checkpoint() so that all writers will wait in log_free_check() until the checkpoint is done. Otherwise, the test innodb.insert_into_empty could occasionally report an error "Crash recovery is broken". log_check_margins(): Replaced by log_free_check(). log_flush_margin(): Removed. This is part of mtr_t::commit() and other operations that write log. log_t::create(), log_t::attach(): Guarantee that buf_free < max_buf_free will always hold on PMEM, to satisfy an assumption of log_t::get_write_target(). log_write_up_to(): Assert lsn!=0. Such calls are not incorrect, but it is cheaper to test that single unlikely condition in mtr_t::commit() rather than test several conditions in log_write_up_to(). innodb_drop_database(), unlock_and_close_files(): Check the LSN before calling log_write_up_to(). ha_innobase::commit_inplace_alter_table(): Remove redundant calls to log_write_up_to() after calling unlock_and_close_files(). Reviewed by: Vladislav Vaintroub Stress tested by: Matthias Leich Performance tested by: Steve Shaw	2 years ago
Marko Mäkelä	eb1f8b2919	MDEV-32027 Opening all .ibd files on InnoDB startup can be slow dict_find_max_space_id(): Return SELECT MAX(SPACE) FROM SYS_TABLES. dict_check_tablespaces_and_store_max_id(): In the normal case (no encryption plugin has been loaded and the change buffer is empty), invoke dict_find_max_space_id() and do not open any .ibd files. If a std::set<uint32_t> has been specified, open the files whose tablespace ID is mentioned. Else, open all data files that are identified by SYS_TABLES records. fil_ibd_open(): Remove a call to os_file_get_last_error() that can report a misleading error, such as EINVAL inside my_realpath() that is not an actual error. This could be invoked when a data file is found but the FSP_SPACE_FLAGS are incorrect, such as is the case for table test.td in ./mtr --mysqld=--innodb-buffer-pool-dump-at-shutdown=0 innodb.table_flags buf_load(): If any tablespaces could not be found, invoke dict_check_tablespaces_and_store_max_id() on the missing tablespaces. dict_load_tablespace(): Try to load the tablespace unless it was found to be futile. This fixes failures related to FTS_*.ibd files for FULLTEXT INDEX. btr_cur_t::search_leaf(): Prevent a crash when the tablespace does not exist. This was caught by the test innodb_fts.fts_concurrent_insert when the change to dict_load_tablespaces() was not present. We modify a few tests to ensure that tables will not be loaded at startup. For some fault injection tests this means that the corrupted tables will not be loaded, because dict_load_tablespace() would perform stricter checks than dict_check_tablespaces_and_store_max_id(). Tested by: Matthias Leich Reviewed by: Thirunarayanan Balathandayuthapani	2 years ago
Marko Mäkelä	1fc2843eee	MDEV-31826: File handle leak on failed IMPORT TABLESPACE fil_space_t::drop(): If the caller is not interested in a detached handle, close it immediately.	2 years ago
Marko Mäkelä	0cc809f91b	MDEV-31826: Memory leak on failed IMPORT TABLESPACE fil_delete_tablespace(): Invoke fil_space_free_low() directly. This fixes up commit `39e3ca8bd2`	2 years ago
Marko Mäkelä	39e3ca8bd2	MDEV-31826 InnoDB may fail to recover after being killed in fil_delete_tablespace() InnoDB was violating the write-ahead-logging protocol when a file was being deleted, like this: 1. fil_delete_tablespace() set the fil_space_t::STOPPING flag 2. The buf_flush_page_cleaner() thread discards some changed pages for this tablespace advances the log checkpoint a little. 3. The server process is killed before fil_delete_tablespace() wrote a FILE_DELETE record. 4. Recovery will try to apply log to pages of the tablespace, because there was no FILE_DELETE record. This will fail, because some pages that had been modified since the latest checkpoint had not been written by the page cleaner. Page writes must not be stopped before a FILE_DELETE record has been durably written. fil_space_t::drop(): Replaces fil_space_t::check_pending_operations(). Add the parameter detached_handle, and return a tablespace pointer if this thread was the first one to stop I/O on the tablespace. mtr_t::commit_file(): Remove the parameter detached_handle, and move some handling to fil_space_t::drop(). fil_space_t: STOPPING_READS, STOPPING_WRITES: Separate flags for STOPPING. We want to stop reads (and encryption) before stopping page writes. fil_space_t::is_stopping_writes(), fil_space_t::get_for_write(): Special accessors for the write path. fil_space_t::flush_low(): Ignore the STOPPING_READS flag and only stop if STOPPING_WRITES is set, to avoid an infinite loop in fil_flush_file_spaces(), which was occasionally repeated by running the test encryption.create_or_replace. Reviewed by: Vladislav Lesin Tested by: Matthias Leich	2 years ago
Marko Mäkelä	cfd1788182	MDEV-32511: Race condition between checkpoint and page write fil_aio_callback(): Invoke fil_node_t::complete_write() before releasing any page latch, so that in case a log checkpoint is executed roughly concurrently with the first write into a file since the previous checkpoint, we will not miss a fdatasync() or fsync() call to make the write durable.	2 years ago
Marko Mäkelä	bf7c6fc20b	MDEV-32511 Assertion !os_aio_pending_writes() failed In MemorySanitizer builds of 10.10 and 10.11, we would rather often have the assertion fail in innodb_init() during mariadb-backup --prepare. The assertion could also fail during InnoDB startup, but less often. Before commit `685d958e38` in 10.8 the log file cleanup after a successfully applied backup is different, and the os_aio_pending_writes() assertion is in srv0start.cc. IORequest::write_complete(): Invoke node->complete_write() before releasing the page latch, so that a log checkpoint that is about to execute concurrently will not miss a fdatasync() or fsync() on the file, in case this was the first write since the last such call. create_log_file(), srv_start(): Replace the debug assertion with a debug check. For all intents and purposes, all writes could have been completed but some write_io_callback() may not have invoked io_slots::release() yet.	2 years ago
Thirunarayanan Balathandayuthapani	cbad0bcd41	MDEV-31098 InnoDB Recovery doesn't display encryption message when no encryption configuration passed - InnoDB fails to report the error when encryption configuration wasn't passed. This patch addresses the issue by adding the error while loading the tablespace and deferring the tablespace creation.	2 years ago
Daniel Black	3b38c2f358	MDEV-18200 MariaBackup full backup failed with InnoDB: Failing assertion: success There are many filesystem related errors that can occur with MariaBackup. These already outputed to stderr with a good description of the error. Many of these are permission or resource (file descriptor) limits where the assertion and resulting core crash doesn't offer developers anything more than the log message. To the user, assertions and core crashes come across as poor error handling. As such we return an error and handle this all the way up the stack.	2 years ago
Daniel Black	c79ca7c7ad	MDEV-18200 MariaBackup full backup failed with InnoDB: Failing assertion: success There are many filesystem related errors that can occur with MariaBackup. These already outputed to stderr with a good description of the error. Many of these are permission or resource (file descriptor) limits where the assertion and resulting core crash doesn't offer developers anything more than the log message. To the user, assertions and core crashes come across as poor error handling. As such we return an error and handle this all the way up the stack.	2 years ago
Daniel Black	ca66a2cbfa	MDEV-18200 MariaBackup full backup failed with InnoDB: Failing assertion: success There are many filesystem related errors that can occur with MariaBackup. These already outputed to stderr with a good description of the error. Many of these are permission or resource (file descriptor) limits where the assertion and resulting core crash doesn't offer developers anything more than the log message. To the user, assertions and core crashes come across as poor error handling. As such we return an error and handle this all the way up the stack.	2 years ago
Marko Mäkelä	72928e640e	MDEV-27593: Crashing on I/O error is unhelpful buf_page_t::write_complete(), buf_page_write_complete(), IORequest::write_complete(): Add a parameter for passing an error code. If an error occurred, we will release the io-fix, buffer-fix and page latch but not reset the oldest_modification field. The block would remain in buf_pool.LRU and possibly buf_pool.flush_list, to be written again later, by buf_flush_page_cleaner(). If all page writes start consistently failing, all write threads should eventually hang in log_free_check() because the log checkpoint cannot be advanced to make room in the circular write-ahead-log ib_logfile0. IORequest::read_complete(): Add a parameter for passing an error code. If a read operation fails, we report the error and discard the page, just like we would do if the page checksum was not validated or the page could not be decrypted. This only affects asynchronous reads, due to linear or random read-ahead or crash recovery. When buf_page_get_low() invokes buf_read_page(), that will be a synchronous read, not involving this code. This was tested by randomly injecting errors in write_io_callback() and read_io_callback(), like this: if (!ut_rnd_interval(100)) cb->m_err= 42;	2 years ago
Marko Mäkelä	bd7908e6ac	MDEV-31568 InnoDB protection against dual processes accessing data insufficient fil_node_open_file_low(): Always acquire an advisory lock on the system tablespace. Originally, we already did this in SysTablespace::open_file(), but SysTablespace::open_or_create() would release those locks when it is closing the file handles. This is a 10.5+ specific follow up to commit `0ee1082bd2` (MDEV-28495). Thanks to Daniel Black for verifying this bug.	2 years ago
Marko Mäkelä	3aea77edeb	MDEV-31347 fil_ibd_create() may hijack the file handle of an old file fil_space_t::add(): If a file handle was passed, invoke fil_node_t::find_metadata() before releasing fil_system.mutex. The call was moved from fil_ibd_create(). This is a 10.5 version of commit `e3b06156c6` from 10.6.	2 years ago
Marko Mäkelä	e3b06156c6	MDEV-31347 fil_ibd_create() may hijack the file handle of an old file fil_ibd_create(): Hold fil_system.mutex until fil_node_t::find_metadata() has completed, so that node->handle cannot be closed by a concurrent thread. This race condition was introduced in commit `10dd290b4b` (MDEV-17380). Tested by: Matthias Leich	2 years ago
Marko Mäkelä	f2c17cc9d9	MDEV-29911 InnoDB recovery and mariadb-backup --prepare fail to report detailed progress This is a 10.6 port of commit `2f9e264781` from MariaDB Server 10.9 that is missing some optimization due to a more complex redo log format and recovery logic (which was simplified in commit `685d958e38`). The progress reporting of InnoDB crash recovery was rather intermittent. Nothing was reported during the single-threaded log record parsing, which could consume minutes when parsing a large log. During log application, there only was progress reporting in background threads that would be invoked on data page read completion. The progress reporting here will be detailed like this: InnoDB: Starting crash recovery from checkpoint LSN=628599973,5653727799 InnoDB: Read redo log up to LSN=1963895808 InnoDB: Multi-batch recovery needed at LSN 2534560930 InnoDB: Read redo log up to LSN=3312233472 InnoDB: Read redo log up to LSN=1599646720 InnoDB: Read redo log up to LSN=2160831488 InnoDB: To recover: LSN 2806789376/2806819840; 195082 pages InnoDB: To recover: LSN 2806789376/2806819840; 63507 pages InnoDB: Read redo log up to LSN=3195776000 InnoDB: Read redo log up to LSN=3687099392 InnoDB: Read redo log up to LSN=4165315584 InnoDB: To recover: LSN 4374395699/4374440960; 241454 pages InnoDB: To recover: LSN 4374395699/4374440960; 123701 pages InnoDB: Read redo log up to LSN=4508724224 InnoDB: Read redo log up to LSN=5094550528 InnoDB: To recover: 205230 pages The previous messages "Starting a batch to recover" or "Starting a final batch to recover" will be replaced by "To recover: ... pages" messages. If a batch lasts longer than 15 seconds, then there will be progress reports every 15 seconds, showing the number of remaining pages. For the non-final batch, the "To recover:" message includes two end LSN: that of the batch, and of the recovered log. This is the primary measure of progress. The batch will end once the number of pages to recover reaches 0. If recovery is possible in a single batch, the output will look like this, with a shorter "To recover:" message that counts only the remaining pages: InnoDB: Starting crash recovery from checkpoint LSN=628599973,5653727799 InnoDB: Read redo log up to LSN=1984539648 InnoDB: Read redo log up to LSN=2710875136 InnoDB: Read redo log up to LSN=3358895104 InnoDB: Read redo log up to LSN=3965299712 InnoDB: Read redo log up to LSN=4557417472 InnoDB: Read redo log up to LSN=5219527680 InnoDB: To recover: 450915 pages We will also speed up recovery by improving the memory management and implementing multi-threaded recovery of data pages that will not need to be read into the buffer pool ("fake read"). Log application in the "fake read" threads will be protected by an atomic being_recovered field and exclusive buf_page_t::lock. Recovery will reserve for data pages two thirds of the buffer pool, or 256 pages, whichever is smaller. Previously, we could only use at most one third of the buffer pool for buffered log records. This would typically mean that with large buffer pools, recovery unnecessary consisted of multiple batches. If recovery runs out of memory, it will "roll back" or "rewind" the current mini-transaction. The recv_sys.recovered_lsn and recv_sys.pages will correspond to the "out of memory LSN", at the end of the previous complete mini-transaction. If recovery runs out of memory while executing the final recovery batch, we can simply invoke recv_sys.apply(false) to make room, and resume parsing. If recovery runs out of memory before the final batch, we will scan the redo log to the end and check for any missing or inconsistent files. In this version of the patch, we will throw away any previously buffered recv_sys.pages and rescan the log from the checkpoint onwards. recv_sys_t::pages_it: A cached iterator to recv_sys.pages. recv_sys_t::is_memory_exhausted(): Remove. We will have out-of-memory handling deep inside recv_sys_t::parse(). recv_sys_t::rewind(), page_recv_t::recs_t::rewind(): Remove all log starting with a specific LSN. IORequest::write_complete(), IORequest::read_complete(): Replaces fil_aio_callback(). read_io_callback(), write_io_callback(): Replaces io_callback(). IORequest::fake_read_complete(), fake_io_callback(), os_fake_read(): Process a "fake read" request for concurrent recovery. recv_sys_t::apply_batch(): Choose a number of successive pages for a recovery batch. recv_sys_t::erase(recv_sys_t::map::iterator): Remove log records for a page whose recovery is not in progress. Log application threads will not invoke this; they will only set being_recovered=-1 to indicate that the entry is no longer needed. recv_sys_t::garbage_collect(): Remove all being_recovered=-1 entries. recv_sys_t::wait_for_pool(): Wait for some space to become available in the buffer pool. mlog_init_t::mark_ibuf_exist(): Avoid calls to recv_sys::recover_low() via ibuf_page_exists() and buf_page_get_low(). Such calls would lead to double locking of recv_sys.mutex, which depending on implementation could cause a deadlock. We will use lower-level calls to look up index pages. buf_LRU_block_remove_hashed(): Disable consistency checks for freed ROW_FORMAT=COMPRESSED pages. Their contents could be uninitialized garbage. This fixes an occasional failure of the test innodb.innodb_bulk_create_index_debug. Tested by: Matthias Leich	2 years ago
Marko Mäkelä	2f9e264781	MDEV-29911 InnoDB recovery and mariadb-backup --prepare fail to report detailed progress The progress reporting of InnoDB crash recovery was rather intermittent. Nothing was reported during the single-threaded log record parsing, which could consume minutes when parsing a large log. During log application, there only was progress reporting in background threads that would be invoked on data page read completion. The progress reporting here will be detailed like this: InnoDB: Starting crash recovery from checkpoint LSN=503549688 InnoDB: Parsed redo log up to LSN=1990840177; to recover: 124806 pages InnoDB: Parsed redo log up to LSN=2729777071; to recover: 186123 pages InnoDB: Parsed redo log up to LSN=3488599173; to recover: 248397 pages InnoDB: Parsed redo log up to LSN=4177856618; to recover: 306469 pages InnoDB: Multi-batch recovery needed at LSN 4189599815 InnoDB: End of log at LSN=4483551634 InnoDB: To recover: LSN 4189599815/4483551634; 307490 pages InnoDB: To recover: LSN 4189599815/4483551634; 197159 pages InnoDB: To recover: LSN 4189599815/4483551634; 67623 pages InnoDB: Parsed redo log up to LSN=4353924218; to recover: 102083 pages ... InnoDB: log sequence number 4483551634 ... The previous messages "Starting a batch to recover" or "Starting a final batch to recover" will be replaced by "To recover: ... pages" messages. If a batch lasts longer than 15 seconds, then there will be progress reports every 15 seconds, showing the number of remaining pages. For the non-final batch, the "To recover:" message includes two end LSN: that of the batch, and of the recovered log. This is the primary measure of progress. The batch will end once the number of pages to recover reaches 0. If recovery is possible in a single batch, the output will look like this, with a shorter "To recover:" message that counts only the remaining pages: InnoDB: Starting crash recovery from checkpoint LSN=503549688 InnoDB: Parsed redo log up to LSN=1998701027; to recover: 125560 pages InnoDB: Parsed redo log up to LSN=2734136874; to recover: 186446 pages InnoDB: Parsed redo log up to LSN=3499505504; to recover: 249378 pages InnoDB: Parsed redo log up to LSN=4183247844; to recover: 306964 pages InnoDB: End of log at LSN=4483551634 ... InnoDB: To recover: 331797 pages ... InnoDB: log sequence number 4483551634 ... We will also speed up recovery by improving the memory management and implementing multi-threaded recovery of data pages that will not need to be read into the buffer pool ("fake read"). Log application in the "fake read" threads will be protected by an atomic being_recovered field and exclusive buf_page_t::latch. Recovery will reserve for data pages two thirds of the buffer pool, or 256 pages, whichever is smaller. Previously, we could only use at most one third of the buffer pool for buffered log records. This would typically mean that with large buffer pools, recovery unnecessary consisted of multiple batches. If recovery runs out of memory, it will "roll back" or "rewind" the current mini-transaction. The recv_sys.lsn and recv_sys.pages will correspond to the "out of memory LSN", at the end of the previous complete mini-transaction. If recovery runs out of memory while executing the final recovery batch, we can simply invoke recv_sys.apply(false) to make room, and resume parsing. If recovery runs out of memory before the final batch, we will scan the redo log to the end (recv_sys.scanned_lsn) and check for any missing or inconsistent files. If recv_init_crash_recovery_spaces() does not report any potentially missing tablespaces, we can make use of the already stored recv_sys.pages and only rewind to the "out of memory LSN". Else, we must keep parsing and invoking recv_validate_tablespace() until an error has been found or everything has been resolved, and ultimatily rewind to to the checkpoint LSN. recv_sys_t::pages_it: A cached iterator to recv_sys.pages recv_sys_t::parse_mtr(): Remove an ATTRIBUTE_NOINLINE that would prevent tail call optimization in recv_sys_t::parse_pmem(). recv_sys_t::parse(), recv_sys_t::parse_mtr(), recv_sys_t::parse_pmem(): Add template<bool store> parameter. Redo log record parsing (store=false) is better specialized from store=true (with bool if_exists) so that we can avoid some conditional branches in frequently invoked low-level code. recv_sys_t::is_memory_exhausted(): Remove. The special parse() status GOT_OOM will report out-of-memory situation at the low level. recv_sys_t::rewind(), page_recv_t::recs_t::rewind(): Remove all log starting with a specific LSN. recv_scan_log(): Separate some code for only parsing, not storing log. In rewound_lsn, remember the LSN at which last_phase=false recovery ran out of memory. This is where the next call to recv_scan_log() will resume storing the log. This replaces recv_sys.last_stored_lsn. recv_sys_t::parse(): Evaluate the template parameter store in a few more cases, to allow dead code to be eliminated at compile time. recv_sys_t::scanned_lsn: The end of the log found by recv_scan_log(). The special value 1 means that recv_sys has been initialized but no log has been parsed. IORequest::write_complete(), IORequest::read_complete(): Replaces fil_aio_callback(). read_io_callback(), write_io_callback(): Replaces io_callback(). IORequest::fake_read_complete(), fake_io_callback(), os_fake_read(): Process a "fake read" request for concurrent recovery. recv_sys_t::apply_batch(): Choose a number of successive pages for a recovery batch. recv_sys_t::erase(recv_sys_t::map::iterator): Remove log records for a page whose recovery is not in progress. Log application threads will not invoke this; they will only set being_recovered=-1 to indicate that the entry is no longer needed. recv_sys_t::garbage_collect(): Remove all being_recovered=-1 entries. recv_sys_t::wait_for_pool(): Wait for some space to become available in the buffer pool. mlog_init_t::mark_ibuf_exist(): Avoid calls to recv_sys::recover_low() via ibuf_page_exists() and buf_page_get_low(). Such calls would lead to double locking of recv_sys.mutex, which depending on implementation could cause a deadlock. We will use lower-level calls to look up index pages. buf_LRU_block_remove_hashed(): Disable consistency checks for freed ROW_FORMAT=COMPRESSED pages. Their contents could be uninitialized garbage. This fixes an occasional failure of the test innodb.innodb_bulk_create_index_debug. Tested by: Matthias Leich	2 years ago
Vlad Lesin	5422784792	MDEV-31256 fil_node_open_file() releases fil_system.mutex allowing other thread to open its file node There is room between mutex_exit(&fil_system.mutex) and mutex_enter(&fil_system.mutex) calls in fil_node_open_file(). During this room another thread can open the node, and ut_ad(!node->is_open()) assertion in fil_node_open_file_low() can fail. The fix is not to open node if it was already opened by another thread.	2 years ago
Marko Mäkelä	5740638c4c	MDEV-31132 Deadlock between DDL and purge of InnoDB history log_free_check(): Assert that the caller must not hold exclusive lock_sys.latch. This was the case for calls from ibuf_delete_for_discarded_space(). This caused a deadlock with another thread that would be holding a latch on a dirty page that would need to be written so that the checkpoint would advance and log_free_check() could return. That other thread was waiting for a shared lock_sys.latch. fil_delete_tablespace(): Do not invoke ibuf_delete_for_discarded_space() because in DDL operations, we will be holding exclusive lock_sys.latch. trx_t::commit(std::vector<pfs_os_file_t>&), innodb_drop_database(), row_purge_remove_clust_if_poss_low(), row_undo_ins_remove_clust_rec(), row_discard_tablespace_for_mysql(): Invoke ibuf_delete_for_discarded_space() on the deleted tablespaces after releasing all latches.	3 years ago
Marko Mäkelä	0cda0e4e15	MDEV-31080 fil_validate() failures during deferred tablespace recovery fil_space_t::create(), fil_space_t::add(): Expect the caller to acquire and release fil_system.mutex. In this way, creating a tablespace and adding the first (usually only) data file will be atomic. recv_sys_t::recover_deferred(): Correctly protect some changes by holding fil_system.mutex. Tested by: Matthias Leich	3 years ago
Vlad Lesin	71f16c836f	MDEV-31049 fil_delete_tablespace() returns wrong file handle if tablespace was closed by parallel thread fil_delete_tablespace() stores file handle in local variable and calls mtr_t::commit_file()=>fil_system_t::detach(..., detach_handle=true), which sets space->chain.start->handle = OS_FILE_CLOSED. fil_system_t::detach() is invoked under fil_system.mutex. But before the mutex is acquired some parallel thread can change space->chain.start->handle. fil_delete_tablespace() returns value, stored in local variable, i.e. wrong value. File handle can be closed, for example, from buf_flush_space() when the limit of innodb_open_files exceded and fil_space_t::get() causes fil_space_t::try_to_close() call. fil_space_t::try_to_close() is executed under fil_system.mutex. And mtr_t::commit_file() locks it for fil_system_t::detach() call. fil_system_t::detach() returns detached file handle if its argument detach_handle is true. The fix is to let mtr_t::commit_file() to pass that detached file handle to fil_delete_tablespace().	3 years ago
Vlad Lesin	0cca8166f3	MDEV-30775 Performance regression in fil_space_t::try_to_close() introduced in MDEV-23855 Post-push fix. 10.5 MDEV-30775 fix inserts just opened tablespace just after the element which fil_system.space_list_last_opened points to. In MDEV-25223 fil_system_t::space_list was changed from UT_LIST to ilist. ilist<...>::insert(iterator pos, reference value) inserts element to list before pos. But it was not taken into account during 10.5->10.6 merge in `85cbfaefee`, and the fix does not work properly, i.e. it inserted just opened tablespace to the position preceding fil_system.space_list_last_opened.	3 years ago
Thirunarayanan Balathandayuthapani	2ddfb83807	MDEV-29273 Race condition between drop table and closing of table - This issue caused by race condition between drop thread and fil_encrypt_thread. fil_encrypt_thread closes the tablespace if the number of opened files exceeds innodb_open_files. fil_node_open_file() closes the tablespace which are open and it doesn't have pending operations. At that time, InnoDB drop tries to write the redo log for the file delete operation. It throws the bad file descriptor error. - When trying to close the file, InnoDB should check whether the table is going to be dropped.	3 years ago
Marko Mäkelä	402f36dd65	MDEV-30936 fixup fil_space_t::~fil_space_t(): Invoke ut_free(name) because doing so in the callers would trip MSAN_OPTIONS=poison_in_dtor=1	3 years ago
Vlad Lesin	4c226c1850	MDEV-29050 mariabackup issues error messages during InnoDB tablespaces export on partial backup preparing The solution is to suppress error messages for missing tablespaces if mariabackup is launched with "--prepare --export" options. "mariabackup --prepare --export" invokes itself with --mysqld parameter. If the parameter is set, then it starts server to feed "FLUSH TABLES ... FOR EXPORT;" queries for exported tablespaces. This is "normal" server start, that's why new srv_operation value is introduced. Reviewed by Marko Makela.	3 years ago
Vlad Lesin	7d6b3d4008	MDEV-30775 Performance regression in fil_space_t::try_to_close() introduced in MDEV-23855 fil_node_open_file_low() tries to close files from the top of fil_system.space_list if the number of opened files is exceeded. It invokes fil_space_t::try_to_close(), which iterates the list searching for the first opened space. Then it just closes the space, leaving it in the same position in fil_system.space_list. On heavy files opening, like during 'SHOW TABLE STATUS ...' execution, if the number of opened files limit is reached, fil_space_t::try_to_close() iterates more and more closed spaces before reaching any opened space for each fil_node_open_file_low() call. What causes performance regression if the number of spaces is big enough. The fix is to keep opened spaces at the top of fil_system.space_list, and move closed files at the end of the list. For this purpose fil_space_t::space_list_last_opened pointer is introduced. It points to the last inserted opened space in fil_space_t::space_list. When space is opened, it's inserted to the position just after the pointer points to in fil_space_t::space_list to preserve the logic, inroduced in MDEV-23855. Any closed space is added to the end of fil_space_t::space_list. As opened spaces are located at the top of fil_space_t::space_list, fil_space_t::try_to_close() finds opened space faster. There can be the case when opened and closed spaces are mixed in fil_space_t::space_list if fil_system.freeze_space_list was set during fil_node_open_file_low() execution. But this should not cause any error, as fil_space_t::try_to_close() still iterates spaces in the list. There is no need in any test case for the fix, as it does not change any functionality, but just fixes performance regression.	3 years ago
Marko Mäkelä	de4030e4d4	MDEV-30400 Assertion height == btr_page_get_level(...) on INSERT This also fixes part of MDEV-29835 Partial server freeze which is caused by violations of the latching order that was defined in https://dev.mysql.com/worklog/task/?id=6326 (WL#6326: InnoDB: fix index->lock contention). Unless the current thread is holding an exclusive dict_index_t::lock, it must acquire page latches in a strict parent-to-child, left-to-right order. Not all cases of MDEV-29835 are fixed yet. Failure to follow the correct latching order will cause deadlocks of threads due to lock order inversion. As part of these changes, the BTR_MODIFY_TREE mode is modified so that an Update latch (U a.k.a. SX) will be acquired on the root page, and eXclusive latches (X) will be acquired on all pages leading to the leaf page, as well as any left and right siblings of the pages along the path. The DEBUG_SYNC test innodb.innodb_wl6326 will be removed, because at the time the DEBUG_SYNC point is hit, the thread is actually holding several page latches that will be blocking a concurrent SELECT statement. We also remove double bookkeeping that was caused due to excessive information hiding in mtr_t::m_memo. We simply let mtr_t::m_memo store information of latched pages, and ensure that mtr_memo_slot_t::object is never a null pointer. The tree_blocks[] and tree_savepoints[] were redundant. buf_page_get_low(): If innodb_change_buffering_debug=1, to avoid a hang, do not try to evict blocks if we are holding a latch on a modified page. The test innodb.innodb-change-buffer-recovery will be removed, because change buffering may no longer be forced by debug injection when the change buffer comprises multiple pages. Remove a debug assertion that could fail when innodb_change_buffering_debug=1 fails to evict a page. For other cases, the assertion is redundant, because we already checked that right after the got_block: label. The test innodb.innodb-change-buffering-recovery will be removed, because due to this change, we will be unable to evict the desired page. mtr_t::lock_register(): Register a change of a page latch on an unmodified buffer-fixed block. mtr_t::x_latch_at_savepoint(), mtr_t::sx_latch_at_savepoint(): Replaced by the use of mtr_t::upgrade_buffer_fix(), which now also handles RW_S_LATCH. mtr_t::set_modified(): For temporary tables, invoke buf_page_t::set_modified() here and not in mtr_t::commit(). We will never set the MTR_MEMO_MODIFY flag on other than persistent data pages, nor set mtr_t::m_modifications when temporary data pages are modified. mtr_t::commit(): Only invoke the buf_flush_note_modification() loop if persistent data pages were modified. mtr_t::get_already_latched(): Look up a latched page in mtr_t::m_memo. This avoids many redundant entries in mtr_t::m_memo, as well as redundant calls to buf_page_get_gen() for blocks that had already been looked up in a mini-transaction. btr_get_latched_root(): Return a pointer to an already latched root page. This replaces btr_root_block_get() in cases where the mini-transaction has already latched the root page. btr_page_get_parent(): Fetch a parent page that was already latched in BTR_MODIFY_TREE, by invoking mtr_t::get_already_latched(). If needed, upgrade the root page U latch to X. This avoids bloating mtr_t::m_memo as well as performing redundant buf_pool.page_hash lookups. For non-QUICK CHECK TABLE as well as for B-tree defragmentation, we will invoke btr_cur_search_to_nth_level(). btr_cur_search_to_nth_level(): This will only be used for non-leaf (level>0) B-tree searches that were formerly named BTR_CONT_SEARCH_TREE or BTR_CONT_MODIFY_TREE. In MDEV-29835, this function could be removed altogether, or retained for the case of CHECK TABLE without QUICK. btr_cur_t::left_block: Remove. btr_pcur_move_backward_from_page() can retrieve the left sibling from the end of mtr_t::m_memo. btr_cur_t::open_leaf(): Some clean-up. btr_cur_t::search_leaf(): Replaces btr_cur_search_to_nth_level() for searches to level=0 (the leaf level). We will never release parent page latches before acquiring leaf page latches. If we need to temporarily release the level=1 page latch in the BTR_SEARCH_PREV or BTR_MODIFY_PREV latch_mode, we will reposition the cursor on the child node pointer so that we will land on the correct leaf page. btr_cur_t::pessimistic_search_leaf(): Implement new BTR_MODIFY_TREE latching logic in the case that page splits or merges will be needed. The parent pages (and their siblings) should already be latched on the first dive to the leaf and be present in mtr_t::m_memo; there should be no need for BTR_CONT_MODIFY_TREE. This pre-latching almost suffices; it must be revised in MDEV-29835 and work-arounds removed for cases where mtr_t::get_already_latched() fails to find a block. rtr_search_to_nth_level(): A SPATIAL INDEX version of btr_search_to_nth_level() that can search to any level (including the leaf level). rtr_search_leaf(), rtr_insert_leaf(): Wrappers for rtr_search_to_nth_level(). rtr_search(): Replaces rtr_pcur_open(). rtr_latch_leaves(): Replaces btr_cur_latch_leaves(). Note that unlike in the B-tree code, there is no error handling in case the sibling pages are corrupted. rtr_cur_restore_position(): Remove an unused constant parameter. btr_pcur_open_on_user_rec(): Remove the constant parameter mode=PAGE_CUR_GE. row_ins_clust_index_entry_low(): Use a new mode=BTR_MODIFY_ROOT_AND_LEAF to gain access to the root page when mode!=BTR_MODIFY_TREE, to write the PAGE_ROOT_AUTO_INC. BTR_SEARCH_TREE, BTR_CONT_SEARCH_TREE: Remove. BTR_CONT_MODIFY_TREE: Note that this is only used by rtr_search_to_nth_level(). btr_pcur_optimistic_latch_leaves(): Replaces btr_cur_optimistic_latch_leaves(). ibuf_delete_rec(): Acquire exclusive ibuf.index->lock in order to avoid a deadlock with ibuf_insert_low(BTR_MODIFY_PREV). btr_blob_log_check_t(): Acquire a U latch on the root page, so that btr_page_alloc() in btr_store_big_rec_extern_fields() will avoid a deadlock. btr_store_big_rec_extern_fields(): Assert that the root page latch is being held. Tested by: Matthias Leich Reviewed by: Vladislav Lesin	3 years ago
Marko Mäkelä	15ab2e122d	MDEV-30132 Crash after recovery, with InnoDB: Tried to read ... os_file_read(): Merged with os_file_read_no_error_handling(). Crashing on a partial page read is as unhelpful as crashing on a corrupted page read (commit `0b47c126e3`). Report the file name if it is available via IORequest.	3 years ago
Marko Mäkelä	1188ef4ade	MDEV-30132 Crash after recovery, with InnoDB: Tried to read ... bytes at offset fil_space_t::prepare_acquired(): Do not attempt to extend (or shrink) files that will be processed by recv_sys_t::recover_deferred().	3 years ago
Marko Mäkelä	b260903832	MDEV-29258 Failing assertion for name length on RENAME TABLE trx_undo_page_report_rename(): Use the correct maximum length of a table name. Both the database name and the table name can be up to NAME_CHAR_LEN (64 characters) times 5 bytes per character in the my_charset_filename encoding. They are not encoded in UTF-8! fil_op_write_log(): Reserve the correct amount of log buffer for a rename operation. The file name will be appended by mlog_catenate_string(). rename_file_ext(): Reserve a large enough buffer for the file names.	3 years ago
Marko Mäkelä	0ee1082bd2	MDEV-28495 InnoDB corruption due to lack of file locking Starting with commit `da094188f6` (MDEV-24393), MariaDB will no longer acquire advisory file locks on InnoDB data files by default, because it would create a large number of entries in Linux /proc/locks. The motivation for acquiring the file locks is to prevent accidental concurrent startup of multiple server processes on the same data files. Such mistake still turns out to be relatively common, based on corruption bug reports from the community. To prevent corruption due to concurrent startup attempts, the Aria storage engine would unconditionally acquire an advisory lock on one of its log files. Solution: InnoDB will always lock its system tablespace files. (Ever since commit `685d958e38` the InnoDB log file will not necessarily be open while the server is running, because it can be accessed via memory-mapped I/O.) If more protection is desired, then the option --external-locking can be used. The mandatory advisory lock also fixes intermittent failures of some crash recovery tests. It turns out that when the mtr test harness kills and restarts the server, it will not actually ensure that the old process has terminated before starting the new one.	3 years ago
Marko Mäkelä	a8a27f1edf	Correct a bogus comment. The logic on Windows was originally simplified in commit `4bca1a786f` for storage/xtradb and later in commit `6304c0bfc7` for storage/innobase.	3 years ago
Marko Mäkelä	a1267724cb	MDEV-26293 InnoDB: Failing assertion: space->is_ready_to_close() ... fil_space_t::acquire_low(): Introduce a parameter that specifies which flags should be avoided. At all times, referenced() must not be incremented if the STOPPING flag is set. When fil_system.mutex is not being held by the current thread, the reference must not be incremented if the CLOSING flag is set (unless NEEDS_FSYNC is set, in fil_space_t::flush()). fil_space_t::acquire(): Invoke acquire_low(STOPPING \| CLOSING). In this way, the reference count cannot be incremented after fil_space_t::try_to_close() invoked fil_space_t::set_closing(). If the CLOSING flag was set, we must retry acquire_low() after acquiring fil_system.mutex. fil_space_t::prepare_acquired(): Replaces prepare(true). fil_space_t::acquire_and_prepare(): Replaces prepare(). This basically retries fil_space_t::acquire() after acquiring fil_system.mutex.	3 years ago
Marko Mäkelä	2e43af69e3	MDEV-28870 InnoDB: Missing FILE_CREATE, FILE_DELETE or FILE_MODIFY before FILE_CHECKPOINT There was a race condition between log_checkpoint_low() and deleting or renaming data files. The scenario is as follows: 1. The buffer pool does not contain dirty pages. 2. A FILE_DELETE or FILE_RENAME record is written. 3. The checkpoint LSN will be moved ahead of the write of the record. 4. The server is killed before the file is actually renamed or deleted. We will prevent this race condition by ensuring that a log checkpoint cannot occur between the durable write and the file system operation: 1. Durably write the FILE_DELETE or FILE_RENAME record. 2. Perform the file system operation. 3. Allow any log checkpoint to proceed. mtr_t::commit_file(): Implement the DELETE or RENAME logic. fil_delete_tablespace(): Delegate some of the logic to mtr_t::commit_file(). fil_space_t::rename(): Delegate some logic to mtr_t::commit_file(). Remove the debug injection point fil_rename_tablespace_failure_2 because we do test RENAME failures without any debug injection. fil_name_write_rename_low(), fil_name_write_rename(): Remove. Tested by Matthias Leich	3 years ago
Marko Mäkelä	77b3959b5c	MDEV-28457 Crash in page_dir_find_owner_slot() A prominent remaining source of crashes on corrupted index pages is page directory corruption. A frequent caller of page_dir_find_owner_slot() is page_rec_get_prev(). Some of those calls can be replaced with simpler logic that is less prone to fail. page_dir_find_owner_slot(), page_rec_get_prev(), page_rec_get_prev_const(), btr_pcur_move_to_prev(), btr_pcur_move_to_prev_on_page(), btr_cur_upd_rec_sys(), page_delete_rec_list_end(), rtr_page_copy_rec_list_end_no_locks(), rtr_page_copy_rec_list_start_no_locks(): Return an error code on failure. fil_space_t::io(), buf_page_get_low(): Use DB_CORRUPTION for out-of-bounds page reads. PageBulk::getSplitRec(), PageBulk::copyOut(): Simplify the code. btr_validate_level(): Prevent some more CHECK TABLE crashes on corrupted pages. btr_block_get(), btr_pcur_move_to_next_page(): Implement some checks that were previously only part of IndexPurge::next(). IndexPurge::next(): Use btr_pcur_move_to_next_page().	3 years ago
Marko Mäkelä	c9498f33de	MDEV-18519: Assertion failure in btr_page_reorganize_low() Even after commit `0b47c126e3` there are a few ib::fatal() calls in non-debug code that can be replaced easily. btr_page_reorganize_low(): On size invariant violation, return an error code instead of crashing. btr_check_blob_fil_page_type(): On an invalid page type, report an error but do not crash. btr_copy_blob_prefix(): Truncate the output if a page type is invalid. dict_load_foreign_cols(): On an error, return DB_CORRUPTION instead of crashing. fil_space_decrypt_full_crc32(), fil_space_decrypt_for_non_full_checksum(): On error, return DB_DECRYPTION_FAILED instead of crashing. fil_set_max_space_id_if_bigger(): Replace ib::fatal() with an equivalent ut_a() assertion.	3 years ago
Marko Mäkelä	4179f93d28	MDEV-18976 Implement OPT_PAGE_CHECKSUM log record for improved validation We will introduce an optional log record OPT_PAGE_CHECKSUM for recording page checksums, so that more inconsistencies on crash recovery may be caught. mtr_t::page_checksum(const buf_page_t&): Write OPT_PAGE_CHECKSUM (currently not for ROW_FORMAT=COMPRESSED pages). mtr_t::do_write(): Write OPT_PAGE_CHECKSUM records for all pages (currently, in debug builds only). mtr_t::is_logged(): Return whether log should be written. mtr_t::set_log_mode_sub(const mtr_t&): Set the logging mode of a sub-minitransaction when another mini-transaction is holding latches on some modified pages. When creating or freeing BLOB pages, we may only write OPT_PAGE_CHECKSUM records in the main mini-transaction, after all changes have been written to the log. MTR_LOG_SUB: Log mode for a sub-mini-transaction. mtr_t::free(): Define non-inline, and invoke MarkFreed. MarkFreed: For any matching page in the mini-transaction log, change the first entry to say MTR_MEMO_PAGE_X_MODIFY and any subsequent entries to MTR_MEMO_PAGE_X_FIX. FindModified: Simplify a condition. MTR_MEMO_MODIFY can only be set if MTR_MEMO_PAGE_X_FIX or MTR_MEMO_PAGE_SX_FIX are set. FindBlockX: Consider also MTR_MEMO_PAGE_X_MODIFY. recv_sys_t::parse(): Store OPT_PAGE_CHECKSUM records. log_phys_t::apply(): Validate OPT_PAGE_CHECKSUM records. log_phys_t::page_checksum(): Validate an OPT_PAGE_CHECKSUM record. Tested by: Matthias Leich	3 years ago
Marko Mäkelä	cc4eabc7b2	MDEV-13542: Implement page read fault injection --debug-dbug=d,intermittent_read_failure is effective after the database has been started up. --debug-dbug=d,intermittent_recovery_failure is always effective, including during recovery.	3 years ago
Marko Mäkelä	0b47c126e3	MDEV-13542: Crashing on corrupted page is unhelpful The approach to handling corruption that was chosen by Oracle in commit `177d8b0c12` is not really useful. Not only did it actually fail to prevent InnoDB from crashing, but it is making things worse by blocking attempts to rescue data from or rebuild a partially readable table. We will try to prevent crashes in a different way: by propagating errors up the call stack. We will never mark the clustered index persistently corrupted, so that data recovery may be attempted by reading from the table, or by rebuilding the table. This should also fix MDEV-13680 (crash on btr_page_alloc() failure); it was extensively tested with innodb_file_per_table=0 and a non-autoextend system tablespace. We should now avoid crashes in many cases, such as when a page cannot be read or allocated, or an inconsistency is detected when attempting to update multiple pages. We will not crash on double-free, such as on the recovery of DDL in system tablespace in case something was corrupted. Crashes on corrupted data are still possible. The fault injection mechanism that is introduced in the subsequent commit may help catch more of them. buf_page_import_corrupt_failure: Remove the fault injection, and instead corrupt some pages using Perl code in the tests. btr_cur_pessimistic_insert(): Always reserve extents (except for the change buffer), in order to prevent a subsequent allocation failure. btr_pcur_open_at_rnd_pos(): Merged to the only caller ibuf_merge_pages(). btr_assert_not_corrupted(), btr_corruption_report(): Remove. Similar checks are already part of btr_block_get(). FSEG_MAGIC_N_BYTES: Replaces FSEG_MAGIC_N_VALUE. dict_hdr_get(), trx_rsegf_get_new(), trx_undo_page_get(), trx_undo_page_get_s_latched(): Replaced with error-checking calls. trx_rseg_t::get(mtr_t): Replaces trx_rsegf_get(). trx_rseg_header_create(): Let the caller update the TRX_SYS page if needed. trx_sys_create_sys_pages(): Merged with trx_sysf_create(). dict_check_tablespaces_and_store_max_id(): Do not access DICT_HDR_MAX_SPACE_ID, because it was already recovered in dict_boot(). Merge dict_check_sys_tables() with this function. dir_pathname(): Replaces os_file_make_new_pathname(). row_undo_ins_remove_sec(): Do not modify the undo page by adding a terminating NUL byte to the record. btr_decryption_failed(): Report decryption failures dict_set_corrupted_by_space(), dict_set_encrypted_by_space(), dict_set_corrupted_index_cache_only(): Remove. dict_set_corrupted(): Remove the constant parameter dict_locked=false. Never flag the clustered index corrupted in SYS_INDEXES, because that would deny further access to the table. It might be possible to repair the table by executing ALTER TABLE or OPTIMIZE TABLE, in case no B-tree leaf page is corrupted. dict_table_skip_corrupt_index(), dict_table_next_uncorrupted_index(), row_purge_skip_uncommitted_virtual_index(): Remove, and refactor the callers to read dict_index_t::type only once. dict_table_is_corrupted(): Remove. dict_index_t::is_btree(): Determine if the index is a valid B-tree. BUF_GET_NO_LATCH, BUF_EVICT_IF_IN_POOL: Remove. UNIV_BTR_DEBUG: Remove. Any inconsistency will no longer trigger assertion failures, but error codes being returned. buf_corrupt_page_release(): Replaced with a direct call to buf_pool.corrupted_evict(). fil_invalid_page_access_msg(): Never crash on an invalid read; let the caller of buf_page_get_gen() decide. btr_pcur_t::restore_position(): Propagate failure status to the caller by returning CORRUPTED. opt_search_plan_for_table(): Simplify the code. row_purge_del_mark(), row_purge_upd_exist_or_extern_func(), row_undo_ins_remove_sec_rec(), row_undo_mod_upd_del_sec(), row_undo_mod_del_mark_sec(): Avoid mem_heap_create()/mem_heap_free() when no secondary indexes exist. row_undo_mod_upd_exist_sec(): Simplify the code. row_upd_clust_step(), dict_load_table_one(): Return DB_TABLE_CORRUPT if the clustered index (and therefore the table) is corrupted, similar to what we do in row_insert_for_mysql(). fut_get_ptr(): Replace with buf_page_get_gen() calls. buf_page_get_gen(): Return nullptr and err=DB_CORRUPTION if the page is marked as freed. For other modes than BUF_GET_POSSIBLY_FREED or BUF_PEEK_IF_IN_POOL this will trigger a debug assertion failure. For BUF_GET_POSSIBLY_FREED, we will return nullptr for freed pages, so that the callers can be simplified. The purge of transaction history will be a new user of BUF_GET_POSSIBLY_FREED, to avoid crashes on corrupted data. buf_page_get_low(): Never crash on a corrupted page, but simply return nullptr. fseg_page_is_allocated(): Replaces fseg_page_is_free(). fts_drop_common_tables(): Return an error if the transaction was rolled back. fil_space_t::set_corrupted(): Report a tablespace as corrupted if it was not reported already. fil_space_t::io(): Invoke fil_space_t::set_corrupted() to report out-of-bounds page access or other errors. Clean up mtr_t::page_lock() buf_page_get_low(): Validate the page identifier (to check for recently read corrupted pages) after acquiring the page latch. buf_page_t::read_complete(): Flag uninitialized (all-zero) pages with DB_FAIL. Return DB_PAGE_CORRUPTED on page number mismatch. mtr_t::defer_drop_ahi(): Renamed from mtr_defer_drop_ahi(). recv_sys_t::free_corrupted_page(): Only set_corrupt_fs() if any log records exist for the page. We do not mind if read-ahead produces corrupted (or all-zero) pages that were not actually needed during recovery. recv_recover_page(): Return whether the operation succeeded. recv_sys_t::recover_low(): Simplify the logic. Check for recovery error. Thanks to Matthias Leich for testing this extensively and to the authors of https://rr-project.org for making it easy to diagnose and fix any failures that were found during the testing.	3 years ago
Marko Mäkelä	b29a8118dd	Cleanup: Remove fil_space_t::magic_n	3 years ago
Marko Mäkelä	5294695ebd	Clean up mtr_t mtr_t::is_empty(): Replaces mtr_t::get_log() and mtr_t::get_memo(). mtr_t::get_log_size(): Replaces mtr_t::get_log(). mtr_t::print(): Remove, unused function. ReleaseBlocks::ReleaseBlocks(): Remove an unused parameter.	3 years ago
Marko Mäkelä	c009ce7dd0	MDEV-27094 Debug builds include useless InnoDB "disabled" options This is a backport of commit `4489a89c71` in order to remove the test innodb.redo_log_during_checkpoint that would cause trouble in the DBUG subsystem invoked by safe_mutex_lock() via log_checkpoint(). Before commit `7cffb5f6e8` these mutexes were of different type. The following options were introduced in commit `2e814d4702` (mariadb-10.2.2) and have little use: innodb_disable_resize_buffer_pool_debug had no effect even in MariaDB 10.2.2 or MySQL 5.7.9. It was introduced in mysql/mysql-server@5c4094cf4971eebab89da4ee4ae92c71f69cd524 to work around a problem that was fixed in mysql/mysql-server@2957ae4f990bf3aed25822b0ce15d3ccad0b54b6 (but the parameter was not removed). innodb_page_cleaner_disabled_debug and innodb_master_thread_disabled_debug are only used by the test innodb.redo_log_during_checkpoint that will be removed as part of this commit. innodb_dict_stats_disabled_debug is only used by that test, and it is redundant because one could simply use innodb_stats_persistent=OFF or the STATS_PERSISTENT=0 attribute of the table in the test to achieve the same effect.	4 years ago
Marko Mäkelä	6948abb94c	Cleanup: Remove fil_names_write_bogus Ever since commit `685d958e38` some Perl code in the test mariabackup.huge_lsn is writing names of non-existing files to the InnoDB redo log and testing the recovery. We do not need any debug instrumentation to duplicate that test.	4 years ago
Marko Mäkelä	fdec842fd7	MDEV-28371 Assertion fold == id.fold() failed in buf_flush_check_neighbor() Due to 32-bit arithmetics, SRV_TMP_SPACE_ID page number 0x200002 would be folded to 0, which is incompatible with the assumption that was made in commit `7cffb5f6e8` (MDEV-23399). page_id_t::fold(): Compute in the native word width instead of uint32_t. On 64-bit platforms, an alternative would be to return the 64-bit m_id directly, but that was measured to cause a performance regression. fil_space_t::open(): Invoke fil_node_t::find_metadata() when the tablespace is being created. In this way, we will actually detect that the temporary tablespace resides on SSD. (During database creation, also the system tablespace will correctly be detected as residing on SSD.)	4 years ago
Marko Mäkelä	a9adfc0f68	MDEV-9948 Failing assertion: new_state->key_version != ENCRYPTION_KEY_VERSION_INVALID in fil0crypt.cc The test encryption.innodb-redo-nokeys did not actually test recovery without valid keys, because due to the setting innodb_encrypt_tables, InnoDB refused to start up at all, without even attempting any crash recovery. fil_ibd_load(): If the encryption key is not available, refuse to load the file.	4 years ago

1 2 3 4 5 ...

719 Commits (3dd7b0a80cb674d0773fd0d25ceb8a5183798597)