mariadb

Commit Graph

Author	SHA1	Message	Date
Marko Mäkelä	312569e2fd	MDEV-21132 Remove buf_page_t::newest_modification At each mini-transaction commit, the log sequence number of the mini-transaction must be written to each modified page, so that it will be available in the FIL_PAGE_LSN field when the page is being read in crash recovery. InnoDB was unnecessarily allocating redundant storage for the field, in buf_page_t::newest_modification. Let us access FIL_PAGE_LSN directly. Furthermore, on ALTER TABLE...IMPORT TABLESPACE, let us write 0 to FIL_PAGE_LSN instead of using log_sys.lsn. buf_flush_init_for_writing(), buf_flush_update_zip_checksum(), fil_encrypt_buf_for_full_crc32(), fil_encrypt_buf(), fil_space_encrypt(): Remove the parameter lsn. buf_page_get_newest_modification(): Merge with the only caller. buf_tmp_reserve_compression_buf(), buf_tmp_page_encrypt(), buf_page_encrypt(): Define static in the same compilation unit with the only caller. PageConverter::m_current_lsn: Remove. Write 0 to FIL_PAGE_LSN on ALTER TABLE...IMPORT TABLESPACE.	6 years ago
Marko Mäkelä	dc8380b65d	MDEV-14602: Cleanup recv_dblwr_t::find_page() Avoid creating std::vector, and use single instead of double traversal.	6 years ago
Marko Mäkelä	b5ef7ffa59	Use uint16_t for FIL_PAGE_TYPE Since commit `5d596064d6` fil_page_type_is_index() expects uint16_t, not ulint.	6 years ago
Marko Mäkelä	51fb39bf17	MDEV-19586: Restore pointer indirection for recv_t::data Something was wrong with the removal of pointer indirection, because on 32-bit Windows we got crash recovery failures. Curiously, WITH_ASAN of a 32-bit debug build did not notice anything. This reverts a part of commit `b7fc2c899e`. We have a 2MiB recv_sys.buf for the initial buffering. The minimum size of log_sys.buf would be 16MiB, and that buffer should be practically unused during recovery. If the buffer pool size is measured in gigabytes, it would indeed make sense to use the buffer pool for the recovered log records, perhaps after we have run out of log_sys.buf. FIXME: allow the entire buf_block_t::frame to be used for buffered log records, and remove MEM_HEAP_FOR_RECV_SYS. For example, use buf_page_t::list or buf_page_t::LRU for keeping track of memory that was allocated for recovery? Most members of buf_block_t (except buf_block_t::frame) are unused when block->page.state == BUF_BLOCK_MEMORY.	6 years ago
Marko Mäkelä	541b00a3ba	MDEV-19586: Clean up recv_t a little recv_t, recv_t::data_t: Define constructors that copy the log records Ideally, we should remove recv_sys.buf, RECV_DATA_BLOCK_SIZE, and recv_sys_justify_left_parsing_buf(), and let the recv_sys.pages point directly to a buffer of parsed redo log records. The RECV_PARSING_BUF_SIZE (size of recv_sys.buf) is only 2MiB, while the minimum innodb_log_buffer_size (size of log_sys.buf) is 16MiB, and log_sys.buf is unused during redo log apply!	6 years ago
Marko Mäkelä	64a02e4fa2	MDEV-19586: Add const qualifiers Except for fil_name_process(), which invokes os_normalize_path(), the redo log record parser will not modify the redo log records. Add const qualifiers accordingly.	6 years ago
Sergei Golubchik	3ef01d1118	compilation failure, gcc 8.3.0 storage/innobase/log/log0recv.cc\|1760 col 35 error\| 'void* memcpy(void, const void, size_t)' writing to an object of type 'struct recv_t' with no trivial copy-assignment [-Werror=class-memaccess]	6 years ago
Marko Mäkelä	e5fed3b94d	MDEV-19586: Avoid std::map::emplace() GCC 4.7 only knows about std::map::insert(), not emplace(). Also, reformat the function in the common style.	6 years ago
Marko Mäkelä	b7fc2c899e	Refactor recv_sys_t::recs_t into page_recv_t page_recv_t: Replaces recv_sys_t::recs_t. page_recv_t::state is not private, even though some accessors exist. page_recv_t::log: A singly-linked list of log_rec_t* with STL decoration and the custom operations trim() and append(). The list members are private. recv_t::data_t: Replaces recv_data_t. recv_t::data: Remove the pointer indirection for the first log chunk, and copy the first chunk directly after the record. Adjust the definition of RECV_DATA_BLOCK_SIZE accordingly.	6 years ago
Marko Mäkelä	2aa1f77ef1	MDEV-19586: Rename recv_sys.empty() to recv_sys.clear() In the collections of Standard Template Library, empty() is a predicate and clear() empties a collection. Let us rename recv_sys.empty() to recv_sys.clear() to avoid confusion.	6 years ago
Marko Mäkelä	0ccfdc8eff	Remove InnoDB wrappers of <string.h> functions ut_strcmp(), ut_strcpy(), ut_strlen(), ut_memcpy(), ut_memcmp(), ut_memmove(): Remove. Invoke the standard library functions directly.	6 years ago
Eugene Kosov	7440e61a64	MDEV-18115: Remove the only async write (of redo log) TODO: do not use fil_* functions for redo log files. log_t::checkpoint_lock: remove this lock which was used to wait for async I/O completion. checkpoint_lock_key checkpoint_lock: remove now unneeded globals log_write_checkpoint_info(): remove sync argument because all checkpoint writes are synchronous now log_write_checkpoint_info(): remove sync argument log_group_checkpoint(): merge with the only caller log_complete_checkpoint(): merge with the only caller log_t::complete_checkpoint(): remove by merging with the only caller.	6 years ago
Marko Mäkelä	d4e1fa39f0	Use C++11 "range for" in crash recovery code	6 years ago
Marko Mäkelä	d67ea8151f	Rename LOG_HEADER_FORMAT_ to log_t::FORMAT_	6 years ago
Marko Mäkelä	b42294bc64	MDEV-19514 Defer change buffer merge until pages are requested We will remove the InnoDB background operation of merging buffered changes to secondary index leaf pages. Changes will only be merged as a result of an operation that accesses a secondary index leaf page, such as a SQL statement that performs a lookup via that index, or is modifying the index. Also ROLLBACK and some background operations, such as purging the history of committed transactions, or computing index cardinality statistics, can cause change buffer merge. Encryption key rotation will not perform change buffer merge. The motivation of this change is to simplify the I/O logic and to allow crash recovery to happen in the background (MDEV-14481). We also hope that this will reduce the number of "mystery" crashes due to corrupted data. Because change buffer merge will typically take place as a result of executing SQL statements, there should be a clearer connection between the crash and the SQL statements that were executed when the server crashed. In many cases, a slight performance improvement was observed. This is joint work with Thirunarayanan Balathandayuthapani and was tested by Axel Schwenke and Matthias Leich. The InnoDB monitor counter innodb_ibuf_merge_usec will be removed. On slow shutdown (innodb_fast_shutdown=0), we will continue to merge all buffered changes (and purge all undo log history). Two InnoDB configuration parameters will be changed as follows: innodb_disable_background_merge: Removed. This parameter existed only in debug builds. All change buffer merges will use synchronous reads. innodb_force_recovery will be changed as follows: * innodb_force_recovery=4 will be the same as innodb_force_recovery=3 (the change buffer merge cannot be disabled; it can only happen as a result of an operation that accesses a secondary index leaf page). The option used to be capable of corrupting secondary index leaf pages. Now that capability is removed, and innodb_force_recovery=4 becomes 'safe'. * innodb_force_recovery=5 (which essentially hard-wires SET GLOBAL TRANSACTION ISOLATION LEVEL READ UNCOMMITTED) becomes safe to use. Bogus data can be returned to SQL, but persistent InnoDB data files will not be corrupted further. * innodb_force_recovery=6 (ignore the redo log files) will be the only option that can potentially cause persistent corruption of InnoDB data files. Code changes: buf_page_t::ibuf_exist: New flag, to indicate whether buffered changes exist for a buffer pool page. Pages with pending changes can be returned by buf_page_get_gen(). Previously, the changes were always merged inside buf_page_get_gen() if needed. ibuf_page_exists(const buf_page_t&): Check if a buffered changes exist for an X-latched or read-fixed page. buf_page_get_gen(): Add the parameter allow_ibuf_merge=false. All callers that know that they may be accessing a secondary index leaf page must pass this parameter as allow_ibuf_merge=true, unless it does not matter for that caller whether all buffered changes have been applied. Assert that whenever allow_ibuf_merge holds, the page actually is a leaf page. Attempt change buffer merge only to secondary B-tree index leaf pages. btr_block_get(): Add parameter 'bool merge'. All callers of btr_block_get() should know whether the page could be a secondary index leaf page. If it is not, we should avoid consulting the change buffer bitmap to even consider a merge. This is the main interface to requesting index pages from the buffer pool. ibuf_merge_or_delete_for_page(), recv_recover_page(): Replace buf_page_get_known_nowait() with much simpler logic, because it is now guaranteed that that the block is x-latched or read-fixed. mlog_init_t::mark_ibuf_exist(): Renamed from mlog_init_t::ibuf_merge(). On crash recovery, we will no longer merge any buffered changes for the pages that we read into the buffer pool during the last batch of applying log records. buf_page_get_gen_known_nowait(), BUF_MAKE_YOUNG, BUF_KEEP_OLD: Remove. btr_search_guess_on_hash(): Merge buf_page_get_gen_known_nowait() to its only remaining caller. buf_page_make_young_if_needed(): Define as an inline function. Add the parameter buf_pool. buf_page_peek_if_young(), buf_page_peek_if_too_old(): Add the parameter buf_pool. fil_space_validate_for_mtr_commit(): Remove a bogus comment about background merge of the change buffer. btr_cur_open_at_rnd_pos_func(), btr_cur_search_to_nth_level_func(), btr_cur_open_at_index_side_func(): Use narrower data types and scopes. ibuf_read_merge_pages(): Replaces buf_read_ibuf_merge_pages(). Merge the change buffer by invoking buf_page_get_gen().	6 years ago
Thirunarayanan Balathandayuthapani	c76873f23d	MDEV-20688 Recovery crashes after unnecessarily reading a corrupted page The test encryption.innodb-redo-badkey was accidentally disabled until commit `23657a2101` enabled it recently. Once it was enabled, it started failing randomly. recv_recover_corrupt_page(): Do not assume that any redo log exists for the page. A page may be unnecessarily read by read-ahead. When noting the corruption, reset recv_addr->state to RECV_PROCESSED, so that even if the same page is re-read again, we will only decrement recv_sys->n_addrs once.	6 years ago
Marko Mäkelä	0c7c61019d	Remove the wrappers ut_time(), ut_difftime(), ib_time_t	6 years ago
Eugene Kosov	53a3594b90	MDEV-19471 Add ASAN-poisoned redzones for mem_heap_t Store REDZONE_SIZE poined bytes before every allocated chunk of memory	6 years ago
Marko Mäkelä	d46db415ce	MDEV-19738: Skip doublewrite on MLOG_ZIP_PAGE_COMPRESS page_zip_compress(), page_zip_compress_write_log(), page_zip_copy_recs(): Replace the parameters page,page_zip with block, and set buf_page_t::init_on_flush on success if innodb_log_optimize_ddl=OFF. page_zip_parse_compress_no_data(): Merge with the only caller recv_parse_or_apply_log_rec_body().	6 years ago
Marko Mäkelä	38018f48b6	MDEV-19586: Remove unnecessary iteration recv_apply_hashed_log_recs(): Refer directly to recs.last->end_lsn instead of iterating the entire list starting from recs.log.	6 years ago
Thirunarayanan Balathandayuthapani	b2f76bac03	MDEV-16866 InnoDB fails to start upon crash recovery with "[ERROR] InnoDB: Redo log crypto: failed to decrypt log block" - Post-push fix to change the copyright of both xtradb and innodb file.	6 years ago
Thirunarayanan Balathandayuthapani	c5fe1b8fc1	MDEV-16866 InnoDB fails to start upon crash recovery with "[ERROR] InnoDB: Redo log crypto: failed to decrypt log block" - If InnoDB encounters garbage or incomplete written log block during recovery then don't throw the error. Treat it as end of the log. - This kind of incomplete or empty block can be result of killing InnoDB when writing the redo log.	6 years ago
Marko Mäkelä	177a571e01	MDEV-19586 Replace recv_sys_t::addr_hash with a std::map InnoDB crash recovery buffers redo log records in a hash table. The function recv_read_in_area() would pick a random hash bucket and then try to submit read requests for a few nearby pages. Let us replace the recv_sys.addr_hash with a std::map, which will automatically be iterated in sorted order. recv_sys_t::pages: Replaces recv_sys_t::addr_hash, recv_sys_t::n_addrs. recv_sys_t::recs: Replaces most of recv_addr_t. recv_t: Encapsulate a raw singly-linked list of records. This reduces overhead compared to std::forward_list. Storage and cache overhead, because the next-element pointer also points to the data payload. Processing overhead, because recv_sys_t::recs_t::last will point to the last record, so that recv_sys_t::add() can append straight to the end of the list. RECV_PROCESSED, RECV_DISCARDED: Remove. When a page is fully processed, it will be deleted from recv_sys.pages. recv_sys_t::trim(): Replaces recv_addr_trim(). recv_sys_t::add(): Use page_id_t for identifying pages. recv_fold(), recv_hash(), recv_get_fil_addr_struct(): Remove. recv_read_in_area(): Simplify the iteration.	6 years ago
Thirunarayanan Balathandayuthapani	b4287ec386	MDEV-19541 InnoDB crashes when trying to recover a corrupted page - Use corrupt page id instead of whole block after releasing it from LRU list.	6 years ago
Thirunarayanan Balathandayuthapani	79b46ab2a6	MDEV-19541 InnoDB crashes when trying to recover a corrupted page - Don't apply redo log for the corrupted page when innodb_force_recovery > 0. - Allow the table to be dropped when index root page is corrupted when innodb_force_recovery > 0.	7 years ago
Marko Mäkelä	5d2619b693	MDEV-19584 Allocate recv_sys statically There is only one InnoDB crash recovery subsystem. Allocating recv_sys statically removes one level of pointer indirection and makes code more readable, and removes the awkward initialization of recv_sys->dblwr. recv_sys_t::create(): Replaces recv_sys_init(). recv_sys_t::debug_free(): Replaces recv_sys_debug_free(). recv_sys_t::close(): Replaces recv_sys_close(). recv_sys_t::add(): Replaces recv_add_to_hash_table(). recv_sys_t::empty(): Replaces recv_sys_empty_hash().	7 years ago
Marko Mäkelä	893472d005	MDEV-19570 Deprecate and ignore innodb_undo_logs, remove innodb_rollback_segments The option innodb_rollback_segments was deprecated already in MariaDB Server 10.0. Its misleadingly named replacement innodb_undo_logs is of very limited use. It makes sense to always create and use the maximum number of rollback segments. Let us remove the deprecated parameter innodb_rollback_segments and deprecate&ignore the parameter innodb_undo_logs (to be removed in a later major release). This work involves some cleanup of InnoDB startup. Similar to other write operations, DROP TABLE will no longer be allowed if innodb_force_recovery is set to a value larger than 3.	7 years ago
Marko Mäkelä	47cede646b	MDEV-19543 Deprecate and ignore innodb_log_checksums The parameter innodb_log_checksums that was introduced in MariaDB 10.2.2 via mysql/mysql-server@af0acedd885eb7103e319f79d25fda7386ef1506 does not make much sense. The original motivation of introducing this parameter (initially called innodb_log_checksum_algorithm in mysql/mysql-server@22ba38218e1d76c24f69b5a5595ad3bf5933acb0) was that the InnoDB redo log used the slow and insecure innodb algorithm. With hardware or SIMD accelerated CRC-32C, there should be no reason to allow checksums to be disabled on the redo log. The parameter innodb_encrypt_log already implies innodb_log_checksums=ON. Let us deprecate the parameter innodb_log_checksums and always compute redo log checksums, even if innodb_log_checksums=OFF is specified. An upgrade from MariaDB 10.2.2 or later will only be possible after using the default value innodb_log_checksums=ON. If the non-default value innodb_log_checksums=OFF was in effect when the server was shut down, a log block checksum mismatch will be reported and the upgraded server will fail to start up.	7 years ago
Vicențiu Ciorbaru	c0ac0b8860	Update FSF address	7 years ago
Marko Mäkelä	3db94d2403	MDEV-19346: Remove dummy InnoDB log checkpoints log_checkpoint(), log_make_checkpoint_at(): Remove the parameter write_always. It seems that the primary purpose of this parameter was to ensure in the function recv_reset_logs() that both checkpoint header pages will be overwritten, when the function is called from the never-enabled function recv_recovery_from_archive_start(). create_log_files(): Merge recv_reset_logs() to its only caller. Debug instrumentation: Prefer to flush the redo log, instead of triggering a redo log checkpoint. page_header_set_field(): Disable a debug assertion that will always fail due to MDEV-19344, now that we no longer initiate a redo log checkpoint before an injected crash. In recv_reset_logs() there used to be two calls to log_make_checkpoint_at(). The apparent purpose of this was to ensure that both InnoDB redo log checkpoint header pages will be initialized or overwritten. The second call was removed (without any explanation) in MySQL 5.6.3: mysql/mysql-server@4ca37968da54ddc6b3b6628f41428ddba1c79bb8 In MySQL 5.6.8 WL#6494, starting with mysql/mysql-server@00a0ba8ad92569fcf08212b3b8cf046dc8a0ce10 the function recv_reset_logs() was not only invoked during InnoDB data file initialization, but also during a regular startup when the redo log is being resized. mysql/mysql-server@45e91679832219e2593c77185342f11f85232b58 in MySQL 5.7.2 removed the UNIV_LOG_ARCHIVE code, but still did not remove the parameter write_always.	7 years ago
Marko Mäkelä	092602ac9b	MDEV-14130 InnoDB messages should not refer to the MySQL 5.7 manual In InnoDB error messages, replace the hyperlink URLs to point to the MariaDB knowledge base.	7 years ago
Marko Mäkelä	169c00994b	MDEV-12699 Improve crash recovery of corrupted data pages InnoDB crash recovery used to read every data page for which redo log exists. This is unnecessary for those pages that are initialized by the redo log. If a newly created page is corrupted, recovery could unnecessarily fail. It would suffice to reinitialize the page based on the redo log records. To add insult to injury, InnoDB crash recovery could hang if it encountered a corrupted page. We will fix also that problem. InnoDB would normally refuse to start up if it encounters a corrupted page on recovery, but that can be overridden by setting innodb_force_recovery=1. Data pages are completely initialized by the records MLOG_INIT_FILE_PAGE2 and MLOG_ZIP_PAGE_COMPRESS. MariaDB 10.4 additionally recognizes MLOG_INIT_FREE_PAGE, which notifies that a page has been freed and its contents can be discarded (filled with zeroes). The record MLOG_INDEX_LOAD notifies that redo logging has been re-enabled after being disabled. We can avoid loading the page if all buffered redo log records predate the MLOG_INDEX_LOAD record. For the internal tables of FULLTEXT INDEX, no MLOG_INDEX_LOAD records were written before commit `aa3f7a107c`. Hence, we will skip these optimizations for tables whose name starts with FTS_. This is joint work with Thirunarayanan Balathandayuthapani. fil_space_t::enable_lsn, file_name_t::enable_lsn: The LSN of the latest recovered MLOG_INDEX_LOAD record for a tablespace. mlog_init: Page initialization operations discovered during redo log scanning. FIXME: This really belongs in recv_sys->addr_hash, and should be removed in MDEV-19176. recv_addr_state: Add the new state RECV_WILL_NOT_READ to indicate that according to mlog_init, the page will be initialized based on redo log record contents. recv_add_to_hash_table(): Set the RECV_WILL_NOT_READ state if appropriate. For now, we do not treat MLOG_ZIP_PAGE_COMPRESS as page initialization. This works around bugs in the crash recovery of ROW_FORMAT=COMPRESSED tables. recv_mark_log_index_load(): Process a MLOG_INDEX_LOAD record by resetting the state to RECV_NOT_PROCESSED and by updating the fil_name_t::enable_lsn. recv_init_crash_recovery_spaces(): Copy fil_name_t::enable_lsn to fil_space_t::enable_lsn. recv_recover_page(): Add the parameter init_lsn, to ignore any log records that precede the page initialization. Add DBUG output about skipped operations. buf_page_create(): Initialize FIL_PAGE_LSN, so that recv_recover_page() will not wrongly skip applying the page-initialization record due to the field containing some newer LSN as a leftover from a different page. Do not invoke ibuf_merge_or_delete_for_page() during crash recovery. recv_apply_hashed_log_recs(): Remove some unnecessary lookups. Note if a corrupted page was found during recovery. After invoking buf_page_create(), do invoke ibuf_merge_or_delete_for_page() via mlog_init.ibuf_merge() in the last recovery batch. ibuf_merge_or_delete_for_page(): Relax a debug assertion. innobase_start_or_create_for_mysql(): Abort startup if a corrupted page was found during recovery. Corrupted pages will not be flagged if innodb_force_recovery is set. However, the recv_sys->found_corrupt_fs flag can be set regardless of innodb_force_recovery if file names are found to be incorrect (for example, multiple files with the same tablespace ID).	7 years ago
Vlad Lesin	05b84b2568	MDEV-14192: Add debug assertions	7 years ago
Marko Mäkelä	1d30b7b1d2	MDEV-12699 preparation: Clean up recv_sys The recv_sys data structures are accessed not only from the thread that executes InnoDB plugin initialization, but also from the InnoDB I/O threads, which can invoke recv_recover_page(). Assert that sufficient concurrency control is in place. Some code was accessing recv_sys data structures without holding recv_sys->mutex. recv_recover_page(bpage): Refactor the call from buf_page_io_complete() into a separate function that performs necessary steps. The main thread was unnecessarily releasing and reacquiring recv_sys->mutex. recv_recover_page(block,mtr,recv_addr): Pass more parameters from the caller. Avoid redundant lookups and computations. Eliminate some redundant variables. recv_get_fil_addr_struct(): Assert that recv_sys->mutex is being held. That was not always the case! recv_scan_log_recs(): Acquire recv_sys->mutex for the whole duration of the function. (While we are scanning and buffering redo log records, no pages can be read in.) recv_read_in_area(): Properly protect access with recv_sys->mutex. recv_apply_hashed_log_recs(): Check recv_addr->state only once, and continuously hold recv_sys->mutex. The mutex will be released and reacquired inside recv_recover_page() and recv_read_in_area(), allowing concurrent processing by buf_page_io_complete() in I/O threads.	7 years ago
Marko Mäkelä	aa3f7a107c	MDEV-12699 preparation: Write MLOG_INDEX_LOAD for FTS_ tables The record MLOG_INDEX_LOAD is supposed to be written to indicate that some page modifications bypassed redo logging, and that redo logging is now re-enabled. It was not written for fulltext indexes during ALTER TABLE. row_merge_write_redo(): Declare globally. Assert that the index is neither a spatial nor fulltext index. recv_mlog_index_load(): Observe a MLOG_INDEX_LOAD operation. recv_parse_log_recs(): Handle MLOG_INDEX_LOAD also in multi-record mini-transactions. Because of this omission, we should keep writing MLOG_INDEX_LOAD in single-record mini-transactions, because older versions of Mariabackup would fail. row_fts_merge_insert(): Write MLOG_INDEX_LOAD for the auxiliary tables of fulltext indexes.	7 years ago
Marko Mäkelä	45d338dca8	MDEV-12699 preparation: Initialize the entire page on MLOG_ZIP_PAGE_COMPRESS The record MLOG_ZIP_PAGE_COMPRESS is similar to MLOG_INIT_FILE_PAGE2 that it contains all the information needed to initialize the page. Like for the other record, do initialize the entire page on recovery.	7 years ago
Marko Mäkelä	1b95118c5f	buf_page_get_gen(): Allow BUF_GET_IF_IN_POOL with a dummy page_size The page_size argument to buf_page_get_gen() only matters when the page is going to be loaded into the buffer pool. Allow callers to pass a dummy parameter when using BUF_GET_IF_IN_POOL (which would return NULL if the block is not in the buffer pool).	7 years ago
Marko Mäkelä	56df18be65	Clean up the parsing of MLOG_INIT_FILE_PAGE2 fsp_apply_init_file_page(): Renamed from fsp_init_file_page_low(). fsp_parse_init_file_page(): Remove. The redo log record has no parameters.	7 years ago
Marko Mäkelä	71f9552fd8	recv_recovery_is_on(): Add UNIV_UNLIKELY Normally, InnoDB is not in the process of executing crash recovery. Provide a hint to the compiler that the recovery-related code paths are rarely executed.	7 years ago
Marko Mäkelä	8650848ec3	MDEV-19128 fil_name_parse() for MLOG_FILE_ is not portable On Microsoft Windows, InnoDB writes the path separator \ to the redo log file, while on all other platforms, / is being used. fil_name_parse(): Normalize the parsed path separators to the native format. This allows backups or data sets to be portable between Windows and other systems.	7 years ago
Marko Mäkelä	7225bef727	Fix crash recovery with small buffer pool Occasionally, the test innodb_gis.rtree_recovery,4k runs out of buffer pool during crash recovery. This test might have started failing in connection with MDEV-18726. Work around it by reserving a little more overhead for the recovery hash tables. FIXME: Propagate the "out of memory" error to recv_add_to_hash_table() and transition to multi-batch recovery with a finer granularity.	7 years ago
Marko Mäkelä	bef947b4c9	MDEV-18902 Uninitialized variable in recv_parse_log_recs() recv_parse_log_recs(): Do not compare type if ptr==end_ptr (we have reached the end of the redo log parsing buffer), because it will not have been correctly initialized in that case.	7 years ago
Thirunarayanan Balathandayuthapani	c0f47a4a58	MDEV-12026: Implement innodb_checksum_algorithm=full_crc32 MariaDB data-at-rest encryption (innodb_encrypt_tables) had repurposed the same unused data field that was repurposed in MySQL 5.7 (and MariaDB 10.2) for the Split Sequence Number (SSN) field of SPATIAL INDEX. Because of this, MariaDB was unable to support encryption on SPATIAL INDEX pages. Furthermore, InnoDB page checksums skipped some bytes, and there are multiple variations and checksum algorithms. By default, InnoDB accepts all variations of all algorithms that ever existed. This unnecessarily weakens the page checksums. We hereby introduce two more innodb_checksum_algorithm variants (full_crc32, strict_full_crc32) that are special in a way: When either setting is active, newly created data files will carry a flag (fil_space_t::full_crc32()) that indicates that all pages of the file will use a full CRC-32C checksum over the entire page contents (excluding the bytes where the checksum is stored, at the very end of the page). Such files will always use that checksum, no matter what the parameter innodb_checksum_algorithm is assigned to. For old files, the old checksum algorithms will continue to be used. The value strict_full_crc32 will be equivalent to strict_crc32 and the value full_crc32 will be equivalent to crc32. ROW_FORMAT=COMPRESSED tables will only use the old format. These tables do not support new features, such as larger innodb_page_size or instant ADD/DROP COLUMN. They may be deprecated in the future. We do not want an unnecessary file format change for them. The new full_crc32() format also cleans up the MariaDB tablespace flags. We will reserve flags to store the page_compressed compression algorithm, and to store the compressed payload length, so that checksum can be computed over the compressed (and possibly encrypted) stream and can be validated without decrypting or decompressing the page. In the full_crc32 format, there no longer are separate before-encryption and after-encryption checksums for pages. The single checksum is computed on the page contents that is written to the file. We do not make the new algorithm the default for two reasons. First, MariaDB 10.4.2 was a beta release, and the default values of parameters should not change after beta. Second, we did not yet implement the full_crc32 format for page_compressed pages. This will be fixed in MDEV-18644. This is joint work with Marko Mäkelä.	7 years ago
Marko Mäkelä	ca76fc4a3a	MDEV-18611: mariabackup silently ended during xtrabackup_copy_logfile() log_group_read_log_seg(): Always return false when returning before reading end_lsn. xtrabackup_copy_logfile(): On error, indicate whether a corrupt log record was encountered. Only xtrabackup_copy_logfile() in Mariabackup cared about the return value of the function. InnoDB crash recovery was not affected by this bug.	7 years ago
Vladislav Vaintroub	568dd5293c	MDEV-18592 Warning if temp-file (with "#sql-" prefix) is missing during prepare	7 years ago
Marko Mäkelä	3c305d3f19	MDEV-15528: Introduce MLOG_INIT_FREE_PAGE This is only a placeholder that allows an implementation later during the development of MariaDB, so that downgrade to an earlier version (with this code) will be possible. We want to be able to zero out freed pages to reduce write amplification, and to scrub old data. Zeroing out the pages is optional, not mandatory for correctness. After all, the MLOG_INIT_FREE_PAGE record can only be emitted for pages that are marked free in the allocation bitmap page.	7 years ago
Marko Mäkelä	ea25bdc135	Do not write MLOG_IBUF_BITMAP_INIT Use ibuf_bitmap_page_init() only during recovery. fsp_fill_free_list(): Initialize the FIL_PAGE_TYPE using MLOG_2BYTES. The page contents will already have been zeroed out by MLOG_INIT_FILE_PAGE2. ibuf_bitmap_init_apply(): Replaces ibuf_parse_bitmap_init().	7 years ago
Marko Mäkelä	0a1c3477bf	MDEV-18493 Remove page_size_t MySQL 5.7 introduced the class page_size_t and increased the size of buffer pool page descriptors by introducing this object to them. Maybe the intention of this exercise was to prepare for a future where the buffer pool could accommodate multiple page sizes. But that future never arrived, not even in MySQL 8.0. It is much easier to manage a pool of a single page size, and typically all storage devices of an InnoDB instance benefit from using the same page size. Let us remove page_size_t from MariaDB Server. This will make it easier to remove support for ROW_FORMAT=COMPRESSED (or make it a compile-time option) in the future, just by removing various occurrences of zip_size.	7 years ago
Thirunarayanan Balathandayuthapani	7c7161a1bd	MDEV-18194 Incremental prepare tries to access page which is out of tablespace bounds Problem: ======= Mariabackup incremental prepare creates new tablespace when it encounter new tablespace. It sets the intial size as FIL_IBD_FILE_INITIAL_SIZE (4). But while applying redo log, it tries to access 5th page and then it leads to out of tablespace error. Fix: === While parsing the redo log record, track FSP_SIZE in recv_spaces for the respective space id. Assign the recv_size for the tablespace when it is loaded. Extend the tablespace depends on recv_size while applying the redo log record.	7 years ago
Marko Mäkelä	8655592ae5	recv_log_recover_10_4(): Fix a trivial bug	7 years ago

1 2 3 4 5 ...

292 Commits (312569e2fd79a6582161900e726847ad688ee3b0)