mariadb

Commit Graph

Author	SHA1	Message	Date
Marko Mäkelä	7f67ef1485	MDEV-16115 Hang after reducing innodb_encryption_threads The test encryption.create_or_replace would occasionally fail, because some fil_space_t::n_pending_ops would never be decremented. fil_crypt_find_space_to_rotate(): If rotate_thread_t::should_shutdown() holds due to innodb_encryption_threads having been reduced, do release the reference. fil_space_remove_from_keyrotation(), fil_space_next(): Declare the functions static, simplify a little, and define in the same compilation unit with the only caller, fil_crypt_find_space_to_rotate(). fil_crypt_key_mutex: Remove (unused).	5 years ago
Marko Mäkelä	57ec42bc32	MDEV-23190 InnoDB data file extension is not crash-safe When InnoDB is extending a data file, it is updating the FSP_SIZE field in the first page of the data file. In commit `8451e09073` (MDEV-11556) we removed a work-around for this bug and made recovery stricter, by making it track changes to FSP_SIZE via redo log records, and extend the data files before any changes are being applied to them. It turns out that the function fsp_fill_free_list() is not crash-safe with respect to this when it is initializing the change buffer bitmap page (page 1, or generally, N*innodb_page_size+1). It uses a separate mini-transaction that is committed (and will be written to the redo log file) before the mini-transaction that actually extended the data file. Hence, recovery can observe a reference to a page that is beyond the current end of the data file. fsp_fill_free_list(): Initialize the change buffer bitmap page in the same mini-transaction. The rest of the changes are fixing a bug that the use of the separate mini-transaction was attempting to work around. Namely, we must ensure that no other thread will access the change buffer bitmap page before our mini-transaction has been committed and all page latches have been released. That is, for read-ahead as well as neighbour flushing, we must avoid accessing pages that might not yet be durably part of the tablespace. fil_space_t::committed_size: The size of the tablespace as persisted by mtr_commit(). fil_space_t::max_page_number_for_io(): Limit the highest page number for I/O batches to committed_size. MTR_MEMO_SPACE_X_LOCK: Replaces MTR_MEMO_X_LOCK for fil_space_t::latch. mtr_x_space_lock(): Replaces mtr_x_lock() for fil_space_t::latch. mtr_memo_slot_release_func(): When releasing MTR_MEMO_SPACE_X_LOCK, copy space->size to space->committed_size. In this way, read-ahead or flushing will never be invoked on pages that do not yet exist according to FSP_SIZE.	5 years ago
Marko Mäkelä	98e2c17e9e	Cleanup: Remove fil_check_adress_in_tablespace()	5 years ago
Eugene Kosov	18d8f06f31	intrusive::list fixes namespace intrusive: removed split class into two: ilist<T> and sized_ilist<T> which has a size field. ilist<T> no more NULLify pointers to bring a slignly better performance. As a consequence, fil_space_t::is_in_unflushed_spaces and fil_space_t::is_in_rotation_list boolean members are needed now.	6 years ago
Marko Mäkelä	6960e9ed24	MDEV-21983: Crash on DROP/RENAME TABLE after DISCARD TABLESPACE fil_delete_tablespace(): Remove the unused parameter drop_ahi, and add the parameter if_exists=false. We want to suppress error messages if we know that the tablespace has been discarded. dict_table_rename_in_cache(): Pass the new parameter to fil_delete_tablespace(), that is, do not complain about missing tablespace if the tablespace has been discarded. row_make_new_pathname(): Declare as static. row_drop_table_for_mysql(): Tolerate !table->data_dir_path when the tablespace has been discarded. row_rename_table_for_mysql(): Skip part of the RENAME TABLE when fil_space_get_first_path() returns NULL.	6 years ago
Eugene Kosov	fd899b3bbd	Lets add another intrusive double linked list! Features: * STL-like interface * Fast modification: no branches on insertion or deletion * Fast iteration: one pointer dereference and one pointer comparison * Your class can be a part of several lists Modeled after std::list<T> but currently has fewer methods (not complete yet) For even more performance it's possible to customize list with templates so it won't have size counter variable or won't NULLify unlinked node. How existing lists differ? No existing lists support STL-like interface. I_List: * slower iteration (one more branch on iteration) * element can't be a part of two lists simultaneously I_P_List: * slower modification (branches, except for the fastest push_back() case) * slower iteration (one more branch on iteration) UT_LIST_BASE_NODE_T: * slower modification (branches) Three UT_LISTs were replaced: two in fil_system_t and one in dyn_buf_t.	6 years ago
Thirunarayanan Balathandayuthapani	88157247fc	MDEV-19509 InnoDB skips the tablespace in rotation list - If one of the encryption threads already started the initialization of the tablespace then don't remove the other uninitialized tablespace from the rotation list. - If there is a change in innodb_encrypt_tables then don't remove the processed tablespace from rotation list.	7 years ago
Vicențiu Ciorbaru	5543b75550	Update FSF Address * Update wrong zip-code	7 years ago
Vicențiu Ciorbaru	c0ac0b8860	Update FSF address	7 years ago
Marko Mäkelä	5b3f7c0c33	MDEV-18220: Remove some redundant data structures fts_state_t, fts_slot_t::state: Remove. Replaced by fts_slot_t::running and fts_slot_t::table_id as follows. FTS_STATE_SUSPENDED: Removed (unused). FTS_STATE_EMPTY: Removed. table_id=0 will denote empty slots. FTS_STATE_RUNNING: Equivalent to running=true. FTS_STATE_LOADED, FTS_STATE_DONE: Equivalent to running=false. fts_slot_t::table: Remove. Tables will be identified by table_id. After opening a table, we will check fil_table_accessible() before accessing the data. fts_optimize_new_table(), fts_optimize_del_table(), fts_optimize_how_many(), fts_is_sync_needed(): Remove the parameter tables, and use the static variable fts_slots (which was introduced in MariaDB 10.2) instead.	7 years ago
Thirunarayanan Balathandayuthapani	ada1074bb1	MDEV-14398 innodb_encryption_rotate_key_age=0 causes innodb_encrypt_tables to be ignored The statement SET GLOBAL innodb_encryption_rotate_key_age=0; would have the unwanted side effect that ENCRYPTION=DEFAULT tablespaces would no longer be encrypted or decrypted according to the setting of innodb_encrypt_tables. We implement a trigger, so that whenever one of the following is executed: SET GLOBAL innodb_encrypt_tables=OFF; SET GLOBAL innodb_encrypt_tables=ON; SET GLOBAL innodb_encrypt_tables=FORCE; all wrong-state ENCRYPTION=DEFAULT tablespaces will be added to fil_system_t::rotation_list, so that the encryption will be added or removed. Note: This will NOT happen automatically after a server restart. Before reading the first page of a data file, InnoDB cannot know the encryption status of the data file. The statement SET GLOBAL innodb_encrypt_tables will have the side effect that all not-yet-read InnoDB data files will be accessed in order to determine the encryption status. innodb_encrypt_tables_validate(): Stop disallowing SET GLOBAL innodb_encrypt_tables when innodb_encryption_rotate_key_age=0. This reverts part of commit `50eb40a2a8` that addressed MDEV-11738 and MDEV-11581. fil_system_t::read_page0(): Trigger a call to fil_node_t::read_page0(). Refactored from fil_space_get_space(). fil_crypt_rotation_list_fill(): If innodb_encryption_rotate_key_age=0, initialize fil_system->rotation_list. This is invoked both on SET GLOBAL innodb_encrypt_tables and on SET GLOBAL innodb_encryption_rotate_key_age=0. fil_space_set_crypt_data(): Remove. fil_parse_write_crypt_data(): Simplify the logic. This is joint work with Marko Mäkelä.	7 years ago
Marko Mäkelä	169c00994b	MDEV-12699 Improve crash recovery of corrupted data pages InnoDB crash recovery used to read every data page for which redo log exists. This is unnecessary for those pages that are initialized by the redo log. If a newly created page is corrupted, recovery could unnecessarily fail. It would suffice to reinitialize the page based on the redo log records. To add insult to injury, InnoDB crash recovery could hang if it encountered a corrupted page. We will fix also that problem. InnoDB would normally refuse to start up if it encounters a corrupted page on recovery, but that can be overridden by setting innodb_force_recovery=1. Data pages are completely initialized by the records MLOG_INIT_FILE_PAGE2 and MLOG_ZIP_PAGE_COMPRESS. MariaDB 10.4 additionally recognizes MLOG_INIT_FREE_PAGE, which notifies that a page has been freed and its contents can be discarded (filled with zeroes). The record MLOG_INDEX_LOAD notifies that redo logging has been re-enabled after being disabled. We can avoid loading the page if all buffered redo log records predate the MLOG_INDEX_LOAD record. For the internal tables of FULLTEXT INDEX, no MLOG_INDEX_LOAD records were written before commit `aa3f7a107c`. Hence, we will skip these optimizations for tables whose name starts with FTS_. This is joint work with Thirunarayanan Balathandayuthapani. fil_space_t::enable_lsn, file_name_t::enable_lsn: The LSN of the latest recovered MLOG_INDEX_LOAD record for a tablespace. mlog_init: Page initialization operations discovered during redo log scanning. FIXME: This really belongs in recv_sys->addr_hash, and should be removed in MDEV-19176. recv_addr_state: Add the new state RECV_WILL_NOT_READ to indicate that according to mlog_init, the page will be initialized based on redo log record contents. recv_add_to_hash_table(): Set the RECV_WILL_NOT_READ state if appropriate. For now, we do not treat MLOG_ZIP_PAGE_COMPRESS as page initialization. This works around bugs in the crash recovery of ROW_FORMAT=COMPRESSED tables. recv_mark_log_index_load(): Process a MLOG_INDEX_LOAD record by resetting the state to RECV_NOT_PROCESSED and by updating the fil_name_t::enable_lsn. recv_init_crash_recovery_spaces(): Copy fil_name_t::enable_lsn to fil_space_t::enable_lsn. recv_recover_page(): Add the parameter init_lsn, to ignore any log records that precede the page initialization. Add DBUG output about skipped operations. buf_page_create(): Initialize FIL_PAGE_LSN, so that recv_recover_page() will not wrongly skip applying the page-initialization record due to the field containing some newer LSN as a leftover from a different page. Do not invoke ibuf_merge_or_delete_for_page() during crash recovery. recv_apply_hashed_log_recs(): Remove some unnecessary lookups. Note if a corrupted page was found during recovery. After invoking buf_page_create(), do invoke ibuf_merge_or_delete_for_page() via mlog_init.ibuf_merge() in the last recovery batch. ibuf_merge_or_delete_for_page(): Relax a debug assertion. innobase_start_or_create_for_mysql(): Abort startup if a corrupted page was found during recovery. Corrupted pages will not be flagged if innodb_force_recovery is set. However, the recv_sys->found_corrupt_fs flag can be set regardless of innodb_force_recovery if file names are found to be incorrect (for example, multiple files with the same tablespace ID).	7 years ago
Marko Mäkelä	4b822111ef	MDEV-8139: Clean up the freeing of B-tree pages btr_page_free(): Renamed from btr_page_free_low(). If scrubbing is enabled, zero out the page with proper redo logging. Only pass ahi=true to fseg_free_page() if the page is actually indexed. fil_space_t::modify_check(): Renamed from fsp_space_modify_check(). fsp_init_file_page(): Define inline.	7 years ago
Marko Mäkelä	7f5849a809	MDEV-18309: Remove unused code	7 years ago
Marko Mäkelä	867617a976	MDEV-18309: InnoDB reports bogus errors about missing #sql-*.ibd on startup This is a follow-up to MDEV-18733. As part of that fix, we made dict_check_sys_tables() skip tables that would be dropped by row_mysql_drop_garbage_tables(). DICT_ERR_IGNORE_DROP: A new mode where the file should not be attempted to be opened. dict_load_tablespace(): Do not try to load the tablespace if DICT_ERR_IGNORE_DROP has been specified. row_mysql_drop_garbage_tables(): Pass the DICT_ERR_IGNORE_DROP mode. fil_space_for_table_exists_in_mem(): Remove a parameter. The only caller that passed print_error_if_does_not_exist=true was row_drop_single_table_tablespace().	7 years ago
Marko Mäkelä	71f9552fd8	recv_recovery_is_on(): Add UNIV_UNLIKELY Normally, InnoDB is not in the process of executing crash recovery. Provide a hint to the compiler that the recovery-related code paths are rarely executed.	7 years ago
Eugene Kosov	31d0727a10	MDEV-18235: Changes related to fsync() Remove fil_node_t::sync_event. I had a discussion with kernel fellows and they said it's safe to call fsync() simultaneously at least on VFS and ext4. So initially I wanted to disable check for recent Linux but than I realized code is buggy. Consider a case when one thread is inside fsync() and two others are waiting inside os_event. First thread after fsync() calls os_event_set() which is a broadcast! So two waiting threads will awake and may call fsync() at the same time. One fix is to add a notify_one() functionality to os_event but I decided to remove incorrect check completely. Note, it works for one waiting thread but not for more than one. IMO it's ok to avoid existing bugs but there is not too much sense in avoiding possible(!) bugs as this code does. fil_space_t::is_in_rotation_list(), fil_space_t::is_in_unflushed_spaces(): Replace redundant bool fields with member functions. fil_node_t::needs_flush: Replaces fil_node_t::modification_counter and fil_node_t::flush_counter. We need to know whether there _are_ some unflushed writes and we do not need to know _how many_ writes. fil_system_t::modification_counter: Remove as not needed. Even if we needed fil_node_t::modification_counter, every file could have its own counter that would be incremented on each write. fil_system_t::modification_counter is a global modification counter for all files. It was incremented on every write. But whether some file was flushed or not is an internal fil_node_t deal/state and this makes fil_system_t::modification_counter useless. Closes #1061	7 years ago
Marko Mäkelä	31d592ba7d	MDEV-18349 InnoDB file size changes are not safe when file system crashes When InnoDB is invoking posix_fallocate() to extend data files, it was missing a call to fsync() to update the file system metadata. If file system recovery is needed, the file size could be incorrect. When the setting innodb_flush_method=O_DIRECT_NO_FSYNC that was introduced in MariaDB 10.0.11 (and MySQL 5.6) is enabled, InnoDB would wrongly skip fsync() after extending files. Furthermore, the merge commit `d8b45b0c00` inadvertently removed XtraDB error checking for posix_fallocate() which this fix is restoring. fil_flush(): Add the parameter bool metadata=false to request that fil_buffering_disabled() be ignored. fil_extend_space_to_desired_size(): Invoke fil_flush() with the extra parameter. After successful posix_fallocate(), invoke os_file_flush(). Note: The bookkeeping for fil_flush() would not be updated the posix_fallocate() code path, so the "redundant" fil_flush() should be a no-op.	7 years ago
Eugene Kosov	a06a3e4670	MDEV-18233 Moving the hash_node_t to improve locality of reference When performing a hash search via HASH_SEARCH we first look at a key of a node and then at its pointer to the next node in chain. If we have those in one cache line instead of a two we reduce memory reads. I found dict_table_t, fil_space_t and buf_page_t suitable for such improvement.	7 years ago
Marko Mäkelä	447e493179	Remove some unnecessary InnoDB #include	7 years ago
Marko Mäkelä	e82e216e37	MDEV-17849 Undo tablespace truncation recovery fails to shrink file fil_space_t::add(): Replaces fil_node_create(), fil_node_create_low(). Let the caller pass fil_node_t::handle, to avoid having to close and re-open files. fil_node_t::read_page0(): Refactored from fil_node_open_file(). Read the first page of a data file. fil_node_open_file(): Open the file only once. srv_undo_tablespace_open(): Set the file handle for the opened undo tablespace. This should ensure that ut_ad(file->is_open()) no longer fails in recv_add_trim(). xtrabackup_backup_func(): Remove some dead code. xb_fil_cur_open(): Open files only if needed. Undo tablespaces should already have been opened.	7 years ago
Marko Mäkelä	eb6364619f	Remove the redundant variable fil_n_file_opened	7 years ago
Marko Mäkelä	b3009059d0	Minor cleanup	7 years ago
Eugene Kosov	14be814380	MDEV-17491 micro optimize page_id_t page_id_t: remove m_fold member various places: pass page_id_t by value instead of by reference	7 years ago
Marko Mäkelä	3448ceb02a	MDEV-13564: Implement innodb_unsafe_truncate=ON for compatibility While MariaDB Server 10.2 is not really guaranteed to be compatible with Percona XtraBackup 2.4 (for example, the MySQL 5.7 undo log format change that could be present in XtraBackup, but was reverted from MariaDB in MDEV-12289), we do not want to disrupt users who have deployed xtrabackup and MariaDB Server 10.2 in their environments. With this change, MariaDB 10.2 will continue to use the backup-unsafe TRUNCATE TABLE code, so that neither the undo log nor the redo log formats will change in an incompatible way. Undo tablespace truncation will keep using the redo log only. Recovery or backup with old code will fail to shrink the undo tablespace files, but the contents will be recovered just fine. In the MariaDB Server 10.2 series only, we introduce the configuration parameter innodb_unsafe_truncate and make it ON by default. To allow MariaDB Backup (mariabackup) to work properly with TRUNCATE TABLE operations, use loose_innodb_unsafe_truncate=OFF. MariaDB Server 10.3.10 and later releases will always use the backup-safe TRUNCATE TABLE, and this parameter will not be added there. recv_recovery_rollback_active(): Skip row_mysql_drop_garbage_tables() unless innodb_unsafe_truncate=OFF. It is too unsafe to drop orphan tables if RENAME operations are not transactional within InnoDB. LOG_HEADER_FORMAT_10_3: Replaces LOG_HEADER_FORMAT_CURRENT. log_init(), log_group_file_header_flush(), srv_prepare_to_delete_redo_log_files(), innobase_start_or_create_for_mysql(): Choose the redo log format and subformat based on the value of innodb_unsafe_truncate.	7 years ago
Marko Mäkelä	75f8e86f57	MDEV-17158 TRUNCATE is not atomic after MDEV-13564 It turned out that ha_innobase::truncate() would prematurely commit the transaction already before the completion of the ha_innobase::create(). All of this must be atomic. innodb.truncate_crash: Use the correct DEBUG_SYNC point, and tolerate non-truncation of the table, because the redo log for the TRUNCATE transaction commit might be flushed due to some InnoDB background activity. dict_build_tablespace_for_table(): Merge to the function dict_build_table_def_step(). dict_build_table_def_step(): If a table is being created during an already started data dictionary transaction (such as TRUNCATE), persistently write the table_id to the undo log header before creating any file. In this way, the recovery of TRUNCATE will be able to delete the new file before rolling back the rename of the original table. dict_table_rename_in_cache(): Add the parameter replace_new_file, used as part of rolling back a TRUNCATE operation. fil_rename_tablespace_check(): Add the parameter replace_new. If the parameter is set and a file identified by new_path exists, remove a possible tablespace and also the file. create_table_info_t::create_table_def(): Remove some debug assertions that no longer hold. During TRUNCATE, the transaction will already have been started (and performed a rename operation) before the table is created. Also, remove a call to dict_build_tablespace_for_table(). create_table_info_t::create_table(): Add the parameter create_fk=true. During TRUNCATE TABLE, do not add FOREIGN KEY constraints to the InnoDB data dictionary, because they will also not be removed. row_table_add_foreign_constraints(): If trx=NULL, do not modify the InnoDB data dictionary, but only load the FOREIGN KEY constraints from the data dictionary. ha_innobase::create(): Lock the InnoDB data dictionary cache only if no transaction was passed by the caller. Unlock it in any case. innobase_rename_table(): Add the parameter commit = true. If !commit, do not lock or unlock the data dictionary cache. ha_innobase::truncate(): Lock the data dictionary before invoking rename or create, and let ha_innobase::create() unlock it and also commit or roll back the transaction. trx_undo_mark_as_dict(): Renamed from trx_undo_mark_as_dict_operation() and declared global instead of static. row_undo_ins_parse_undo_rec(): If table_id is set, this must be rolling back the rename operation in TRUNCATE TABLE, and therefore replace_new_file=true.	7 years ago
Marko Mäkelä	cf2a4426a2	MDEV-14717 RENAME TABLE in InnoDB is not crash-safe This is a backport of commit `0bc36758ba` and commit `9eb3fcc9fb`. InnoDB in MariaDB 10.2 appears to only write MLOG_FILE_RENAME2 redo log records during table-rebuilding ALGORITHM=INPLACE operations. We must write the records for any .ibd file renames, so that the operations are crash-safe. If InnoDB is killed during a RENAME TABLE operation, it can happen that the transaction for updating the data dictionary will be rolled back. But, nothing will roll back the renaming of the .ibd file (the MLOG_FILE_RENAME2 only guarantees roll-forward), or for that matter, the renaming of the dict_table_t::name in the dict_sys cache. We introduce the undo log record TRX_UNDO_RENAME_TABLE to fix this. fil_space_for_table_exists_in_mem(): Remove the parameters adjust_space, table_id and some code that was trying to work around these deficiencies. fil_name_write_rename(): Write a MLOG_FILE_RENAME2 record. dict_table_rename_in_cache(): Invoke fil_name_write_rename(). trx_undo_rec_copy(): Set the first 2 bytes to the length of the copied undo log record. trx_undo_page_report_rename(), trx_undo_report_rename(): Write a TRX_UNDO_RENAME_TABLE record with the old table name. row_rename_table_for_mysql(): Invoke trx_undo_report_rename() before modifying any data dictionary tables. row_undo_ins_parse_undo_rec(): Roll back TRX_UNDO_RENAME_TABLE by invoking dict_table_rename_in_cache(), which will take care of both renaming the table and the file. ha_innobase::truncate(): Remove a work-around.	7 years ago
Marko Mäkelä	055a3334ad	MDEV-13564 Mariabackup does not work with TRUNCATE Implement undo tablespace truncation via normal redo logging. Implement TRUNCATE TABLE as a combination of RENAME to #sql-ib name, CREATE, and DROP. Note: Orphan #sql-ib.ibd may be left behind if MariaDB Server 10.2 is killed before the DROP operation is committed. If MariaDB Server 10.2 is killed during TRUNCATE, it is also possible that the old table was renamed to #sql-ib.ibd but the data dictionary will refer to the table using the original name. In MariaDB Server 10.3, RENAME inside InnoDB is transactional, and #sql-* tables will be dropped on startup. So, this new TRUNCATE will be fully crash-safe in 10.3. ha_mroonga::wrapper_truncate(): Pass table options to the underlying storage engine, now that ha_innobase::truncate() will need them. rpl_slave_state::truncate_state_table(): Before truncating mysql.gtid_slave_pos, evict any cached table handles from the table definition cache, so that there will be no stale references to the old table after truncating. == TRUNCATE TABLE == WL#6501 in MySQL 5.7 introduced separate log files for implementing atomic and crash-safe TRUNCATE TABLE, instead of using the InnoDB undo and redo log. Some convoluted logic was added to the InnoDB crash recovery, and some extra synchronization (including a redo log checkpoint) was introduced to make this work. This synchronization has caused performance problems and race conditions, and the extra log files cannot be copied or applied by external backup programs. In order to support crash-upgrade from MariaDB 10.2, we will keep the logic for parsing and applying the extra log files, but we will no longer generate those files in TRUNCATE TABLE. A prerequisite for crash-safe TRUNCATE is a crash-safe RENAME TABLE (with full redo and undo logging and proper rollback). This will be implemented in MDEV-14717. ha_innobase::truncate(): Invoke RENAME, create(), delete_table(). Because RENAME cannot be fully rolled back before MariaDB 10.3 due to missing undo logging, add some explicit rename-back in case the operation fails. ha_innobase::delete(): Introduce a variant that takes sqlcom as a parameter. In TRUNCATE TABLE, we do not want to touch any FOREIGN KEY constraints. ha_innobase::create(): Add the parameters file_per_table, trx. In TRUNCATE, the new table must be created in the same transaction that renames the old table. create_table_info_t::create_table_info_t(): Add the parameters file_per_table, trx. row_drop_table_for_mysql(): Replace a bool parameter with sqlcom. row_drop_table_after_create_fail(): New function, wrapping row_drop_table_for_mysql(). dict_truncate_index_tree_in_mem(), fil_truncate_tablespace(), fil_prepare_for_truncate(), fil_reinit_space_header_for_table(), row_truncate_table_for_mysql(), TruncateLogger, row_truncate_prepare(), row_truncate_rollback(), row_truncate_complete(), row_truncate_fts(), row_truncate_update_system_tables(), row_truncate_foreign_key_checks(), row_truncate_sanity_checks(): Remove. row_upd_check_references_constraints(): Remove a check for TRUNCATE, now that the table is no longer truncated in place. The new test innodb.truncate_foreign uses DEBUG_SYNC to cover some race-condition like scenarios. The test innodb-innodb.truncate does not use any synchronization. We add a redo log subformat to indicate backup-friendly format. MariaDB 10.4 will remove support for the old TRUNCATE logging, so crash-upgrade from old 10.2 or 10.3 to 10.4 will involve limitations. == Undo tablespace truncation == MySQL 5.7 implements undo tablespace truncation. It is only possible when innodb_undo_tablespaces is set to at least 2. The logging is implemented similar to the WL#6501 TRUNCATE, that is, using separate log files and a redo log checkpoint. We can simply implement undo tablespace truncation within a single mini-transaction that reinitializes the undo log tablespace file. Unfortunately, due to the redo log format of some operations, currently, the total redo log written by undo tablespace truncation will be more than the combined size of the truncated undo tablespace. It should be acceptable to have a little more than 1 megabyte of log in a single mini-transaction. This will be fixed in MDEV-17138 in MariaDB Server 10.4. recv_sys_t: Add truncated_undo_spaces[] to remember for which undo tablespaces a MLOG_FILE_CREATE2 record was seen. namespace undo: Remove some unnecessary declarations. fil_space_t::is_being_truncated: Document that this flag now only applies to undo tablespaces. Remove some references. fil_space_t::is_stopping(): Do not refer to is_being_truncated. This check is for tablespaces of tables. Potentially used tablespaces are never truncated any more. buf_dblwr_process(): Suppress the out-of-bounds warning for undo tablespaces. fil_truncate_log(): Write a MLOG_FILE_CREATE2 with a nonzero page number (new size of the tablespace in pages) to inform crash recovery that the undo tablespace size has been reduced. fil_op_write_log(): Relax assertions, so that MLOG_FILE_CREATE2 can be written for undo tablespaces (without .ibd file suffix) for a nonzero page number. os_file_truncate(): Add the parameter allow_shrink=false so that undo tablespaces can actually be shrunk using this function. fil_name_parse(): For undo tablespace truncation, buffer MLOG_FILE_CREATE2 in truncated_undo_spaces[]. recv_read_in_area(): Avoid reading pages for which no redo log records remain buffered, after recv_addr_trim() removed them. trx_rseg_header_create(): Add a FIXME comment that we could write much less redo log. trx_undo_truncate_tablespace(): Reinitialize the undo tablespace in a single mini-transaction, which will be flushed to the redo log before the file size is trimmed. recv_addr_trim(): Discard any redo logs for pages that were logged after the new end of a file, before the truncation LSN. If the rec_list becomes empty, reduce n_addrs. After removing any affected records, actually truncate the file. recv_apply_hashed_log_recs(): Invoke recv_addr_trim() right before applying any log records. The undo tablespace files must be open at this point. buf_flush_or_remove_pages(), buf_flush_dirty_pages(), buf_LRU_flush_or_remove_pages(): Add a parameter for specifying the number of the first page to flush or remove (default 0). trx_purge_initiate_truncate(): Remove the log checkpoints, the extra logging, and some unnecessary crash points. Merge the code from trx_undo_truncate_tablespace(). First, flush all to-be-discarded pages (beyond the new end of the file), then trim the space->size to make the page allocation deterministic. At the only remaining crash injection point, flush the redo log, so that the recovery can be tested.	7 years ago
Marko Mäkelä	2ca904f0ca	MDEV-13103 Deal with page_compressed page corruption fil_page_decompress(): Replaces fil_decompress_page(). Allow the caller detect errors. Remove duplicated code. Use the "safe" instead of "fast" variants of decompression routines. fil_page_compress(): Replaces fil_compress_page(). The length of the input buffer always was srv_page_size (innodb_page_size). Remove printouts, and remove the fil_space_t* parameter. buf_tmp_buffer_t::reserved: Make private; the accessors acquire() and release() will use atomic memory access. buf_pool_reserve_tmp_slot(): Make static. Remove the second parameter. Do not acquire any mutex. Remove the allocation of the buffers. buf_tmp_reserve_crypt_buf(), buf_tmp_reserve_compression_buf(): Refactored away from buf_pool_reserve_tmp_slot(). buf_page_decrypt_after_read(): Make static, and simplify the logic. Use the encryption buffer also for decompressing. buf_page_io_complete(), buf_dblwr_process(): Check more failures. fil_space_encrypt(): Simplify the debug checks. fil_space_t::printed_compression_failure: Remove. fil_get_compression_alg_name(): Remove. fil_iterate(): Allocate a buffer for compression and decompression only once, instead of allocating and freeing it for every page that uses compression, during IMPORT TABLESPACE. Also, validate the page checksum before decryption, and reduce the scope of some variables. fil_page_is_index_page(), fil_page_is_lzo_compressed(): Remove (unused). AbstractCallback::operator()(): Remove the parameter 'offset'. The check for it in FetchIndexRootPages::operator() was basically redundant and dead code since the previous refactoring.	8 years ago
Marko Mäkelä	f5eb37129f	MDEV-13103 Deal with page_compressed page corruption fil_page_decompress(): Replaces fil_decompress_page(). Allow the caller detect errors. Remove duplicated code. Use the "safe" instead of "fast" variants of decompression routines. fil_page_compress(): Replaces fil_compress_page(). The length of the input buffer always was srv_page_size (innodb_page_size). Remove printouts, and remove the fil_space_t* parameter. buf_tmp_buffer_t::reserved: Make private; the accessors acquire() and release() will use atomic memory access. buf_pool_reserve_tmp_slot(): Make static. Remove the second parameter. Do not acquire any mutex. Remove the allocation of the buffers. buf_tmp_reserve_crypt_buf(), buf_tmp_reserve_compression_buf(): Refactored away from buf_pool_reserve_tmp_slot(). buf_page_decrypt_after_read(): Make static, and simplify the logic. Use the encryption buffer also for decompressing. buf_page_io_complete(), buf_dblwr_process(): Check more failures. fil_space_encrypt(): Simplify the debug checks. fil_space_t::printed_compression_failure: Remove. fil_get_compression_alg_name(): Remove. fil_iterate(): Allocate a buffer for compression and decompression only once, instead of allocating and freeing it for every page that uses compression, during IMPORT TABLESPACE. fil_node_get_space_id(), fil_page_is_index_page(), fil_page_is_lzo_compressed(): Remove (unused code).	8 years ago
Marko Mäkelä	3fcc11fbb4	Remove traces of the non-working MDEV-6354 MariaDB never supported the MySQL 5.7 compression format. FIL_PAGE_TYPE_COMPRESSED: Remove. This was originally added as FIL_PAGE_COMPRESSED.	8 years ago
Marko Mäkelä	55abcfa7b7	MDEV-16124 fil_rename_tablespace() times out and crashes server during table-rebuilding ALTER TABLE InnoDB insisted on closing the file handle before renaming a file. Renaming a file should never be a problem on POSIX systems. Also on Windows it should work if the file was opened in FILE_SHARE_DELETE mode. fil_space_t::stop_ios: Remove. We no longer need to stop file access during rename operations. fil_mutex_enter_and_prepare_for_io(): Remove the wait for stop_ios. fil_rename_tablespace(): Remove the retry logic; do not close the file handle. Remove the unused fault injection that was added along with the DATA DIRECTORY functionality (MySQL WL#5980). os_file_create_simple_func(), os_file_create_func(), os_file_create_simple_no_error_handling_func(): Include FILE_SHARE_DELETE in the share_mode. (We will still prevent multiple InnoDB instances from using the same files by not setting FILE_SHARE_WRITE.)	8 years ago
Marko Mäkelä	6247c64c2a	MDEV-12396 IMPORT TABLESPACE cleanup Reduce unnecessary inter-module calls for IMPORT TABLESPACE. Move some IMPORT-related code from fil0fil.cc to row0import.cc. PageCallback: Remove. Make AbstractCallback the base class. PageConverter: Define some member functions inline.	8 years ago
Marko Mäkelä	112df06996	MDEV-15529 IMPORT TABLESPACE unnecessarily uses the doublewrite buffer fil_space_t::atomic_write_supported: Always set this flag for TEMPORARY TABLESPACE and during IMPORT TABLESPACE. The page writes during these operations are by definition not crash-safe because they are not written to the redo log. fil_space_t::use_doublewrite(): Determine if doublewrite should be used. buf_dblwr_update(): Add assertions, and let the caller check whether doublewrite buffering is desired. buf_flush_write_block_low(): Disable the doublewrite buffer for the temporary tablespace and for IMPORT TABLESPACE. fil_space_set_imported(), fil_node_open_file(), fil_space_create(): Initialize or revise the space->atomic_write_supported flag. buf_page_io_complete(), buf_flush_write_complete(): Add the parameter dblwr, to indicate whether doublewrite was used for writes. buf_dblwr_sync_datafiles(): Remove an unnecessary flush of persistent tablespaces when flushing temporary tablespaces. (Move the call to buf_dblwr_flush_buffered_writes().)	8 years ago
Marko Mäkelä	431607237d	MDEV-12173 "Error: trying to do an operation on a dropped tablespace" InnoDB is issuing a 'noise' message that is not a sign of abnormal operation. The only issuers of it are the debug function lock_rec_block_validate() and the change buffer merge. While the error should ideally never occur in transactional locking, we happen to know that DISCARD TABLESPACE and TRUNCATE TABLE and possibly DROP TABLE are breaking InnoDB table locks. When it comes to the change buffer merge, the message simply is useless noise. We know perfectly well that a tablespace can be dropped while a change buffer merge is pending. And the code is prepared to handle that, which is demonstrated by the fact that whenever the message was issued, InnoDB did not crash. fil_inc_pending_ops(): Remove the parameter print_err.	8 years ago
Marko Mäkelä	ec062c6181	MDEV-12121 follow-up: Unbreak the WITH_INNODB_AHI=OFF build	8 years ago
Marko Mäkelä	51b4366bfb	MDEV-13328 ALTER TABLE…DISCARD TABLESPACE takes a lot of time With a big buffer pool that contains many data pages, DISCARD TABLESPACE took a long time, because it would scan the entire buffer pool to remove any pages that belong to the tablespace. With a large buffer pool, this would take a lot of time, especially when the table-to-discard is empty. The minimum amount of work that DISCARD TABLESPACE must do is to remove the pages of the to-be-discarded table from the buf_pool->flush_list because any writes to the data file must be prevented before the file is deleted. If DISCARD TABLESPACE does not evict the pages from the buffer pool, then IMPORT TABLESPACE must do it, because we must prevent pre-DISCARD, not-yet-evicted pages from being mistaken for pages of the imported tablespace. It would not be a useful fix to simply move the buffer pool scan to the IMPORT TABLESPACE step. What we can do is to actively evict those pages that could be mistaken for imported pages. In this way, when importing a small table into a big buffer pool, the import should still run relatively fast. Import is bypassing the buffer pool when reading pages for the adjustment phase. In the adjustment phase, if a page exists in the buffer pool, we could replace it with the page from the imported file. Unfortunately I did not get this to work properly, so instead we will simply evict any matching page from the buffer pool. buf_page_get_gen(): Implement BUF_EVICT_IF_IN_POOL, a new mode where the requested page will be evicted if it is found. There must be no unwritten changes for the page. buf_remove_t: Remove. Instead, use trx!=NULL to signify that a write to file is desired, and use a separate parameter bool drop_ahi. buf_LRU_flush_or_remove_pages(), fil_delete_tablespace(): Replace buf_remove_t. buf_LRU_remove_pages(), buf_LRU_remove_all_pages(): Remove. PageConverter::m_mtr: A dummy mini-transaction buffer PageConverter::PageConverter(): Complete the member initialization list. PageConverter::operator()(): Evict any 'shadow' pages from the buffer pool so that pre-existing (garbage) pages cannot be mistaken for pages that exist in the being-imported file. row_discard_tablespace(): Remove a bogus comment that seems to refer to IMPORT TABLESPACE, not DISCARD TABLESPACE.	8 years ago
Marko Mäkelä	dd35fb35fa	Return uint16_t instead of ulint	8 years ago
Jan Lindström	aa22981dd2	MDEV-12741: innodb.ibuf_not_empty failed in buildbot with "InnoDB: Trying to do I/O to a tablespace which does not exist" Background thread is doing ibuf merge, in buf0rea.cc buf_read_ibuf_merge_pages(). It first tries to get page_size and if space is not found it deletes them, but as we do not hold any mutexes, space can be marked as stopped between that and buf_read_page_low() for same space. This naturally leads seen error message on log. buf_read_page_low(): Add parameter ignore_missing_space = false that is passed to fil_io() buf_read_ibuf_merge_pages(): call buf_read_page_low with ignore_missing_space = true, this function will handle missing space error code after buf_read_page_low returns. fil_io(): if ignore_missing_space = true do not print error message about trying to do I/0 for missing space, just return correct error code that is handled later.	9 years ago
Jan Lindström	61096ff214	MDEV-13591: InnoDB: Database page corruption on disk or a failed file read and assertion failure Problem is that page 0 and its possible enrryption information is not read for undo tablespaces. fil_crypt_get_latest_key_version(): Do not send event to encryption threads if event does not yet exists. Seen on regression testing. fil_read_first_page: Add new parameter does page belong to undo tablespace and if it does, we do not read FSP_HEADER. srv_undo_tablespace_open : Read first page of the tablespace to get crypt_data if it exists and pass it to fil_space_create. Tested using innodb_encryption with combinations with innodb-undo-tablespaces.	8 years ago
Thirunarayanan Balathandayuthapani	9d57468dde	Bug #25357789 INNODB: LATCH ORDER VIOLATION DURING TRUNCATE TABLE IF INNODB_SYNC_DEBUG ENABLED Analysis: ======== (1) During TRUNCATE of file_per_table tablespace, dict_operation_lock is released before eviction of dirty pages of a tablespace from the buffer pool. After eviction, we try to re-acquire dict_operation_lock (higher level latch) but we already hold lower level latch (index->lock). This causes latch order violation (2) Deadlock issue is present if child table is being truncated and it holds index lock. At the same time, cascade dml happens and it took dict_operation_lock and waiting for index lock. Fix: ==== 1) Release the indexes lock before releasing the dict operation lock. 2) Ignore the cascading dml operation on the parent table, for the cascading foreign key, if the child table is truncated or if it is in the process of being truncated. Reviewed-by: Jimmy Yang <jimmy.yang@oracle.com> Reviewed-by: Kevin Lewis <kevin.lewis@oracle.com> RB: 16122	9 years ago
Jan Lindström	34eef269eb	MDEV-11939: innochecksum mistakes a file for an encrypted one (page 0 invalid) Always read full page 0 to determine does tablespace contain encryption metadata. Tablespaces that are page compressed or page compressed and encrypted do not compare checksum as it does not exists. For encrypted tables use checksum verification written for encrypted tables and normal tables use normal method. buf_page_is_checksum_valid_crc32 buf_page_is_checksum_valid_innodb buf_page_is_checksum_valid_none Modify Innochecksum logging to file to avoid compilation warnings. fil0crypt.cc fil0crypt.h Modify to be able to use in innochecksum compilation and move fil_space_verify_crypt_checksum to end of the file. Add innochecksum logging to file. univ.i Add innochecksum strict_verify, log_file and cur_page_num variables as extern. page_zip_verify_checksum Add innochecksum logging to file and remove unnecessary code. innochecksum.cc Lot of changes most notable able to read encryption metadata from page 0 of the tablespace. Added test case where we corrupt intentionally FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION (encryption key version) FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION+4 (post encryption checksum) FIL_DATA+10 (data)	8 years ago
Jan Lindström	8b019f87dd	MDEV-11939: innochecksum mistakes a file for an encrypted one (page 0 invalid) Always read full page 0 to determine does tablespace contain encryption metadata. Tablespaces that are page compressed or page compressed and encrypted do not compare checksum as it does not exists. For encrypted tables use checksum verification written for encrypted tables and normal tables use normal method. buf_page_is_checksum_valid_crc32 buf_page_is_checksum_valid_innodb buf_page_is_checksum_valid_none Add Innochecksum logging to file buf_page_is_corrupted Remove ib_logf and page_warn_strict_checksum calls in innochecksum compilation. Add innochecksum logging to file. fil0crypt.cc fil0crypt.h Modify to be able to use in innochecksum compilation and move fil_space_verify_crypt_checksum to end of the file. Add innochecksum logging to file. univ.i Add innochecksum strict_verify, log_file and cur_page_num variables as extern. page_zip_verify_checksum Add innochecksum logging to file. innochecksum.cc Lot of changes most notable able to read encryption metadata from page 0 of the tablespace. Added test case where we corrupt intentionally FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION (encryption key version) FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION+4 (post encryption checksum) FIL_DATA+10 (data)	8 years ago
Marko Mäkelä	8c71c6aa8b	MDEV-12548 Initial implementation of Mariabackup for MariaDB 10.2 InnoDB I/O and buffer pool interfaces and the redo log format have been changed between MariaDB 10.1 and 10.2, and the backup code has to be adjusted accordingly. The code has been simplified, and many memory leaks have been fixed. Instead of the file name xtrabackup_logfile, the file name ib_logfile0 is being used for the copy of the redo log. Unnecessary InnoDB startup and shutdown and some unnecessary threads have been removed. Some help was provided by Vladislav Vaintroub. Parameters have been cleaned up and aligned with those of MariaDB 10.2. The --dbug option has been added, so that in debug builds, --dbug=d,ib_log can be specified to enable diagnostic messages for processing redo log entries. By default, innodb_doublewrite=OFF, so that --prepare works faster. If more crash-safety for --prepare is needed, double buffering can be enabled. The parameter innodb_log_checksums=OFF can be used to ignore redo log checksums in --backup. Some messages have been cleaned up. Unless --export is specified, Mariabackup will not deal with undo log. The InnoDB mini-transaction redo log is not only about user-level transactions; it is actually about mini-transactions. To avoid confusion, call it the redo log, not transaction log. We disable any undo log processing in --prepare. Because MariaDB 10.2 supports indexed virtual columns, the undo log processing would need to be able to evaluate virtual column expressions. To reduce the amount of code dependencies, we will not process any undo log in prepare. This means that the --export option must be disabled for now. This also means that the following options are redundant and have been removed: xtrabackup --apply-log-only innobackupex --redo-only In addition to disabling any undo log processing, we will disable any further changes to data pages during --prepare, including the change buffer merge. This means that restoring incremental backups should reliably work even when change buffering is being used on the server. Because of this, preparing a backup will not generate any further redo log, and the redo log file can be safely deleted. (If the --export option is enabled in the future, it must generate redo log when processing undo logs and buffered changes.) In --prepare, we cannot easily know if a partial backup was used, especially when restoring a series of incremental backups. So, we simply warn about any missing files, and ignore the redo log for them. FIXME: Enable the --export option. FIXME: Improve the handling of the MLOG_INDEX_LOAD record, and write a test that initiates a backup while an ALGORITHM=INPLACE operation is creating indexes or rebuilding a table. An error should be detected when preparing the backup. FIXME: In --incremental --prepare, xtrabackup_apply_delta() should ensure that if FSP_SIZE is modified, the file size will be adjusted accordingly.	9 years ago
Marko Mäkelä	58f87a41bd	Remove some fields from dict_table_t dict_table_t::thd: Remove. This was only used by btr_root_block_get() for reporting decryption failures, and it was only assigned by ha_innobase::open(), and never cleared. This could mean that if a connection is closed, the pointer would become stale, and the server could crash while trying to report the error. It could also mean that an error is being reported to the wrong client. It is better to use current_thd in this case, even though it could mean that if the code is invoked from an InnoDB background operation, there would be no connection to which to send the error message. Remove dict_table_t::crypt_data and dict_table_t::page_0_read. These fields were never read. fil_open_single_table_tablespace(): Remove the parameter "table".	9 years ago
Jan Lindström	58c56dd7f8	MDEV-12610: MariaDB start is slow Problem appears to be that the function fsp_flags_try_adjust() is being unconditionally invoked on every .ibd file on startup. Based on performance investigation also the top function fsp_header_get_crypt_offset() needs to addressed. Ported implementation of fsp_header_get_encryption_offset() function from 10.2 to fsp_header_get_crypt_offset(). Introduced a new function fil_crypt_read_crypt_data() to read page 0 if it is not yet read. fil_crypt_find_space_to_rotate(): Now that page 0 for every .ibd file is not read on startup we need to check has page 0 read from space that we investigate for key rotation, if it is not read we read it. fil_space_crypt_get_status(): Now that page 0 for every .ibd file is not read on startup here also we need to read page 0 if it is not yet read it. This is needed as tests use IS query to wait until background encryption or decryption has finished and this function is used to produce results. fil_crypt_thread(): Add is_stopping condition for tablespace so that we do not rotate pages if usage of tablespace should be stopped. This was needed for failure seen on regression testing. fil_space_create: Remove page_0_crypt_read and extra unnecessary info output. fil_open_single_table_tablespace(): We call fsp_flags_try_adjust only when when no errors has happened and server was not started on read only mode and tablespace validation was requested or flags contain other table options except low order bits to FSP_FLAGS_POS_PAGE_SSIZE position. fil_space_t::page_0_crypt_read removed. Added test case innodb-first-page-read to test startup when encryption is on and when encryption is off to check that not for all tables page 0 is read on startup.	9 years ago
Jan Lindström	1af8bf39ca	MDEV-12113: install_db shows corruption for rest encryption with innodb_data_file_path=ibdata1:3M; Problem was that FIL_PAGE_FLUSH_LSN_OR_KEY_VERSION field that for encrypted pages even in system datafiles should contain key_version except very first page (0:0) is after encryption overwritten with flush lsn. Ported WL#7990 Repurpose FIL_PAGE_FLUSH_LSN to 10.1 The field FIL_PAGE_FLUSH_LSN_OR_KEY_VERSION is consulted during InnoDB startup. At startup, InnoDB reads the FIL_PAGE_FLUSH_LSN_OR_KEY_VERSION from the first page of each file in the InnoDB system tablespace. If there are multiple files, the minimum and maximum LSN can differ. These numbers are passed to InnoDB startup. Having the number in other files than the first file of the InnoDB system tablespace is not providing much additional value. It is conflicting with other use of the field, such as on InnoDB R-tree index pages and encryption key_version. This worklog will stop writing FIL_PAGE_FLUSH_LSN_OR_KEY_VERSION to other files than the first file of the InnoDB system tablespace (page number 0:0) when system tablespace is encrypted. If tablespace is not encrypted we continue writing FIL_PAGE_FLUSH_LSN_OR_KEY_VERSION to all first pages of system tablespace to avoid unnecessary warnings on downgrade. open_or_create_data_files(): pass only one flushed_lsn parameter xb_load_tablespaces(): pass only one flushed_lsn parameter. buf_page_create(): Improve comment about where FIL_PAGE_FIL_FLUSH_LSN_OR_KEY_VERSION is set. fil_write_flushed_lsn(): A new function, merged from fil_write_lsn_and_arch_no_to_file() and fil_write_flushed_lsn_to_data_files(). Only write to the first page of the system tablespace (page 0:0) if tablespace is encrypted, or write all first pages of system tablespace and invoke fil_flush_file_spaces(FIL_TYPE_TABLESPACE) afterwards. fil_read_first_page(): read flush_lsn and crypt_data only from first datafile. fil_open_single_table_tablespace(): Remove output of LSN, because it was only valid for the system tablespace and the undo tablespaces, not user tablespaces. fil_validate_single_table_tablespace(): Remove output of LSN. checkpoint_now_set(): Use fil_write_flushed_lsn and output a error if operation fails. Remove lsn variable from fsp_open_info. recv_recovery_from_checkpoint_start(): Remove unnecessary second flush_lsn parameter. log_empty_and_mark_files_at_shutdown(): Use fil_writte_flushed_lsn and output error if it fails. open_or_create_data_files(): Pass only one flushed_lsn variable.	9 years ago
Marko Mäkelä	e22d86a3eb	fil_create_new_single_table_tablespace(): Correct a bogus nonnull attribute The parameter path can be passed as NULL. This error was reported by GCC 7.1.0 when compiling CMAKE_BUILD_TYPE=Debug with -O3.	9 years ago
Vicențiu Ciorbaru	0af9818240	5.6.36	9 years ago
Marko Mäkelä	021d636551	Fix some integer type mismatch. Use uint32_t for the encryption key_id. When filling unsigned integer values into INFORMATION_SCHEMA tables, use the method Field::store(longlong, bool unsigned) instead of using Field::store(double). Fix also some miscellanous type mismatch related to ulint (size_t).	9 years ago

1 2 3 4

179 Commits (3b6dadb5ebedab71bf1870579745ff3cd05e498a)