mariadb

Tree: 8e3d85e112

Author	SHA1	Message	Date
Nikita Malyavin	350e46a8b5	MDEV-18546 ASAN heap-use-after-free in innobase_get_computed_value / row_purge the bug was already fixed in MDEV-17005, so now only test is added	6 years ago
Marko Mäkelä	4afe9d4b6d	MDEV-16222: Move the test to gcol.innodb_virtual_debug_purge	6 years ago
Marko Mäkelä	055a3334ad	MDEV-13564 Mariabackup does not work with TRUNCATE Implement undo tablespace truncation via normal redo logging. Implement TRUNCATE TABLE as a combination of RENAME to #sql-ib name, CREATE, and DROP. Note: Orphan #sql-ib.ibd may be left behind if MariaDB Server 10.2 is killed before the DROP operation is committed. If MariaDB Server 10.2 is killed during TRUNCATE, it is also possible that the old table was renamed to #sql-ib.ibd but the data dictionary will refer to the table using the original name. In MariaDB Server 10.3, RENAME inside InnoDB is transactional, and #sql-* tables will be dropped on startup. So, this new TRUNCATE will be fully crash-safe in 10.3. ha_mroonga::wrapper_truncate(): Pass table options to the underlying storage engine, now that ha_innobase::truncate() will need them. rpl_slave_state::truncate_state_table(): Before truncating mysql.gtid_slave_pos, evict any cached table handles from the table definition cache, so that there will be no stale references to the old table after truncating. == TRUNCATE TABLE == WL#6501 in MySQL 5.7 introduced separate log files for implementing atomic and crash-safe TRUNCATE TABLE, instead of using the InnoDB undo and redo log. Some convoluted logic was added to the InnoDB crash recovery, and some extra synchronization (including a redo log checkpoint) was introduced to make this work. This synchronization has caused performance problems and race conditions, and the extra log files cannot be copied or applied by external backup programs. In order to support crash-upgrade from MariaDB 10.2, we will keep the logic for parsing and applying the extra log files, but we will no longer generate those files in TRUNCATE TABLE. A prerequisite for crash-safe TRUNCATE is a crash-safe RENAME TABLE (with full redo and undo logging and proper rollback). This will be implemented in MDEV-14717. ha_innobase::truncate(): Invoke RENAME, create(), delete_table(). Because RENAME cannot be fully rolled back before MariaDB 10.3 due to missing undo logging, add some explicit rename-back in case the operation fails. ha_innobase::delete(): Introduce a variant that takes sqlcom as a parameter. In TRUNCATE TABLE, we do not want to touch any FOREIGN KEY constraints. ha_innobase::create(): Add the parameters file_per_table, trx. In TRUNCATE, the new table must be created in the same transaction that renames the old table. create_table_info_t::create_table_info_t(): Add the parameters file_per_table, trx. row_drop_table_for_mysql(): Replace a bool parameter with sqlcom. row_drop_table_after_create_fail(): New function, wrapping row_drop_table_for_mysql(). dict_truncate_index_tree_in_mem(), fil_truncate_tablespace(), fil_prepare_for_truncate(), fil_reinit_space_header_for_table(), row_truncate_table_for_mysql(), TruncateLogger, row_truncate_prepare(), row_truncate_rollback(), row_truncate_complete(), row_truncate_fts(), row_truncate_update_system_tables(), row_truncate_foreign_key_checks(), row_truncate_sanity_checks(): Remove. row_upd_check_references_constraints(): Remove a check for TRUNCATE, now that the table is no longer truncated in place. The new test innodb.truncate_foreign uses DEBUG_SYNC to cover some race-condition like scenarios. The test innodb-innodb.truncate does not use any synchronization. We add a redo log subformat to indicate backup-friendly format. MariaDB 10.4 will remove support for the old TRUNCATE logging, so crash-upgrade from old 10.2 or 10.3 to 10.4 will involve limitations. == Undo tablespace truncation == MySQL 5.7 implements undo tablespace truncation. It is only possible when innodb_undo_tablespaces is set to at least 2. The logging is implemented similar to the WL#6501 TRUNCATE, that is, using separate log files and a redo log checkpoint. We can simply implement undo tablespace truncation within a single mini-transaction that reinitializes the undo log tablespace file. Unfortunately, due to the redo log format of some operations, currently, the total redo log written by undo tablespace truncation will be more than the combined size of the truncated undo tablespace. It should be acceptable to have a little more than 1 megabyte of log in a single mini-transaction. This will be fixed in MDEV-17138 in MariaDB Server 10.4. recv_sys_t: Add truncated_undo_spaces[] to remember for which undo tablespaces a MLOG_FILE_CREATE2 record was seen. namespace undo: Remove some unnecessary declarations. fil_space_t::is_being_truncated: Document that this flag now only applies to undo tablespaces. Remove some references. fil_space_t::is_stopping(): Do not refer to is_being_truncated. This check is for tablespaces of tables. Potentially used tablespaces are never truncated any more. buf_dblwr_process(): Suppress the out-of-bounds warning for undo tablespaces. fil_truncate_log(): Write a MLOG_FILE_CREATE2 with a nonzero page number (new size of the tablespace in pages) to inform crash recovery that the undo tablespace size has been reduced. fil_op_write_log(): Relax assertions, so that MLOG_FILE_CREATE2 can be written for undo tablespaces (without .ibd file suffix) for a nonzero page number. os_file_truncate(): Add the parameter allow_shrink=false so that undo tablespaces can actually be shrunk using this function. fil_name_parse(): For undo tablespace truncation, buffer MLOG_FILE_CREATE2 in truncated_undo_spaces[]. recv_read_in_area(): Avoid reading pages for which no redo log records remain buffered, after recv_addr_trim() removed them. trx_rseg_header_create(): Add a FIXME comment that we could write much less redo log. trx_undo_truncate_tablespace(): Reinitialize the undo tablespace in a single mini-transaction, which will be flushed to the redo log before the file size is trimmed. recv_addr_trim(): Discard any redo logs for pages that were logged after the new end of a file, before the truncation LSN. If the rec_list becomes empty, reduce n_addrs. After removing any affected records, actually truncate the file. recv_apply_hashed_log_recs(): Invoke recv_addr_trim() right before applying any log records. The undo tablespace files must be open at this point. buf_flush_or_remove_pages(), buf_flush_dirty_pages(), buf_LRU_flush_or_remove_pages(): Add a parameter for specifying the number of the first page to flush or remove (default 0). trx_purge_initiate_truncate(): Remove the log checkpoints, the extra logging, and some unnecessary crash points. Merge the code from trx_undo_truncate_tablespace(). First, flush all to-be-discarded pages (beyond the new end of the file), then trim the space->size to make the page allocation deterministic. At the only remaining crash injection point, flush the redo log, so that the recovery can be tested.	7 years ago
Thirunarayanan Balathandayuthapani	8b0d4cff07	MDEV-15855 Deadlock between purge thread and DDL statement Problem: ======== Truncate operation holds MDL on the table (t1) and tries to acquire InnoDB dict_operation_lock. Purge holds dict_operation_lock and tries to acquire MDL on the table (t1) to evaluate virtual column expressions for indexed virtual columns. It leads to deadlock of purge and truncate table (DDL). Solution: ========= If purge tries to acquire MDL on the table then it should do the following: i) Purge should release all innodb latches (including dict_operation_lock) before acquiring metadata lock on the table. ii) After acquiring metadata lock on the table, it should check whether the table was dropped or renamed. If the table is dropped then purge should ignore the undo log record. If the table is renamed then it should release the old MDL and acquire MDL on the new name. iii) Once purge acquires MDL, it should use the SQL table handle for all the remaining virtual index for the purge record. purge_node_t: Introduce new virtual column information to know whether the MDL was acquired successfully. This is joint work with Marko Mäkelä.	7 years ago
Marko Mäkelä	fdb9e66fee	Implement a parameter for wait_all_purged.inc	7 years ago
Marko Mäkelä	44314c768f	MDEV-15165 InnoDB purge for index on virtual column is trying to access an incomplete record The algorithm change is based on a MySQL 8.0 fix for BUG #26818787: ASSERTION: DATA0DATA.IC:430:TUPLE by Krzysztof Kapuścik `ee606e62bb` If a record had been inserted in place of a delete-marked purgeable record by modifying that record, and purge was accessing that record before the off-page columns were written, row_build_index_entry() would have returned NULL, causing a crash. row_vers_non_virtual_fields_equal(): Check whether all non-virtual fields of an index are equal. Replaces row_vers_non_vc_match(). A more complex version of this function was called row_vers_non_vc_index_entry_match() in the MySQL 8.0 fix. row_vers_impl_x_locked_low(): This change is not directly related to the reported problem, but apparently to the removal of the function row_vers_non_vc_match(). This function checks if a secondary index record was modified by a transaction that has not been committed yet. For comparing the non-virtual columns, construct a secondary index tuple from the table row. row_vers_vc_matches_cluster(): Replace row_vers_non_vc_match() with code that is equivalent to the row_vers_non_vc_index_entry_match() in the MySQL 8.0 fix. Also, deduplicate some code by using goto.	8 years ago
Marko Mäkelä	7a9fee9853	MDEV-13568 gcol.innodb_virtual_debug_purge failed in buildbot with wrong result The InnoDB background tasks can modify tables while LOCK TABLES...WRITE is in effect. The purge of InnoDB history always worked like this in MariaDB, but in MySQL 5.7 it sometimes yields to LOCK TABLES. Also, make gcol.innodb_virtual_index run the purge for an UPDATE before DROP TABLE is executed.	8 years ago
Marko Mäkelä	03f3bdce5f	Deterministically wait for purge using wait_all_purged.inc	8 years ago
Sergei Golubchik	8b3b6dc377	test how MDL blocks InnoDB purge it must, because purge opens a table that might be being altered right now	9 years ago
Sergei Golubchik	1cae1af6f9	MDEV-5800 InnoDB support for indexed vcols * remove old 5.2+ InnoDB support for virtual columns * enable corresponding parts of the innodb-5.7 sources * copy corresponding test cases from 5.7 * copy detailed Alter_inplace_info::HA_ALTER_FLAGS flags from 5.7 - and more detailed detection of changes in fill_alter_inplace_info() * more "innodb compatibility hooks" in sql_class.cc to - create/destroy/reset a THD (used by background purge threads) - find a prelocked table by name - open a table (from a background purge thread) * different from 5.7: - new service thread "thd_destructor_proxy" to make sure all THDs are destroyed at the correct point in time during the server shutdown - proper opening/closing of tables for vcol evaluations in + FK checks (use already opened prelocked tables) + purge threads (open the table, MDLock it, add it to tdc, close when not needed) - cache open tables in vc_templ - avoid unnecessary allocations, reuse table->record[0] and table->s->default_values - not needed in 5.7, because it overcalculates: + tell the server to calculate vcols for an on-going inline ADD INDEX + calculate vcols for correct error messages * update other engines (mroonga/tokudb) accordingly	9 years ago

14 Commits (8e3d85e112964c4f7c2d07710a3c8ee9d5041685)