mariadb

Commit Graph

Author	SHA1	Message	Date
Thirunarayanan Balathandayuthapani	4b80c11f52	MDEV-15250 UPSERT during ALTER TABLE results in 'Duplicate entry' error for alter - InnoDB DDL results in `Duplicate entry' if concurrent DML throws duplicate key error. The following scenario explains the problem connection con1: ALTER TABLE t1 FORCE; connection con2: INSERT INTO t1(pk, uk) VALUES (2, 2), (3, 2); In connection con2, InnoDB throws the 'DUPLICATE KEY' error because of unique index. Alter operation will throw the error when applying the concurrent DML log. - Inserting the duplicate key for unique index logs the insert operation for online ALTER TABLE. When insertion fails, transaction does rollback and it leads to logging of delete operation for online ALTER TABLE. While applying the insert log entries, alter operation encounters 'DUPLICATE KEY' error. - To avoid the above fake duplicate scenario, InnoDB should not write any log for online ALTER TABLE before DML transaction commit. - User thread which does DML can apply the online log if InnoDB ran out of online log and index is marked as completed. Set online log error if apply phase encountered any error. It can also clear all other indexes log, marks the newly added indexes as corrupted. - Removed the old online code which was a part of DML operations commit_inplace_alter_table() : Does apply the online log for the last batch of secondary index log and does frees the log for the completed index. trx_t::apply_online_log: Set to true while writing the undo log if the modified table has active DDL trx_t::apply_log(): Apply the DML changes to online DDL tables dict_table_t::is_active_ddl(): Returns true if the table has an active DDL dict_index_t::online_log_make_dummy(): Assign dummy value for clustered index online log to indicate the secondary indexes are being rebuild. dict_index_t::online_log_is_dummy(): Check whether the online log has dummy value ha_innobase_inplace_ctx::log_failure(): Handle the apply log failure for online DDL transaction row_log_mark_other_online_index_abort(): Clear out all other online index log after encountering the error during row_log_apply() row_log_get_error(): Get the error happened during row_log_apply() row_log_online_op(): Does apply the online log if index is completed and ran out of memory. Returns false if apply log fails UndorecApplier: Introduced a class to maintain the undo log record, latched undo buffer page, parse the undo log record, maintain the undo record type, info bits and update vector UndorecApplier::get_old_rec(): Get the correct version of the clustered index record that was modified by the current undo log record UndorecApplier::clear_undo_rec(): Clear the undo log related information after applying the undo log record UndorecApplier::log_update(): Handle the update, delete undo log and apply it on online indexes UndorecApplier::log_insert(): Handle the insert undo log and apply it on online indexes UndorecApplier::is_same(): Check whether the given roll pointer is generated by the current undo log record information trx_t::rollback_low(): Set apply_online_log for the transaction after partially rollbacked transaction has any active DDL prepare_inplace_alter_table_dict(): After allocating the online log, InnoDB does create fulltext common tables. Fulltext index doesn't allow the index to be online. So removed the dead code of online log removal Thanks to Marko Mäkelä for providing the initial prototype and Matthias Leich for testing the issue patiently.	4 years ago
Marko Mäkelä	d1edb011ee	Cleanup: Remove os0thread Let us use the common pthread_t wrapper for Microsoft Windows. This fixes up commit `dbe941e06f`	4 years ago
Marko Mäkelä	840bab858b	MDEV-28289 fts_optimize_sync_table() is acquiring dict_sys.latch while holding it dict_acquire_mdl_shared(): Invoke the correct variant dict_table_t::parse_name<true>() also when trylock=true, that is, we are being called from fts_optimize_sync_table(). Ever since commit `82b7c561b7` (MDEV-24258) this code was prone to a hang. If another thread requested an exclusive dict_sys.latch between the time dict_acquire_mdl_shared<trylock=true>() acquired a shared dict_sys.latch and dict_table_t::parse_name<false>() was trying to acquire another shared dict_sys.latch, InnoDB would get into an infinite livelock of threads, because the futex-based srw-lock implementation prioritizes exclusive latch requests. dict_table_t::parse_name(): Rename the template parameter dict_locked into dict_frozen.	4 years ago
Eugene Kosov	e128d852e8	MDEV-27272 Crash on EXPORT/IMPORT tablespace with column added in the middle dict_index_t::reconstruct_fields(): add input validation by replacing some assertions handle_instant_metadata(): fix nullptr dereference	4 years ago
Vladislav Vaintroub	47e18af906	MDEV-27494 Rename .ic files to .inl	4 years ago
Marko Mäkelä	1f02280904	MDEV-26769 InnoDB does not support hardware lock elision This implements memory transaction support for: * Intel Restricted Transactional Memory (RTM), also known as TSX-NI (Transactional Synchronization Extensions New Instructions) * POWER v2.09 Hardware Trace Monitor (HTM) on GNU/Linux transactional_lock_guard, transactional_shared_lock_guard: RAII lock guards that try to elide the lock acquisition when transactional memory is available. buf_pool.page_hash: Try to elide latches whenever feasible. Related to the InnoDB change buffer and ROW_FORMAT=COMPRESSED tables, this is not always possible. In buf_page_get_low(), memory transactions only work reasonably well for validating a guessed block address. TMLockGuard, TMLockTrxGuard, TMLockMutexGuard: RAII lock guards that try to elide lock_sys.latch and related latches.	4 years ago
Eugene Kosov	5b0a76078a	MDEV-26621 assertion failue "index->table->persistent_autoinc" in /storage/innobase/btr/btr0btr.cc during IMPORT dict_index_t::clear_instant_alter(): when searhing for an AUTO_INCREMENT column don't skip the beginning of the list because the field can be at the beginning of the list	4 years ago
Marko Mäkelä	277ba134ad	MDEV-26467: Avoid futile spin loops Typically, index_lock and fil_space_t::latch will be held for a longer time than the spin loop in latch acquisition would be waiting for. Let us avoid spin loops for those as well as dict_sys.latch, which could be held in exclusive mode for a longer time (while loading metadata into the buffer pool and the dictionary cache). Performance testing on a dual Intel Xeon E5-2630 v4 (2 NUMA nodes) suggests that the buffer pool page latch (block_lock) benefits from a spin loop in both read-only and read-write workloads where the working set is slightly larger than the buffer pool. Presumably, most contention would occur on leaf page latches. Contention on upper level pages in the buffer pool should intuitively last longer. We introduce srw_spin_lock and srw_spin_mutex to allow users of srw_lock or srw_mutex to opt in for the spin loop. On Microsoft Windows, a spin loop variant was and will not be available; srw_mutex and srw_lock will simply wrap SRWLOCK. That is, on Microsoft Windows, the parameters innodb_sync_spin_loops and innodb_spin_wait_delay will only affect block_lock.	4 years ago
Marko Mäkelä	45a05fda27	MDEV-25919: Replace dict_table_t::stats_bg_flag with MDL The purpose of dict_table_t::stats_bg_flag was to prevent race conditions between DDL operations and a background thread that updates persistent statistics for InnoDB tables. Now that with the parent commit, we started to acquire a shared meta-data lock (MDL) on the InnoDB persistent statistics tables in background tasks that access them, we may easily acquire MDL on the table for which the statistics are being updated. This will by design prevent race conditions with any DDL operations on that table, and the stats_bg_flag may be removed. dict_stats_process_entry_from_recalc_pool(): Complete rewrite. During the processing, retain the entry in recalc_pool, so that dict_stats_recalc_pool_del() will be able to request deletion of the entry, or delete the entry if its caller is holding MDL_EXCLUSIVE while we are waiting for MDL. recalc_pool: In addition to the table ID, store a state for inter-thread communication, so that dict_stats_recalc_pool_del() can wait until all processing is finished. Reviewed by: Thirunarayanan Balathandayuthapani	4 years ago
Marko Mäkelä	c5fd9aa562	MDEV-25919: Lock tables before acquiring dict_sys.latch In commit `1bd681c8b3` (MDEV-25506 part 3) we introduced a "fake instant timeout" when a transaction would wait for a table or record lock while holding dict_sys.latch. This prevented a deadlock of the server but could cause bogus errors for operations on the InnoDB persistent statistics tables. A better fix is to ensure that whenever a transaction is being executed in the InnoDB internal SQL parser (which will for now require dict_sys.latch to be held), it will already have acquired all locks that could be required for the execution. So, we will acquire the following locks upfront, before acquiring dict_sys.latch: (1) MDL on the affected user table (acquired by the SQL layer) (2) If applicable (not for RENAME TABLE): InnoDB table lock (3) If persistent statistics are going to be modified: (3.a) MDL_SHARED on mysql.innodb_table_stats, mysql.innodb_index_stats (3.b) exclusive table locks on the statistics tables (4) Exclusive table locks on the InnoDB data dictionary tables (not needed in ANALYZE TABLE and the like) Note: Acquiring exclusive locks on the statistics tables may cause more locking conflicts between concurrent DDL operations. Notably, RENAME TABLE will lock the statistics tables even if no persistent statistics are enabled for the table. DROP DATABASE will only acquire locks on statistics tables if persistent statistics are enabled for the tables on which the SQL layer is invoking ha_innobase::delete_table(). For any "garbage collection" in innodb_drop_database(), a timeout while acquiring locks on the statistics tables will result in any statistics not being deleted for any tables that the SQL layer did not know about. If innodb_defragment=ON, information may be written to the statistics tables even for tables for which InnoDB persistent statistics are disabled. But, DROP TABLE will no longer attempt to delete that information if persistent statistics are not enabled for the table. This change should also fix the hangs related to InnoDB persistent statistics and STATS_AUTO_RECALC (MDEV-15020) as well as a bug that running ALTER TABLE on the statistics tables concurrently with running ALTER TABLE on InnoDB tables could cause trouble. lock_rec_enqueue_waiting(), lock_table_enqueue_waiting(): Do not issue a fake instant timeout error when the transaction is holding dict_sys.latch. Instead, assert that the dict_sys.latch is never being held here. lock_sys_tables(): A new function to acquire exclusive locks on all dictionary tables, in case DROP TABLE or similar operation is being executed. Locking non-hard-coded tables is optional to avoid a crash in row_merge_drop_temp_indexes(). The SYS_VIRTUAL table was introduced in MySQL 5.7 and MariaDB Server 10.2. Normally, we require all these dictionary tables to exist before executing any DDL, but the function row_merge_drop_temp_indexes() is an exception. When upgrading from MariaDB Server 10.1 or MySQL 5.6 or earlier, the table SYS_VIRTUAL would not exist at this point. ha_innobase::commit_inplace_alter_table(): Invoke log_write_up_to() while not holding dict_sys.latch. dict_sys_t::remove(), dict_table_close(): No longer try to drop index stubs that were left behind by aborted online ADD INDEX. Such indexes should be dropped from the InnoDB data dictionary by row_merge_drop_indexes() as part of the failed DDL operation. Stubs for aborted indexes may only be left behind in the data dictionary cache. dict_stats_fetch_from_ps(): Use a normal read-only transaction. ha_innobase::delete_table(), ha_innobase::truncate(), fts_lock_table(): While waiting for purge to stop using the table, do not hold dict_sys.latch. ha_innobase::delete_table(): Implement a work-around for the rollback of ALTER TABLE...ADD PARTITION. MDL_EXCLUSIVE would not be held if ALTER TABLE hits lock_wait_timeout while trying to upgrade the MDL due to a conflicting LOCK TABLES, such as in the first ALTER TABLE in the test case of Bug#53676 in parts.partition_special_innodb. Therefore, we must explicitly stop purge, because it would not be stopped by MDL. dict_stats_func(), btr_defragment_chunk(): Allocate a THD so that we can acquire MDL on the InnoDB persistent statistics tables. mysqltest_embedded: Invoke ha_pre_shutdown() before free_used_memory() in order to avoid ASAN heap-use-after-free related to acquire_thd(). trx_t::dict_operation_lock_mode: Changed the type to bool. row_mysql_lock_data_dictionary(), row_mysql_unlock_data_dictionary(): Implemented as macros. rollback_inplace_alter_table(): Apply an infinite timeout to lock waits. innodb_thd_increment_pending_ops(): Wrapper for thd_increment_pending_ops(). Never attempt async operation for InnoDB background threads, such as the trx_t::commit() in dict_stats_process_entry_from_recalc_pool(). lock_sys_t::cancel(trx_t*): Make dictionary transactions immune to KILL. lock_wait(): Make dictionary transactions immune to KILL, and to lock wait timeout when waiting for locks on dictionary tables. parts.partition_special_innodb: Use lock_wait_timeout=0 to instantly get ER_LOCK_WAIT_TIMEOUT. main.mdl: Filter out MDL on InnoDB persistent statistics tables Reviewed by: Thirunarayanan Balathandayuthapani	4 years ago
Marko Mäkelä	82b7c561b7	MDEV-24258 Merge dict_sys.mutex into dict_sys.latch In the parent commit, dict_sys.latch could theoretically have been replaced with a mutex. But, we can do better and merge dict_sys.mutex into dict_sys.latch. Generally, every occurrence of dict_sys.mutex_lock() will be replaced with dict_sys.lock(). The PERFORMANCE_SCHEMA instrumentation for dict_sys_mutex will be removed along with dict_sys.mutex. The dict_sys.latch will remain instrumented as dict_operation_lock. Some use of dict_sys.lock() will be replaced with dict_sys.freeze(), which we will reintroduce for the new shared mode. Most notably, concurrent table lookups are possible as long as the tables are present in the dict_sys cache. In particular, this will allow more concurrency among InnoDB purge workers. Because dict_sys.mutex will no longer 'throttle' the threads that purge InnoDB transaction history, a performance degradation may be observed unless innodb_purge_threads=1. The table cache eviction policy will become FIFO-like, similar to what happened to fil_system.LRU in commit `45ed9dd957`. The name of the list dict_sys.table_LRU will become somewhat misleading; that list contains tables that may be evicted, even though the eviction policy no longer is least-recently-used but first-in-first-out. (Note: Tables can never be evicted as long as locks exist on them or the tables are in use by some thread.) As demonstrated by the test perfschema.sxlock_func, there will be less contention on dict_sys.latch, because some previous use of exclusive latches will be replaced with shared latches. fts_parse_sql_no_dict_lock(): Replaced with pars_sql(). fts_get_table_name_prefix(): Merged to fts_optimize_create(). dict_stats_update_transient_for_index(): Deduplicated some code. ha_innobase::info_low(), dict_stats_stop_bg(): Use a combination of dict_sys.latch and table->stats_mutex_lock() to cover the changes of BG_STAT_SHOULD_QUIT, because the flag is being read in dict_stats_update_persistent() while not holding dict_sys.latch. row_discard_tablespace_for_mysql(): Protect stats_bg_flag by exclusive dict_sys.latch, like most other code does. row_quiesce_table_has_fts_index(): Remove unnecessary mutex acquisition. FLUSH TABLES...FOR EXPORT is protected by MDL. row_import::set_root_by_heuristic(): Remove unnecessary mutex acquisition. ALTER TABLE...IMPORT TABLESPACE is protected by MDL. row_ins_sec_index_entry_low(): Replace a call to dict_set_corrupted_index_cache_only(). Reads of index->type were not really protected by dict_sys.mutex, and writes (flagging an index corrupted) should be extremely rare. dict_stats_process_entry_from_defrag_pool(): Only freeze the dictionary, do not lock it exclusively. dict_stats_wait_bg_to_stop_using_table(), DICT_BG_YIELD: Remove trx. We can simply invoke dict_sys.unlock() and dict_sys.lock() directly. dict_acquire_mdl_shared()<trylock=false>: Assert that dict_sys.latch is only held in shared more, not exclusive mode. Only acquire it in exclusive mode if the table needs to be loaded to the cache. dict_sys_t::acquire(): Remove. Relocating elements in dict_sys.table_LRU would require holding an exclusive latch, which we want to avoid for performance reasons. dict_sys_t::allow_eviction(): Add the table first to dict_sys.table_LRU, to compensate for the removal of dict_sys_t::acquire(). This function is only invoked by INFORMATION_SCHEMA.INNODB_SYS_TABLESTATS. dict_table_open_on_id(), dict_table_open_on_name(): If dict_locked=false, try to acquire dict_sys.latch in shared mode. Only acquire the latch in exclusive mode if the table is not found in the cache. Reviewed by: Thirunarayanan Balathandayuthapani	4 years ago
Marko Mäkelä	ca89489716	MDEV-26383 fixup: Consistently protect freed_indexes with autoinc_mutex To avoid potential race conditions between concurrent access to dict_table_t::freed_indexes, let us consistently use dict_table_t::autoinc_mutex. dict_table_remove_from_cache_low(): To avoid extensive hold time of table->autoinc_mutex, unconditionally free the FTS data structures.	4 years ago
Thirunarayanan Balathandayuthapani	08e5a3d2e3	MDEV-26383 ASAN heap-use-after-free failure in btr_search_lazy_free Problem: ======= The last AHI page for two indexes of an dropped table is being freed at the same time by two threads. One thread frees the table heap and other thread tries to access table heap again. It leads to asan failure in btr_search_lazy_free(). Solution: ======== InnoDB uses autoinc_mutex to avoid the race condition in btr_search_lazy_free()	4 years ago
Marko Mäkelä	e305493b1c	MDEV-21175 follow-up: Remove redundant locking; rely on MDL Before entering DML or DDL execution in the storage engine, the SQL layer will have acquired metadata lock (MDL) on the current table name as well as the names of FOREIGN KEY (grand)child tables (that is, tables whose REFERENCES clauses point to the current table). The MDL prevents any metadata changes to these tables, such as RENAME, TRUNCATE, DROP, ALTER. While the MDL on the current table prevents dict_table_t::foreign_set from being modified, it does not prevent the table metadata that the stored pointers are pointing to from being modified. The MDL on the child tables will prevent both dict_table_t::referenced_set as well as the pointed child table metadata from being modified. wsrep_row_upd_index_is_foreign(): Do not unnecessarily acquire the data dictionary latch if Galera replication is not enabled. ha_innobase::can_switch_engines(): Rely on MDL. We are not dereferencing any pointers stored in the sets. row_mysql_freeze_data_dictionary(), row_mysql_unfreeze_data_dictionary(): Remove. row_update_for_mysql(): Call init_fts_doc_id_for_ref() only once. In ALTER TABLE...IMPORT TABLESPACE and FLUSH TABLES...FOR EXPORT the SQL layer is protecting the current table with MDL. We do not need InnoDB latches.	4 years ago
Marko Mäkelä	1bd681c8b3	MDEV-25506 (3 of 3): Do not delete .ibd files before commit This is a complete rewrite of DROP TABLE, also as part of other DDL, such as ALTER TABLE, CREATE TABLE...SELECT, TRUNCATE TABLE. The background DROP TABLE queue hack is removed. If a transaction needs to drop and create a table by the same name (like TRUNCATE TABLE does), it must first rename the table to an internal #sql-ib name. No committed version of the data dictionary will include any #sql-ib tables, because whenever a transaction renames a table to a #sql-ib name, it will also drop that table. Either the rename will be rolled back, or the drop will be committed. Data files will be unlinked after the transaction has been committed and a FILE_RENAME record has been durably written. The file will actually be deleted when the detached file handle returned by fil_delete_tablespace() will be closed, after the latches have been released. It is possible that a purge of the delete of the SYS_INDEXES record for the clustered index will execute fil_delete_tablespace() concurrently with the DDL transaction. In that case, the thread that arrives later will wait for the other thread to finish. HTON_TRUNCATE_REQUIRES_EXCLUSIVE_USE: A new handler flag. ha_innobase::truncate() now requires that all other references to the table be released in advance. This was implemented by Monty. ha_innobase::delete_table(): If CREATE TABLE..SELECT is detected, we will "hijack" the current transaction, drop the table in the current transaction and commit the current transaction. This essentially fixes MDEV-21602. There is a FIXME comment about making the check less failure-prone. ha_innobase::truncate(), ha_innobase::delete_table(): Implement a fast path for temporary tables. We will no longer allow temporary tables to use the adaptive hash index. dict_table_t::mdl_name: The original table name for the purpose of acquiring MDL in purge, to prevent a race condition between a DDL transaction that is dropping a table, and purge processing undo log records of DML that had executed before the DDL operation. For #sql-backup- tables during ALTER TABLE...ALGORITHM=COPY, the dict_table_t::mdl_name will differ from dict_table_t::name. dict_table_t::parse_name(): Use mdl_name instead of name. dict_table_rename_in_cache(): Update mdl_name. For the internal FTS_ tables of FULLTEXT INDEX, purge would acquire MDL on the FTS_ table name, but not on the main table, and therefore it would be able to run concurrently with a DDL transaction that is dropping the table. Previously, the DROP TABLE queue hack prevented a race between purge and DDL. For now, we introduce purge_sys.stop_FTS() to prevent purge from opening any table, while a DDL transaction that may drop FTS_ tables is in progress. The function fts_lock_table(), which will be invoked before the dictionary is locked, will wait for purge to release any table handles. trx_t::drop_table_statistics(): Drop statistics for the table. This replaces dict_stats_drop_index(). We will drop or rename persistent statistics atomically as part of DDL transactions. On lock conflict for dropping statistics, we will fail instantly with DB_LOCK_WAIT_TIMEOUT, because we will be holding the exclusive data dictionary latch. trx_t::commit_cleanup(): Separated from trx_t::commit_in_memory(). Relax an assertion around fts_commit() and allow DB_LOCK_WAIT_TIMEOUT in addition to DB_DUPLICATE_KEY. The call to fts_commit() is entirely misplaced here and may obviously break the consistency of transactions that affect FULLTEXT INDEX. It needs to be fixed separately. dict_table_t::n_foreign_key_checks_running: Remove (MDEV-21175). The counter was a work-around for missing meta-data locking (MDL) on the SQL layer, and not really needed in MariaDB. ER_TABLE_IN_FK_CHECK: Replaced with ER_UNUSED_28. HA_ERR_TABLE_IN_FK_CHECK: Remove. row_ins_check_foreign_constraints(): Do not acquire dict_sys.latch either. The SQL-layer MDL will protect us. This was reviewed by Thirunarayanan Balathandayuthapani and tested by Matthias Leich.	5 years ago
Marko Mäkelä	49e2c8f0a6	MDEV-25743: Unnecessary copying of table names in InnoDB dictionary Many InnoDB data dictionary cache operations require that the table name be copied so that it will be NUL terminated. (For example, SYS_TABLES.NAME is not guaranteed to be NUL-terminated.) dict_table_t::is_garbage_name(): Check if a name belongs to the background drop table queue. dict_check_if_system_table_exists(): Remove. dict_sys_t::load_sys_tables(): Load the non-hard-coded system tables SYS_FOREIGN, SYS_FOREIGN_COLS, SYS_VIRTUAL on startup. dict_sys_t::create_or_check_sys_tables(): Replaces dict_create_or_check_foreign_constraint_tables() and dict_create_or_check_sys_virtual(). dict_sys_t::load_table(): Replaces dict_table_get_low() and dict_load_table(). dict_sys_t::find_table(): Renamed from get_table(). dict_sys_t::sys_tables_exist(): Check whether all the non-hard-coded tables SYS_FOREIGN, SYS_FOREIGN_COLS, SYS_VIRTUAL exist. trx_t::has_stats_table_lock(): Moved to dict0stats.cc. Some error messages will now report table names in the internal databasename/tablename format, instead of `databasename`.`tablename`.	5 years ago
Monty	7762ee5dbe	MDEV-25180 Atomic ALTER TABLE MDEV-25604 Atomic DDL: Binlog event written upon recovery does not have default database The purpose of this task is to ensure that ALTER TABLE is atomic even if the MariaDB server would be killed at any point of the alter table. This means that either the ALTER TABLE succeeds (including that triggers, the status tables and the binary log are updated) or things should be reverted to their original state. If the server crashes before the new version is fully up to date and commited, it will revert to the original table and remove all temporary files and tables. If the new version is commited, crash recovery will use the new version, and update triggers, the status tables and the binary log. The one execption is ALTER TABLE .. RENAME .. where no changes are done to table definition. This one will work as RENAME and roll back unless the whole statement completed, including updating the binary log (if enabled). Other changes: - Added handlerton->check_version() function to allow the ddl recovery code to check, in case of inplace alter table, if the table in the storage engine is of the new or old version. - Added handler->table_version() so that an engine can report the current version of the table. This should be changed each time the table definition changes. - Added ha_signal_ddl_recovery_done() and handlerton::signal_ddl_recovery_done() to inform all handlers when ddl recovery has been done. (Needed by InnoDB). - Added handlerton call inplace_alter_table_committed, to signal engine that ddl_log has been closed for the alter table query. - Added new handerton flag HTON_REQUIRES_NOTIFY_TABLEDEF_CHANGED_AFTER_COMMIT to signal when we should call hton->notify_tabledef_changed() during mysql_inplace_alter_table. This was required as MyRocks and InnoDB needed the call at different times. - Added function server_uuid_value() to be able to generate a temporary xid when ddl recovery writes the query to the binary log. This is needed to be able to handle crashes during ddl log recovery. - Moved freeing of the frm definition to end of mysql_alter_table() to remove duplicate code and have a common exit strategy. ------- InnoDB part of atomic ALTER TABLE (Implemented by Marko Mäkelä) innodb_check_version(): Compare the saved dict_table_t::def_trx_id to determine whether an ALTER TABLE operation was committed. We must correctly recover dict_table_t::def_trx_id for this to work. Before purge removes any trace of DB_TRX_ID from system tables, it will make an effort to load the user table into the cache, so that the dict_table_t::def_trx_id can be recovered. ha_innobase::table_version(): return garbage, or the trx_id that would be used for committing an ALTER TABLE operation. In InnoDB, table names starting with #sql-ib will remain special: they will be dropped on startup. This may be revisited later in MDEV-18518 when we implement proper undo logging and rollback for creating or dropping multiple tables in a transaction. Table names starting with #sql will retain some special meaning: dict_table_t::parse_name() will not consider such names for MDL acquisition, and dict_table_rename_in_cache() will treat such names specially when handling FOREIGN KEY constraints. Simplify InnoDB DROP INDEX. Prevent purge wakeup To ensure that dict_table_t::def_trx_id will be recovered correctly in case the server is killed before ddl_log_complete(), we will block the purge of any history in SYS_TABLES, SYS_INDEXES, SYS_COLUMNS between ha_innobase::commit_inplace_alter_table(commit=true) (purge_sys.stop_SYS()) and purge_sys.resume_SYS(). The completion callback purge_sys.resume_SYS() must be between ddl_log_complete() and MDL release. -------- MyRocks support for atomic ALTER TABLE (Implemented by Sergui Petrunia) Implement these SE API functions: - ha_rocksdb::table_version() - hton->check_version = rocksdb_check_versionMyRocks data dictionary now stores table version for each table. (Absence of table version record is interpreted as table_version=0, that is, which means no upgrade changes are needed) - For inplace alter table of a partitioned table, call the underlying handlerton when checking if the table is ok. This assumes that the partition engine commits all changes at once.	5 years ago
Marko Mäkelä	895c126a23	MDEV-15528 fixup: Remove references to background scrubbing	5 years ago
Marko Mäkelä	c366845a0b	MDEV-25691: Simplify handlerton::drop_database for InnoDB The implementation of handlerton::drop_database in InnoDB is unnecessarily complex. The minimal implementation should check that no conflicting locks or references exist on the tables, delete all table metadata in a single transaction, and finally delete the tablespaces. Note: DROP DATABASE will delete each individual table that the SQL layer knows about, one table per transaction. The handlerton::drop_database is basically a final cleanup step for removing any garbage that could have been left behind in InnoDB due to some bug, or not having atomic DDL in the past. hash_node_t: Remove. Use the proper data type name in pointers. dict_drop_index_tree(): Do not take the table as a parameter. Instead, return the tablespace ID if the tablespace should be dropped (we are dropping a clustered index tree). fil_delete_tablespace(), fil_system_t::detach(): Return a single detached file handle. Multi-file tablespaces cannot be deleted via this interface. ha_innobase::delete_table(): Remove a work-around for non-atomic DDL and do not try to drop tables with similar-looking name. innodb_drop_database(): Complete rewrite. innobase_drop_database(), dict_get_first_table_name_in_db(), row_drop_database_for_mysql(), drop_all_foreign_keys_in_db(): Remove. row_purge_remove_clust_if_poss_low(), row_undo_ins_remove_clust_rec(): If the tablespace is to be deleted, try to evict the table definition from the cache. Failing that, set dict_table_t::space to nullptr. lock_release_on_rollback(): On the rollback of CREATE TABLE, release all locks that the transaction had on the table, to avoid heap-use-after-free.	5 years ago
Marko Mäkelä	b9a2e4609f	MDEV-25594: Assertion failure in DeadlockChecker::check_and_resolve() ha_innobase::index_read(): If an autocommit non-locking transaction was already started, refuse to access a SPATIAL INDEX. Once a non-locking autocommit transaction has started, it must remain in that mode (not acquire any locks). This should fix one cause of the assertion failure that would occur in DeadlockChecker::check_and_resolve() under heavy load, presumably due to concurrent execution of trx_commit_in_memory().	5 years ago
Marko Mäkelä	54e2e70194	MDEV-25524 heap-use-after-free in fil_space_t::rename() In commit `91599701d0` (MDEV-25312) some recovery code for TRUNCATE TABLE was broken causing a regression in a case where undo log for a RENAME TABLE operation had been durably written but the tablespace had not been renamed yet. row_rename_table_for_mysql(): Add a DEBUG_SYNC point for the test case, and simplify the logic and trim the error messages. fil_space_t::rename(): Simplify the operation. Merge the necessary part of fil_rename_tablespace_check(). If there is no change to the file name, do nothing. dict_table_t::rename_tablespace(): Refactored from dict_table_rename_in_cache(). row_undo_ins_parse_undo_rec(): On rolling back TRX_UNDO_RENAME_TABLE, invoke dict_table_t::rename_tablespace() even if the table name matches. os_file_rename_func(): Temporarily relax an assertion that would fail during the recovery in the test innodb.truncate_crash.	5 years ago
Thirunarayanan Balathandayuthapani	b862377c3e	MDEV-25503 InnoDB hangs on startup during recovery InnoDB startup hangs if a DDL transaction needs to be rolled back and a recovered transaction on statistics tables exists. In that case, InnoDB should rollback the transaction which holds locks on innodb_table_stats or innodb_index_stats during trx_rollback_or_clean_recovered().	5 years ago
Marko Mäkelä	b8c8692fd9	MDEV-24620 ASAN heap-buffer-overflow in btr_pcur_restore_position() Between btr_pcur_store_position() and btr_pcur_restore_position() it is possible that purge empties a table and enlarges index->n_core_fields and index->n_core_null_bytes. Therefore, we must cache index->n_core_fields in btr_pcur_t::old_n_core_fields so that btr_pcur_t::old_rec can be parsed correctly. Unfortunately, this is a huge change, because we will replace "bool leaf" parameters with "ulint n_core" (passing index->n_core_fields, or 0 for non-leaf pages). For special cases where we know that index->is_instant() cannot hold, we may also pass index->n_fields.	5 years ago
Thirunarayanan Balathandayuthapani	cf2c6b7f8d	MDEV-24971 InnoDB access freed virtual column after rollback of secondary index Problem: ======== InnoDB fails to clean the index stub if it fails to add the virtual index which contains new virtual column. But it clears the newly virtual column from index in clear_added_indexes() during inplace_alter_table. On commit, InnoDB evicts and reload the table. In case of rollback, it doesn't happen. InnoDB clears the ABORTED index while opening the table or doing the DDL. In the mean time, InnoDB can access the dropped virtual index columns while creating prebuilt or rollback of concurrent DML. Solution: ========== (1) InnoDB should maintain newly added virtual column while rollbacking the newly added virtual index. (2) InnoDB must not defer the index removal if the alter table is executed with LOCK=EXCLUSIVE. (3) For LOCK=SHARED, InnoDB should check whether the table has any other transaction lock other than alter transaction before deferring the index stub. Replaced has_new_v_col with dict_add_vcol_info in dict_index_t to indicate whether the index has any new virtual column. dict_index_t::has_new_v_col(): Returns whether the index has newly added virtual column, it doesn't say which columns are newly added virtual column ha_innobase_inplace_ctx::is_new_vcol(): Return whether the given column is added as a part of the current alter. ha_innobase_inplace_ctx::clean_new_vcol_index(): Copy the newly added virtual column to new_vcol_info in dict_index_t. Replace the column in the index fields with virtual column stored in new_vcol_info. dict_index_t::assign_new_v_col(): Store the number of virtual column added in index as a part of alter table. dict_index_t::get_n_new_vcol(): Get the number of newly added virtual column dict_index_t::assign_drop_v_col(): Allocate the memory for adding new virtual column in new_vcol_info. dict_index_t::add_drop_v_col(): Add the newly added virtual column in new_vcol_info. dict_table_t::has_lock_for_other_trx(): Whether the table has any other transaction lock than given transaction. row_merge_drop_indexes(): Add parameter alter_trx and check whether the table has any other lock than alter transaction.	5 years ago
Marko Mäkelä	1fde581237	MDEV-25315 Crash in SHOW ENGINE INNODB STATUS In commit `8ea923f55b` (MDEV-24818) when we optimized multi-statement INSERT into an empty table, we would sometimes wrongly enable bulk insert into a table that is actually already using row-level locking and undo logging. trx_has_lock_x(): New predicate, to check if the transaction of the current thread is holding an exclusive lock on a table. trx_undo_report_row_operation(): Only invoke trx_mod_table_time_t::start_bulk_insert() if trx_has_lock_x() holds.	5 years ago
Krunal Bauskar	33cf577ad8	MDEV-25029: Reduce dict_sys mutex contention for read-only workload In theory, read-only workload shouldn't have anything to do with dict_sys mutex. dict_sys mutex is meant to protect database metadata. But then why does dict_sys mutex shows up as part of a read-only workload? This workload needs to fetch stats for query processing and while reading these stats dict_sys mutex is taken. Is this really needed? No. For the traditional reasons, it was the default global mutex used. Based on 10.6 changes, flow can now use table->lock_mutex to protect update/access of these stats. table mutexes being table specific global contention arising out of dict_sys is reduced. Thanks to Marko Makela for his early suggestion around proposed alternative and review of the draft patch.	5 years ago
Marko Mäkelä	26d6224dd6	MDEV-20612: Enable concurrent lock_release() lock_release_try(): Try to release locks while only holding shared lock_sys.latch. lock_release(): If 5 attempts of lock_release_try() fail, proceed to acquire exclusive lock_sys.latch.	5 years ago
Marko Mäkelä	b08448de64	MDEV-20612: Partition lock_sys.latch We replace the old lock_sys.mutex (which was renamed to lock_sys.latch) with a combination of a global lock_sys.latch and table or page hash lock mutexes. The global lock_sys.latch can be acquired in exclusive mode, or it can be acquired in shared mode and another mutex will be acquired to protect the locks for a particular page or a table. This is inspired by mysql/mysql-server@1d259b87a63defa814e19a7534380cb43ee23c48 but the optimization of lock_release() will be done in the next commit. Also, we will interleave mutexes with the hash table elements, similar to how buf_pool.page_hash was optimized in commit `5155a300fa` (MDEV-22871). dict_table_t::autoinc_trx: Use Atomic_relaxed. dict_table_t::autoinc_mutex: Use srw_mutex in order to reduce the memory footprint. On 64-bit Linux or OpenBSD, both this and the new dict_table_t::lock_mutex should be 32 bits and be stored in the same 64-bit word. On Microsoft Windows, the underlying SRWLOCK is 32 or 64 bits, and on other systems, sizeof(pthread_mutex_t) can be much larger. ib_lock_t::trx_locks, trx_lock_t::trx_locks: Document the new rules. Writers must assert lock_sys.is_writer() \|\| trx->mutex_is_owner(). LockGuard: A RAII wrapper for acquiring a page hash table lock. LockGGuard: Like LockGuard, but when Galera Write-Set Replication is enabled, we must acquire all shards, for updating arbitrary trx_locks. LockMultiGuard: A RAII wrapper for acquiring two page hash table locks. lock_rec_create_wsrep(), lock_table_create_wsrep(): Special Galera conflict resolution in non-inlined functions in order to keep the common code paths shorter. lock_sys_t::prdt_page_free_from_discard(): Refactored from lock_prdt_page_free_from_discard() and lock_rec_free_all_from_discard_page(). trx_t::commit_tables(): Replaces trx_update_mod_tables_timestamp(). lock_release(): Let trx_t::commit_tables() invalidate the query cache for those tables that were actually modified by the transaction. Merge lock_check_dict_lock() to lock_release(). We must never release lock_sys.latch while holding any lock_sys_t::hash_latch. Failure to do that could lead to memory corruption if the buffer pool is resized between the time lock_sys.latch is released and the hash_latch is released.	5 years ago
Marko Mäkelä	b01d8e1a33	MDEV-20612: Replace lock_sys.mutex with lock_sys.latch For now, we will acquire the lock_sys.latch only in exclusive mode, that is, use it as a mutex. This is preparation for the next commit where we will introduce a less intrusive alternative, combining a shared lock_sys.latch with dict_table_t::lock_mutex or a mutex embedded in lock_sys.rec_hash, lock_sys.prdt_hash, or lock_sys.prdt_page_hash.	5 years ago
Marko Mäkelä	121d0f7f53	MDEV-20612: Speed up lock_table_other_has_incompatible() dict_table_t::n_lock_x_or_s: Keep track of LOCK_S or LOCK_X on the table. lock_table_other_has_incompatible(): In the likely case that no transaction is waiting for or holding LOCK_S or LOCK_X on the table, return early: conflicts cannot exist. This is based on the idea of Zhai Weixiang, who reported MySQL Bug #72948. lock_table_has_to_wait_in_queue(), lock_table_dequeue(): Extend the optimization, inspired by mysql/mysql-server@bb7191d6cbe47e15923143e194c03406cff9024b by Jakub Łopuszański.	5 years ago
Marko Mäkelä	3cef4f8f0f	MDEV-515 Reduce InnoDB undo logging for insert into empty table We implement an idea that was suggested by Michael 'Monty' Widenius in October 2017: When InnoDB is inserting into an empty table or partition, we can write a single undo log record TRX_UNDO_EMPTY, which will cause ROLLBACK to clear the table. For this to work, the insert into an empty table or partition must be covered by an exclusive table lock that will be held until the transaction has been committed or rolled back, or the INSERT operation has been rolled back (and the table is empty again), in lock_table_x_unlock(). Clustered index records that are covered by the TRX_UNDO_EMPTY record will carry DB_TRX_ID=0 and DB_ROLL_PTR=1<<55, and thus they cannot be distinguished from what MDEV-12288 leaves behind after purging the history of row-logged operations. Concurrent non-locking reads must be adjusted: If the read view was created before the INSERT into an empty table, then we must continue to imagine that the table is empty, and not try to read any records. If the read view was created after the INSERT was committed, then all records must be visible normally. To implement this, we introduce the field dict_table_t::bulk_trx_id. This special handling only applies to the very first INSERT statement of a transaction for the empty table or partition. If a subsequent statement in the transaction is modifying the initially empty table again, we must enable row-level undo logging, so that we will be able to roll back to the start of the statement in case of an error (such as duplicate key). INSERT IGNORE will continue to use row-level logging and locking, because implementing it would require the ability to roll back the latest row. Since the undo log that we write only allows us to roll back the entire statement, we cannot support INSERT IGNORE. We will introduce a handler::extra() parameter HA_EXTRA_IGNORE_INSERT to indicate to storage engines that INSERT IGNORE is being executed. In many test cases, we add an extra record to the table, so that during the 'interesting' part of the test, row-level locking and logging will be used. Replicas will continue to use row-level logging and locking until MDEV-24622 has been addressed. Likewise, this optimization will be disabled in Galera cluster until MDEV-24623 enables it. dict_table_t::bulk_trx_id: The latest active or committed transaction that initiated an insert into an empty table or partition. Protected by exclusive table lock and a clustered index leaf page latch. ins_node_t::bulk_insert: Whether bulk insert was initiated. trx_t::mod_tables: Use C++11 style accessors (emplace instead of insert). Unlike earlier, this collection will cover also temporary tables. trx_mod_table_time_t: Add start_bulk_insert(), end_bulk_insert(), is_bulk_insert(), was_bulk_insert(). trx_undo_report_row_operation(): Before accessing any undo log pages, invoke trx->mod_tables.emplace() in order to determine whether undo logging was disabled, or whether this is the first INSERT and we are supposed to write a TRX_UNDO_EMPTY record. row_ins_clust_index_entry_low(): If we are inserting into an empty clustered index leaf page, set the ins_node_t::bulk_insert flag for the subsequent trx_undo_report_row_operation() call. lock_rec_insert_check_and_lock(), lock_prdt_insert_check_and_lock(): Remove the redundant parameter 'flags' that can be checked in the caller. btr_cur_ins_lock_and_undo(): Simplify the logic. Correctly write DB_TRX_ID,DB_ROLL_PTR after invoking trx_undo_report_row_operation(). trx_mark_sql_stat_end(), ha_innobase::extra(HA_EXTRA_IGNORE_INSERT), ha_innobase::external_lock(): Invoke trx_t::end_bulk_insert() so that the next statement will not be covered by table-level undo logging. ReadView::changes_visible(trx_id_t) const: New accessor for the case where the trx_id_t is not read from a potentially corrupted index page but directly from the memory. In this case, we can skip a sanity check. row_sel(), row_sel_try_search_shortcut(), row_search_mvcc(): row_sel_try_search_shortcut_for_mysql(), row_merge_read_clustered_index(): Check dict_table_t::bulk_trx_id. row_sel_clust_sees(): Replaces lock_clust_rec_cons_read_sees(). lock_sec_rec_cons_read_sees(): Replaced with lower-level code. btr_root_page_init(): Refactored from btr_create(). dict_index_t::clear(), dict_table_t::clear(): Empty an index or table, for the ROLLBACK of an INSERT operation. ROW_T_EMPTY, ROW_OP_EMPTY: Note a concurrent ROLLBACK of an INSERT into an empty table. This is joint work with Thirunarayanan Balathandayuthapani, who created a working prototype. Thanks to Matthias Leich for extensive testing.	5 years ago
Marko Mäkelä	03ca6495df	MDEV-24142: Replace InnoDB rw_lock_t with sux_lock InnoDB buffer pool block and index tree latches depend on a special kind of read-update-write lock that allows reentrant (recursive) acquisition of the 'update' and 'write' locks as well as an upgrade from 'update' lock to 'write' lock. The 'update' lock allows any number of reader locks from other threads, but no concurrent 'update' or 'write' lock. If there were no requirement to support an upgrade from 'update' to 'write', we could compose the lock out of two srw_lock (implemented as any type of native rw-lock, such as SRWLOCK on Microsoft Windows). Removing this requirement is very difficult, so in commit `f7e7f487d4` we implemented an 'update' mode to our srw_lock. Re-entrant or recursive locking is mostly needed when writing or freeing BLOB pages, but also in crash recovery or when merging buffered changes to an index page. The re-entrancy allows us to attach a previously acquired page to a sub-mini-transaction that will be committed before whatever else is holding the page latch. The SUX lock supports Shared ('read'), Update, and eXclusive ('write') locking modes. The S latches are not re-entrant, but a single S latch may be acquired even if the thread already holds an U latch. The idea of the U latch is to allow a write of something that concurrent readers do not care about (such as the contents of BTR_SEG_LEAF, BTR_SEG_TOP and other page allocation metadata structures, or the MDEV-6076 PAGE_ROOT_AUTO_INC). (The PAGE_ROOT_AUTO_INC field is only updated when a dict_table_t for the table exists, and only read when a dict_table_t for the table is being added to dict_sys.) block_lock::u_lock_try(bool for_io=true) is used in buf_flush_page() to allow concurrent readers but no concurrent modifications while the page is being written to the data file. That latch will be released by buf_page_write_complete() in a different thread. Hence, we use the special lock owner value FOR_IO. The index_lock::u_lock() improves concurrency on operations that involve non-leaf index pages. The interface has been cleaned up a little. We will use x_lock_recursive() instead of x_lock() when we know that a lock is already held by the current thread. Similarly, a lock upgrade from U to X is only allowed via u_x_upgrade() or x_lock_upgraded() but not via x_lock(). We will disable the LatchDebug and sync_array interfaces to InnoDB rw-locks. The SEMAPHORES section of SHOW ENGINE INNODB STATUS output will no longer include any information about InnoDB rw-locks, only TTASEventMutex (cmake -DMUTEXTYPE=event) waits. This will make a part of the 'innotop' script dead code. The block_lock buf_block_t::lock will not be covered by any PERFORMANCE_SCHEMA instrumentation. SHOW ENGINE INNODB MUTEX and INFORMATION_SCHEMA.INNODB_MUTEXES will no longer output source code file names or line numbers. The dict_index_t::lock will be identified by index and table names, which should be much more useful. PERFORMANCE_SCHEMA is lumping information about all dict_index_t::lock together as event_name='wait/synch/sxlock/innodb/index_tree_rw_lock'. buf_page_free(): Remove the file,line parameters. The sux_lock will not store such diagnostic information. buf_block_dbg_add_level(): Define as empty macro, to be removed in a subsequent commit. Unless the build was configured with cmake -DPLUGIN_PERFSCHEMA=NO the index_lock dict_index_t::lock will be instrumented via PERFORMANCE_SCHEMA. Similar to commit `1669c8890c` we will distinguish lock waits by registering shared_lock,exclusive_lock events instead of try_shared_lock,try_exclusive_lock. Actual 'try' operations will not be instrumented at all. rw_lock_list: Remove. After MDEV-24167, this only covered buf_block_t::lock and dict_index_t::lock. We will output their information by traversing buf_pool or dict_sys.	5 years ago
Eugene Kosov	afc9d00c66	MDEV-23991 dict_table_stats_lock() has unnecessarily long scope Patch removes dict_index_t::stats_latch. Table/index statistics now protected with dict_sys->mutex. That way statistics computation can happen in parallel in several threads and dict_sys->mutex will be locked only for a short period of time. This patch is a joint work with Marko Mäkelä dict_index_t:🔒 make mutable which allows to pass const pointer when only lock is touched in an object btr_height_get() btr_get_size(): make index argument const for better type safety btr_estimate_number_of_different_key_vals(): now returns computed values instead of setting fields in dict_index_t directly remove everything related to dict_index_t::stats_latch dict_stats_index_set_n_diff(): now returns computed values instead of setting fields in dict_index_t directly dict_stats_analyze_index(): now returns computed values instead of setting fields in dict_index_t directly Reviewed by: Marko Mäkelä	5 years ago
Marko Mäkelä	118e258aaa	MDEV-23855: Shrink fil_space_t Merge n_pending_ios, n_pending_ops to std::atomic<uint32_t> n_pending. Change some more fil_space_t members to uint32_t to reduce the memory footprint. fil_space_t::add(), fil_ibd_create(): Attach the already opened handle to the tablespace, and enforce the fil_system.n_open limit. dict_boot(): Initialize fil_system.max_assigned_id. srv_boot(): Call srv_thread_pool_init() before anything else, so that files should be opened in the correct mode on Windows. fil_ibd_create(): Create the file in OS_FILE_AIO mode, just like fil_node_open_file_low() does it. dict_table_t::is_accessible(): Replaces fil_table_accessible(). Reviewed by: Vladislav Vaintroub	5 years ago
Thirunarayanan Balathandayuthapani	3ba8f619e4	MDEV-23370 innodb_fts.innodb_fts_misc failed in buildbot, server crashed in dict_table_autoinc_destroy This issue is caused by MDEV-22456 `ad6171b91c`. Fix involves the backported version of 10.4 patch MDEV-22778 `5f2628d1ee` and few parts of MDEV-17441 (`e9a5f288f2`). dict_table_t::stats_latch_created: Removed dict_table_t::stats_latch: make value member and always lock it for simplicity even for stats cloned table. zip_pad_info_t::mutex_created: Removed zip_pad_info_t::mutex: make member value instead of pointer os0once.h: Removed dict_table_remove_from_cache_low(): Ensure that fts_free() is always called, even if dict_mem_table_free() is deferred until btr_search_lazy_free(). InnoDB would always zip_pad_info_t::mutex and dict_table_t::autoinc_mutex, even for tables are not in ROW_FORMAT=COMPRESSED nor include any AUTO_INCREMENT column.	5 years ago
Aleksey Midenkov	7eda556196	MDEV-23672 Assertion `v.v_indexes.empty()' failed in dict_table_t::instant_column dict_v_idx_t node was shared between two dict_v_col_t objects because of wrong object copy. Replace memory plain copy with copy constructor. Tha patch also removes n_v_indexes property and improves "page full" judgements for trx_undo_log_v_idx().	5 years ago
Thirunarayanan Balathandayuthapani	6504d3d229	MDEV-23722 InnoDB: Assertion: result != FTS_INVALID in fts_trx_row_get_new_state Marking of deletion of row in fts index happens twice in self-referential foreign key relation. So while performing referential checks of foreign key, InnoDB can avoid updating of fts index if the foreign key has self-referential relationship. Reviewed-by: Marko Mäkelä	5 years ago
Marko Mäkelä	f99cace77f	MDEV-22924 Corruption in MVCC read via secondary index An unsafe optimization was introduced by commit `2347ffd843` (MDEV-20301) which is based on mysql/mysql-server@3f3136188f1bd383f77f97823cf6ebd72d5e4d7e or mysql/mysql-server@647a3814a91c3d3bffc70ddff5513398e3f37bd4 in MySQL 8.0.12 or MySQL 8.0.13 (which in turn is based on the contribution in MySQL Bug #84958). Row_sel_get_clust_rec_for_mysql::operator(): In addition to checking that the pointer to the record matches, also check the latest modification of the page (FIL_PAGE_LSN) as well as the page identifier. Only if all three match, it is safe to reuse cached_old_vers. Row_sel_get_clust_rec_for_mysql::check_eq(): Assert that the PRIMARY KEY of the cached old version of the record corresponds to the latest version. We got a test case where CHECK TABLE, UPDATE and purge would be hammering on the same table (with only 6 rows) and a pointer that was originally pointing to a record pk=2 would match a cached_clust_rec that was pointing to a record pk=1. In the diagnosed `rr replay` trace, we would wrongly return an old cached version of the pk=1 record, instead of retrieving the correct version of the pk=2 record. Because of this, CHECK TABLE would fail to count one of the records in a secondary index, and report failure. This bug appears to affect MVCC reads via secondary indexes only. The purge of history in secondary indexes uses a different code path, and so do checks for implicit record locks.	5 years ago
Marko Mäkelä	064bfbaf06	MDEV-23499 Assertion c.same_type(*o) failed dict_col_t::same_encoding(), dict_col_t::same_format(): Allow an instantaneous change of a column to a compatible encoding, just like ha_innobase::can_convert_string() and similar functions do.	5 years ago
Thirunarayanan Balathandayuthapani	362b18c536	MDEV-23380 InnoDB reads a page from disk despite parsing MLOG_INIT_FILE_PAGE2 record This problem is caused by `6697135c6d` (MDEV-21572). During recovery, InnoDB prefetches the siblings of change buffer index leaf page. It does asynchronous page read and recovery scenario wasn't handled in buf_read_page_background(). It leads to the refusal of startup of the server. Solution: ========= InnoDB shouldn't allow the change buffer index page siblings to be prefetched.	5 years ago
Marko Mäkelä	5f2628d1ee	MDEV-22778 Slow InnoDB shutdown on large instance Starting with MDEV-17441 we would no longer have os_once, and we would always initialize zip_pad_info_t::mutex and dict_table_t::autoinc_mutex, even for tables are not in ROW_FORMAT=COMPRESSED nor include any AUTO_INCREMENT column. mutex_free() on those unnecessary objects would make shutdown very slow compared to older versions. Let us use std::mutex for those two mutexes, to reduce the overhead. The critical sections protected by these mutexes is very small, and therefore contention or the need for any instrumentation should be unlikely.	5 years ago
Eugene Kosov	0e86254b05	MDEV-22930 Unnecessary contention on rw_lock_list_mutex in ibuf_dummy_index_create() 1. Do not initialize dict_table_t::stats_latch in ibuf 2. Remove overengineering in GenericPolicy to speed up things dict_mem_table_create(): add new argument init_stats_latch ibuf_dummy_index_create(): do not initialize dict_table_t::stats_latch GenericPolicy: add new members m_filename and m_line sync_file_create_register() sync_file_created_deregister() sync_file_created_get() CreateTracker: remove rw_lock_t::created: a new debug member	5 years ago
Marko Mäkelä	431200090e	MDEV-22867 Assertion instant.n_core_fields == n_core_fields failed This is a race condition where a table on which a 10.3-style instant ADD COLUMN is emptied during the execution of ALTER TABLE ... DROP COLUMN ..., DROP INDEX ..., ALGORITHM=NOCOPY. In commit `2c4844c9e7` the function instant_metadata_lock() would prevent this race condition. But, it would also hold a page latch on the leftmost leaf page of clustered index for the duration of a possible DROP INDEX operation. The race could be fixed by restoring the function instant_metadata_lock() that was removed in commit `ea37b14409` but it would be more future-proof to prevent the dict_index_t::clear_instant_add() call from being issued at all. We at some point support DROP COLUMN ..., ADD INDEX ..., ALGORITHM=NOCOPY and that would spend a non-trivial amount of execution time in ha_innobase::inplace_alter(), making a server hang possible. Currently this is not supported and our added test case will notice when the support is introduced. dict_index_t::must_avoid_clear_instant_add(): Determine if a call to clear_instant_add() must be avoided. btr_discard_only_page_on_level(): Preserve the metadata record if must_avoid_clear_instant_add() holds. btr_cur_optimistic_delete_func(), btr_cur_pessimistic_delete(): Do not remove the metadata record even if the table becomes empty but must_avoid_clear_instant_add() holds. btr_pcur_store_position(): Relax a debug assertion. This is joint work with Thirunarayanan Balathandayuthapani.	6 years ago
Thirunarayanan Balathandayuthapani	fdb37d084b	MDEV-21787 Alter table failure tries to access uninitialized column - Problem is that failure of inplace DDL tries to access the uninitialized column. This is caused by MDEV-19606 (commit `0274ab1de3`). Fix is that InnoDB should use column while freeing the index when index is completely initialized.	6 years ago
Marko Mäkelä	cb437417d2	MDEV-19114 gcol.innodb_virtual_debug: Assertion n_fields>0 failed This is a regression due to MDEV-16376 commit `8dc70c862b`. To make dict_index_t::detach_columns() idempotent, we cleared dict_index_t::n_fields. But, this could cause trouble with purge after a secondary index creation failed (not even involving virtual columns). A better way is to clear the dict_field_t::col pointers that point to virtual columns that are being freed due to aborting index creation on an index that depends on a virtual column. Note: the v_cols[] of an existing dict_table_t object will never be modified. If any virtual columns are added or removed, ha_innobase::commit_inplace_alter_table() would invoke dict_table_remove_from_cache() and reload the table to dict_sys. Index creation is a special case where the dict_index_t points to virtual columns that do not yet exist in dict_table_t.	6 years ago
Marko Mäkelä	386f168ab3	MDEV-22456 after-merge fix: introduce Atomic_relaxed In the merge `9e6e43551f` we made Atomic_counter a more generic wrapper of std::atomic so that dict_index_t would support the implicit assignment operator. It is better to revert the changes to Atomic_counter and instead introduce Atomic_relaxed as a generic wrapper to std::atomic. Unlike Atomic_counter, we will not define operator++, operator+= or similar, because we want to make the operations more explicit in the users of Atomic_wrapper, because unlike loads and stores, atomic read-modify-write operations always incur some overhead.	6 years ago
Marko Mäkelä	ad6171b91c	MDEV-22456 Dropping the adaptive hash index may cause DDL to lock up InnoDB If the InnoDB buffer pool contains many pages for a table or index that is being dropped or rebuilt, and if many of such pages are pointed to by the adaptive hash index, dropping the adaptive hash index may consume a lot of time. The time-consuming operation of dropping the adaptive hash index entries is being executed while the InnoDB data dictionary cache dict_sys is exclusively locked. It is not actually necessary to drop all adaptive hash index entries at the time a table or index is being dropped or rebuilt. We can let the LRU replacement policy of the buffer pool take care of this gradually. For this to work, we must detach the dict_table_t and dict_index_t objects from the main dict_sys cache, and once the last adaptive hash index entry for the detached table is removed (when the garbage page is evicted from the buffer pool) we can free the dict_table_t and dict_index_t object. Related to this, in MDEV-16283, we made ALTER TABLE...DISCARD TABLESPACE skip both the buffer pool eviction and the drop of the adaptive hash index. We shifted the burden to ALTER TABLE...IMPORT TABLESPACE or DROP TABLE. We can remove the eviction from DROP TABLE. We must retain the eviction in the ALTER TABLE...IMPORT TABLESPACE code path, so that in case the discarded table is being re-imported with the same tablespace identifier, the fresh data from the imported tablespace will replace any stale pages in the buffer pool. rpl.rpl_failed_drop_tbl_binlog: Remove the test. DROP TABLE can no longer be interrupted inside InnoDB. fseg_free_page(), fseg_free_step(), fseg_free_step_not_header(), fseg_free_page_low(), fseg_free_extent(): Remove the parameter that specifies whether the adaptive hash index should be dropped. btr_search_lazy_free(): Lazily free an index when the last reference to it is dropped from the adaptive hash index. buf_pool_clear_hash_index(): Declare static, and move to the same compilation unit with the bulk of the adaptive hash index code. dict_index_t::clone(), dict_index_t::clone_if_needed(): Clone an index that is being rebuilt while adaptive hash index entries exist. The original index will be inserted into dict_table_t::freed_indexes and dict_index_t::set_freed() will be called. dict_index_t::set_freed(), dict_index_t::freed(): Note that or check whether the index has been freed. We will use the impossible page number 1 to denote this condition. dict_index_t::n_ahi_pages(): Replaces btr_search_info_get_ref_count(). dict_index_t::detach_columns(): Move the assignment n_fields=0 to ha_innobase_inplace_ctx::clear_added_indexes(). We must have access to the columns when freeing the adaptive hash index. Note: dict_table_t::v_cols[] will remain valid. If virtual columns are dropped or added, the table definition will be reloaded in ha_innobase::commit_inplace_alter_table(). buf_page_mtr_lock(): Drop a stale adaptive hash index if needed. We will also reduce the number of btr_get_search_latch() calls and enclose some more code inside #ifdef BTR_CUR_HASH_ADAPT in order to benefit cmake -DWITH_INNODB_AHI=OFF.	6 years ago
Marko Mäkelä	ba3d58ad4c	MDEV-22523 index->rtr_ssn.mutex is wasting memory As part of the SPATIAL INDEX implementation in InnoDB, dict_index_t was expanded by a rtr_ssn_t field. There are only 3 operations for this field, all protected by rtr_ssn_t::mutex: * btr_cur_search_to_nth_level() stores the least significant 32 bits of the 64-bit value that is stored in the index root page. (This would better be done when the table is opened for the very first time.) * rtr_get_new_ssn_id() increments the value by 1. * rtr_get_current_ssn_id() reads the current value. All these operations can be implemented equally safely by using atomic memory access operations.	6 years ago
Thirunarayanan Balathandayuthapani	f7599f4799	MDEV-21792 Server aborts upon attempt to create foreign key on spatial field - mbmaxlen is always 0 for binary field. Ignore the assert of checking field->prefix_len % field->col->mbmaxlen == 0.	6 years ago
Marko Mäkelä	f224525204	MDEV-21907: InnoDB: Enable -Wconversion on clang and GCC The -Wconversion in GCC seems to be stricter than in clang. GCC at least since version 4.4.7 issues truncation warnings for assignments to bitfields, while clang 10 appears to only issue warnings when the sizes in bytes rounded to the nearest integer powers of 2 are different. Before GCC 10.0.0, -Wconversion required more casts and would not allow some operations, such as x<<=1 or x+=1 on a data type that is narrower than int. GCC 5 (but not GCC 4, GCC 6, or any later version) is complaining about x\|=y even when x and y are compatible types that are narrower than int. Hence, we must rewrite some x\|=y as x=static_cast<byte>(x\|y) or similar, or we must disable -Wconversion. In GCC 6 and later, the warning for assigning wider to bitfields that are narrower than 8, 16, or 32 bits can be suppressed by applying a bitwise & with the exact bitmask of the bitfield. For older GCC, we must disable -Wconversion for GCC 4 or 5 in such cases. The bitwise negation operator appears to promote short integers to a wider type, and hence we must add explicit truncation casts around them. Microsoft Visual C does not allow a static_cast to truncate a constant, such as static_cast<byte>(1) truncating int. Hence, we will use the constructor-style cast byte(~1) for such cases. This has been tested at least with GCC 4.8.5, 5.4.0, 7.4.0, 9.2.1, 10.0.0, clang 9.0.1, 10.0.0, and MSVC 14.22.27905 (Microsoft Visual Studio 2019) on 64-bit and 32-bit targets (IA-32, AMD64, POWER 8, POWER 9, ARMv8).	6 years ago

1 2 3 4 5 ...

377 Commits (4b80c11f52a3da189bafd7a772bcbf3519ceb41e)