mariadb

Commit Graph

Author	SHA1	Message	Date
Marko Mäkelä	d98922789e	Experiment: Remove handler::unlock_row() and friends Failing test(s): innodb.innodb main.partition_innodb_semi_consistent innodb.innodb-semi-consistent innodb.innodb_lock_wait_timeout_1 main.concurrent_innodb_unsafelog main.unsafe_binlog_innodb innodb.create_table_insert_skip_locked rpl.rpl_unsafe_statements innodb.partition_locking innodb.skip_locked_nowait innodb.deadlock_wait_lock_race	3 years ago
Oleg Smirnov	ddcdb1a4fc	MDEV-16232: Use fewer mini-transactions SQL Layer support #3. Add checks of applicability for UPDATEs and DELETEs. Rename "operations batch" to "mini transactions" to disambiguate the term "batch" which is already used. Temporary disable mini-transactions for SELECTs.	3 years ago
Sergei Petrunia	e4d704b54e	MDEV-16232: Use fewer mini-transactions SQL layer support part #2. The main idea is: employ this optimization only when the query is using just one table. (the scope is too limited but the goal is to get sysbench to work). Make UPDATE, DELETE, and SELECT codepaths check that the query has just one table before the optimization is employed. Also: in DELETE code, the start_operations_batch/end_operations_batch were in the "Direct DELETE" branch. I think it's a mistake as InnoDB doesn't use Direct DELETE optimization. Removed the calls from there and added them to the regular DELETE codepath.	4 years ago
Thirunarayanan Balathandayuthapani	ec16b35db2	MDEV-16232 Use fewer mini-transactions - For update operation, InnoDB uses separate mtr for the following: 1) searching the key in primary key leaf page, stores the position 2) exclusive lock the primary key leaf page 3) Allocate the undo log page, write the undo log record 4) Modify the PRIMARY KEY index change InnoDB should use the single mtr for the all above conditions.	3 years ago
Vicențiu Ciorbaru	08c852026d	Apply clang-tidy to remove empty constructors / destructors This patch is the result of running run-clang-tidy -fix -header-filter=.* -checks='-,modernize-use-equals-default' . Code style changes have been done on top. The result of this change leads to the following improvements: 1. Binary size reduction. For a -DBUILD_CONFIG=mysql_release build, the binary size is reduced by ~400kb. * A raw -DCMAKE_BUILD_TYPE=Release reduces the binary size by ~1.4kb. 2. Compiler can better understand the intent of the code, thus it leads to more optimization possibilities. Additionally it enabled detecting unused variables that had an empty default constructor but not marked so explicitly. Particular change required following this patch in sql/opt_range.cc result_keys, an unused template class Bitmap now correctly issues unused variable warnings. Setting Bitmap template class constructor to default allows the compiler to identify that there are no side-effects when instantiating the class. Previously the compiler could not issue the warning as it assumed Bitmap class (being a template) would not be performing a NO-OP for its default constructor. This prevented the "unused variable warning".	3 years ago
Alexander Barkov	a27b8b2683	MDEV-27653 long uniques don't work with unicode collations	3 years ago
Alexander Barkov	284ac6f2b7	MDEV-27653 long uniques don't work with unicode collations	3 years ago
Monty	981a6b7044	MDEV-30395 Wrong result with semijoin and Federated as outer table The problem was that federated engine does not support comparable rowids which was not taken into account by semijoin code. Fixed by checking that we don't use semijoin with tables that does not support comparable rowids. Other things: - Fixed some typos in the code comments	3 years ago
Aleksey Midenkov	5d506ac201	MDEV-25004 vers_force_trx option to force transactional System Versioning Works like vers_force but forces trx_id-based system-versioned tables if the storage supports it (currently InnoDB-only). Otherwise creates timestamp-based system-versioned table.	3 years ago
Monty	dd5f4b3625	Fixed bug in Aria when used with enterprise mariadb-backup If the backup finished in the middle of a Aria bulk load insert, which could happen with LOAD DATA INFILE, CREATE ... SELECT etc) there was a chance that Aria recovery would fail on the backup. Fixed by ensuring that bulk load operations for Aria are not allowed under BACKUP LOCK. I also changed so that the table TRN is updated just before truncate which ensures that old redo's for the table are ignored. I also enabled Aria redo for DDL's to be able to repeat REPAIR commands. Without this change recovery would not work on repaired tables. Notes: - We take the backup lock protection at the end of bulk insert (as we don't want to keep the lock over a very long running insert). If mariadb-backup keeps the backup lock too long, this may fail with a lock timeout. In this case the batch insert will fail and the table will be truncated (set to it's original state).	3 years ago
Sergei Golubchik	2bd41fc5bf	Revert MDEV-25292 Atomic CREATE OR REPLACE TABLE Specifically: Revert "MDEV-29664 Assertion `!n_mysql_tables_in_use' failed in innobase_close_connection" This reverts commit `ba875e9396`. Revert "MDEV-29620 Assertion `next_insert_id == 0' failed in handler::ha_external_lock" This reverts commit `aa08a7442a`. Revert "MDEV-29628 Memory leak after CREATE OR REPLACE with foreign key" This reverts commit `c579d66ba6`. Revert "MDEV-29609 create_not_windows test fails with different result" This reverts commit `cb583b2f1b`. Revert "MDEV-29544 SIGSEGV in HA_CREATE_INFO::finalize_locked_tables" This reverts commit `dcd66c3814`. Revert "MDEV-28933 CREATE OR REPLACE fails to recreate same constraint name" This reverts commit `cf6c517632`. Revert "MDEV-28933 Moved RENAME_CONSTRAINT_IDS to include/sql_funcs.h" This reverts commit `f1e1c1335b`. Revert "MDEV-28956 Locking is broken if CREATE OR REPLACE fails under LOCK TABLES" This reverts commit `a228ec80e3`. Revert "MDEV-25292 gcol.gcol_bugfixes --ps fix" This reverts commit `24fff8267d`. Revert "MDEV-25292 Disable atomic replace for slave-generated or-replace" This reverts commit `2af15914cb`. Revert "MDEV-25292 backup_log improved" This reverts commit `34398a20b5`. Revert "MDEV-25292 Atomic CREATE OR REPLACE TABLE" This reverts commit `93c8252f02`. Revert "MDEV-25292 Table_name class for (db, table_name, alias)" This reverts commit `d145dda9c7`. Revert "MDEV-25292 ha_table_exists() cleanup and improvement" This reverts commit `409b8a86de`. Revert "MDEV-25292 Cleanups" This reverts commit `595dad83ad`. Revert "MDEV-25292 Refactoring: moved select_field_count into Alter_info." This reverts commit `f02af1d229`.	3 years ago
Alexander Barkov	ce443c8554	MDEV-29495 Generalize can_convert_xxx() hook engine API to cover any arbitrary data type	3 years ago
Igor Babaev	58cd0bd59e	MDEV-28846 Poor performance when rowid filter contains no elements When a range rowid filter was used with an index ref access the cost of accessing the index entries for the records rejected by the filter was not taken into account. For a ref access by an index with big average number of records per key this led to poor execution plans if selectivity of the used filter was high. The patch resolves this problem. It also introduces a minor optimization that skips look-ups into a filter that turns out to be empty. With this patch the output of ANALYZE stmt reports the number of look-ups into used rowid filters. The patch also back-ports from 10.5 the code that properly sets the field TABLE::file::table for opened temporary tables. The test cases that were supposed to use rowid filters have been adjusted in order to use similar execution plans after this fix. Approved by Oleksandr Byelkin <sanja@mariadb.com>	3 years ago
Aleksey Midenkov	a228ec80e3	MDEV-28956 Locking is broken if CREATE OR REPLACE fails under LOCK TABLES add_back_last_deleted_lock() was called when the lock was never removed. Lock is removed in finalize_atomic_replace() in close_all_tables_for_name(). finalize_atomic_replace() is done only for successful operation. In non-atomic codepath it drops the table first, if anything fails later we don't need to return back the lock since there is no table now. So the fix is required as well.	3 years ago
Aleksey Midenkov	2af15914cb	MDEV-25292 Disable atomic replace for slave-generated or-replace Also fixes wrong rollback of ddl_log_state_rm in case of non-atomic replace. There is no backup to remove in that case.	3 years ago
Aleksey Midenkov	93c8252f02	MDEV-25292 Atomic CREATE OR REPLACE TABLE Atomic CREATE OR REPLACE allows to keep an old table intact if the command fails or during the crash. That is done through creating a table with a temporary name and filling it with the data (for CREATE OR REPLACE .. SELECT), then renaming the original table to another temporary (backup) name and renaming the replacement table to original table. The backup table is kept until the last chance of failure and if that happens, the replacement table is thrown off and backup recovered. When the command is complete and logged the backup table is deleted. Atomic replace algorithm Two DDL chains are used for CREATE OR REPLACE: ddl_log_state_create (C) and ddl_log_state_rm (D). 1. (C) Log CREATE_TABLE_ACTION of TMP table (drops TMP table); 2. Create new table as TMP; 3. Do everything with TMP (like insert data); finalize_atomic_replace(): 4. Link chains: (D) is executed only if (C) is closed; 5. (D) Log DROP_ACTION of BACKUP; 6. (C) Log RENAME_TABLE_ACTION from ORIG to BACKUP (replays BACKUP -> ORIG); 7. Rename ORIG to BACKUP; 8. (C) Log CREATE_TABLE_ACTION of ORIG (drops ORIG); 9. Rename TMP to ORIG; finalize_ddl() in case of success: 10. Close (C); 11. Replay (D): BACKUP is dropped. finalize_ddl() in case of error: 10. Close (D); 11. Replay (C): 1) ORIG is dropped (only after finalize_atomic_replace()); 2) BACKUP renamed to ORIG (only after finalize_atomic_replace()); 3) drop TMP. If crash happens (C) or (D) is replayed in reverse order. (C) is replayed if crash happens before it is closed, otherwise (D) is replayed. Temporary table for CREATE OR REPLACE Before dropping "old" table, CREATE OR REPLACE creates "tmp" table. ddl_log_state_create holds the drop of the "tmp" table. When everything is OK (data is inserted, "tmp" is ready) ddl_log_state_rm is written to replace "old" with "tmp". Until ddl_log_state_create is closed ddl_log_state_rm is not executed. After the binlogging is done ddl_log_state_create is closed. At that point ddl_log_state_rm is executed and "tmp" is replaced with "old". That is: final rename is done by the DDL log. With that important role of DDL log for CREATE OR REPLACE operation replay of ddl_log_state_rm must fail at the first hit error and print the error message if possible. F.ex. foreign key error is discovered at this phase: InnoDB rejects to drop the "old" table and returns corresponding foreign key error code. Additional notes - CREATE TABLE without REPLACE is not affected by this commit. - Engines having HTON_EXPENSIVE_RENAME flag set are not affected by this commit. - CREATE TABLE .. SELECT XID usage is fixed and now there is no need to log DROP TABLE via DDL_CREATE_TABLE_PHASE_LOG (see comments in do_postlock()). XID is now correctly updated so it disables DDL_LOG_DROP_TABLE_ACTION. Note that binary log is flushed at the final stage when the table is ready. So if we have XID in the binary log we don't need to drop the table. - Three variations of CREATE OR REPLACE handled: 1. CREATE OR REPLACE TABLE t1 (..); 2. CREATE OR REPLACE TABLE t1 LIKE t2; 3. CREATE OR REPLACE TABLE t1 SELECT ..; - Test case uses 6 combinations for engines (aria, aria_notrans, myisam, ib, lock_tables, expensive_rename) and 2 combinations for binlog types (row, stmt). Combinations help to check differences between the results. Error failures are tested for the above three variations. - expensive_rename tests CREATE OR REPLACE without atomic replace. The effect should be the same as with the old behaviour before this commit. - Triggers mechanism is unaffected by this change. This is tested in create_replace.test. - LOCK TABLES is affected. Lock restoration must be done after "rm" chain is replayed. - Moved ddl_log_complete() from send_eof() to finalize_ddl(). This checkpoint was not executed before for normal CREATE TABLE but is executed now. - CREATE TABLE will now rollback also if writing to the binary logging failed. See rpl_gtid_strict.test Rename and drop via DDL log We replay ddl_log_state_rm to drop the old table and rename the temporary table. In that case we must throw the correct error message if ddl_log_revert() fails (f.ex. on FK error). If table is deleted earlier and not via DDL log and the crash happened, the create chain is not closed. Linked drop chain is not executed and the new table is not installed. But the old table is already deleted. ddl_log.cc changes Now we can place action before DDL_LOG_DROP_INIT_ACTION and it will be replayed after DDL_LOG_DROP_TABLE_ACTION. report_error parameter for ddl_log_revert() allows to fail at first error and print the error message if possible. ddl_log_execute_action() now can print error message. Since we now can handle errors from ddl_log_execute_action() (in case of non-recovery execution) unconditional setting "error= TRUE" is wrong (it was wrong anyway because it was overwritten at the end of the function). On XID usage Like with all other atomic DDL operations XID is used to avoid inconsistency between master and slave in the case of a crash after binary log is written and before ddl_log_state_create is closed. On recovery XIDs are taken from binary log and corresponding DDL log events get disabled. That is done by ddl_log_close_binlogged_events(). On linking two chains together Chains are executed in the ascending order of entry_pos of execute entries. But entry_pos assignment order is undefined: it may assign bigger number for the first chain and then smaller number for the second chain. So the execution order in that case will be reverse: second chain will be executed first. To avoid that we link one chain to another. While the base chain (ddl_log_state_create) is active the secondary chain (ddl_log_state_rm) is not executed. That is: only one chain can be executed in two linked chains. The interface ddl_log_link_chains() was done in "MDEV-22166 ddl_log_write_execute_entry() extension". More on CREATE OR REPLACE .. SELECT We use create_and_open_tmp_table() like in ALTER TABLE to create temporary TABLE object (tmp_table is (NON_)TRANSACTIONAL_TMP_TABLE). After we created such TABLE object we use create_info->tmp_table() instead of table->s->tmp_table when we need to check for parser-requested tmp-table. External locking is required for temporary table created by create_and_open_tmp_table(). F.ex. that disables logging for Aria transactional tables and without that (when no mysql_lock_tables() is done) it cannot work correctly. For making external lock the patch requires Aria table to work in non-transactional mode. That is usually done by ha_enable_transaction(false). But we cannot disable transaction completely because: 1. binlog rollback removes pending row events (binlog_remove_pending_rows_event()). The row events are added during CREATE .. SELECT data insertion phase. 2. replication slave highly depends on transaction and cannot work without it. So we put temporary Aria table into non-transactional mode with "thd->transaction->on hack". See comment for on_save variable. Note that Aria table has internal_table mode. But we cannot use it because: if (!internal_table) { mysql_mutex_lock(&THR_LOCK_myisam); old_info= test_if_reopen(name_buff); } For internal_table test_if_reopen() is not called and we get a new MARIA_SHARE for each file handler. In that case duplicate errors are missed because insert and lookup in CREATE .. SELECT is done via two different handlers (see create_lookup_handler()). For temporary table before dropping TABLE_SHARE by drop_temporary_table() we must do ha_reset(). ha_reset() releases storage share. Without that the share is kept and the second CREATE OR REPLACE .. SELECT fails with: HA_ERR_TABLE_EXIST (156): MyISAM table '#sql-create-b5377-4-t2' is in use (most likely by a MERGE table). Try FLUSH TABLES. HA_EXTRA_PREPARE_FOR_DROP also removes MYISAM_SHARE, but that is not needed as ha_reset() does the job. ha_reset() is usually done by mark_tmp_table_as_free_for_reuse(). But we don't need that mechanism for our temporary table. Atomic_info in HA_CREATE_INFO Many functions in CREATE TABLE pass the same parameters. These parameters are part of table creation info and should be in HA_CREATE_INFO (or whatever). Passing parameters via single structure is much easier for adding new data and refactoring. InnoDB changes (revised by Marko Mäkelä) row_rename_table_for_mysql(): Specify the treatment of FOREIGN KEY constraints in a 4-valued enum parameter. In cases where FOREIGN KEY constraints cannot exist (partitioned tables, or internal tables of FULLTEXT INDEX), we can use the mode RENAME_IGNORE_FK. The mod RENAME_REBUILD is for any DDL operation that rebuilds the table inside InnoDB, such as TRUNCATE and native ALTER TABLE (or OPTIMIZE TABLE). The mode RENAME_ALTER_COPY is used solely during non-native ALTER TABLE in ha_innobase::rename_table(). Normal ha_innobase::rename_table() will use the mode RENAME_FK. CREATE OR REPLACE will rename the old table (if one exists) along with its FOREIGN KEY constraints into a temporary name. The replacement table will be initially created with another temporary name. Unlike in ALTER TABLE, all FOREIGN KEY constraints must be renamed and not inherited as part of these operations, using the mode RENAME_FK. dict_get_referenced_table(): Let the callers convert names when needed. create_table_info_t::create_foreign_keys(): CREATE OR REPLACE creates the replacement table with a temporary name table, so for self-references foreign->referenced_table will be a table with temporary name and charset conversion must be skipped for it. Reviewed by: Michael Widenius <monty@mariadb.org>	3 years ago
Aleksey Midenkov	32d88faec5	MDEV-25292 Refactoring: removed TABLEOP_HOOKS TABLEOP_HOOKS is a strange interface: proxy interface calls virtual interface. Since it is used only for select_create::prepare() such complexity is overwhelming.	3 years ago
Aleksey Midenkov	409b8a86de	MDEV-25292 ha_table_exists() cleanup and improvement Removed default values for arguments, added flags argument to specify filename flags (FN_TO_IS_TMP, FN_FROM_IS_TMP).	3 years ago
Aleksey Midenkov	f02af1d229	MDEV-25292 Refactoring: moved select_field_count into Alter_info. There is a need in MDEV-25292 to have both C_ALTER_TABLE and select_field_count in one call. Semantically creation mode and field count are two different things. Making creation mode negative constants and field count positive variable into one parameter seems to be a lazy hack for not making the second parameter. select_count does not make sense without alter_info->create_list, so the natural way is to hold it in Alter_info too. select_count is now stored in member select_field_count.	3 years ago
Alexander Barkov	208addf484	Main patch MDEV-27896 Wrong result upon `COLLATE latin1_bin CHARACTER SET latin1` on the table or the database level Also fixes MDEV-27782 Wrong columns when using table level `CHARACTER SET utf8mb4 COLLATE DEFAULT` MDEV-28644 Unexpected error on ALTER TABLE t1 CONVERT TO CHARACTER SET utf8mb3, DEFAULT CHARACTER SET utf8mb4	4 years ago
Andrei	98ca71ab28	MDEV-28461 semisync-slave server recovery fails to rollback prepared transaction that is not in binlog. Post-crash recovery of --rpl-semi-sync-slave-enabled server failed to recognize a transaction in-doubt that needed rolled back. A prepared-but-not-in-binlog transaction gets committed instead to possibly create inconsistency with a master (e.g the way it was observed in the bug report). The semisync recovery is corrected now with initializing binlog coordinates of any transaction in-doubt to the maximum offset which is unreachable. In effect when a prepared transaction that is not found in binlog it will be decided to rollback because it's guaranteed to reside in a truncated tail area of binlog. Mtr tests are reinforced to cover the described scenario.	4 years ago
Sergei Golubchik	29c07643a1	enable -Wenum-compare -Wenum-conversion to make the all headers -std=c++20 clean for those, who need c++20 (some plugins)	4 years ago
Aleksey Midenkov	92bfc0e8c4	MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT :: Syntax change :: Keyword AUTO enables history partition auto-creation. Examples: CREATE TABLE t1 (x int) WITH SYSTEM VERSIONING PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR AUTO; CREATE TABLE t1 (x int) WITH SYSTEM VERSIONING PARTITION BY SYSTEM_TIME INTERVAL 1 MONTH STARTS '2021-01-01 00:00:00' AUTO PARTITIONS 12; CREATE TABLE t1 (x int) WITH SYSTEM VERSIONING PARTITION BY SYSTEM_TIME LIMIT 1000 AUTO; Or with explicit partitions: CREATE TABLE t1 (x int) WITH SYSTEM VERSIONING PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR AUTO (PARTITION p0 HISTORY, PARTITION pn CURRENT); To disable or enable auto-creation one can use ALTER TABLE by adding or removing AUTO from partitioning specification: CREATE TABLE t1 (x int) WITH SYSTEM VERSIONING PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR AUTO; # Disables auto-creation: ALTER TABLE t1 PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR; # Enables auto-creation: ALTER TABLE t1 PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR AUTO; If the rest of partitioning specification is identical to CREATE TABLE no repartitioning will be done (for details see MDEV-27328). :: Description :: Before executing history-generating DML command (see the list of commands below) add N history partitions, so that N would be sufficient for potentially generated history. N > 1 may be required when history partitions are switched by INTERVAL and current_timestamp is N times further than the interval boundary of the last history partition. If the last history partition equals or exceeds LIMIT records then new history partition is created and selected as the working partition. According to MDEV-28411 partitions cannot be switched (or created) while the command is running. Thus LIMIT does not carry strict limitation and the history partition size must be planned as LIMIT value plus average number of history one DML command can generate. Auto-creation is implemented by synchronous fast_alter_partition_table() call from the thread of the executed DML command before the command itself is run (by the fallback and retry mechanism similar to Discovery feature, see Open_table_context). The name for newly added partitions are generated like default partition names with extension of MDEV-22155 (which avoids name clashes by extending assignment counter to next free-enough gap). These DML commands can trigger auto-creation: DELETE (including multitable DELETE, excluding DELETE HISTORY) UPDATE (including multitable UPDATE) REPLACE (including REPLACE .. SELECT) INSERT .. ON DUPLICATE KEY UPDATE (including INSERT .. SELECT .. ODKU) LOAD DATA .. REPLACE :: Bug fixes :: MDEV-23642 Locking timeout caused by auto-creation affects original DML The reasons for this are: - Do not disrupt main business process (the history is auxiliary service); - Consequences are non-fatal (history is not lost, but comes into wrong partition; fixed by partitioning rebuild); - There is more freedom for application to fail in this case or not: it may read warning info and find corresponding error number. - While non-failing command is easy to handle by an application and fail it, the opposite is hard to handle: there is no automatic actions to fix failed command and retry, DBA intervention is required and until then application is non-functioning. MDEV-23639 Auto-create does not work under LOCK TABLES or inside triggers Don't do tdc_remove_table() for OT_ADD_HISTORY_PARTITION because it is not possible in locked tables mode. LTM_LOCK_TABLES mode (and LTM_PRELOCKED_UNDER_LOCK_TABLES) works out of the box as fast_alter_partition_table() can reopen tables via locked_tables_list. In LTM_PRELOCKED we reopen and relock table manually. :: More fixes :: * some_table_marked_for_reopen flag fix some_table_marked_for_reopen affets only reopen of m_locked_tables. I.e. Locked_tables_list::reopen_tables() reopens only tables from m_locked_tables. * Unused can_recover_from_failed_open() condition Is recover_from_failed_open() can be really used after open_and_process_routine()? :: Reviewed by :: Sergei Golubchik <serg@mariadb.org>	4 years ago
Oleg Smirnov	d1a1ad4c28	MDEV-27021 Add support of FORMAT=JSON for SHOW ANALYZE	4 years ago
Oleg Smirnov	e7fcd496d4	MDEV-27021 Implement SHOW ANALYZE command	4 years ago
Oleg Smirnov	328684833b	MDEV-10000 Add EXPLAIN [FORMAT=JSON] FOR CONNECTION syntax support EXPLAIN FOR CONNECTION is a MySQL-compatible syntax for SHOW EXPLAIN. This commit also adds support for FORMAT=JSON to SHOW EXPLAIN, so the possible options to get JSON-formatted output are: - SHOW EXPLAIN FORMAT=JSON FOR $con - EXPLAIN FORMAT=JSON FOR CONNECTION $con	4 years ago
Marko Mäkelä	e9735a8185	MDEV-25975 innodb_disallow_writes causes shutdown to hang We will remove the parameter innodb_disallow_writes because it is badly designed and implemented. The parameter was never allowed at startup. It was only internally used by Galera snapshot transfer. If a user executed SET GLOBAL innodb_disallow_writes=ON; the server could hang even on subsequent read operations. During Galera snapshot transfer, we will block writes to implement an rsync friendly snapshot, as follows: sst_flush_tables() will acquire a global lock by executing FLUSH TABLES WITH READ LOCK, which will block any writes at the high level. sst_disable_innodb_writes(), invoked via ha_disable_internal_writes(true), will suspend or disable InnoDB background tasks or threads that could initiate writes. As part of this, log_make_checkpoint() will be invoked to ensure that anything in the InnoDB buf_pool.flush_list will be written to the data files. This has the nice side effect that the Galera joiner will avoid crash recovery. The changes to sql/wsrep.cc and to the tests are based on a prototype that was developed by Jan Lindström. Reviewed by: Jan Lindström	4 years ago
Marko Mäkelä	35425cfc55	Cleanup: Remove some unused functions	4 years ago
Marko Mäkelä	118826d173	Fix gcc-12 -O2 -Warray-bounds	4 years ago
Hugo Wen	dafc5fb9c1	MDEV-27342: Fix issue of recovery failure using new server id Commit `6c39eaeb1` made the crash recovery dependent on server_id. The crash recovery could fail when restoring a new instance from original crashed data directory USING A NEW SERVER ID. The issue doesn't exist in previous major versions before 10.6. Root cause is when generating the input XID to be searched in the hash, server id is populated with the current server id. So if the server id changed when recovering, the XID couldn't be found in the hash due to server id doesn't match. This fix is to use original server id when creating the input XID object in function `xarecover_do_commit_or_rollback`. All new code of the whole pull request, including one or several files that are either new files or modified ones, are contributed under the BSD-new license. I am contributing on behalf of my employer Amazon Web Services, Inc.	4 years ago
Vlad Lesin	1766a18e06	MDEV-19577 Replication does not work with innodb_autoinc_lock_mode=2 The first step for deprecating innodb_autoinc_lock_mode(see MDEV-27844) is: - to switch statement binlog format to ROW if binlog format is MIXED and the statement changes autoincremented fields - issue warnings if innodb_autoinc_lock_mode == 2 and binlog format is STATEMENT	4 years ago
Igor Babaev	8265d6d9f6	MDEV-22846 Server crashes in handler_index_cond_check on SELECT If the optimizer decides to rewrites a NOT IN predicand of the form outer_expr IN (SELECT inner_col FROM ... WHERE subquery_where) into the EXISTS subquery EXISTS (SELECT 1 FROM ... WHERE subquery_where AND (outer_expr=inner_col OR inner_col IS NULL)) then the pushed equality predicate outer_expr=inner_col can be used for ref[or_null] access if inner_col is a reference to an indexed column. In this case if there is a selective range condition over this column then a Rowid filter may be employed coupled the with ref[or_null] access. The filter is 'pushed' into the engine and in InnoDB currently it cannot be used with index look-ups by primary key. The ref[or_null] access can be used only when outer_expr is not NULL. Otherwise the original predicand is evaluated to TRUE only if the result set returned by the query SELECT 1 FROM ... WHERE subquery_where is empty. When performing this evaluation the executor switches to the table scan by primary key. Before this patch the pushed filter still remained marked as active and the engine tried to apply the filter. This was incorrect and in InnoDB this attempt to use the filter led to an assertion failure. This patch fixes the problem by disabling usage of the filter when outer_expr is evaluated to NULL.	4 years ago
Monty	c18896f9c1	MDEV-14907 FEDERATEDX doesn't respect DISTINCT Federated and Federatex cannot be used with ROR scans Federated::position() and Federatex::position() is storing in 'ref' a pointer into a local result set buffer. This means that one cannot compare 'ref' from different handler instances to see if they point to the same physical record. This bug caused federated.federatedx to return wrong results when the optimizer tried to use index_merge to resolve some queries. Fixed by introducing table flag HA_NON_COMPARABLE_ROWID and using this with the above handlers. Todo: - Fix multi_delete(), multi_update and read_records() to use primary key instead of 'ref' if case HA_NON_COMPARABLE_ROWID is set. The current code only works if we have only one range (like table scan) for the tables that will be updated in the second pass. - Enable DBUG_ASSERT() in ha_federated::cmp_ref() and ha_federatedx::cmp_ref().	4 years ago
Thirunarayanan Balathandayuthapani	d270525dfd	MDEV-23805 Make Online DDL to Instant DDL when table is empty - In ha_innobase::prepare_inplace_alter_table(), InnoDB should check whether the table is empty. If the table is empty then server should avoid downgrading the MDL after prepare phase. It is more like instant alter, does change only in dicationary and metadata. - Changed few debug test case to make non-empty DDL table	4 years ago
Andrei Elkin	561b6c7e51	MDEV-26833 Missed statement rollback in case transaction drops or create temporary table When transaction creates or drops temporary tables and afterward its statement faces an error even the transactional table statement's cached ROW format events get involved into binlog and are visible after the transaction's commit. Fixed with proper analysis of whether the errored-out statement needs to be rolled back in binlog. For instance a fact of already cached CREATE or DROP for temporary tables by previous statements alone does not cause to retain the being errored-out statement events in the cache. Conversely, if the statement creates or drops a temporary table itself it can't be rolled back - this rule remains.	4 years ago
Aleksey Midenkov	5cae401b00	MDEV-25555 Server crashes in tree_record_pos after INPLACE-recreating index on HEAP table Drop and add same key is considered rename (look ALTER_RENAME_INDEX in fill_alter_inplace_info()). But in this case order of keys may be changed, because mysql_prepare_alter_table() yet does not know about rename and treats 2 operations: drop and add. In that case we disable inplace algorithm for such engines as Memory, MyISAM and Aria with ALTER_INDEX_ORDER flag. These engines have no specialized check_if_supported_inplace_alter() and default handler::check_if_supported_inplace_alter() sees an unknown flag and returns HA_ALTER_INPLACE_NOT_SUPPORTED. ha_innobase::check_if_supported_inplace_alter() works differently and inplace is not disabled (with the help of modified INNOBASE_INPLACE_IGNORE). add_drop_v_cols fork was also tweaked as it wrongly failed with MSG_UNSUPPORTED_ALTER_ONLINE_ON_VIRTUAL_COLUMN when it seen ALTER_INDEX_ORDER. No-op operation must be still no-op no matter of ALTER_INDEX_ORDER presence, so we tweek its condition as well.	4 years ago
Aleksey Midenkov	b3bdc1c142	MDEV-25803 Inplace ALTER breaks MyISAM/Aria table when order of keys is changed mysql_prepare_create_table() does my_qsort(sort_keys) on key info. This sorting is indeterministic: a table is created with one order and inplace alter may overwrite frm with another order. Since inplace alter does nothing about key info for MyISAM/Aria storage engines this results in discrepancy between frm and storage engine key definitions. The fix avoids the sorting of keys when no new keys added by ALTER (and this is ok for MyISAM/Aria since it cannot add new keys inplace). There is a case when implicit primary key may be changed when removing NOT NULL from the part of unique key. In that case we update modified_primary_key which is then used to not skip key sorting. According to is_candidate_key() there is no other cases when primary key may be changed implicitly. Notes: mi_keydef_write()/mi_keyseg_write() are used only in mi_create(). They should be used in ha_inplace_alter_table() as well. Aria corruption detection is unimplemented: maria_check_definition() is never used! MySQL 8.0 has this bug as well as of 8.0.26.	4 years ago
Aleksey Midenkov	c6207ecba4	MDEV-25803 innodb.alter_candidate_key fix There is a case when implicit primary key may be changed when removing NOT NULL from the part of unique key. In that case we update modified_primary_key which is then used to not skip key sorting. According to is_candidate_key() there is no other cases when primary kay may be changed implicitly.	4 years ago
Andrei Elkin	42ae765960	MDEV-26833 Missed statement rollback in case transaction drops or create temporary table When transaction creates or drops temporary tables and afterward its statement faces an error even the transactional table statement's cached ROW format events get involved into binlog and are visible after the transaction's commit. Fixed with proper analysis of whether the errored-out statement needs to be rolled back in binlog. For instance a fact of already cached CREATE or DROP for temporary tables by previous statements alone does not cause to retain the being errored-out statement events in the cache. Conversely, if the statement creates or drops a temporary table itself it can't be rolled back - this rule remains.	4 years ago
Aleksey Midenkov	69724805bc	MDEV-22165 CONVERT TABLE: move in partition from existing table Syntax for CONVERT TABLE ALTER TABLE tbl_name CONVERT TABLE tbl_name TO PARTITION partition_name partition_spec Examples: ALTER TABLE t1 CONVERT TABLE tp2 TO PARTITION p2 VALUES LESS THAN MAX_VALUE(); New ALTER_PARTITION_CONVERT_IN command for fast_alter_partition_table() is done in alter_partition_convert_in() function which basically does ha_rename_table(). Table structure and data check is basically the same as in EXCHANGE PARTITION command. And these are done by compare_table_with_partition() and check_table_data(). Atomic DDL is done by the scheme from MDEV-22166 (see the corresponding commit message). The only differnce is that it also has to drop source table frm and that is done by WFRM_DROP_CONVERTED_FROM. Initial patch was done by Dmitry Shulga <dmitry.shulga@mariadb.com>	4 years ago
Aleksey Midenkov	b7bba721ee	MDEV-22166 CONVERT PARTITION: move out partition into a table Syntax for CONVERT keyword ALTER TABLE tbl_name [alter_option [, alter_option] ...] \| [partition_options] partition_option: { ... \| CONVERT PARTITION partition_name TO TABLE tbl_name } Examples: ALTER TABLE t1 CONVERT PARTITION p2 TO TABLE tp2; New ALTER_PARTITION_CONVERT_OUT command for fast_alter_partition_table() is done in alter_partition_convert_out() function which basically does ha_rename_table(). Partition to extract is marked with the same flag as dropped partition: PART_TO_BE_DROPPED. Note that we cannot have multiple partitioning commands in one ALTER. For DDL logging basically the principle is the same as for other fast_alter_partition_table() commands. The only difference is that it integrates late Atomic DDL functions and introduces additional phase of WFRM_BACKUP_ORIGINAL. That is required for binlog consistency because otherwise we could not revert back after WFRM_INSTALL_SHADOW is done. And before DDL log is complete if we crash or fail the altered table will be already new but binlog will miss that ALTER command. Note that this is different from all other atomic DDL in that it rolls back until the ddl_log_complete() is done even if everything was done fully before the crash. Test cases added to: parts.alter_table \ parts.partition_debug \ versioning.partition \ atomic.alter_partition	4 years ago
Monty	8d08971c84	Removed CREATE/DROP TABLESPACE and related commands - DISCARD/IMPORT TABLESPACE are the only tablespace commands left - TABLESPACE arguments for CREATE TABLE and ALTER ... ADD PARTITION are ignored. - Tablespace names are not shown anymore in .frm and not shown in information schema Other things - Removed end spaces from sql/CMakeList.txt	4 years ago
xing-zhi, jiang	768c51880a	MDEV-25129 Add KEYWORDS view to the INFORMATION_SCHEMA Add KEYWORDS table and SQL_FUNCTIONS table to INFORMATION_SCHEMA. This commits needs some minor changes when propagated upwards (e.g. func_array in item_create.cc has a termination element that doesn't exist in later versions of MariaDB)	4 years ago
Andrei Elkin	e95f78f475	MDEV-21117 post-push to cover a "custom" xid format Due to wsrep uses its own xid format for its recovery, the xid hashing has to be refined. When a xid object is not in the server "mysql" format, the hash record made to contain the xid also in the full format.	4 years ago
Andrei Elkin	79a2dbc879	MDEV-21117 post-push fixes 1. work around MDEV-25912 to not apply assert at wsrep running time; 2. handle wsrep mode of the server recovery 3. convert hton calls to static binlog_commit ones. 4. satisfy MSAN complain on uninitialized std::pair	5 years ago
Sujatha	6c39eaeb12	MDEV-21117: refine the server binlog-based recovery for semisync Problem: ======= When the semisync master is crashed and restarted as slave it could recover transactions that former slaves may never have seen. A known method existed to clear out all prepared transactions with --tc-heuristic-recover=rollback does not care to adjust binlog accordingly. Fix: === The binlog-based recovery is made to concern of the slave semisync role of post-crash restarted server. No changes in behavior is done to the "normal" binloggging server and the semisync master. When the restarted server is configured with --rpl-semi-sync-slave-enabled=1 the refined recovery attempts to roll back prepared transactions and truncate binlog accordingly. In case of a partially committed (that is committed at least in one of the engine participants) such transaction gets committed. It's guaranteed no (partially as well) committed transactions exist beyond the truncate position. In case there exists a non-transactional replication event (being in a way a committed transaction) past the computed truncate position the recovery ends with an error. As after master crash and failover to slave, the demoted-to-slave ex-master must be ready to face and accept its own (generated by) events, without generally necessary --replicate-same-server-id. So the acceptance conditions are relaxed for the semisync slave to accept own events without that option. While gtid_strict_mode ON ensures no duplicate transaction can be (re-)executed the master_use_gtid=none slave has to be configured with --replicate-same-server-id. NOTE for reviewers. This patch does not handle the user XA which is done in next git commit.	6 years ago
Marko Mäkelä	1bd681c8b3	MDEV-25506 (3 of 3): Do not delete .ibd files before commit This is a complete rewrite of DROP TABLE, also as part of other DDL, such as ALTER TABLE, CREATE TABLE...SELECT, TRUNCATE TABLE. The background DROP TABLE queue hack is removed. If a transaction needs to drop and create a table by the same name (like TRUNCATE TABLE does), it must first rename the table to an internal #sql-ib name. No committed version of the data dictionary will include any #sql-ib tables, because whenever a transaction renames a table to a #sql-ib name, it will also drop that table. Either the rename will be rolled back, or the drop will be committed. Data files will be unlinked after the transaction has been committed and a FILE_RENAME record has been durably written. The file will actually be deleted when the detached file handle returned by fil_delete_tablespace() will be closed, after the latches have been released. It is possible that a purge of the delete of the SYS_INDEXES record for the clustered index will execute fil_delete_tablespace() concurrently with the DDL transaction. In that case, the thread that arrives later will wait for the other thread to finish. HTON_TRUNCATE_REQUIRES_EXCLUSIVE_USE: A new handler flag. ha_innobase::truncate() now requires that all other references to the table be released in advance. This was implemented by Monty. ha_innobase::delete_table(): If CREATE TABLE..SELECT is detected, we will "hijack" the current transaction, drop the table in the current transaction and commit the current transaction. This essentially fixes MDEV-21602. There is a FIXME comment about making the check less failure-prone. ha_innobase::truncate(), ha_innobase::delete_table(): Implement a fast path for temporary tables. We will no longer allow temporary tables to use the adaptive hash index. dict_table_t::mdl_name: The original table name for the purpose of acquiring MDL in purge, to prevent a race condition between a DDL transaction that is dropping a table, and purge processing undo log records of DML that had executed before the DDL operation. For #sql-backup- tables during ALTER TABLE...ALGORITHM=COPY, the dict_table_t::mdl_name will differ from dict_table_t::name. dict_table_t::parse_name(): Use mdl_name instead of name. dict_table_rename_in_cache(): Update mdl_name. For the internal FTS_ tables of FULLTEXT INDEX, purge would acquire MDL on the FTS_ table name, but not on the main table, and therefore it would be able to run concurrently with a DDL transaction that is dropping the table. Previously, the DROP TABLE queue hack prevented a race between purge and DDL. For now, we introduce purge_sys.stop_FTS() to prevent purge from opening any table, while a DDL transaction that may drop FTS_ tables is in progress. The function fts_lock_table(), which will be invoked before the dictionary is locked, will wait for purge to release any table handles. trx_t::drop_table_statistics(): Drop statistics for the table. This replaces dict_stats_drop_index(). We will drop or rename persistent statistics atomically as part of DDL transactions. On lock conflict for dropping statistics, we will fail instantly with DB_LOCK_WAIT_TIMEOUT, because we will be holding the exclusive data dictionary latch. trx_t::commit_cleanup(): Separated from trx_t::commit_in_memory(). Relax an assertion around fts_commit() and allow DB_LOCK_WAIT_TIMEOUT in addition to DB_DUPLICATE_KEY. The call to fts_commit() is entirely misplaced here and may obviously break the consistency of transactions that affect FULLTEXT INDEX. It needs to be fixed separately. dict_table_t::n_foreign_key_checks_running: Remove (MDEV-21175). The counter was a work-around for missing meta-data locking (MDL) on the SQL layer, and not really needed in MariaDB. ER_TABLE_IN_FK_CHECK: Replaced with ER_UNUSED_28. HA_ERR_TABLE_IN_FK_CHECK: Remove. row_ins_check_foreign_constraints(): Do not acquire dict_sys.latch either. The SQL-layer MDL will protect us. This was reviewed by Thirunarayanan Balathandayuthapani and tested by Matthias Leich.	5 years ago
Marko Mäkelä	ab87fc6c7a	Cleanup: Remove handler::update_table_comment() The only call of the virtual member function handler::update_table_comment() was removed in commit `82d28fada7` (MySQL 5.5.53) but the implementation was not removed. The only non-trivial implementation was for InnoDB. The information is now returned via handler::get_foreign_key_create_info() and ha_statistics::delete_length.	5 years ago
Monty	83e529eced	MDEV-18465 Logging of DDL statements during backup Many of the changes was needed to be able to collect and print engine name and table version id's in the ddl log.	5 years ago
Monty	7762ee5dbe	MDEV-25180 Atomic ALTER TABLE MDEV-25604 Atomic DDL: Binlog event written upon recovery does not have default database The purpose of this task is to ensure that ALTER TABLE is atomic even if the MariaDB server would be killed at any point of the alter table. This means that either the ALTER TABLE succeeds (including that triggers, the status tables and the binary log are updated) or things should be reverted to their original state. If the server crashes before the new version is fully up to date and commited, it will revert to the original table and remove all temporary files and tables. If the new version is commited, crash recovery will use the new version, and update triggers, the status tables and the binary log. The one execption is ALTER TABLE .. RENAME .. where no changes are done to table definition. This one will work as RENAME and roll back unless the whole statement completed, including updating the binary log (if enabled). Other changes: - Added handlerton->check_version() function to allow the ddl recovery code to check, in case of inplace alter table, if the table in the storage engine is of the new or old version. - Added handler->table_version() so that an engine can report the current version of the table. This should be changed each time the table definition changes. - Added ha_signal_ddl_recovery_done() and handlerton::signal_ddl_recovery_done() to inform all handlers when ddl recovery has been done. (Needed by InnoDB). - Added handlerton call inplace_alter_table_committed, to signal engine that ddl_log has been closed for the alter table query. - Added new handerton flag HTON_REQUIRES_NOTIFY_TABLEDEF_CHANGED_AFTER_COMMIT to signal when we should call hton->notify_tabledef_changed() during mysql_inplace_alter_table. This was required as MyRocks and InnoDB needed the call at different times. - Added function server_uuid_value() to be able to generate a temporary xid when ddl recovery writes the query to the binary log. This is needed to be able to handle crashes during ddl log recovery. - Moved freeing of the frm definition to end of mysql_alter_table() to remove duplicate code and have a common exit strategy. ------- InnoDB part of atomic ALTER TABLE (Implemented by Marko Mäkelä) innodb_check_version(): Compare the saved dict_table_t::def_trx_id to determine whether an ALTER TABLE operation was committed. We must correctly recover dict_table_t::def_trx_id for this to work. Before purge removes any trace of DB_TRX_ID from system tables, it will make an effort to load the user table into the cache, so that the dict_table_t::def_trx_id can be recovered. ha_innobase::table_version(): return garbage, or the trx_id that would be used for committing an ALTER TABLE operation. In InnoDB, table names starting with #sql-ib will remain special: they will be dropped on startup. This may be revisited later in MDEV-18518 when we implement proper undo logging and rollback for creating or dropping multiple tables in a transaction. Table names starting with #sql will retain some special meaning: dict_table_t::parse_name() will not consider such names for MDL acquisition, and dict_table_rename_in_cache() will treat such names specially when handling FOREIGN KEY constraints. Simplify InnoDB DROP INDEX. Prevent purge wakeup To ensure that dict_table_t::def_trx_id will be recovered correctly in case the server is killed before ddl_log_complete(), we will block the purge of any history in SYS_TABLES, SYS_INDEXES, SYS_COLUMNS between ha_innobase::commit_inplace_alter_table(commit=true) (purge_sys.stop_SYS()) and purge_sys.resume_SYS(). The completion callback purge_sys.resume_SYS() must be between ddl_log_complete() and MDL release. -------- MyRocks support for atomic ALTER TABLE (Implemented by Sergui Petrunia) Implement these SE API functions: - ha_rocksdb::table_version() - hton->check_version = rocksdb_check_versionMyRocks data dictionary now stores table version for each table. (Absence of table version record is interpreted as table_version=0, that is, which means no upgrade changes are needed) - For inplace alter table of a partitioned table, call the underlying handlerton when checking if the table is ok. This assumes that the partition engine commits all changes at once.	5 years ago

1 2 3 4 5 ...

2097 Commits (d98922789ed8421b6e60770c53f36daa70367a8a)