mariadb

Commit Graph

Author	SHA1	Message	Date
Marko Mäkelä	1eb59c307d	MDEV-24340 Unique final message of InnoDB during shutdown innobase_space_shutdown(): Remove. We want this step to be executed before the message "InnoDB: Shutdown completed; log sequence number " is output by innodb_shutdown(). It used to be executed after that step. innodb_shutdown(): Duplicate the code that used to live in innobase_space_shutdown(). innobase_init_abort(): Merge with innobase_space_shutdown().	5 years ago
Marko Mäkelä	309302a3da	MDEV-23475 InnoDB performance regression for write-heavy workloads In commit `fe39d02f51` (MDEV-20638) we removed some wake-up signaling of the master thread that should have been there, to ensure a steady log checkpointing workload. Common sense suggests that the commit omitted some necessary calls to srv_inc_activity_count(). But, an attempt to add the call to trx_flush_log_if_needed_low() as well as to reinstate the function innobase_active_small() did not restore the performance for the case where sync_binlog=1 is set. Therefore, we will revert the entire commit in MariaDB Server 10.2. In MariaDB Server 10.5, adding a srv_inc_activity_count() call to trx_flush_log_if_needed_low() did restore the performance, so we will not revert MDEV-20638 across all versions.	5 years ago
Thirunarayanan Balathandayuthapani	fe39d02f51	MDEV-20638 Remove the deadcode from srv_master_thread() and srv_active_wake_master_thread_low() - Due to commit `fe95cb2e40` (MDEV-16125), InnoDB master thread does not need to call srv_resume_thread() and therefore there is no need to wake up the thread. Due to the above patch, InnoDB should remove the following dead code. srv_check_activity(): Makes the parameter as in,out and returns the recent activity value innobase_active_small(): Removed srv_active_wake_master_thread(): Removed srv_wake_master_thread(): Removed srv_active_wake_master_thread_low(): Removed Simplify srv_master_thread() and remove switch cases, added the assert. Replace srv_wake_master_thread() with srv_inc_activity_count() INNOBASE_WAKE_INTERVAL: Removed	5 years ago
Marko Mäkelä	57ec42bc32	MDEV-23190 InnoDB data file extension is not crash-safe When InnoDB is extending a data file, it is updating the FSP_SIZE field in the first page of the data file. In commit `8451e09073` (MDEV-11556) we removed a work-around for this bug and made recovery stricter, by making it track changes to FSP_SIZE via redo log records, and extend the data files before any changes are being applied to them. It turns out that the function fsp_fill_free_list() is not crash-safe with respect to this when it is initializing the change buffer bitmap page (page 1, or generally, N*innodb_page_size+1). It uses a separate mini-transaction that is committed (and will be written to the redo log file) before the mini-transaction that actually extended the data file. Hence, recovery can observe a reference to a page that is beyond the current end of the data file. fsp_fill_free_list(): Initialize the change buffer bitmap page in the same mini-transaction. The rest of the changes are fixing a bug that the use of the separate mini-transaction was attempting to work around. Namely, we must ensure that no other thread will access the change buffer bitmap page before our mini-transaction has been committed and all page latches have been released. That is, for read-ahead as well as neighbour flushing, we must avoid accessing pages that might not yet be durably part of the tablespace. fil_space_t::committed_size: The size of the tablespace as persisted by mtr_commit(). fil_space_t::max_page_number_for_io(): Limit the highest page number for I/O batches to committed_size. MTR_MEMO_SPACE_X_LOCK: Replaces MTR_MEMO_X_LOCK for fil_space_t::latch. mtr_x_space_lock(): Replaces mtr_x_lock() for fil_space_t::latch. mtr_memo_slot_release_func(): When releasing MTR_MEMO_SPACE_X_LOCK, copy space->size to space->committed_size. In this way, read-ahead or flushing will never be invoked on pages that do not yet exist according to FSP_SIZE.	5 years ago
Marko Mäkelä	f458b40f66	MDEV-22827 InnoDB: Failing assertion: purge_sys->n_stop == 0 When MDEV-22769 introduced srv_shutdown_state=SRV_SHUTDOWN_INITIATED in commit `efc70da5fd` we forgot to adjust a few checks for SRV_SHUTDOWN_NONE. In the initial shutdown step, we are waiting for the background DROP TABLE queue to be processed or discarded. At that time, some background tasks (such as buffer pool resizing or dumping or encryption key rotation) may be terminated, but others must remain running normally. srv_purge_coordinator_suspend(), srv_purge_coordinator_thread(), srv_start_wait_for_purge_to_start(): Treat SRV_SHUTDOWN_NONE and SRV_SHUTDOWN_INITIATED equally.	5 years ago
Marko Mäkelä	efc70da5fd	MDEV-22769 Shutdown hang or crash due to XA breaking locks The background drop table queue in InnoDB is a work-around for cases where the SQL layer is requesting DDL on tables on which transactional locks exist. One such case are XA transactions. Our test case exploits the fact that the recovery of XA PREPARE transactions will only resurrect InnoDB table locks, but not MDL that should block any concurrent DDL. srv_shutdown_t: Introduce the srv_shutdown_state=SRV_SHUTDOWN_INITIATED for the initial part of shutdown, to wait for the background drop table queue to be emptied. srv_shutdown_bg_undo_sources(): Assign srv_shutdown_state=SRV_SHUTDOWN_INITIATED before waiting for the background drop table queue to be emptied. row_drop_tables_for_mysql_in_background(): On slow shutdown, if no active transactions exist (excluding ones that are in XA PREPARE state), skip any tables on which locks exist. row_drop_table_for_mysql(): Do not unnecessarily attempt to drop InnoDB persistent statistics for tables that have already been added to the background drop table queue. row_mysql_close(): Relax an assertion, and free all memory even if innodb_force_recovery=2 would prevent the background drop table queue from being emptied.	5 years ago
Marko Mäkelä	eba2d10ac5	MDEV-22721 Remove bloat caused by InnoDB logger class Introduce a new ATTRIBUTE_NOINLINE to ib::logger member functions, and add UNIV_UNLIKELY hints to callers. Also, remove some crash reporting output. If needed, the information will be available using debugging tools. Furthermore, remove some fts_enable_diag_print output that included indexed words in raw form. The code seemed to assume that words are NUL-terminated byte strings. It is not clear whether a NUL terminator is always guaranteed to be present. Also, UCS2 or UTF-16 strings would typically contain many NUL bytes.	5 years ago
Thirunarayanan Balathandayuthapani	ad2bf1129c	MDEV-22646 Assertion `table2->cached' failed in dict_table_t::add_to_cache Problem: ======== During buffer pool resizing, InnoDB recreates the dictionary hash tables. Dictionary hash table reuses the heap of AHI hash tables. It leads to memory corruption. Fix: ==== - While disabling AHI, free the heap and AHI hash tables. Recreate the AHI hash tables and assign new heap when AHI is enabled. - btr_blob_free() access invalid page if page was reallocated during buffer poolresizing. So btr_blob_free() should get the page from buf_pool instead of using existing block. - btr_search_enabled and block->index should be checked after acquiring the btr_search_sys latch - Moved the buffer_pool_scan debug sync to earlier before accessing the btr_search_sys latches to avoid the hang of truncate_purge_debug test case - srv_printf_innodb_monitor() should acquire btr_search_sys latches before AHI hash tables.	5 years ago
Vlad Lesin	218d20ffe3	MDEV-22398: mariabackup.innodb_xa_rollback fails on repeat Flush LSN to system tablespace on innodb shutdown if XA is rolled back by mariabackup.	6 years ago
Vlad Lesin	5836191c8f	MDEV-21168: Active XA transactions stop slave from working after backup was restored. Optionally rollback prepared XA's on "mariabackup --prepare". The fix MUST NOT be ported on 10.5+, as MDEV-742 fix solves the issue for slaves.	6 years ago
Marko Mäkelä	6fde0073bf	Rename log_make_checkpoint_at() to log_make_checkpoint() The function was always called with lsn=LSN_MAX. Remove that redundant parameter. Spotted by Thirunarayanan Balathandayuthapani.	6 years ago
Marko Mäkelä	f13471c9fe	MDEV-20060: Remove bogus unit from error message	6 years ago
Vlad Lesin	d39d5dd2bc	MDEV-20060: Failing assertion: srv_log_file_size <= 512ULL << 30 while preparing backup The general reason why innodb redo log file is limited by 512G is that log_block_convert_lsn_to_no() returns value limited by 1G. But there is no need to have unique log block numbers in log group. The fix removes 512G limit and limits log group size by (uint32_t maximum value) * (minimum page size), which, in turns, can be removed if fil_io() is no longer used for innodb redo log io.	6 years ago
Marko Mäkelä	a7e9395f9d	fts_sync_table(), fts_sync() dead code removal fts_sync(): Remove the constant parameter has_dict=false. fts_sync_table(): Remove the constant parameter has_dict=false, and the redundant parameter unlock_cache = !wait. Make wait=true the default parameter.	6 years ago
Marko Mäkelä	f6ea0389a4	Replace ut_timer() with my_interval_timer() The function pointer ut_timer() was only used by the InnoDB defragmenting thread. Let InnoDB use a single monotonic high-precision timer, my_interval_timer() [in nanoseconds], occasionally wrapped by microsecond_interval_timer(). srv_defragment_interval: Change from "timer" units to nanoseconds. This concludes the InnoDB time function cleanup that was motivated by MDEV-14154. Only ut_time_ms() will remain for now, wrapping my_interval_timer().	6 years ago
Thirunarayanan Balathandayuthapani	dca9792a24	MDEV-17228 Encrypted temporary tables are not encrypted - log_crypt_init() should be called from create_log_files(). Because it should re-create the new random nonce compared to the old ones.	6 years ago
Thirunarayanan Balathandayuthapani	723a4b1d78	MDEV-17228 Encrypted temporary tables are not encrypted - Introduce a new variable called innodb_encrypt_temporary_tables which is a boolean variable. It decides whether to encrypt the temporary tablespace. - Encrypts the temporary tablespace based on full checksum format. - Introduced a new counter to track encrypted and decrypted temporary tablespace pages. - Warnings issued if temporary table creation has conflict value with innodb_encrypt_temporary_tables - Added a new test case which reads and writes the pages from/to temporary tablespace.	6 years ago
Vicențiu Ciorbaru	c0ac0b8860	Update FSF address	7 years ago
Marko Mäkelä	3db94d2403	MDEV-19346: Remove dummy InnoDB log checkpoints log_checkpoint(), log_make_checkpoint_at(): Remove the parameter write_always. It seems that the primary purpose of this parameter was to ensure in the function recv_reset_logs() that both checkpoint header pages will be overwritten, when the function is called from the never-enabled function recv_recovery_from_archive_start(). create_log_files(): Merge recv_reset_logs() to its only caller. Debug instrumentation: Prefer to flush the redo log, instead of triggering a redo log checkpoint. page_header_set_field(): Disable a debug assertion that will always fail due to MDEV-19344, now that we no longer initiate a redo log checkpoint before an injected crash. In recv_reset_logs() there used to be two calls to log_make_checkpoint_at(). The apparent purpose of this was to ensure that both InnoDB redo log checkpoint header pages will be initialized or overwritten. The second call was removed (without any explanation) in MySQL 5.6.3: mysql/mysql-server@4ca37968da54ddc6b3b6628f41428ddba1c79bb8 In MySQL 5.6.8 WL#6494, starting with mysql/mysql-server@00a0ba8ad92569fcf08212b3b8cf046dc8a0ce10 the function recv_reset_logs() was not only invoked during InnoDB data file initialization, but also during a regular startup when the redo log is being resized. mysql/mysql-server@45e91679832219e2593c77185342f11f85232b58 in MySQL 5.7.2 removed the UNIV_LOG_ARCHIVE code, but still did not remove the parameter write_always.	7 years ago
Marko Mäkelä	d315b4ff39	Remove IBUF_COUNT_DEBUG The compile-time option IBUF_COUNT_DEBUG has not been used for years. It would only work with up to 3 created .ibd files, with no buffered changes existing while InnoDB is started up.	7 years ago
Marko Mäkelä	169c00994b	MDEV-12699 Improve crash recovery of corrupted data pages InnoDB crash recovery used to read every data page for which redo log exists. This is unnecessary for those pages that are initialized by the redo log. If a newly created page is corrupted, recovery could unnecessarily fail. It would suffice to reinitialize the page based on the redo log records. To add insult to injury, InnoDB crash recovery could hang if it encountered a corrupted page. We will fix also that problem. InnoDB would normally refuse to start up if it encounters a corrupted page on recovery, but that can be overridden by setting innodb_force_recovery=1. Data pages are completely initialized by the records MLOG_INIT_FILE_PAGE2 and MLOG_ZIP_PAGE_COMPRESS. MariaDB 10.4 additionally recognizes MLOG_INIT_FREE_PAGE, which notifies that a page has been freed and its contents can be discarded (filled with zeroes). The record MLOG_INDEX_LOAD notifies that redo logging has been re-enabled after being disabled. We can avoid loading the page if all buffered redo log records predate the MLOG_INDEX_LOAD record. For the internal tables of FULLTEXT INDEX, no MLOG_INDEX_LOAD records were written before commit `aa3f7a107c`. Hence, we will skip these optimizations for tables whose name starts with FTS_. This is joint work with Thirunarayanan Balathandayuthapani. fil_space_t::enable_lsn, file_name_t::enable_lsn: The LSN of the latest recovered MLOG_INDEX_LOAD record for a tablespace. mlog_init: Page initialization operations discovered during redo log scanning. FIXME: This really belongs in recv_sys->addr_hash, and should be removed in MDEV-19176. recv_addr_state: Add the new state RECV_WILL_NOT_READ to indicate that according to mlog_init, the page will be initialized based on redo log record contents. recv_add_to_hash_table(): Set the RECV_WILL_NOT_READ state if appropriate. For now, we do not treat MLOG_ZIP_PAGE_COMPRESS as page initialization. This works around bugs in the crash recovery of ROW_FORMAT=COMPRESSED tables. recv_mark_log_index_load(): Process a MLOG_INDEX_LOAD record by resetting the state to RECV_NOT_PROCESSED and by updating the fil_name_t::enable_lsn. recv_init_crash_recovery_spaces(): Copy fil_name_t::enable_lsn to fil_space_t::enable_lsn. recv_recover_page(): Add the parameter init_lsn, to ignore any log records that precede the page initialization. Add DBUG output about skipped operations. buf_page_create(): Initialize FIL_PAGE_LSN, so that recv_recover_page() will not wrongly skip applying the page-initialization record due to the field containing some newer LSN as a leftover from a different page. Do not invoke ibuf_merge_or_delete_for_page() during crash recovery. recv_apply_hashed_log_recs(): Remove some unnecessary lookups. Note if a corrupted page was found during recovery. After invoking buf_page_create(), do invoke ibuf_merge_or_delete_for_page() via mlog_init.ibuf_merge() in the last recovery batch. ibuf_merge_or_delete_for_page(): Relax a debug assertion. innobase_start_or_create_for_mysql(): Abort startup if a corrupted page was found during recovery. Corrupted pages will not be flagged if innodb_force_recovery is set. However, the recv_sys->found_corrupt_fs flag can be set regardless of innodb_force_recovery if file names are found to be incorrect (for example, multiple files with the same tablespace ID).	7 years ago
Marko Mäkelä	cad56fbaba	MDEV-18733 MariaDB slow start after crash recovery If InnoDB crash recovery was needed, the InnoDB function srv_start() would invoke extra validation, reading something from every InnoDB data file. This should be unnecessary now that MDEV-14717 made RENAME operations crash-safe inside InnoDB (which can be disabled in MariaDB 10.2 by setting innodb_safe_truncate=OFF). dict_check_sys_tables(): Skip tables that would be dropped by row_mysql_drop_garbage_tables(). Perform extra validation only if innodb_safe_truncate=OFF, innodb_force_recovery=0 and crash recovery was needed. dict_load_table_one(): Validate the root page of the table. In this way, we can deny access to corrupted or mismatching tables not only after crash recovery, but also after a clean shutdown.	7 years ago
Marko Mäkelä	b761211685	MDEV-18659: Fix string truncation/overflow in InnoDB and XtraDB Fix the warnings issued by GCC 8 -Wstringop-truncation and -Wstringop-overflow in InnoDB and XtraDB. This work is motivated by Jan Lindström. The patch mainly differs from his original one as follows: (1) We remove explicit initialization of stack-allocated string buffers. The minimum amount of initialization that is needed is a terminating NUL character. (2) GCC issues a warning for invoking strncpy(dest, src, sizeof dest) because if strlen(src) >= sizeof dest, there would be no terminating NUL byte in dest. We avoid this problem by invoking strncpy() with a limit that is 1 less than the buffer size, and by always writing NUL to the last byte of the buffer. (3) We replace strncpy() with memcpy() or strcpy() in those cases when the result is functionally equivalent. Note: fts_fetch_index_words() never deals with len==UNIV_SQL_NULL. This was enforced by an assertion that limits the maximum length to FTS_MAX_WORD_LEN. Also, the encoding that InnoDB uses for the compressed fulltext index is not byte-order agnostic, that is, InnoDB data files that use FULLTEXT INDEX are not portable between big-endian and little-endian systems.	7 years ago
Sergei Golubchik	2a0f1d6132	Bug#28867993: POSSIBLE ISSUE WITH MYSQL SERVER RESTART on startup innodb is checking whether files "ib_logfileN" (for N from 1 to 100) exist, and whether they're readable. A non-existent file aborted the scan. A directory instead of a file made InnoDB to fail. Now it treats "directory exists" as "file doesn't exist".	7 years ago
Marko Mäkelä	e82e216e37	MDEV-17849 Undo tablespace truncation recovery fails to shrink file fil_space_t::add(): Replaces fil_node_create(), fil_node_create_low(). Let the caller pass fil_node_t::handle, to avoid having to close and re-open files. fil_node_t::read_page0(): Refactored from fil_node_open_file(). Read the first page of a data file. fil_node_open_file(): Open the file only once. srv_undo_tablespace_open(): Set the file handle for the opened undo tablespace. This should ensure that ut_ad(file->is_open()) no longer fails in recv_add_trim(). xtrabackup_backup_func(): Remove some dead code. xb_fil_cur_open(): Open files only if needed. Undo tablespaces should already have been opened.	7 years ago
Marko Mäkelä	ff88e4bb8a	Remove many redundant #include from InnoDB	7 years ago
Eugene Kosov	14be814380	MDEV-17491 micro optimize page_id_t page_id_t: remove m_fold member various places: pass page_id_t by value instead of by reference	7 years ago
Marko Mäkelä	81a5b6ccd5	MDEV-17433 Allow InnoDB start up with empty ib_logfile0 from mariabackup --prepare A prepared backup from Mariabackup does not really need to contain any redo log file, because all log will have been applied to the data files. When the user copies a prepared backup to a data directory (overwriting existing files), it could happen that the data directory already contained redo log files from the past. mariabackup --copy-back) would delete the old redo log files, but a user’s own copying script might not do that. To prevent corruption caused by mixing an old redo log file with data files from a backup, starting with MDEV-13311, Mariabackup would create a zero-length ib_logfile0 that would prevent startup. Actually, there is no need to prevent InnoDB from starting up when a single zero-length file ib_logfile0 is present. Only if there exist multiple data files of different lengths, then we should refuse to start up due to inconsistency. A single zero-length ib_logfile0 should be treated as if the log files were missing: create new log files according to the configuration. open_log_file(): Remove. There is no need to open the log files at this point, because os_file_get_status() already determined the size of the file. innobase_start_or_create_for_mysql(): Move the creation of new log files a little later, not when finding out that the first log file does not exist, but after finding out that it does not exist or it exists as a zero-length file.	7 years ago
Marko Mäkelä	b8944e8972	Fix a sign mismatch	7 years ago
Marko Mäkelä	6319c0b541	MDEV-13564: Replace innodb_unsafe_truncate with innodb_safe_truncate Rename the 10.2-specific configuration option innodb_unsafe_truncate to innodb_safe_truncate, and invert its value. The default (for now) is innodb_safe_truncate=OFF, to avoid disrupting users with an undo and redo log format change within a Generally Available (GA) release series.	7 years ago
Marko Mäkelä	3448ceb02a	MDEV-13564: Implement innodb_unsafe_truncate=ON for compatibility While MariaDB Server 10.2 is not really guaranteed to be compatible with Percona XtraBackup 2.4 (for example, the MySQL 5.7 undo log format change that could be present in XtraBackup, but was reverted from MariaDB in MDEV-12289), we do not want to disrupt users who have deployed xtrabackup and MariaDB Server 10.2 in their environments. With this change, MariaDB 10.2 will continue to use the backup-unsafe TRUNCATE TABLE code, so that neither the undo log nor the redo log formats will change in an incompatible way. Undo tablespace truncation will keep using the redo log only. Recovery or backup with old code will fail to shrink the undo tablespace files, but the contents will be recovered just fine. In the MariaDB Server 10.2 series only, we introduce the configuration parameter innodb_unsafe_truncate and make it ON by default. To allow MariaDB Backup (mariabackup) to work properly with TRUNCATE TABLE operations, use loose_innodb_unsafe_truncate=OFF. MariaDB Server 10.3.10 and later releases will always use the backup-safe TRUNCATE TABLE, and this parameter will not be added there. recv_recovery_rollback_active(): Skip row_mysql_drop_garbage_tables() unless innodb_unsafe_truncate=OFF. It is too unsafe to drop orphan tables if RENAME operations are not transactional within InnoDB. LOG_HEADER_FORMAT_10_3: Replaces LOG_HEADER_FORMAT_CURRENT. log_init(), log_group_file_header_flush(), srv_prepare_to_delete_redo_log_files(), innobase_start_or_create_for_mysql(): Choose the redo log format and subformat based on the value of innodb_unsafe_truncate.	7 years ago
Marko Mäkelä	980d1bf1a9	MDEV-14717: Prevent crash-downgrade to earlier MariaDB 10.2 A crash-downgrade of a RENAME (or TRUNCATE or table-rebuilding ALTER TABLE or OPTIMIZE TABLE) operation to an earlier 10.2 version would trigger a debug assertion failure during rollback, in trx_roll_pop_top_rec_of_trx(). In a non-debug build, the TRX_UNDO_RENAME_TABLE record would be misinterpreted as an update_undo log record, and typically the file name would be interpreted as DB_TRX_ID,DB_ROLL_PTR,PRIMARY KEY. If a matching record would be found, row_undo_mod() would hit ut_error in switch (node->rec_type). Typically, ut_a(table2 == NULL) would fail when opening the table from SQL. Because of this, we prevent a crash-downgrade to earlier MariaDB 10.2 versions by changing the InnoDB redo log format identifier to the 10.3 identifier, and by introducing a subformat identifier so that 10.2 can continue to refuse crash-downgrade from 10.3 or later. After a clean shutdown, a downgrade to MariaDB 10.2.13 or later would still be possible thanks to MDEV-14909. A downgrade to older 10.2 versions is only possible after removing the log files (not recommended). LOG_HEADER_FORMAT_CURRENT: Change to 103 (originally the 10.3 format). log_group_t: Add subformat. For 10.2, we will use subformat 1, and will refuse crash recovery from any other subformat of the 10.3 format, that is, a genuine 10.3 redo log. recv_find_max_checkpoint(): Allow startup after clean shutdown from a future LOG_HEADER_FORMAT_10_4 (unencrypted only). We cannot handle the encrypted 10.4 redo log block format, which was introduced in MDEV-12041. Allow crash recovery from the original 10.2 format as well as the new format. In Mariabackup --backup, do not allow any startup from 10.3 or 10.4 redo logs. recv_recovery_from_checkpoint_start(): Skip redo log apply for clean 10.3 redo log, but not for the new 10.2 redo log (10.3 format, subformat 1). srv_prepare_to_delete_redo_log_files(): On format or subformat mismatch, set srv_log_file_size = 0, so that we will display the correct message. innobase_start_or_create_for_mysql(): Check for format or subformat mismatch. xtrabackup_backup_func(): Remove debug assertions that were made redundant by the code changes in recv_find_max_checkpoint().	7 years ago
Marko Mäkelä	055a3334ad	MDEV-13564 Mariabackup does not work with TRUNCATE Implement undo tablespace truncation via normal redo logging. Implement TRUNCATE TABLE as a combination of RENAME to #sql-ib name, CREATE, and DROP. Note: Orphan #sql-ib.ibd may be left behind if MariaDB Server 10.2 is killed before the DROP operation is committed. If MariaDB Server 10.2 is killed during TRUNCATE, it is also possible that the old table was renamed to #sql-ib.ibd but the data dictionary will refer to the table using the original name. In MariaDB Server 10.3, RENAME inside InnoDB is transactional, and #sql-* tables will be dropped on startup. So, this new TRUNCATE will be fully crash-safe in 10.3. ha_mroonga::wrapper_truncate(): Pass table options to the underlying storage engine, now that ha_innobase::truncate() will need them. rpl_slave_state::truncate_state_table(): Before truncating mysql.gtid_slave_pos, evict any cached table handles from the table definition cache, so that there will be no stale references to the old table after truncating. == TRUNCATE TABLE == WL#6501 in MySQL 5.7 introduced separate log files for implementing atomic and crash-safe TRUNCATE TABLE, instead of using the InnoDB undo and redo log. Some convoluted logic was added to the InnoDB crash recovery, and some extra synchronization (including a redo log checkpoint) was introduced to make this work. This synchronization has caused performance problems and race conditions, and the extra log files cannot be copied or applied by external backup programs. In order to support crash-upgrade from MariaDB 10.2, we will keep the logic for parsing and applying the extra log files, but we will no longer generate those files in TRUNCATE TABLE. A prerequisite for crash-safe TRUNCATE is a crash-safe RENAME TABLE (with full redo and undo logging and proper rollback). This will be implemented in MDEV-14717. ha_innobase::truncate(): Invoke RENAME, create(), delete_table(). Because RENAME cannot be fully rolled back before MariaDB 10.3 due to missing undo logging, add some explicit rename-back in case the operation fails. ha_innobase::delete(): Introduce a variant that takes sqlcom as a parameter. In TRUNCATE TABLE, we do not want to touch any FOREIGN KEY constraints. ha_innobase::create(): Add the parameters file_per_table, trx. In TRUNCATE, the new table must be created in the same transaction that renames the old table. create_table_info_t::create_table_info_t(): Add the parameters file_per_table, trx. row_drop_table_for_mysql(): Replace a bool parameter with sqlcom. row_drop_table_after_create_fail(): New function, wrapping row_drop_table_for_mysql(). dict_truncate_index_tree_in_mem(), fil_truncate_tablespace(), fil_prepare_for_truncate(), fil_reinit_space_header_for_table(), row_truncate_table_for_mysql(), TruncateLogger, row_truncate_prepare(), row_truncate_rollback(), row_truncate_complete(), row_truncate_fts(), row_truncate_update_system_tables(), row_truncate_foreign_key_checks(), row_truncate_sanity_checks(): Remove. row_upd_check_references_constraints(): Remove a check for TRUNCATE, now that the table is no longer truncated in place. The new test innodb.truncate_foreign uses DEBUG_SYNC to cover some race-condition like scenarios. The test innodb-innodb.truncate does not use any synchronization. We add a redo log subformat to indicate backup-friendly format. MariaDB 10.4 will remove support for the old TRUNCATE logging, so crash-upgrade from old 10.2 or 10.3 to 10.4 will involve limitations. == Undo tablespace truncation == MySQL 5.7 implements undo tablespace truncation. It is only possible when innodb_undo_tablespaces is set to at least 2. The logging is implemented similar to the WL#6501 TRUNCATE, that is, using separate log files and a redo log checkpoint. We can simply implement undo tablespace truncation within a single mini-transaction that reinitializes the undo log tablespace file. Unfortunately, due to the redo log format of some operations, currently, the total redo log written by undo tablespace truncation will be more than the combined size of the truncated undo tablespace. It should be acceptable to have a little more than 1 megabyte of log in a single mini-transaction. This will be fixed in MDEV-17138 in MariaDB Server 10.4. recv_sys_t: Add truncated_undo_spaces[] to remember for which undo tablespaces a MLOG_FILE_CREATE2 record was seen. namespace undo: Remove some unnecessary declarations. fil_space_t::is_being_truncated: Document that this flag now only applies to undo tablespaces. Remove some references. fil_space_t::is_stopping(): Do not refer to is_being_truncated. This check is for tablespaces of tables. Potentially used tablespaces are never truncated any more. buf_dblwr_process(): Suppress the out-of-bounds warning for undo tablespaces. fil_truncate_log(): Write a MLOG_FILE_CREATE2 with a nonzero page number (new size of the tablespace in pages) to inform crash recovery that the undo tablespace size has been reduced. fil_op_write_log(): Relax assertions, so that MLOG_FILE_CREATE2 can be written for undo tablespaces (without .ibd file suffix) for a nonzero page number. os_file_truncate(): Add the parameter allow_shrink=false so that undo tablespaces can actually be shrunk using this function. fil_name_parse(): For undo tablespace truncation, buffer MLOG_FILE_CREATE2 in truncated_undo_spaces[]. recv_read_in_area(): Avoid reading pages for which no redo log records remain buffered, after recv_addr_trim() removed them. trx_rseg_header_create(): Add a FIXME comment that we could write much less redo log. trx_undo_truncate_tablespace(): Reinitialize the undo tablespace in a single mini-transaction, which will be flushed to the redo log before the file size is trimmed. recv_addr_trim(): Discard any redo logs for pages that were logged after the new end of a file, before the truncation LSN. If the rec_list becomes empty, reduce n_addrs. After removing any affected records, actually truncate the file. recv_apply_hashed_log_recs(): Invoke recv_addr_trim() right before applying any log records. The undo tablespace files must be open at this point. buf_flush_or_remove_pages(), buf_flush_dirty_pages(), buf_LRU_flush_or_remove_pages(): Add a parameter for specifying the number of the first page to flush or remove (default 0). trx_purge_initiate_truncate(): Remove the log checkpoints, the extra logging, and some unnecessary crash points. Merge the code from trx_undo_truncate_tablespace(). First, flush all to-be-discarded pages (beyond the new end of the file), then trim the space->size to make the page allocation deterministic. At the only remaining crash injection point, flush the redo log, so that the recovery can be tested.	7 years ago
Sergei Golubchik	ef64856b97	don't crash on innodb_undo_tablespaces=1	7 years ago
Thirunarayanan Balathandayuthapani	fe95cb2e40	MDEV-16125 Shutdown crash when innodb_force_recovery >= 2 Problem: ======= InnoDB master thread encounters the shutdown state as SRV_SHUTDOWN_FLUSH_PHASE when innodb_force_recovery >=2 and slow scheduling of master thread during shutdown. Fix: ==== There is no need for master thread itself if innodb_force_recovery >=2. Don't create the master thread if innodb_force_recovery >= 2	8 years ago
Marko Mäkelä	6e04af1b78	Remove dead code HAVE_LZO1X MariaDB uses HAVE_LZO, not HAVE_LZO1X (which was never defined). Also, the variable srv_lzo_disabled was never defined or read (only declared and assigned to, in unreachable code).	8 years ago
Marko Mäkelä	4d9969c216	MDEV-15719 ALTER TABLE…ALGORITHM=INPLACE is unnecessarily refused due to innodb_force_recovery ha_innobase::check_if_supported_inplace_alter(): Only check for high_level_read_only. Do not unnecessarily refuse ALTER TABLE...ALGORITHM=INPLACE if innodb_force_recovery was specified as 1, 2, or 3. innobase_start_or_create_for_mysql(): Block all writes from SQL if the system tablespace was initialized with 'newraw'.	8 years ago
Sergey Vojtovich	e147a4a067	Fixed build failure	8 years ago
Eugene Kosov	8d32959b09	fix data races srv_last_monitor_time: make all accesses relaxed atomical WARNING: ThreadSanitizer: data race (pid=12041) Write of size 8 at 0x000003949278 by thread T26 (mutexes: write M226445748578513120): #0 thd_destructor_proxy storage/innobase/handler/ha_innodb.cc:314:14 (mysqld+0x19b5505) Previous read of size 8 at 0x000003949278 by main thread: #0 innobase_init(void) storage/innobase/handler/ha_innodb.cc:4180:11 (mysqld+0x1a03404) #1 ha_initialize_handlerton(st_plugin_int) sql/handler.cc:522:31 (mysqld+0xc5ec73) #2 plugin_initialize(st_mem_root, st_plugin_int, int, char, bool) sql/sql_plugin.cc:1447:9 (mysqld+0x134908d) #3 plugin_init(int, char, int) sql/sql_plugin.cc:1729:15 (mysqld+0x13484f0) #4 init_server_components() sql/mysqld.cc:5345:7 (mysqld+0xbf720f) #5 mysqld_main(int, char) sql/mysqld.cc:5940:7 (mysqld+0xbf107d) #6 main sql/main.cc:25:10 (mysqld+0xbe971b) Location is global 'srv_running' of size 8 at 0x000003949278 (mysqld+0x000003949278) WARNING: ThreadSanitizer: data race (pid=27869) Atomic write of size 4 at 0x7b4800000c00 by thread T8: #0 __tsan_atomic32_exchange llvm/projects/compiler-rt/lib/tsan/rtl/tsan_interface_atomic.cc:589 (mysqld+0xbd4eac) #1 TTASEventMutex<GenericPolicy>::exit() storage/innobase/include/ib0mutex.h:467:7 (mysqld+0x1a8d4cb) #2 PolicyMutex<TTASEventMutex<GenericPolicy> >::exit() storage/innobase/include/ib0mutex.h:609:10 (mysqld+0x1a7839e) #3 fil_validate() storage/innobase/fil/fil0fil.cc:5535:2 (mysqld+0x1abd913) #4 fil_validate_skip() storage/innobase/fil/fil0fil.cc:204:9 (mysqld+0x1aba601) #5 fil_aio_wait(unsigned long) storage/innobase/fil/fil0fil.cc:5296:2 (mysqld+0x1abbae6) #6 io_handler_thread storage/innobase/srv/srv0start.cc:340:3 (mysqld+0x21abe1e) Previous read of size 4 at 0x7b4800000c00 by main thread (mutexes: write M1273, write M1271): #0 TTASEventMutex<GenericPolicy>::state() const storage/innobase/include/ib0mutex.h:530:10 (mysqld+0x21c66e2) #1 sync_array_detect_deadlock(sync_array_t, sync_cell_t, sync_cell_t, unsigned long) storage/innobase/sync/sync0arr.cc:746:14 (mysqld+0x21c1c7a) #2 sync_array_wait_event(sync_array_t, sync_cell_t&) storage/innobase/sync/sync0arr.cc:465:6 (mysqld+0x21c1708) #3 TTASEventMutex<GenericPolicy>::enter(unsigned int, unsigned int, char const, unsigned int) storage/innobase/include/ib0mutex.h:516:6 (mysqld+0x1a8c206) #4 PolicyMutex<TTASEventMutex<GenericPolicy> >::enter(unsigned int, unsigned int, char const, unsigned int) storage/innobase/include/ib0mutex.h:635:10 (mysqld+0x1a782c3) #5 fil_mutex_enter_and_prepare_for_io(unsigned long) storage/innobase/fil/fil0fil.cc:1131:3 (mysqld+0x1a9a92e) #6 fil_io(IORequest const&, bool, page_id_t const&, page_size_t const&, unsigned long, unsigned long, void, void, bool) storage/innobase/fil/fil0fil.cc:5082:2 (mysqld+0x1ab8de2) #7 buf_flush_write_block_low(buf_page_t, buf_flush_t, bool) storage/innobase/buf/buf0flu.cc:1112:3 (mysqld+0x1cb970a) #8 buf_flush_page(buf_pool_t, buf_page_t, buf_flush_t, bool) storage/innobase/buf/buf0flu.cc:1270:3 (mysqld+0x1cb7d70) #9 buf_flush_try_neighbors(page_id_t const&, buf_flush_t, unsigned long, unsigned long) storage/innobase/buf/buf0flu.cc:1493:9 (mysqld+0x1cc9674) #10 buf_flush_page_and_try_neighbors(buf_page_t, buf_flush_t, unsigned long, unsigned long) storage/innobase/buf/buf0flu.cc:1565:13 (mysqld+0x1cbadf3) #11 buf_do_flush_list_batch(buf_pool_t, unsigned long, unsigned long) storage/innobase/buf/buf0flu.cc:1825:3 (mysqld+0x1cbbcb8) #12 buf_flush_batch(buf_pool_t, buf_flush_t, unsigned long, unsigned long, flush_counters_t) storage/innobase/buf/buf0flu.cc:1895:16 (mysqld+0x1cbb459) #13 buf_flush_do_batch(buf_pool_t, buf_flush_t, unsigned long, unsigned long, flush_counters_t) storage/innobase/buf/buf0flu.cc:2065:2 (mysqld+0x1cbcfe1) #14 buf_flush_lists(unsigned long, unsigned long, unsigned long) storage/innobase/buf/buf0flu.cc:2167:8 (mysqld+0x1cbd5a3) #15 log_preflush_pool_modified_pages(unsigned long) storage/innobase/log/log0log.cc:1400:13 (mysqld+0x1eefc3b) #16 log_make_checkpoint_at(unsigned long, bool) storage/innobase/log/log0log.cc:1751:10 (mysqld+0x1eefb16) #17 buf_dblwr_create() storage/innobase/buf/buf0dblwr.cc:335:2 (mysqld+0x1cd2141) #18 innobase_start_or_create_for_mysql() storage/innobase/srv/srv0start.cc:2539:10 (mysqld+0x21b4d8e) #19 innobase_init(void) storage/innobase/handler/ha_innodb.cc:4193:8 (mysqld+0x1a5e3d7) #20 ha_initialize_handlerton(st_plugin_int) sql/handler.cc:522:31 (mysqld+0xc74d33) #21 plugin_initialize(st_mem_root, st_plugin_int, int, char, bool) sql/sql_plugin.cc:1447:9 (mysqld+0x1376d5d) #22 plugin_init(int, char, int) sql/sql_plugin.cc:1729:15 (mysqld+0x13761c0) #23 init_server_components() sql/mysqld.cc:5348:7 (mysqld+0xc0d0ff) #24 mysqld_main(int, char) sql/mysqld.cc:5943:7 (mysqld+0xc06f9d) #25 main sql/main.cc:25:10 (mysqld+0xbff71b) WARNING: ThreadSanitizer: data race (pid=29031) Write of size 8 at 0x0000039e48e0 by thread T15: #0 srv_monitor_thread storage/innobase/srv/srv0srv.cc:1699:24 (mysqld+0x21a254e) Previous write of size 8 at 0x0000039e48e0 by thread T14: #0 srv_refresh_innodb_monitor_stats() storage/innobase/srv/srv0srv.cc:1165:24 (mysqld+0x21a3124) #1 srv_error_monitor_thread storage/innobase/srv/srv0srv.cc:1836:3 (mysqld+0x21a2d40) Location is global 'srv_last_monitor_time' of size 8 at 0x0000039e48e0 (mysqld+0x0000039e48e0)	8 years ago
Marko Mäkelä	03400d974d	Dead code removal: sess_t The session object is not really needed for anything. We can directly create and free the dummy purge_sys->query->trx.	8 years ago
Marko Mäkelä	27ea2963fc	Dead code removal: sess_t The session object is not really needed for anything. We can directly create and free the dummy purge_sys->query->trx.	8 years ago
Marko Mäkelä	f44017384a	MDEV-14968 On upgrade, InnoDB reports "started; log sequence number 0" srv_prepare_to_delete_redo_log_files(): Initialize srv_start_lsn.	8 years ago
Marko Mäkelä	be85c2dc88	Mariabackup --prepare: Do not access transactions or data dictionary innobase_start_or_create_for_mysql(): Only start the data dictionary and transaction subsystems in normal server startup and during mariabackup --export.	8 years ago
Marko Mäkelä	d1cf9b167c	MDEV-14909 MariaDB 10.2 refuses to start up after clean shutdown of MariaDB 10.3 recv_log_recover_10_3(): Determine if a log from MariaDB 10.3 is clean. recv_find_max_checkpoint(): Allow startup with a clean 10.3 redo log. srv_prepare_to_delete_redo_log_files(): When starting up with a 10.3 log, display a "Downgrading redo log" message instead of "Upgrading".	8 years ago
Marko Mäkelä	ce64a65f27	MDEV-14310 Possible corruption by table-rebuilding or index-creating ALTER TABLE…ALGORITHM=INPLACE Also, MDEV-14317 When ALTER TABLE is aborted, do not write garbage pages to data files As pointed out by Shaohua Wang, the merge of MDEV-13328 from MariaDB 10.1 (based on MySQL 5.6) to 10.2 (based on 5.7) was performed incorrectly. Let us always pass a non-NULL FlushObserver* when writing to data files is desired. FlushObserver::is_partial_flush(): Check if this is a bulk-load (partial flush of the tablespace). FlushObserver::is_interrupted(): Check for interrupt status. buf_LRU_flush_or_remove_pages(): Instead of trx_t, take FlushObserver as a parameter. buf_flush_or_remove_pages(): Remove the parameters flush, trx. If observer!=NULL, write out the data pages. Use the new predicate observer->is_partial() to distinguish a partial tablespace flush (after bulk-loading) from a full tablespace flush (export). Return a bool (whether all pages were removed from the flush_list). buf_flush_dirty_pages(): Remove the parameter trx.	8 years ago
Marko Mäkelä	c19ef508b8	InnoDB: Remove ut_snprintf() and the use of my_snprintf(); use snprintf()	8 years ago
Marko Mäkelä	1b478a7aba	MDEV-13311 Presence of old logs in 10.2.7 will corrupt restored instance (change in behavior) Mariabackup 10.2.7 would delete the redo log files after a successful --prepare operation. If the user is manually copying the prepared files instead of using the --copy-back option, it could happen that some old redo log file would be preserved in the restored location. These old redo log files could cause corruption of the restored data files when the server is started up. We prevent this scenario by creating a "poisoned" redo log file ib_logfile0 at the end of the --prepare step. The poisoning consists of simply truncating the file to an empty file. InnoDB will refuse to start up on an empty redo log file. copy_back(): Delete all redo log files in the target if the source file ib_logfile0 is empty. (Previously we did this if the source file is missing.) SRV_OPERATION_RESTORE_EXPORT: A new variant of SRV_OPERATION_RESTORE when the --export option is specified. In this mode, we will keep deleting all redo log files, instead of truncating the first one. delete_log_files(): Add a parameter for the first file to delete, to be passed as 0 or 1. innobase_start_or_create_for_mysql(): In mariabackup --prepare, tolerate an empty ib_logfile0 file. Otherwise, require the first redo log file to be longer than 4 blocks (2048 bytes). Unless --export was specified, truncate the first log file at the end of --prepare.	8 years ago
Vladislav Vaintroub	b731a5bcf2	Innodb : Refactor os_file_set_size() to be compatible 10.1 The last parameter to this function is now,"bool is_sparse", like in 10.1 rather than the unused/useless "bool is_readonly", merged from MySQL 5.7 Like in 10.1, this function now supports sparse files, and efficient platform specific mechanisms for file extension os_file_set_size() is now consistenly used in all places where innodb files are extended.	8 years ago
Vladislav Vaintroub	7354dc6773	MDEV-13384 - misc Windows warnings fixed	8 years ago
Marko Mäkelä	c53f57c390	Remove DICT_UNIVERSAL The only universal index in InnoDB was the change buffer. It suffices to keep the DICT_IBUF flag (which, like DICT_UNIVERSAL, is not written to any persistent data structure).	8 years ago

1 2 3 4 5

237 Commits (1eb59c307de163b507efade1fc372d8ff2bb94b7)