mariadb

Commit Graph

Author	SHA1	Message	Date
Eugene Kosov	e9de6386ad	MDEV-18115 remove now unneeded constraint log_group_max_size: is not needed because redo log do not use fil_io() now	6 years ago
Eugene Kosov	6f2e228529	MDEV-21382 use fdatasync() for redo log where appropriate log_t::files::fdatasync(): syncs only data for every log file os_file_flush_data() pfs_os_file_flush_data_func(): syncs only data for a given file	6 years ago
Eugene Kosov	562c037b48	MDEV-18115 Remove dummy tablespace for the redo log Redo log subsystem was decoupled from tablespace subsystem. It now manages file descriptors for redo log files by itself. FIL_TYPE_LOG: removed, code in various places was simplified SRV_LOG_SPACE_FIRST_ID: renamed to SRV_SPACE_ID_UPPER_BOUND to better match its purpose. Code in various places was simplified fil_n_log_flushes: replaced with log_sys::flushes fil_n_pending_log_flushes: replaced with log_sys::pending_flushes log_t::files::files: redo log file descriptors log_t::files::file_names: redo log file names log_t::files::set_file_names(): set file names without opening them log_t::files::open_files(): opens redo log files log_t::files::read(): treats several files as one big log_t::files::write(): treats several files as one big log_t::files::fsync(): flushes page cache to disk log_t::files::close_files(): closes redo log files fil_open_log_and_system_tablespace_files(): renamed to fil_open_system_tablespace_files() and obviously it now doesn't open redo log files global files[1000]: removed. Why it was needed at all?	6 years ago
Eugene Kosov	7440e61a64	MDEV-18115: Remove the only async write (of redo log) TODO: do not use fil_* functions for redo log files. log_t::checkpoint_lock: remove this lock which was used to wait for async I/O completion. checkpoint_lock_key checkpoint_lock: remove now unneeded globals log_write_checkpoint_info(): remove sync argument because all checkpoint writes are synchronous now log_write_checkpoint_info(): remove sync argument log_group_checkpoint(): merge with the only caller log_complete_checkpoint(): merge with the only caller log_t::complete_checkpoint(): remove by merging with the only caller.	6 years ago
Marko Mäkelä	d67ea8151f	Rename LOG_HEADER_FORMAT_ to log_t::FORMAT_	6 years ago
Marko Mäkelä	6fde0073bf	Rename log_make_checkpoint_at() to log_make_checkpoint() The function was always called with lsn=LSN_MAX. Remove that redundant parameter. Spotted by Thirunarayanan Balathandayuthapani.	6 years ago
Vlad Lesin	d39d5dd2bc	MDEV-20060: Failing assertion: srv_log_file_size <= 512ULL << 30 while preparing backup The general reason why innodb redo log file is limited by 512G is that log_block_convert_lsn_to_no() returns value limited by 1G. But there is no need to have unique log block numbers in log group. The fix removes 512G limit and limits log group size by (uint32_t maximum value) * (minimum page size), which, in turns, can be removed if fil_io() is no longer used for innodb redo log io.	6 years ago
Marko Mäkelä	47cede646b	MDEV-19543 Deprecate and ignore innodb_log_checksums The parameter innodb_log_checksums that was introduced in MariaDB 10.2.2 via mysql/mysql-server@af0acedd885eb7103e319f79d25fda7386ef1506 does not make much sense. The original motivation of introducing this parameter (initially called innodb_log_checksum_algorithm in mysql/mysql-server@22ba38218e1d76c24f69b5a5595ad3bf5933acb0) was that the InnoDB redo log used the slow and insecure innodb algorithm. With hardware or SIMD accelerated CRC-32C, there should be no reason to allow checksums to be disabled on the redo log. The parameter innodb_encrypt_log already implies innodb_log_checksums=ON. Let us deprecate the parameter innodb_log_checksums and always compute redo log checksums, even if innodb_log_checksums=OFF is specified. An upgrade from MariaDB 10.2.2 or later will only be possible after using the default value innodb_log_checksums=ON. If the non-default value innodb_log_checksums=OFF was in effect when the server was shut down, a log block checksum mismatch will be reported and the upgraded server will fail to start up.	6 years ago
Eugene Kosov	56976e60f5	MDEV-13080 [ERROR] InnoDB: Missing MLOG_CHECKPOINT between the checkpoint x and the end y log_buffer_extend(): Do not write to disk. Just allocate new bigger buffer and copy contents of old one to it. Do not acquire write_mutex. log_t::is_extending: Removed as unneeded now. LOG_BUFFER_SIZE: Removed to make the dependence on srv_log_buffer_size visible.	7 years ago
Vicențiu Ciorbaru	15f1e03d46	Follow-up to changing FSF address Some places didn't match the previous rules, making the Floor address wrong. Additional sed rules: sed -i -e 's/Place.Suite ., Boston/Street, Fifth Floor, Boston/g' sed -i -e 's/Suite .*, Boston/Fifth Floor, Boston/g'	7 years ago
Marko Mäkelä	3db94d2403	MDEV-19346: Remove dummy InnoDB log checkpoints log_checkpoint(), log_make_checkpoint_at(): Remove the parameter write_always. It seems that the primary purpose of this parameter was to ensure in the function recv_reset_logs() that both checkpoint header pages will be overwritten, when the function is called from the never-enabled function recv_recovery_from_archive_start(). create_log_files(): Merge recv_reset_logs() to its only caller. Debug instrumentation: Prefer to flush the redo log, instead of triggering a redo log checkpoint. page_header_set_field(): Disable a debug assertion that will always fail due to MDEV-19344, now that we no longer initiate a redo log checkpoint before an injected crash. In recv_reset_logs() there used to be two calls to log_make_checkpoint_at(). The apparent purpose of this was to ensure that both InnoDB redo log checkpoint header pages will be initialized or overwritten. The second call was removed (without any explanation) in MySQL 5.6.3: mysql/mysql-server@4ca37968da54ddc6b3b6628f41428ddba1c79bb8 In MySQL 5.6.8 WL#6494, starting with mysql/mysql-server@00a0ba8ad92569fcf08212b3b8cf046dc8a0ce10 the function recv_reset_logs() was not only invoked during InnoDB data file initialization, but also during a regular startup when the redo log is being resized. mysql/mysql-server@45e91679832219e2593c77185342f11f85232b58 in MySQL 5.7.2 removed the UNIV_LOG_ARCHIVE code, but still did not remove the parameter write_always.	7 years ago
Vlad Lesin	05b84b2568	MDEV-14192: Add debug assertions	7 years ago
Eugene Kosov	cbdc2d9489	Replace log_group_t::file_header_bufs with local array log_group_file_header_flush(): Use a stack-local variable instead of the heap-allocated buffers. Closes #1060	7 years ago
Marko Mäkelä	447e493179	Remove some unnecessary InnoDB #include	7 years ago
Marko Mäkelä	3448ceb02a	MDEV-13564: Implement innodb_unsafe_truncate=ON for compatibility While MariaDB Server 10.2 is not really guaranteed to be compatible with Percona XtraBackup 2.4 (for example, the MySQL 5.7 undo log format change that could be present in XtraBackup, but was reverted from MariaDB in MDEV-12289), we do not want to disrupt users who have deployed xtrabackup and MariaDB Server 10.2 in their environments. With this change, MariaDB 10.2 will continue to use the backup-unsafe TRUNCATE TABLE code, so that neither the undo log nor the redo log formats will change in an incompatible way. Undo tablespace truncation will keep using the redo log only. Recovery or backup with old code will fail to shrink the undo tablespace files, but the contents will be recovered just fine. In the MariaDB Server 10.2 series only, we introduce the configuration parameter innodb_unsafe_truncate and make it ON by default. To allow MariaDB Backup (mariabackup) to work properly with TRUNCATE TABLE operations, use loose_innodb_unsafe_truncate=OFF. MariaDB Server 10.3.10 and later releases will always use the backup-safe TRUNCATE TABLE, and this parameter will not be added there. recv_recovery_rollback_active(): Skip row_mysql_drop_garbage_tables() unless innodb_unsafe_truncate=OFF. It is too unsafe to drop orphan tables if RENAME operations are not transactional within InnoDB. LOG_HEADER_FORMAT_10_3: Replaces LOG_HEADER_FORMAT_CURRENT. log_init(), log_group_file_header_flush(), srv_prepare_to_delete_redo_log_files(), innobase_start_or_create_for_mysql(): Choose the redo log format and subformat based on the value of innodb_unsafe_truncate.	7 years ago
Marko Mäkelä	5567a8c936	MDEV-17138 Reduce redo log volume for undo tablespace initialization Implement a 10.4 redo log format, which extends the 10.3 format by introducing the MLOG_MEMSET record. MLOG_MEMSET: A new redo log record type for filling an area with a byte. mlog_memset(): Write the MLOG_MEMSET record. mlog_parse_nbytes(): Handle MLOG_MEMSET as well. trx_rseg_header_create(): Reduce the redo log volume by making use of mlog_memset() and the zero-initialization that happens inside page allocation. fil_addr_null: Remove. flst_init(): Create a variant that takes a zero-initialized buf_block_t* as a parameter, and only writes the FIL_NULL using mlog_memset(). flst_zero_addr(): A variant of flst_write_addr() that writes a null address using mlog_memset() for the FIL_NULL. The following fixes are replacing some use of MLOG_WRITE_STRING with the more compact MLOG_MEMSET record, or eliminating redundant redo log writes: btr_store_big_rec_extern_fields(): Invoke mlog_memset() for zero-initializing the tail of the ROW_FORMAT=COMPRESSED BLOB page. trx_sysf_create(), trx_rseg_format_upgrade(): Invoke mlog_memset() for zero-initializing the page trailer. fsp_header_init(), trx_rseg_header_create(): Remove redundant zero-initializations.	7 years ago
Marko Mäkelä	980d1bf1a9	MDEV-14717: Prevent crash-downgrade to earlier MariaDB 10.2 A crash-downgrade of a RENAME (or TRUNCATE or table-rebuilding ALTER TABLE or OPTIMIZE TABLE) operation to an earlier 10.2 version would trigger a debug assertion failure during rollback, in trx_roll_pop_top_rec_of_trx(). In a non-debug build, the TRX_UNDO_RENAME_TABLE record would be misinterpreted as an update_undo log record, and typically the file name would be interpreted as DB_TRX_ID,DB_ROLL_PTR,PRIMARY KEY. If a matching record would be found, row_undo_mod() would hit ut_error in switch (node->rec_type). Typically, ut_a(table2 == NULL) would fail when opening the table from SQL. Because of this, we prevent a crash-downgrade to earlier MariaDB 10.2 versions by changing the InnoDB redo log format identifier to the 10.3 identifier, and by introducing a subformat identifier so that 10.2 can continue to refuse crash-downgrade from 10.3 or later. After a clean shutdown, a downgrade to MariaDB 10.2.13 or later would still be possible thanks to MDEV-14909. A downgrade to older 10.2 versions is only possible after removing the log files (not recommended). LOG_HEADER_FORMAT_CURRENT: Change to 103 (originally the 10.3 format). log_group_t: Add subformat. For 10.2, we will use subformat 1, and will refuse crash recovery from any other subformat of the 10.3 format, that is, a genuine 10.3 redo log. recv_find_max_checkpoint(): Allow startup after clean shutdown from a future LOG_HEADER_FORMAT_10_4 (unencrypted only). We cannot handle the encrypted 10.4 redo log block format, which was introduced in MDEV-12041. Allow crash recovery from the original 10.2 format as well as the new format. In Mariabackup --backup, do not allow any startup from 10.3 or 10.4 redo logs. recv_recovery_from_checkpoint_start(): Skip redo log apply for clean 10.3 redo log, but not for the new 10.2 redo log (10.3 format, subformat 1). srv_prepare_to_delete_redo_log_files(): On format or subformat mismatch, set srv_log_file_size = 0, so that we will display the correct message. innobase_start_or_create_for_mysql(): Check for format or subformat mismatch. xtrabackup_backup_func(): Remove debug assertions that were made redundant by the code changes in recv_find_max_checkpoint().	7 years ago
Marko Mäkelä	055a3334ad	MDEV-13564 Mariabackup does not work with TRUNCATE Implement undo tablespace truncation via normal redo logging. Implement TRUNCATE TABLE as a combination of RENAME to #sql-ib name, CREATE, and DROP. Note: Orphan #sql-ib.ibd may be left behind if MariaDB Server 10.2 is killed before the DROP operation is committed. If MariaDB Server 10.2 is killed during TRUNCATE, it is also possible that the old table was renamed to #sql-ib.ibd but the data dictionary will refer to the table using the original name. In MariaDB Server 10.3, RENAME inside InnoDB is transactional, and #sql-* tables will be dropped on startup. So, this new TRUNCATE will be fully crash-safe in 10.3. ha_mroonga::wrapper_truncate(): Pass table options to the underlying storage engine, now that ha_innobase::truncate() will need them. rpl_slave_state::truncate_state_table(): Before truncating mysql.gtid_slave_pos, evict any cached table handles from the table definition cache, so that there will be no stale references to the old table after truncating. == TRUNCATE TABLE == WL#6501 in MySQL 5.7 introduced separate log files for implementing atomic and crash-safe TRUNCATE TABLE, instead of using the InnoDB undo and redo log. Some convoluted logic was added to the InnoDB crash recovery, and some extra synchronization (including a redo log checkpoint) was introduced to make this work. This synchronization has caused performance problems and race conditions, and the extra log files cannot be copied or applied by external backup programs. In order to support crash-upgrade from MariaDB 10.2, we will keep the logic for parsing and applying the extra log files, but we will no longer generate those files in TRUNCATE TABLE. A prerequisite for crash-safe TRUNCATE is a crash-safe RENAME TABLE (with full redo and undo logging and proper rollback). This will be implemented in MDEV-14717. ha_innobase::truncate(): Invoke RENAME, create(), delete_table(). Because RENAME cannot be fully rolled back before MariaDB 10.3 due to missing undo logging, add some explicit rename-back in case the operation fails. ha_innobase::delete(): Introduce a variant that takes sqlcom as a parameter. In TRUNCATE TABLE, we do not want to touch any FOREIGN KEY constraints. ha_innobase::create(): Add the parameters file_per_table, trx. In TRUNCATE, the new table must be created in the same transaction that renames the old table. create_table_info_t::create_table_info_t(): Add the parameters file_per_table, trx. row_drop_table_for_mysql(): Replace a bool parameter with sqlcom. row_drop_table_after_create_fail(): New function, wrapping row_drop_table_for_mysql(). dict_truncate_index_tree_in_mem(), fil_truncate_tablespace(), fil_prepare_for_truncate(), fil_reinit_space_header_for_table(), row_truncate_table_for_mysql(), TruncateLogger, row_truncate_prepare(), row_truncate_rollback(), row_truncate_complete(), row_truncate_fts(), row_truncate_update_system_tables(), row_truncate_foreign_key_checks(), row_truncate_sanity_checks(): Remove. row_upd_check_references_constraints(): Remove a check for TRUNCATE, now that the table is no longer truncated in place. The new test innodb.truncate_foreign uses DEBUG_SYNC to cover some race-condition like scenarios. The test innodb-innodb.truncate does not use any synchronization. We add a redo log subformat to indicate backup-friendly format. MariaDB 10.4 will remove support for the old TRUNCATE logging, so crash-upgrade from old 10.2 or 10.3 to 10.4 will involve limitations. == Undo tablespace truncation == MySQL 5.7 implements undo tablespace truncation. It is only possible when innodb_undo_tablespaces is set to at least 2. The logging is implemented similar to the WL#6501 TRUNCATE, that is, using separate log files and a redo log checkpoint. We can simply implement undo tablespace truncation within a single mini-transaction that reinitializes the undo log tablespace file. Unfortunately, due to the redo log format of some operations, currently, the total redo log written by undo tablespace truncation will be more than the combined size of the truncated undo tablespace. It should be acceptable to have a little more than 1 megabyte of log in a single mini-transaction. This will be fixed in MDEV-17138 in MariaDB Server 10.4. recv_sys_t: Add truncated_undo_spaces[] to remember for which undo tablespaces a MLOG_FILE_CREATE2 record was seen. namespace undo: Remove some unnecessary declarations. fil_space_t::is_being_truncated: Document that this flag now only applies to undo tablespaces. Remove some references. fil_space_t::is_stopping(): Do not refer to is_being_truncated. This check is for tablespaces of tables. Potentially used tablespaces are never truncated any more. buf_dblwr_process(): Suppress the out-of-bounds warning for undo tablespaces. fil_truncate_log(): Write a MLOG_FILE_CREATE2 with a nonzero page number (new size of the tablespace in pages) to inform crash recovery that the undo tablespace size has been reduced. fil_op_write_log(): Relax assertions, so that MLOG_FILE_CREATE2 can be written for undo tablespaces (without .ibd file suffix) for a nonzero page number. os_file_truncate(): Add the parameter allow_shrink=false so that undo tablespaces can actually be shrunk using this function. fil_name_parse(): For undo tablespace truncation, buffer MLOG_FILE_CREATE2 in truncated_undo_spaces[]. recv_read_in_area(): Avoid reading pages for which no redo log records remain buffered, after recv_addr_trim() removed them. trx_rseg_header_create(): Add a FIXME comment that we could write much less redo log. trx_undo_truncate_tablespace(): Reinitialize the undo tablespace in a single mini-transaction, which will be flushed to the redo log before the file size is trimmed. recv_addr_trim(): Discard any redo logs for pages that were logged after the new end of a file, before the truncation LSN. If the rec_list becomes empty, reduce n_addrs. After removing any affected records, actually truncate the file. recv_apply_hashed_log_recs(): Invoke recv_addr_trim() right before applying any log records. The undo tablespace files must be open at this point. buf_flush_or_remove_pages(), buf_flush_dirty_pages(), buf_LRU_flush_or_remove_pages(): Add a parameter for specifying the number of the first page to flush or remove (default 0). trx_purge_initiate_truncate(): Remove the log checkpoints, the extra logging, and some unnecessary crash points. Merge the code from trx_undo_truncate_tablespace(). First, flush all to-be-discarded pages (beyond the new end of the file), then trim the space->size to make the page allocation deterministic. At the only remaining crash injection point, flush the redo log, so that the recovery can be tested.	7 years ago
Marko Mäkelä	f6d4f624eb	MDEV-12041: innodb_encrypt_log key rotation This will change the InnoDB encrypted redo log format only. Unencrypted redo log will keep using the MariaDB 10.3 format. In the new encrypted redo log format, 4 additional bytes will be reserved in the redo log block trailer for storing the encryption key version. For performance reasons, the encryption key rotation (checking if the latest encryption key version is being used) is only done at log_checkpoint(). LOG_HEADER_FORMAT_CURRENT: Remove. LOG_HEADER_FORMAT_ENC_10_4: The encrypted 10.4 format. LOG_BLOCK_KEY: The encryption key version field. LOG_BLOCK_TRL_SIZE: Remove. log_t: Add accessors framing_size(), payload_size(), trailer_offset(), to be used instead of referring to LOG_BLOCK_TRL_SIZE. log_crypt_t: An operation passed to log_crypt(). log_crypt(): Perform decryption, encryption, or encryption with key rotation. Return an error if key rotation at decryption fails. On encryption, keep using the previous key if the rotation fails. At startup, old-format encrypted redo log may be written before the redo log is upgraded (rebuilt) to the latest format. log_write_up_to(): Add the parameter rotate_key=false. log_checkpoint(): Invoke log_write_up_to() with rotate_key=true.	7 years ago
Marko Mäkelä	340c8a2a32	Remove set-but-not-used log_sys.log.state	7 years ago
Marko Mäkelä	a80e410438	Correct comments	8 years ago
Marko Mäkelä	2b27ac8282	Fix many -Wunused-parameter Remove unused InnoDB function parameters and functions. i_s_sys_virtual_fill_table(): Do not allocate heap memory. mtr_is_block_fix(): Replace with mtr_memo_contains(). mtr_is_page_fix(): Replace with mtr_memo_contains_page().	8 years ago
Marko Mäkelä	baa5a43d8c	MDEV-16045: Replace log_group_t with log_t::files There is only one log_sys and only one log_sys.log. log_t::files::create(): Replaces log_init(). log_t::files::close(): Replaces log_group_close(), log_group_close_all(). fil_close_log_files(): if (free) log_sys.log_close(); The callers that passed free=true used to call log_group_close_all(). log_header_read(): Replaces log_group_header_read(). log_t::files::file_header_bufs_ptr: Use a single allocation. log_t::files::file_header_bufs[]: Statically allocate the pointers. log_t::files::set_fields(): Replaces log_group_set_fields(). log_t::files::calc_lsn_offset(): Replaces log_group_calc_lsn_offset(). Simplify the computation by using fewer variables. log_t::files::read_log_seg(): Replaces log_group_read_log_seg(). log_sys_t::complete_checkpoint(): Replaces log_io_complete_checkpoint(). fil_aio_wait(): Move the logic from log_io_complete().	8 years ago
Marko Mäkelä	d73a898d64	MDEV-16045: Allocate log_sys statically There is only one redo log subsystem in InnoDB. Allocate the object statically, to avoid unnecessary dereferencing of the pointer. log_t::create(): Renamed from log_sys_init(). log_t::close(): Renamed from log_shutdown(). log_t::checkpoint_buf_ptr: Remove. Allocate log_t::checkpoint_buf statically.	8 years ago
Marko Mäkelä	715e4f4320	MDEV-12218 Clean up InnoDB parameter validation Bind more InnoDB parameters directly to MYSQL_SYSVAR and remove "shadow variables". innodb_change_buffering: Declare as ENUM, not STRING. innodb_flush_method: Declare as ENUM, not STRING. innodb_log_buffer_size: Bind directly to srv_log_buffer_size, without rounding it to a multiple of innodb_page_size. LOG_BUFFER_SIZE: Remove. SysTablespace::normalize_size(): Renamed from normalize(). innodb_init_params(): A new function to initialize and validate InnoDB startup parameters. innodb_init(): Renamed from innobase_init(). Invoke innodb_init_params() before actually trying to start up InnoDB. srv_start(bool): Renamed from innobase_start_or_create_for_mysql(). Added the input parameter create_new_db. SRV_ALL_O_DIRECT_FSYNC: Define only for _WIN32. xb_normalize_init_values(): Merge to innodb_init_param().	8 years ago
Marko Mäkelä	9ed2b2b2b8	Do not divide or multiply by srv_page_size Instead, shift by srv_page_size_shift.	8 years ago
Marko Mäkelä	a90100d756	Replace univ_page_size and UNIV_PAGE_SIZE Try to use one variable (srv_page_size) for innodb_page_size. Also, replace UNIV_PAGE_SIZE_SHIFT with srv_page_size_shift.	8 years ago
Daniel Black	b600f30786	MDEV-10814: Innodb large allocations - madvise - Don't dump Note: Linux only Core dumps of large buffer pool pages take time and space and pose potential data expose in scenarios where data-at-rest encryption is deployed. Here we use madvise(MADV_DONT_DUMP) on large memory allocations used by the innodb buffer pool, log_sys and recv_sys. The effect of this system call is that these memory areas will not appear in a core dump. Data from these buffers is rarely useful in fault diagnosis. log_sys and recv_sys structures now use large memory allocations for their large buffer. Debug builds don't include the madvise syscall and as such will include full core dumps. A function, buf_madvise_do_dump, is added but never called. It is there to be called from a debugger to re-enable the core dumping of all of these pages if for some reason the entire contents of these buffers are needed. Idea thanks to Hartmut Holzgraefe	9 years ago
Marko Mäkelä	d1cf9b167c	MDEV-14909 MariaDB 10.2 refuses to start up after clean shutdown of MariaDB 10.3 recv_log_recover_10_3(): Determine if a log from MariaDB 10.3 is clean. recv_find_max_checkpoint(): Allow startup with a clean 10.3 redo log. srv_prepare_to_delete_redo_log_files(): When starting up with a 10.3 log, display a "Downgrading redo log" message instead of "Upgrading".	8 years ago
Marko Mäkelä	bae0844f65	Introduce a new InnoDB redo log format for MariaDB 10.3.1 The redo log format will be changed by MDEV-12288, and it could be changed further during MariaDB 10.3 development. We will allow startup from a clean redo log from any earlier InnoDB version (up to MySQL 5.7 or MariaDB 10.3), but we will refuse to do crash recovery from older-format redo logs. recv_log_format_0_recover(): Remove a reference to MySQL documentation, which may be misleading when it comes to MariaDB. recv_log_recover_10_2(): Check if a MariaDB 10.2.2/MySQL 5.7.9 redo log is clean. recv_find_max_checkpoint(): Invoke recv_log_recover_10_2() if the redo log is in the MariaDB 10.2.2 or MySQL 5.7.9 format.	8 years ago
Marko Mäkelä	8c71c6aa8b	MDEV-12548 Initial implementation of Mariabackup for MariaDB 10.2 InnoDB I/O and buffer pool interfaces and the redo log format have been changed between MariaDB 10.1 and 10.2, and the backup code has to be adjusted accordingly. The code has been simplified, and many memory leaks have been fixed. Instead of the file name xtrabackup_logfile, the file name ib_logfile0 is being used for the copy of the redo log. Unnecessary InnoDB startup and shutdown and some unnecessary threads have been removed. Some help was provided by Vladislav Vaintroub. Parameters have been cleaned up and aligned with those of MariaDB 10.2. The --dbug option has been added, so that in debug builds, --dbug=d,ib_log can be specified to enable diagnostic messages for processing redo log entries. By default, innodb_doublewrite=OFF, so that --prepare works faster. If more crash-safety for --prepare is needed, double buffering can be enabled. The parameter innodb_log_checksums=OFF can be used to ignore redo log checksums in --backup. Some messages have been cleaned up. Unless --export is specified, Mariabackup will not deal with undo log. The InnoDB mini-transaction redo log is not only about user-level transactions; it is actually about mini-transactions. To avoid confusion, call it the redo log, not transaction log. We disable any undo log processing in --prepare. Because MariaDB 10.2 supports indexed virtual columns, the undo log processing would need to be able to evaluate virtual column expressions. To reduce the amount of code dependencies, we will not process any undo log in prepare. This means that the --export option must be disabled for now. This also means that the following options are redundant and have been removed: xtrabackup --apply-log-only innobackupex --redo-only In addition to disabling any undo log processing, we will disable any further changes to data pages during --prepare, including the change buffer merge. This means that restoring incremental backups should reliably work even when change buffering is being used on the server. Because of this, preparing a backup will not generate any further redo log, and the redo log file can be safely deleted. (If the --export option is enabled in the future, it must generate redo log when processing undo logs and buffered changes.) In --prepare, we cannot easily know if a partial backup was used, especially when restoring a series of incremental backups. So, we simply warn about any missing files, and ignore the redo log for them. FIXME: Enable the --export option. FIXME: Improve the handling of the MLOG_INDEX_LOAD record, and write a test that initiates a backup while an ALGORITHM=INPLACE operation is creating indexes or rebuilding a table. An error should be detected when preparing the backup. FIXME: In --incremental --prepare, xtrabackup_apply_delta() should ensure that if FSP_SIZE is modified, the file size will be adjusted accordingly.	8 years ago
Marko Mäkelä	84e4e4506f	Reduce the granularity of innodb_log_file_size In Mariabackup, we would want the backed-up redo log file size to be a multiple of 512 bytes, or OS_FILE_LOG_BLOCK_SIZE. However, at startup, InnoDB would be picky, requiring the file size to be a multiple of innodb_page_size. Furthermore, InnoDB would require the parameter to be a multiple of one megabyte, while the minimum granularity is 512 bytes. Because the data-file-oriented fil_io() API is being used for writing the InnoDB redo log, writes will for now require innodb_log_file_size to be a multiple of the maximum innodb_page_size (65536 bytes). To complicate matters, InnoDB startup divided srv_log_file_size by UNIV_PAGE_SIZE, so that initially, the unit was bytes, and later it was innodb_page_size. We will simplify this and keep srv_log_file_size in bytes at all times. innobase_log_file_size: Remove. Remove some obsolete checks against overflow on 32-bit systems. srv_log_file_size is always 64 bits, and the maximum size 512GiB in multiples of innodb_page_size always fits in ulint (which is 32 or 64 bits). 512GiB would be 8,388,60864KiB or 134,217,7284KiB. log_init(): Remove the parameter file_size that was always passed as srv_log_file_size. log_set_capacity(): Add a parameter for passing the requested file size. srv_log_file_size_requested: Declare static in srv0start.cc. create_log_file(), create_log_files(), innobase_start_or_create_for_mysql(): Invoke fil_node_create() with srv_log_file_size expressed in multiples of innodb_page_size. innobase_start_or_create_for_mysql(): Require the redo log file sizes to be multiples of 512 bytes.	8 years ago
Marko Mäkelä	859714e73d	Simplify up InnoDB redo log system startup and shutdown recv_sys_init(): Remove the parameter. recv_sys_create(): Merge to recv_sys_init(). recv_sys_mem_free(): Merge to recv_sys_close(). log_mem_free(): Merge to log_shutdown().	8 years ago
Marko Mäkelä	86927cc712	Remove traces of multiple InnoDB redo logs InnoDB never supported more than one copy of a redo log. There were provisions to do that. For Mariabackup, let us clean up this code. log_sys_init(): Renamed from log_init(). log_set_capacity(): Renamed from log_calc_max_ages(). log_init(): Renamed from log_group_init(). Remove the parameters id, space_id. Let the caller invoke log_set_capacity() when needed. log_group_t: Remove id, space_id, log_groups. log_t: Replace log_groups with a single log. recv_find_max_checkpoint(): Declare globally. Remove the first parameter. xtrabackup_choose_lsn_offset(): Remove (dead code).	9 years ago
Thirunarayanan Balathandayuthapani	2ef1baa75f	Bug #24793413 LOG PARSING BUFFER OVERFLOW Problem: ======== During checkpoint, we are writing all MLOG_FILE_NAME records in one mtr and parse buffer can't be processed till MLOG_MULTI_REC_END. Eventually parse buffer exceeds the RECV_PARSING_BUF_SIZE and eventually it overflows. Fix: === 1) Break the large mtr if it exceeds LOG_CHECKPOINT_FREE_PER_THREAD into multiple mtr during checkpoint. 2) Move the parsing buffer if we are encountering only MLOG_FILE_NAME records. So that it will never exceed the RECV_PARSING_BUF_SIZE. Reviewed-by: Debarun Bannerjee <debarun.bannerjee@oracle.com> Reviewed-by: Rahul M Malik <rahul.m.malik@oracle.com> RB: 14743	9 years ago
Marko Mäkelä	97acc4a1c3	MDEV-12270 Port MySQL 8.0 Bug#21141390 REMOVE UNUSED FUNCTIONS AND CONVERT GLOBAL SYMBOLS TO STATIC InnoDB defines some functions that are not called at all. Other functions are called, but only from the same compilation unit. Remove some function declarations and definitions, and add 'static' keywords. Some symbols must be kept for separately compiled tools, such as innochecksum.	11 years ago
Marko Mäkelä	4e1116b2c6	MDEV-12271 Port MySQL 8.0 Bug#23150562 REMOVE UNIV_MUST_NOT_INLINE AND UNIV_NONINL Also, remove empty .ic files that were not removed by my MySQL commit. Problem: InnoDB used to support a compilation mode that allowed to choose whether the function definitions in .ic files are to be inlined or not. This stopped making sense when InnoDB moved to C++ in MySQL 5.6 (and ha_innodb.cc started to #include .ic files), and more so in MySQL 5.7 when inline methods and functions were introduced in .h files. Solution: Remove all references to UNIV_NONINL and UNIV_MUST_NOT_INLINE from all files, assuming that the symbols are never defined. Remove the files fut0fut.cc and ut0byte.cc which only mattered when UNIV_NONINL was defined.	10 years ago
Marko Mäkelä	ab8199f38e	MDEV-12103 Reduce the time of looking for MLOG_CHECKPOINT during crash recovery This fixes MySQL Bug#80788 in MariaDB 10.2.5. When I made the InnoDB crash recovery more robust by implementing WL#7142, I also introduced an extra redo log scan pass that can be shortened. This fix will slightly extend the InnoDB redo log format that I introduced in MySQL 5.7.9 by writing the start LSN of the MLOG_CHECKPOINT mini-transaction to the end of the log checkpoint page, so that recovery can jump straight to it without scanning all the preceding redo log. LOG_CHECKPOINT_END_LSN: At the end of the checkpoint page, the start LSN of the MLOG_CHECKPOINT mini-transaction. Previously, these bytes were written as 0. log_write_checkpoint_info(), log_group_checkpoint(): Add the parameter end_lsn for writing LOG_CHECKPOINT_END_LSN. log_checkpoint(): Remember the LSN at which the MLOG_CHECKPOINT mini-transaction is starting (or at which the redo log ends on shutdown). recv_init_crash_recovery(): Remove. recv_group_scan_log_recs(): Add the parameter checkpoint_lsn. recv_recovery_from_checkpoint_start(): Read LOG_CHECKPOINT_END_LSN and if it is set, start the first scan from it instead of the checkpoint LSN. Improve some messages and remove bogus assertions. recv_parse_log_recs(): Do not skip DBUG_PRINT("ib_log") for some file-level redo log records. recv_parse_or_apply_log_rec_body(): If we have not parsed all redo log between the checkpoint and the corresponding MLOG_CHECKPOINT record, defer the check for MLOG_FILE_DELETE or MLOG_FILE_NAME records to recv_init_crash_recovery_spaces(). recv_init_crash_recovery_spaces(): Refuse recovery if MLOG_FILE_NAME or MLOG_FILE_DELETE records are missing.	9 years ago
Marko Mäkelä	a13a636c74	MDEV-11802 innodb.innodb_bug14676111 fails The function trx_purge_stop() was calling os_event_reset(purge_sys->event) before calling rw_lock_x_lock(&purge_sys->latch). The os_event_set() call in srv_purge_coordinator_suspend() is protected by that X-latch. It would seem a good idea to consistently protect both os_event_set() and os_event_reset() calls with a common mutex or rw-lock in those cases where os_event_set() and os_event_reset() are used like condition variables, tied to changes of shared state. For each os_event_t, we try to document the mutex or rw-lock that is being used. For some events, frequent calls to os_event_set() seem to try to avoid hangs. Some events are never waited for infinitely, only timed waits, and os_event_set() is used for early termination of these waits. os_aio_simulated_put_read_threads_to_sleep(): Define as a null macro on other systems than Windows. TODO: remove this altogether and disable innodb_use_native_aio on Windows. os_aio_segment_wait_events[]: Initialize only if innodb_use_native_aio=0. log_write_flush_to_disk_low(): Invoke log_mutex_enter() at the end, to avoid race conditions when changing the system state. (No potential race condition existed before MySQL 5.7.)	9 years ago
Marko Mäkelä	13493078e9	MDEV-11802 innodb.innodb_bug14676111 fails The function trx_purge_stop() was calling os_event_reset(purge_sys->event) before calling rw_lock_x_lock(&purge_sys->latch). The os_event_set() call in srv_purge_coordinator_suspend() is protected by that X-latch. It would seem a good idea to consistently protect both os_event_set() and os_event_reset() calls with a common mutex or rw-lock in those cases where os_event_set() and os_event_reset() are used like condition variables, tied to changes of shared state. For each os_event_t, we try to document the mutex or rw-lock that is being used. For some events, frequent calls to os_event_set() seem to try to avoid hangs. Some events are never waited for infinitely, only timed waits, and os_event_set() is used for early termination of these waits. os_aio_simulated_put_read_threads_to_sleep(): Define as a null macro on other systems than Windows. TODO: remove this altogether and disable innodb_use_native_aio on Windows. os_aio_segment_wait_events[]: Initialize only if innodb_use_native_aio=0.	9 years ago
Marko Mäkelä	2af28a363c	MDEV-11782: Redefine the innodb_encrypt_log format Write only one encryption key to the checkpoint page. Use 4 bytes of nonce. Encrypt more of each redo log block, only skipping the 4-byte field LOG_BLOCK_HDR_NO which the initialization vector is derived from. Issue notes, not warning messages for rewriting the redo log files. recv_recovery_from_checkpoint_finish(): Do not generate any redo log, because we must avoid that before rewriting the redo log files, or otherwise a crash during a redo log rewrite (removing or adding encryption) may end up making the database unrecoverable. Instead, do these tasks in innobase_start_or_create_for_mysql(). Issue a firm "Missing MLOG_CHECKPOINT" error message. Remove some unreachable code and duplicated error messages for log corruption. LOG_HEADER_FORMAT_ENCRYPTED: A flag for identifying an encrypted redo log format. log_group_t::is_encrypted(), log_t::is_encrypted(): Determine if the redo log is in encrypted format. recv_find_max_checkpoint(): Interpret LOG_HEADER_FORMAT_ENCRYPTED. srv_prepare_to_delete_redo_log_files(): Display NOTE messages about adding or removing encryption. Do not issue warnings for redo log resizing any more. innobase_start_or_create_for_mysql(): Rebuild the redo logs also when the encryption changes. innodb_log_checksums_func_update(): Always use the CRC-32C checksum if innodb_encrypt_log. If needed, issue a warning that innodb_encrypt_log implies innodb_log_checksums. log_group_write_buf(): Compute the checksum on the encrypted block contents, so that transmission errors or incomplete blocks can be detected without decrypting. Rewrite most of the redo log encryption code. Only remember one encryption key at a time (but remember up to 5 when upgrading from the MariaDB 10.1 format.)	9 years ago
Marko Mäkelä	b40a1fbc93	MDEV-11782 WIP: Clean up the code, and add a test. LOG_CHECKPOINT_ARRAY_END, LOG_CHECKPOINT_SIZE: Remove. Change some error messages to refer to MariaDB 10.2.2 instead of MySQL 5.7.9. recv_find_max_checkpoint_0(): Do not abort when decrypting one of the checkpoint pages fails.	9 years ago
Marko Mäkelä	719321e78e	MDEV-11638 Encryption causes race conditions in InnoDB shutdown InnoDB shutdown failed to properly take fil_crypt_thread() into account. The encryption threads were signalled to shut down together with other non-critical tasks. This could be much too early in case of slow shutdown, which could need minutes to complete the purge. Furthermore, InnoDB failed to wait for the fil_crypt_thread() to actually exit before proceeding to the final steps of shutdown, causing the race conditions. Furthermore, the log_scrub_thread() was shut down way too early. Also it should remain until the SRV_SHUTDOWN_FLUSH_PHASE. fil_crypt_threads_end(): Remove. This would cause the threads to be terminated way too early. srv_buf_dump_thread_active, srv_dict_stats_thread_active, lock_sys->timeout_thread_active, log_scrub_thread_active, srv_monitor_active, srv_error_monitor_active: Remove a race condition between startup and shutdown, by setting these in the startup thread that creates threads, not in each created thread. In this way, once the flag is cleared, it will remain cleared during shutdown. srv_n_fil_crypt_threads_started, fil_crypt_threads_event: Declare in global rather than static scope. log_scrub_event, srv_log_scrub_thread_active, log_scrub_thread(): Declare in static rather than global scope. Let these be created by log_init() and freed by log_shutdown(). rotate_thread_t::should_shutdown(): Do not shut down before the SRV_SHUTDOWN_FLUSH_PHASE. srv_any_background_threads_are_active(): Remove. These checks now exist in logs_empty_and_mark_files_at_shutdown(). logs_empty_and_mark_files_at_shutdown(): Shut down the threads in the proper order. Keep fil_crypt_thread() and log_scrub_thread() alive until SRV_SHUTDOWN_FLUSH_PHASE, and check that they actually terminate.	9 years ago
Marko Mäkelä	63574f1275	MDEV-11690 Remove UNIV_HOTBACKUP The InnoDB source code contains quite a few references to a closed-source hot backup tool which was originally called InnoDB Hot Backup (ibbackup) and later incorporated in MySQL Enterprise Backup. The open source backup tool XtraBackup uses the full database for recovery. So, the references to UNIV_HOTBACKUP are only cluttering the source code.	9 years ago
Marko Mäkelä	2c9bb42d90	MDEV-11432 Change the informational redo log format tag to "MariaDB 10.2.3" MariaDB 10.2 incorporates MySQL 5.7. MySQL 5.7.9 (the first GA release of the series) introduced an informational field to the InnoDB redo log header, which identifies the server version where the redo log files were created (initialized, resized or updated), in WL#8845: InnoDB: Redo log format version identifier. The informational message would be displayed to the user, for example if someone tries to start up MySQL 8.0 after killing a MariaDB 10.2 server. In the current MariaDB 10.2 source code, the identifier string would misleadingly say "MySQL 5.7.14" (using the hard-coded version number in univ.i) instead of "MariaDB 10.2.3" (using the contents of the VERSION file, the build system copies to config.h and my_config.h). This is only a cosmetic change. The compatibility check is based on a numeric identifier. We should probably also change the numeric identifier and some logic around it. MariaDB 10.2 should refuse to recover from a crashed MySQL 5.7 instance, because the redo log might contain references to shared tablespaces, which are not supported by MariaDB 10.2. Also, when MariaDB 10.2 creates an encrypted redo log, there should be a redo log format version tag that will prevent MySQL 5.7 or 8.0 from starting up.	9 years ago
Jan Lindström	fec844aca8	Merge InnoDB 5.7 from mysql-5.7.14. Contains also: MDEV-10549 mysqld: sql/handler.cc:2692: int handler::ha_index_first(uchar*): Assertion `table_share->tmp_table != NO_TMP_TABLE \|\| m_lock_type != 2' failed. (branch bb-10.2-jan) Unlike MySQL, InnoDB still uses THR_LOCK in MariaDB MDEV-10548 Some of the debug sync waits do not work with InnoDB 5.7 (branch bb-10.2-jan) enable tests that were fixed in MDEV-10549 MDEV-10548 Some of the debug sync waits do not work with InnoDB 5.7 (branch bb-10.2-jan) fix main.innodb_mysql_sync - re-enable online alter for partitioned innodb tables	9 years ago
Jan Lindström	2e814d4702	Merge InnoDB 5.7 from mysql-5.7.9. Contains also MDEV-10547: Test multi_update_innodb fails with InnoDB 5.7 The failure happened because 5.7 has changed the signature of the bool handler::primary_key_is_clustered() const virtual function ("const" was added). InnoDB was using the old signature which caused the function not to be used. MDEV-10550: Parallel replication lock waits/deadlock handling does not work with InnoDB 5.7 Fixed mutexing problem on lock_trx_handle_wait. Note that rpl_parallel and rpl_optimistic_parallel tests still fail. MDEV-10156 : Group commit tests fail on 10.2 InnoDB (branch bb-10.2-jan) Reason: incorrect merge MDEV-10550: Parallel replication can't sync with master in InnoDB 5.7 (branch bb-10.2-jan) Reason: incorrect merge	9 years ago
Alexey Kopytov	49ad084503	Few improvements related to CPU cache line size and padding: Bug #79636: CACHE_LINE_SIZE should be 128 on AArch64 Bug #79637: Hard-coded cache line size Bug #79638: Reconcile CACHE_LINE_SIZE with CPU_LEVEL1_DCACHE_LINESIZE Bug #79652: Suspicious padding in srv_conc_t - changed CPU_LEVEL1_DCACHE_LINESIZE to default to 128 bytes on POWER and AArch64 architectures in cases when no value could be detected by CMake using getconf - changed CACHE_LINE_SIZE definition in ut0counter.h to be an alias of CPU_LEVEL1_DCACHE_LINESIZE - changed a number of hard-coded 64-byte cache line size values in the InnoDB code - fixed insufficient padding for srv_conc members in srv0conc.cc Ported to Mariadb by Daniel Black <daniel.black@au.ibm.com> Added s390 cache size of 256 at same time.	10 years ago
Hyeonseok Oh	500b03fe30	Fix spelling: shhould -> should	10 years ago
Jan Lindström	d259376fd8	MDEV-8041: InnoDB redo log encryption Merged new version of InnoDB/XtraDB redo log encryption from Google provided by Jonas Oreland.	11 years ago

1 2 3

106 Commits (e9de6386ad9d53a3e6280722427b26123acc805a)