mariadb

Commit Graph

Author	SHA1	Message	Date
Marko Mäkelä	8a9e9d896e	MDEV-13570 Assertion failure !srv_read_only_mode in --innodb-read-only shutdown when buf_resize_thread is active logs_empty_and_mark_files_at_shutdown(): Skip the debug assertion when the buf_resize_thread is active.	8 years ago
Marko Mäkelä	dcdc1c6d09	MDEV-13452 Assertion `!recv_no_log_write' failed in log_reserve_and_open() The debug flag recv_no_log_write prohibits writes of redo log records for modifying page data. The debug assertion was failing when fil_names_clear() was writing the informative MLOG_FILE_NAME and MLOG_CHECKPOINT records which do not modify any data. log_reserve_and_open(), log_write_low(): Remove the debug assertion. log_pad_current_log_block(), mtr_write_log(), mtr_t::Command::prepare_write(): Add the debug assertion.	8 years ago
Marko Mäkelä	72a2de92a1	Avoid a hang when InnoDB startup is aborted during redo log apply buf_flush_page_cleaner_coordinator: In the first loop, use an appropriate termination condition, waiting for !recv_writer_thread_active. logs_empty_and_mark_files_at_shutdown(): Signal recv_sys->flush_start in case the recv_writer_thread was never started, or buf_flush_page_cleaner_coordinator failed to notice its termination. innobase_start_or_create_for_mysql(): Remove a redundant, unreachable condition, and properly release resources when aborting startup due to recv_sys->found_corrupt_log.	8 years ago
Marko Mäkelä	8c71c6aa8b	MDEV-12548 Initial implementation of Mariabackup for MariaDB 10.2 InnoDB I/O and buffer pool interfaces and the redo log format have been changed between MariaDB 10.1 and 10.2, and the backup code has to be adjusted accordingly. The code has been simplified, and many memory leaks have been fixed. Instead of the file name xtrabackup_logfile, the file name ib_logfile0 is being used for the copy of the redo log. Unnecessary InnoDB startup and shutdown and some unnecessary threads have been removed. Some help was provided by Vladislav Vaintroub. Parameters have been cleaned up and aligned with those of MariaDB 10.2. The --dbug option has been added, so that in debug builds, --dbug=d,ib_log can be specified to enable diagnostic messages for processing redo log entries. By default, innodb_doublewrite=OFF, so that --prepare works faster. If more crash-safety for --prepare is needed, double buffering can be enabled. The parameter innodb_log_checksums=OFF can be used to ignore redo log checksums in --backup. Some messages have been cleaned up. Unless --export is specified, Mariabackup will not deal with undo log. The InnoDB mini-transaction redo log is not only about user-level transactions; it is actually about mini-transactions. To avoid confusion, call it the redo log, not transaction log. We disable any undo log processing in --prepare. Because MariaDB 10.2 supports indexed virtual columns, the undo log processing would need to be able to evaluate virtual column expressions. To reduce the amount of code dependencies, we will not process any undo log in prepare. This means that the --export option must be disabled for now. This also means that the following options are redundant and have been removed: xtrabackup --apply-log-only innobackupex --redo-only In addition to disabling any undo log processing, we will disable any further changes to data pages during --prepare, including the change buffer merge. This means that restoring incremental backups should reliably work even when change buffering is being used on the server. Because of this, preparing a backup will not generate any further redo log, and the redo log file can be safely deleted. (If the --export option is enabled in the future, it must generate redo log when processing undo logs and buffered changes.) In --prepare, we cannot easily know if a partial backup was used, especially when restoring a series of incremental backups. So, we simply warn about any missing files, and ignore the redo log for them. FIXME: Enable the --export option. FIXME: Improve the handling of the MLOG_INDEX_LOAD record, and write a test that initiates a backup while an ALGORITHM=INPLACE operation is creating indexes or rebuilding a table. An error should be detected when preparing the backup. FIXME: In --incremental --prepare, xtrabackup_apply_delta() should ensure that if FSP_SIZE is modified, the file size will be adjusted accordingly.	8 years ago
Marko Mäkelä	84e4e4506f	Reduce the granularity of innodb_log_file_size In Mariabackup, we would want the backed-up redo log file size to be a multiple of 512 bytes, or OS_FILE_LOG_BLOCK_SIZE. However, at startup, InnoDB would be picky, requiring the file size to be a multiple of innodb_page_size. Furthermore, InnoDB would require the parameter to be a multiple of one megabyte, while the minimum granularity is 512 bytes. Because the data-file-oriented fil_io() API is being used for writing the InnoDB redo log, writes will for now require innodb_log_file_size to be a multiple of the maximum innodb_page_size (65536 bytes). To complicate matters, InnoDB startup divided srv_log_file_size by UNIV_PAGE_SIZE, so that initially, the unit was bytes, and later it was innodb_page_size. We will simplify this and keep srv_log_file_size in bytes at all times. innobase_log_file_size: Remove. Remove some obsolete checks against overflow on 32-bit systems. srv_log_file_size is always 64 bits, and the maximum size 512GiB in multiples of innodb_page_size always fits in ulint (which is 32 or 64 bits). 512GiB would be 8,388,60864KiB or 134,217,7284KiB. log_init(): Remove the parameter file_size that was always passed as srv_log_file_size. log_set_capacity(): Add a parameter for passing the requested file size. srv_log_file_size_requested: Declare static in srv0start.cc. create_log_file(), create_log_files(), innobase_start_or_create_for_mysql(): Invoke fil_node_create() with srv_log_file_size expressed in multiples of innodb_page_size. innobase_start_or_create_for_mysql(): Require the redo log file sizes to be multiples of 512 bytes.	8 years ago
Marko Mäkelä	859714e73d	Simplify up InnoDB redo log system startup and shutdown recv_sys_init(): Remove the parameter. recv_sys_create(): Merge to recv_sys_init(). recv_sys_mem_free(): Merge to recv_sys_close(). log_mem_free(): Merge to log_shutdown().	8 years ago
Marko Mäkelä	d1e182d603	MDEV-12975 InnoDB redo log minimum size check uses detected file size instead of requested innodb_log_file_size log_calc_max_ages(): Use the requested size in the check, instead of the detected redo log size. The redo log will be resized at startup if it differs from what has been requested.	8 years ago
Marko Mäkelä	5e4f4ec821	MDEV-12975 InnoDB redo log minimum size check uses detected file size instead of requested innodb_log_file_size log_calc_max_ages(): Use the requested size in the check, instead of the detected redo log size. The redo log will be resized at startup if it differs from what has been requested.	8 years ago
Marko Mäkelä	86927cc712	Remove traces of multiple InnoDB redo logs InnoDB never supported more than one copy of a redo log. There were provisions to do that. For Mariabackup, let us clean up this code. log_sys_init(): Renamed from log_init(). log_set_capacity(): Renamed from log_calc_max_ages(). log_init(): Renamed from log_group_init(). Remove the parameters id, space_id. Let the caller invoke log_set_capacity() when needed. log_group_t: Remove id, space_id, log_groups. log_t: Replace log_groups with a single log. recv_find_max_checkpoint(): Declare globally. Remove the first parameter. xtrabackup_choose_lsn_offset(): Remove (dead code).	9 years ago
Jan Lindström	1af8bf39ca	MDEV-12113: install_db shows corruption for rest encryption with innodb_data_file_path=ibdata1:3M; Problem was that FIL_PAGE_FLUSH_LSN_OR_KEY_VERSION field that for encrypted pages even in system datafiles should contain key_version except very first page (0:0) is after encryption overwritten with flush lsn. Ported WL#7990 Repurpose FIL_PAGE_FLUSH_LSN to 10.1 The field FIL_PAGE_FLUSH_LSN_OR_KEY_VERSION is consulted during InnoDB startup. At startup, InnoDB reads the FIL_PAGE_FLUSH_LSN_OR_KEY_VERSION from the first page of each file in the InnoDB system tablespace. If there are multiple files, the minimum and maximum LSN can differ. These numbers are passed to InnoDB startup. Having the number in other files than the first file of the InnoDB system tablespace is not providing much additional value. It is conflicting with other use of the field, such as on InnoDB R-tree index pages and encryption key_version. This worklog will stop writing FIL_PAGE_FLUSH_LSN_OR_KEY_VERSION to other files than the first file of the InnoDB system tablespace (page number 0:0) when system tablespace is encrypted. If tablespace is not encrypted we continue writing FIL_PAGE_FLUSH_LSN_OR_KEY_VERSION to all first pages of system tablespace to avoid unnecessary warnings on downgrade. open_or_create_data_files(): pass only one flushed_lsn parameter xb_load_tablespaces(): pass only one flushed_lsn parameter. buf_page_create(): Improve comment about where FIL_PAGE_FIL_FLUSH_LSN_OR_KEY_VERSION is set. fil_write_flushed_lsn(): A new function, merged from fil_write_lsn_and_arch_no_to_file() and fil_write_flushed_lsn_to_data_files(). Only write to the first page of the system tablespace (page 0:0) if tablespace is encrypted, or write all first pages of system tablespace and invoke fil_flush_file_spaces(FIL_TYPE_TABLESPACE) afterwards. fil_read_first_page(): read flush_lsn and crypt_data only from first datafile. fil_open_single_table_tablespace(): Remove output of LSN, because it was only valid for the system tablespace and the undo tablespaces, not user tablespaces. fil_validate_single_table_tablespace(): Remove output of LSN. checkpoint_now_set(): Use fil_write_flushed_lsn and output a error if operation fails. Remove lsn variable from fsp_open_info. recv_recovery_from_checkpoint_start(): Remove unnecessary second flush_lsn parameter. log_empty_and_mark_files_at_shutdown(): Use fil_writte_flushed_lsn and output error if it fails. open_or_create_data_files(): Pass only one flushed_lsn variable.	9 years ago
Marko Mäkelä	2f29fc3c1c	10.1 additions for MDEV-12052 Shutdown crash presumably due to master thread activity btr_defragment_thread(): Create the thread in the same place as other threads. Do not invoke btr_defragment_shutdown(), because row_drop_tables_for_mysql_in_background() in the master thread can still keep invoking btr_defragment_remove_table(). logs_empty_and_mark_files_at_shutdown(): Wait for btr_defragment_thread() to exit. innobase_start_or_create_for_mysql(), innobase_shutdown_for_mysql(): Skip encryption and scrubbing in innodb_read_only_mode. srv_export_innodb_status(): Do not export encryption or scrubbing statistics in innodb_read_only mode, because the threads will not be running.	9 years ago
Marko Mäkelä	a4d4a5fe82	After-merge fix for MDEV-11638 In commit `360a4a0372` some debug assertions were introduced to the page flushing code in XtraDB. Add these assertions to InnoDB as well, and adjust the InnoDB shutdown so that these assertions will not fail. logs_empty_and_mark_files_at_shutdown(): Advance srv_shutdown_state from the first phase SRV_SHUTDOWN_CLEANUP only after no page-dirtying activity is possible (well, except by srv_master_do_shutdown_tasks(), which will be fixed separately in MDEV-12052). rotate_thread_t::should_shutdown(): Already exit the key rotation threads at the first phase of shutdown (SRV_SHUTDOWN_CLEANUP). page_cleaner_sleep_if_needed(): Do not sleep during shutdown. This change is originally from XtraDB.	9 years ago
Vicențiu Ciorbaru	0af9818240	5.6.36	9 years ago
Marko Mäkelä	67e9c4cf6c	Adapt the test case for Oracle Bug#25385590 buf_chunk_not_freed(), logs_empty_and_mark_files_at_shutdown(): Relax debug assertions when innodb_force_recovery=6 implies innodb_read_only.	9 years ago
Thirunarayanan Balathandayuthapani	2ef1baa75f	Bug #24793413 LOG PARSING BUFFER OVERFLOW Problem: ======== During checkpoint, we are writing all MLOG_FILE_NAME records in one mtr and parse buffer can't be processed till MLOG_MULTI_REC_END. Eventually parse buffer exceeds the RECV_PARSING_BUF_SIZE and eventually it overflows. Fix: === 1) Break the large mtr if it exceeds LOG_CHECKPOINT_FREE_PER_THREAD into multiple mtr during checkpoint. 2) Move the parsing buffer if we are encountering only MLOG_FILE_NAME records. So that it will never exceed the RECV_PARSING_BUF_SIZE. Reviewed-by: Debarun Bannerjee <debarun.bannerjee@oracle.com> Reviewed-by: Rahul M Malik <rahul.m.malik@oracle.com> RB: 14743	9 years ago
Marko Mäkelä	5684aa220c	MDEV-12488 Remove type mismatch in InnoDB printf-like calls Alias the InnoDB ulint and lint data types to size_t and ssize_t, which are the standard names for the machine-word-width data types. Correspondingly, define ULINTPF as "%zu" and introduce ULINTPFx as "%zx". In this way, better compiler warnings for type mismatch are possible. Furthermore, use PRIu64 for that 64-bit format, and define the feature macro __STDC_FORMAT_MACROS to enable it on Red Hat systems. Fix some errors in error messages, and replace some error messages with assertions. Most notably, an IMPORT TABLESPACE error message in InnoDB was displaying the number of columns instead of the mismatching flags.	9 years ago
Jon Olav Hauglid	a77ac6513e	Bug#21153166: REMOVE UNUSED VARIABLES AND CONVERT GLOBAL SYMBOLS TO STATIC Follow-up to Bug#21141390: REMOVE UNUSED FUNCTIONS AND CONVERT GLOBAL SYMBOLS TO STATIC but for variables instead of functions. Was identified with the -Wmissing-variable-declarations compiler warning option supported by Clang 3.6. Reviewed-by: Marko Mäkelä <marko.makela@oracle.com> RB: 9070	11 years ago
Marko Mäkelä	97acc4a1c3	MDEV-12270 Port MySQL 8.0 Bug#21141390 REMOVE UNUSED FUNCTIONS AND CONVERT GLOBAL SYMBOLS TO STATIC InnoDB defines some functions that are not called at all. Other functions are called, but only from the same compilation unit. Remove some function declarations and definitions, and add 'static' keywords. Some symbols must be kept for separately compiled tools, such as innochecksum.	11 years ago
Marko Mäkelä	4e1116b2c6	MDEV-12271 Port MySQL 8.0 Bug#23150562 REMOVE UNIV_MUST_NOT_INLINE AND UNIV_NONINL Also, remove empty .ic files that were not removed by my MySQL commit. Problem: InnoDB used to support a compilation mode that allowed to choose whether the function definitions in .ic files are to be inlined or not. This stopped making sense when InnoDB moved to C++ in MySQL 5.6 (and ha_innodb.cc started to #include .ic files), and more so in MySQL 5.7 when inline methods and functions were introduced in .h files. Solution: Remove all references to UNIV_NONINL and UNIV_MUST_NOT_INLINE from all files, assuming that the symbols are never defined. Remove the files fut0fut.cc and ut0byte.cc which only mattered when UNIV_NONINL was defined.	10 years ago
Marko Mäkelä	e5b155a4e5	MDEV-12091 Shutdown fails to wait for rollback of recovered transactions to finish In the 10.1 InnoDB Plugin, a call os_event_free(buf_flush_event) was misplaced. The event could be triggered by rollback of resurrected transactions while shutdown was in progress. This bug was caught by cmake -DWITH_ASAN testing. This call was only present in the 10.1 InnoDB Plugin, not in other versions, or in XtraDB. That said, the bug affects all InnoDB versions. Shutdown assumes the cessation of any page-dirtying activity, including the activity of the background rollback thread. InnoDB only waited for the background rollback to finish as part of a slow shutdown (innodb_fast_shutdown=0). The default is a clean shutdown (innodb_fast_shutdown=1). In a scenario where InnoDB is killed, restarted, and shut down soon enough, the data files could become corrupted. logs_empty_and_mark_files_at_shutdown(): Wait for the rollback to finish, except if innodb_fast_shutdown=2 (crash-like shutdown) was requested. trx_rollback_or_clean_recovered(): Before choosing the next recovered transaction to roll back, terminate early if non-slow shutdown was initiated. Roll back everything on slow shutdown (innodb_fast_shutdown=0). srv_innodb_monitor_mutex: Declare as static, because the mutex is only used within one module. In 10.2, os_event_destroy() sets the event to a NULL pointer, while os_event_free() in earlier versions did not do that.	9 years ago
Vladislav Vaintroub	7c512138a1	Revert MySQL commit that disables writing on Windows while flush is in progress Reason : after running sysbench tests (rw,update_no_index), in all cases the throughout was considerably (approx 15%) better with the patch removed. Reverted commit info commit `8dc03bee3a` Author: Yasufumi Kinoshita <yasufumi.kinoshita@oracle.com> Date: Wed Nov 20 17:02:24 2013 +0900 Bug #17824101 : WL#7050 CAUSES RW PERFORMANCE REGRESSION AT SOME WINDOWS ENVIRONMENT	9 years ago
Marko Mäkelä	032678ad18	MDEV-12091 Shutdown fails to wait for rollback of recovered transactions to finish In the 10.1 InnoDB Plugin, a call os_event_free(buf_flush_event) was misplaced. The event could be signalled by rollback of resurrected transactions while shutdown was in progress. This bug was caught by cmake -DWITH_ASAN testing. This call was only present in the 10.1 InnoDB Plugin, not in other versions, or in XtraDB. That said, the bug affects all InnoDB versions. Shutdown assumes the cessation of any page-dirtying activity, including the activity of the background rollback thread. InnoDB only waited for the background rollback to finish as part of a slow shutdown (innodb_fast_shutdown=0). The default is a clean shutdown (innodb_fast_shutdown=1). In a scenario where InnoDB is killed, restarted, and shut down soon enough, the data files could become corrupted. logs_empty_and_mark_files_at_shutdown(): Wait for the rollback to finish, except if innodb_fast_shutdown=2 (crash-like shutdown) was requested. trx_rollback_or_clean_recovered(): Before choosing the next recovered transaction to roll back, terminate early if non-slow shutdown was initiated. Roll back everything on slow shutdown (innodb_fast_shutdown=0). srv_innodb_monitor_mutex: Declare as static, because the mutex is only used within one module. After each call to os_event_free(), ensure that the freed event is not reachable via global variables, by setting the relevant variables to NULL.	9 years ago
Vladislav Vaintroub	9928dbe5f6	Add SRV_ALL_O_DIRECT_FSYNC to switch(srv_flush_method) in log0log.c. Also in log_checkpoint(), where fil_flush_file_spaces() was not done on Windows, it is not done.	9 years ago
Vladislav Vaintroub	a98009ab02	MDEV-12201 innodb_flush_method are not available on Windows Remove srv_win_file_flush_method - Rename srv_unix_file_flush_method to srv_file_flush_method, and rename constants to remove UNIX from them, i.e SRV_UNIX_FSYNC=>SRV_FSYNC - Add SRV_ALL_O_DIRECT_FSYNC corresponding to current Windows default (no buffering for either log or data, flush on both log and data) - change os_file_open on Windows to behave identically to Unix wrt O_DIRECT and O_DSYNC settings. map O_DIRECT to FILE_FLAG_NO_BUFFERING and O_DSYNC to FILE_FLAG_WRITE_THROUGH - remove various #ifdef _WIN32	9 years ago
Marko Mäkelä	29a980cf5c	MDEV-11688 follow-up: More robust shutdown after aborted startup. After starting MariaDB 10.2 with an invalid value of --innodb-flush-method= (the empty string), shutdown would attempt to dereference some NULL pointers. This was probably broken in commit `81b7fe9d38` which implemented shutdown after aborted startup. logs_empty_and_mark_files_at_shutdown(): Allow shutdown even if lock_sys, log_sys, or fil_system is NULL. os_aio_free(): Tolerate os_aio_segment_wait_events==NULL. innobase_start_or_create_for_mysql(): Do not invoke srv_init_abort() before initializing all mutexes for the temporary files. innodb_shutdown(): Tolerate buf_pool_ptr==NULL.	9 years ago
Marko Mäkelä	5da6bd7b95	MDEV-11027 InnoDB log recovery is too noisy Provide more useful progress reporting of crash recovery. recv_sys_t::progress_time: The time of the last report. recv_sys_t::report(ib_time_t): Determine whether progress should be reported. recv_scan_print_counter: Remove. log_group_read_log_seg(): After after each I/O request, invoke recv_sys_t::report() and report progress if needed. recv_apply_hashed_log_recs(): Change the return type back to void (DB_SUCCESS was always returned), and rename the parameter to last_batch. At the start of each batch, if there are pages to be recovered, issue a message.	9 years ago
Marko Mäkelä	ab8199f38e	MDEV-12103 Reduce the time of looking for MLOG_CHECKPOINT during crash recovery This fixes MySQL Bug#80788 in MariaDB 10.2.5. When I made the InnoDB crash recovery more robust by implementing WL#7142, I also introduced an extra redo log scan pass that can be shortened. This fix will slightly extend the InnoDB redo log format that I introduced in MySQL 5.7.9 by writing the start LSN of the MLOG_CHECKPOINT mini-transaction to the end of the log checkpoint page, so that recovery can jump straight to it without scanning all the preceding redo log. LOG_CHECKPOINT_END_LSN: At the end of the checkpoint page, the start LSN of the MLOG_CHECKPOINT mini-transaction. Previously, these bytes were written as 0. log_write_checkpoint_info(), log_group_checkpoint(): Add the parameter end_lsn for writing LOG_CHECKPOINT_END_LSN. log_checkpoint(): Remember the LSN at which the MLOG_CHECKPOINT mini-transaction is starting (or at which the redo log ends on shutdown). recv_init_crash_recovery(): Remove. recv_group_scan_log_recs(): Add the parameter checkpoint_lsn. recv_recovery_from_checkpoint_start(): Read LOG_CHECKPOINT_END_LSN and if it is set, start the first scan from it instead of the checkpoint LSN. Improve some messages and remove bogus assertions. recv_parse_log_recs(): Do not skip DBUG_PRINT("ib_log") for some file-level redo log records. recv_parse_or_apply_log_rec_body(): If we have not parsed all redo log between the checkpoint and the corresponding MLOG_CHECKPOINT record, defer the check for MLOG_FILE_DELETE or MLOG_FILE_NAME records to recv_init_crash_recovery_spaces(). recv_init_crash_recovery_spaces(): Refuse recovery if MLOG_FILE_NAME or MLOG_FILE_DELETE records are missing.	9 years ago
Marko Mäkelä	fc673a2c12	MDEV-12127 InnoDB: Assertion failure loop_count < 5 in file log0log.cc As suggested in MySQL Bug#58536, increase the limit in this debug assertion in order to avoid false positives on heavily loaded systems.	9 years ago
Marko Mäkelä	a13a636c74	MDEV-11802 innodb.innodb_bug14676111 fails The function trx_purge_stop() was calling os_event_reset(purge_sys->event) before calling rw_lock_x_lock(&purge_sys->latch). The os_event_set() call in srv_purge_coordinator_suspend() is protected by that X-latch. It would seem a good idea to consistently protect both os_event_set() and os_event_reset() calls with a common mutex or rw-lock in those cases where os_event_set() and os_event_reset() are used like condition variables, tied to changes of shared state. For each os_event_t, we try to document the mutex or rw-lock that is being used. For some events, frequent calls to os_event_set() seem to try to avoid hangs. Some events are never waited for infinitely, only timed waits, and os_event_set() is used for early termination of these waits. os_aio_simulated_put_read_threads_to_sleep(): Define as a null macro on other systems than Windows. TODO: remove this altogether and disable innodb_use_native_aio on Windows. os_aio_segment_wait_events[]: Initialize only if innodb_use_native_aio=0. log_write_flush_to_disk_low(): Invoke log_mutex_enter() at the end, to avoid race conditions when changing the system state. (No potential race condition existed before MySQL 5.7.)	9 years ago
Marko Mäkelä	13493078e9	MDEV-11802 innodb.innodb_bug14676111 fails The function trx_purge_stop() was calling os_event_reset(purge_sys->event) before calling rw_lock_x_lock(&purge_sys->latch). The os_event_set() call in srv_purge_coordinator_suspend() is protected by that X-latch. It would seem a good idea to consistently protect both os_event_set() and os_event_reset() calls with a common mutex or rw-lock in those cases where os_event_set() and os_event_reset() are used like condition variables, tied to changes of shared state. For each os_event_t, we try to document the mutex or rw-lock that is being used. For some events, frequent calls to os_event_set() seem to try to avoid hangs. Some events are never waited for infinitely, only timed waits, and os_event_set() is used for early termination of these waits. os_aio_simulated_put_read_threads_to_sleep(): Define as a null macro on other systems than Windows. TODO: remove this altogether and disable innodb_use_native_aio on Windows. os_aio_segment_wait_events[]: Initialize only if innodb_use_native_aio=0.	9 years ago
Marko Mäkelä	2af28a363c	MDEV-11782: Redefine the innodb_encrypt_log format Write only one encryption key to the checkpoint page. Use 4 bytes of nonce. Encrypt more of each redo log block, only skipping the 4-byte field LOG_BLOCK_HDR_NO which the initialization vector is derived from. Issue notes, not warning messages for rewriting the redo log files. recv_recovery_from_checkpoint_finish(): Do not generate any redo log, because we must avoid that before rewriting the redo log files, or otherwise a crash during a redo log rewrite (removing or adding encryption) may end up making the database unrecoverable. Instead, do these tasks in innobase_start_or_create_for_mysql(). Issue a firm "Missing MLOG_CHECKPOINT" error message. Remove some unreachable code and duplicated error messages for log corruption. LOG_HEADER_FORMAT_ENCRYPTED: A flag for identifying an encrypted redo log format. log_group_t::is_encrypted(), log_t::is_encrypted(): Determine if the redo log is in encrypted format. recv_find_max_checkpoint(): Interpret LOG_HEADER_FORMAT_ENCRYPTED. srv_prepare_to_delete_redo_log_files(): Display NOTE messages about adding or removing encryption. Do not issue warnings for redo log resizing any more. innobase_start_or_create_for_mysql(): Rebuild the redo logs also when the encryption changes. innodb_log_checksums_func_update(): Always use the CRC-32C checksum if innodb_encrypt_log. If needed, issue a warning that innodb_encrypt_log implies innodb_log_checksums. log_group_write_buf(): Compute the checksum on the encrypted block contents, so that transmission errors or incomplete blocks can be detected without decrypting. Rewrite most of the redo log encryption code. Only remember one encryption key at a time (but remember up to 5 when upgrading from the MariaDB 10.1 format.)	9 years ago
Marko Mäkelä	b40a1fbc93	MDEV-11782 WIP: Clean up the code, and add a test. LOG_CHECKPOINT_ARRAY_END, LOG_CHECKPOINT_SIZE: Remove. Change some error messages to refer to MariaDB 10.2.2 instead of MySQL 5.7.9. recv_find_max_checkpoint_0(): Do not abort when decrypting one of the checkpoint pages fails.	9 years ago
Marko Mäkelä	2f00b73a4b	MDEV-11985 Make innodb_read_only shutdown more robust If InnoDB is started in innodb_read_only mode such that recovered incomplete transactions exist at startup (but the redo logs are clean), an assertion will fail at shutdown, because there would exist some non-prepared transactions. logs_empty_and_mark_files_at_shutdown(): Do not wait for incomplete transactions to finish if innodb_read_only or innodb_force_recovery>=3. Wait for purge to finish in only one place. trx_sys_close(): Relax the assertion that would fail first. trx_free_prepared(): Also free recovered TRX_STATE_ACTIVE transactions if innodb_read_only or innodb_force_recovery>=3. Also, revert my earlier fix to MySQL 5.7 because this fix is more generic: Bug#20874411 INNODB SHUTDOWN HANGS IF INNODB_FORCE_RECOVERY>=3 SKIPPED ANY ROLLBACK trx_undo_fake_prepared(): Remove. trx_sys_any_active_transactions(): Revert the changes.	9 years ago
Marko Mäkelä	20e8347447	MDEV-11985 Make innodb_read_only shutdown more robust If InnoDB is started in innodb_read_only mode such that recovered incomplete transactions exist at startup (but the redo logs are clean), an assertion will fail at shutdown, because there would exist some non-prepared transactions. logs_empty_and_mark_files_at_shutdown(): Do not wait for incomplete transactions to finish if innodb_read_only or innodb_force_recovery>=3. Wait for purge to finish in only one place. trx_sys_close(): Relax the assertion that would fail first. trx_free_prepared(): Also free recovered TRX_STATE_ACTIVE transactions if innodb_read_only or innodb_force_recovery>=3.	9 years ago
Marko Mäkelä	81b7fe9d38	Shut down InnoDB after aborted startup. This fixes memory leaks in tests that cause InnoDB startup to fail. buf_pool_free_instance(): Also free buf_pool->flush_rbt, which would normally be freed when crash recovery finishes. fil_node_close_file(), fil_space_free_low(), fil_close_all_files(): Relax some debug assertions to tolerate !srv_was_started. innodb_shutdown(): Renamed from innobase_shutdown_for_mysql(). Changed the return type to void. Do not assume that all subsystems were started. que_init(), que_close(): Remove (empty functions). srv_init(), srv_general_init(): Remove as global functions. srv_free(): Allow srv_sys=NULL. srv_get_active_thread_type(): Only return SRV_PURGE if purge really is running. srv_shutdown_all_bg_threads(): Do not reset srv_start_state. It will be needed by innodb_shutdown(). innobase_start_or_create_for_mysql(): Always call srv_boot() so that innodb_shutdown() can assume that it was called. Make more subsystems dependent on SRV_START_STATE_STAT. srv_shutdown_bg_undo_sources(): Require SRV_START_STATE_STAT. trx_sys_close(): Do not assume purge_sys!=NULL. Do not call buf_dblwr_free(), because the doublewrite buffer can exist while the transaction system does not. logs_empty_and_mark_files_at_shutdown(): Do a faster shutdown if !srv_was_started. recv_sys_close(): Invoke dblwr.pages.clear() which would normally be invoked by buf_dblwr_process(). recv_recovery_from_checkpoint_start(): Always release log_sys->mutex. row_mysql_close(): Allow the subsystem not to exist.	9 years ago
Jan Lindström	71495a1748	MDEV-11849: Fix storage/innobase/* compile warnings	9 years ago
Jan Lindström	6495806e59	MDEV-11254: innodb-use-trim has no effect in 10.2 Problem was that implementation merged from 10.1 was incompatible with InnoDB 5.7. buf0buf.cc: Add functions to return should we punch hole and how big. buf0flu.cc: Add written page to IORequest fil0fil.cc: Remove unneeded status call and add test is sparse files and punch hole supported by file system when tablespace is created. Add call to get file system block size. Used file node is added to IORequest. Added functions to check is punch hole supported and setting punch hole. ha_innodb.cc: Remove unneeded status variables (trim512-32768) and trim_op_saved. Deprecate innodb_use_trim and set it ON by default. Add function to set innodb-use-trim dynamically. dberr.h: Add error code DB_IO_NO_PUNCH_HOLE if punch hole operation fails. fil0fil.h: Add punch_hole variable to fil_space_t and block size to fil_node_t. os0api.h: Header to helper functions on buf0buf.cc and fil0fil.cc for os0file.h os0file.h: Remove unneeded m_block_size from IORequest and add bpage to IORequest to know actual size of the block and m_fil_node to know tablespace file system block size and does it support punch hole. os0file.cc: Add function punch_hole() to IORequest to do punch_hole operation, get the file system block size and determine does file system support sparse files (for punch hole). page0size.h: remove implicit copy disable and use this implicit copy to implement copy_from() function. buf0dblwr.cc, buf0flu.cc, buf0rea.cc, fil0fil.cc, fil0fil.h, os0file.h, os0file.cc, log0log.cc, log0recv.cc: Remove unneeded write_size parameter from fil_io calls. srv0mon.h, srv0srv.h, srv0mon.cc: Remove unneeded trim512-trim32678 status variables. Removed these from monitor tests.	9 years ago
Marko Mäkelä	30f27b0de0	Post-merge fix for MDEV-11638. logs_empty_and_mark_files_at_shutdown(): Wait for the log_scrub_thread to exit.	9 years ago
Marko Mäkelä	719321e78e	MDEV-11638 Encryption causes race conditions in InnoDB shutdown InnoDB shutdown failed to properly take fil_crypt_thread() into account. The encryption threads were signalled to shut down together with other non-critical tasks. This could be much too early in case of slow shutdown, which could need minutes to complete the purge. Furthermore, InnoDB failed to wait for the fil_crypt_thread() to actually exit before proceeding to the final steps of shutdown, causing the race conditions. Furthermore, the log_scrub_thread() was shut down way too early. Also it should remain until the SRV_SHUTDOWN_FLUSH_PHASE. fil_crypt_threads_end(): Remove. This would cause the threads to be terminated way too early. srv_buf_dump_thread_active, srv_dict_stats_thread_active, lock_sys->timeout_thread_active, log_scrub_thread_active, srv_monitor_active, srv_error_monitor_active: Remove a race condition between startup and shutdown, by setting these in the startup thread that creates threads, not in each created thread. In this way, once the flag is cleared, it will remain cleared during shutdown. srv_n_fil_crypt_threads_started, fil_crypt_threads_event: Declare in global rather than static scope. log_scrub_event, srv_log_scrub_thread_active, log_scrub_thread(): Declare in static rather than global scope. Let these be created by log_init() and freed by log_shutdown(). rotate_thread_t::should_shutdown(): Do not shut down before the SRV_SHUTDOWN_FLUSH_PHASE. srv_any_background_threads_are_active(): Remove. These checks now exist in logs_empty_and_mark_files_at_shutdown(). logs_empty_and_mark_files_at_shutdown(): Shut down the threads in the proper order. Keep fil_crypt_thread() and log_scrub_thread() alive until SRV_SHUTDOWN_FLUSH_PHASE, and check that they actually terminate.	9 years ago
Jan Lindström	403f6e9607	MDEV-11705: InnoDB: Failing assertion: (&log_sys->mutex)->is_owned() if server started with innodb-scrub-log Problem was that log_scrub function did not take required log_sys mutex. Background: Unused space in log blocks are padded with MLOG_DUMMY_RECORD if innodb-scrub-log is enabled. As log files are written on circular fashion old log blocks can be reused later for new redo-log entries. Scrubbing pads unused space in log blocks to avoid visibility of the possible old redo-log contents. log_scrub(): Take log_sys mutex log_pad_current_log_block(): Increase srv_stats.n_log_scrubs if padding is done. srv0srv.cc: Set srv_stats.n_log_scrubs to export vars innodb_scrub_log ha_innodb.cc: Export innodb_scrub_log to global status.	9 years ago
Marko Mäkelä	63574f1275	MDEV-11690 Remove UNIV_HOTBACKUP The InnoDB source code contains quite a few references to a closed-source hot backup tool which was originally called InnoDB Hot Backup (ibbackup) and later incorporated in MySQL Enterprise Backup. The open source backup tool XtraBackup uses the full database for recovery. So, the references to UNIV_HOTBACKUP are only cluttering the source code.	9 years ago
Marko Mäkelä	d50cf42bc0	MDEV-9282 Debian: the Lintian complains about "shlib-calls-exit" in ha_innodb.so Replace all exit() calls in InnoDB with abort() [possibly via ut_a()]. Calling exit() in a multi-threaded program is problematic also for the reason that other threads could see corrupted data structures while some data structures are being cleaned up by atexit() handlers or similar. In the long term, all these calls should be replaced with something that returns an error all the way up the call stack.	9 years ago
Sergei Golubchik	1cae1af6f9	MDEV-5800 InnoDB support for indexed vcols * remove old 5.2+ InnoDB support for virtual columns * enable corresponding parts of the innodb-5.7 sources * copy corresponding test cases from 5.7 * copy detailed Alter_inplace_info::HA_ALTER_FLAGS flags from 5.7 - and more detailed detection of changes in fill_alter_inplace_info() * more "innodb compatibility hooks" in sql_class.cc to - create/destroy/reset a THD (used by background purge threads) - find a prelocked table by name - open a table (from a background purge thread) * different from 5.7: - new service thread "thd_destructor_proxy" to make sure all THDs are destroyed at the correct point in time during the server shutdown - proper opening/closing of tables for vcol evaluations in + FK checks (use already opened prelocked tables) + purge threads (open the table, MDLock it, add it to tdc, close when not needed) - cache open tables in vc_templ - avoid unnecessary allocations, reuse table->record[0] and table->s->default_values - not needed in 5.7, because it overcalculates: + tell the server to calculate vcols for an on-going inline ADD INDEX + calculate vcols for correct error messages * update other engines (mroonga/tokudb) accordingly	9 years ago
Sergey Vojtovich	5608a737ea	MDEV-10813 - Clean-up InnoDB atomics, memory barriers and mutexes No point to issue RELEASE memory barrier in os_thread_create_func(): thread creation is full memory barrier. No point to issue os_wmb in rw_lock_set_waiter_flag() and rw_lock_reset_waiter_flag(): this is deadcode and it is unlikely operational anyway. If atomic builtins are unavailable - memory barriers are most certainly unavailable too. RELEASE memory barrier is definitely abused in buf_pool_withdraw_blocks(): most probably it was supposed to commit volatile variable update, which is not what memory barriers actually do. To operate properly it needs corresponding ACQUIRE barrier without an associated atomic operation anyway. ACQUIRE memory barrier is definitely abused in log_write_up_to(): most probably it was supposed to synchronize dirty read of log_sys->write_lsn. To operate properly it needs corresponding RELEASE barrier without an associated atomic operation anyway. Removed a bunch of ACQUIRE memory barriers from InnoDB rwlocks. They're meaningless without corresponding RELEASE memory barriers. Valid usage example of memory barriers without an associated atomic operation: http://en.cppreference.com/w/cpp/atomic/atomic_thread_fence	9 years ago
Jan Lindström	fec844aca8	Merge InnoDB 5.7 from mysql-5.7.14. Contains also: MDEV-10549 mysqld: sql/handler.cc:2692: int handler::ha_index_first(uchar*): Assertion `table_share->tmp_table != NO_TMP_TABLE \|\| m_lock_type != 2' failed. (branch bb-10.2-jan) Unlike MySQL, InnoDB still uses THR_LOCK in MariaDB MDEV-10548 Some of the debug sync waits do not work with InnoDB 5.7 (branch bb-10.2-jan) enable tests that were fixed in MDEV-10549 MDEV-10548 Some of the debug sync waits do not work with InnoDB 5.7 (branch bb-10.2-jan) fix main.innodb_mysql_sync - re-enable online alter for partitioned innodb tables	9 years ago
Jan Lindström	2e814d4702	Merge InnoDB 5.7 from mysql-5.7.9. Contains also MDEV-10547: Test multi_update_innodb fails with InnoDB 5.7 The failure happened because 5.7 has changed the signature of the bool handler::primary_key_is_clustered() const virtual function ("const" was added). InnoDB was using the old signature which caused the function not to be used. MDEV-10550: Parallel replication lock waits/deadlock handling does not work with InnoDB 5.7 Fixed mutexing problem on lock_trx_handle_wait. Note that rpl_parallel and rpl_optimistic_parallel tests still fail. MDEV-10156 : Group commit tests fail on 10.2 InnoDB (branch bb-10.2-jan) Reason: incorrect merge MDEV-10550: Parallel replication can't sync with master in InnoDB 5.7 (branch bb-10.2-jan) Reason: incorrect merge	9 years ago
Monty	addb38f476	More DBUG_PRINT's to make it easier to debug parallel replication	9 years ago
Sergei Golubchik	720e04ff67	5.6.31	9 years ago
Jan Lindström	f448a800e1	MDEV-9422: Checksum errors on restart when killing busy instance that uses encrypted XtraDB tables Analysis: -- InnoDB has n (>0) redo-log files. -- In the first page of redo-log there is 2 checkpoint records on fixed location (checkpoint is not encrypted) -- On every checkpoint record there is up to 5 crypt_keys containing the keys used for encryption/decryption -- On crash recovery we read all checkpoints on every file -- Recovery starts by reading from the latest checkpoint forward -- Problem is that latest checkpoint might not always contain the key we need to decrypt all the redo-log blocks (see MDEV-9422 for one example) -- Furthermore, there is no way to identify is the log block corrupted or encrypted For example checkpoint can contain following keys : write chk: 4 [ chk key ]: [ 5 1 ] [ 4 1 ] [ 3 1 ] [ 2 1 ] [ 1 1 ] so over time we could have a checkpoint write chk: 13 [ chk key ]: [ 14 1 ] [ 13 1 ] [ 12 1 ] [ 11 1 ] [ 10 1 ] killall -9 mysqld causes crash recovery and on crash recovery we read as many checkpoints as there is log files, e.g. read [ chk key ]: [ 13 1 ] [ 12 1 ] [ 11 1 ] [ 10 1 ] [ 9 1 ] read [ chk key ]: [ 14 1 ] [ 13 1 ] [ 12 1 ] [ 11 1 ] [ 10 1 ] [ 9 1 ] This is problematic, as we could still scan log blocks e.g. from checkpoint 4 and we do not know anymore the correct key. CRYPT INFO: for checkpoint 14 search 4 CRYPT INFO: for checkpoint 13 search 4 CRYPT INFO: for checkpoint 12 search 4 CRYPT INFO: for checkpoint 11 search 4 CRYPT INFO: for checkpoint 10 search 4 CRYPT INFO: for checkpoint 9 search 4 (NOTE: NOT FOUND) For every checkpoint, code generated a new encrypted key based on key from encryption plugin and random numbers. Only random numbers are stored on checkpoint. Fix: Generate only one key for every log file. If checkpoint contains only one key, use that key to encrypt/decrypt all log blocks. If checkpoint contains more than one key (this is case for databases created using MariaDB server version 10.1.0 - 10.1.12 if log encryption was used). If looked checkpoint_no is found from keys on checkpoint we use that key to decrypt the log block. For encryption we use always the first key. If the looked checkpoint_no is not found from keys on checkpoint we use the first key. Modified code also so that if log is not encrypted, we do not generate any empty keys. If we have a log block and no keys is found from checkpoint we assume that log block is unencrypted. Log corruption or missing keys is found by comparing log block checksums. If we have a keys but current log block checksum is correct we again assume log block to be unencrypted. This is because current implementation stores checksum only before encryption and new checksum after encryption but before disk write is not stored anywhere.	10 years ago
Sergei Golubchik	86ff4da14d	5.6.27	10 years ago

1 2

86 Commits (8a9e9d896ece93902425d419bb75b38820e7a9b9)