mariadb

Commit Graph

Author	SHA1	Message	Date
Vladislav Vaintroub	5e62b6a5e0	MDEV-16264 Use threadpool for Innodb background work. Almost all threads have gone - the "ticking" threads, that sleep a while then do some work) (srv_monitor_thread, srv_error_monitor_thread, srv_master_thread) were replaced with timers. Some timers are periodic, e.g the "master" timer. - The btr_defragment_thread is also replaced by a timer , which reschedules it self when current defragment "item" needs throttling - the buf_resize_thread and buf_dump_threads are substitutes with tasks Ditto with page cleaner workers. - purge workers threads are not tasks as well, and purge cleaner coordinator is a combination of a task and timer. - All AIO is outsourced to tpool, Innodb just calls thread_pool::submit_io() and provides the callback. - The srv_slot_t was removed, and innodb_debug_sync used in purge is currently not working, and needs reimplementation.	6 years ago
Eugene Kosov	7440e61a64	MDEV-18115: Remove the only async write (of redo log) TODO: do not use fil_* functions for redo log files. log_t::checkpoint_lock: remove this lock which was used to wait for async I/O completion. checkpoint_lock_key checkpoint_lock: remove now unneeded globals log_write_checkpoint_info(): remove sync argument because all checkpoint writes are synchronous now log_write_checkpoint_info(): remove sync argument log_group_checkpoint(): merge with the only caller log_complete_checkpoint(): merge with the only caller log_t::complete_checkpoint(): remove by merging with the only caller.	6 years ago
Marko Mäkelä	38736928e7	Fix -std=c++98 -Wzero-length-array This is another follow-up fix to commit `b393e2cb0c` which turned out to be still broken. Replace the C++11 keyword 'constexpr' with #define. debug_sync_t::str: Remove the zero-length array. Replace sync->str with reinterpret_cast<char*>(&sync[1]).	6 years ago
Marko Mäkelä	1fd1ef25c2	Fix CMAKE_BUILD_TYPE=Debug Remove unused variables and type mismatch that was introduced in commit `b393e2cb0c` Also, fix a typo in the documentation of the parameter, and update the test.	6 years ago
Marko Mäkelä	b42294bc64	MDEV-19514 Defer change buffer merge until pages are requested We will remove the InnoDB background operation of merging buffered changes to secondary index leaf pages. Changes will only be merged as a result of an operation that accesses a secondary index leaf page, such as a SQL statement that performs a lookup via that index, or is modifying the index. Also ROLLBACK and some background operations, such as purging the history of committed transactions, or computing index cardinality statistics, can cause change buffer merge. Encryption key rotation will not perform change buffer merge. The motivation of this change is to simplify the I/O logic and to allow crash recovery to happen in the background (MDEV-14481). We also hope that this will reduce the number of "mystery" crashes due to corrupted data. Because change buffer merge will typically take place as a result of executing SQL statements, there should be a clearer connection between the crash and the SQL statements that were executed when the server crashed. In many cases, a slight performance improvement was observed. This is joint work with Thirunarayanan Balathandayuthapani and was tested by Axel Schwenke and Matthias Leich. The InnoDB monitor counter innodb_ibuf_merge_usec will be removed. On slow shutdown (innodb_fast_shutdown=0), we will continue to merge all buffered changes (and purge all undo log history). Two InnoDB configuration parameters will be changed as follows: innodb_disable_background_merge: Removed. This parameter existed only in debug builds. All change buffer merges will use synchronous reads. innodb_force_recovery will be changed as follows: * innodb_force_recovery=4 will be the same as innodb_force_recovery=3 (the change buffer merge cannot be disabled; it can only happen as a result of an operation that accesses a secondary index leaf page). The option used to be capable of corrupting secondary index leaf pages. Now that capability is removed, and innodb_force_recovery=4 becomes 'safe'. * innodb_force_recovery=5 (which essentially hard-wires SET GLOBAL TRANSACTION ISOLATION LEVEL READ UNCOMMITTED) becomes safe to use. Bogus data can be returned to SQL, but persistent InnoDB data files will not be corrupted further. * innodb_force_recovery=6 (ignore the redo log files) will be the only option that can potentially cause persistent corruption of InnoDB data files. Code changes: buf_page_t::ibuf_exist: New flag, to indicate whether buffered changes exist for a buffer pool page. Pages with pending changes can be returned by buf_page_get_gen(). Previously, the changes were always merged inside buf_page_get_gen() if needed. ibuf_page_exists(const buf_page_t&): Check if a buffered changes exist for an X-latched or read-fixed page. buf_page_get_gen(): Add the parameter allow_ibuf_merge=false. All callers that know that they may be accessing a secondary index leaf page must pass this parameter as allow_ibuf_merge=true, unless it does not matter for that caller whether all buffered changes have been applied. Assert that whenever allow_ibuf_merge holds, the page actually is a leaf page. Attempt change buffer merge only to secondary B-tree index leaf pages. btr_block_get(): Add parameter 'bool merge'. All callers of btr_block_get() should know whether the page could be a secondary index leaf page. If it is not, we should avoid consulting the change buffer bitmap to even consider a merge. This is the main interface to requesting index pages from the buffer pool. ibuf_merge_or_delete_for_page(), recv_recover_page(): Replace buf_page_get_known_nowait() with much simpler logic, because it is now guaranteed that that the block is x-latched or read-fixed. mlog_init_t::mark_ibuf_exist(): Renamed from mlog_init_t::ibuf_merge(). On crash recovery, we will no longer merge any buffered changes for the pages that we read into the buffer pool during the last batch of applying log records. buf_page_get_gen_known_nowait(), BUF_MAKE_YOUNG, BUF_KEEP_OLD: Remove. btr_search_guess_on_hash(): Merge buf_page_get_gen_known_nowait() to its only remaining caller. buf_page_make_young_if_needed(): Define as an inline function. Add the parameter buf_pool. buf_page_peek_if_young(), buf_page_peek_if_too_old(): Add the parameter buf_pool. fil_space_validate_for_mtr_commit(): Remove a bogus comment about background merge of the change buffer. btr_cur_open_at_rnd_pos_func(), btr_cur_search_to_nth_level_func(), btr_cur_open_at_index_side_func(): Use narrower data types and scopes. ibuf_read_merge_pages(): Replaces buf_read_ibuf_merge_pages(). Merge the change buffer by invoking buf_page_get_gen().	6 years ago
Nikita Malyavin	b393e2cb0c	add innodb_debug_sync var to support DEBUG_SYNC from purge threads	6 years ago
Marko Mäkelä	6fde0073bf	Rename log_make_checkpoint_at() to log_make_checkpoint() The function was always called with lsn=LSN_MAX. Remove that redundant parameter. Spotted by Thirunarayanan Balathandayuthapani.	6 years ago
Eugene Kosov	29df1003d9	MDEV-20184 data race at global counter btr_cur_n_non_sea Make all accesses to btr_cur_n_non_sea atomic.	6 years ago
Marko Mäkelä	0c7c61019d	Remove the wrappers ut_time(), ut_difftime(), ib_time_t	6 years ago
Marko Mäkelä	ab6dd77408	MDEV-14154: Remove ut_time_us() Use microsecond_interval_timer() or my_interval_timer() [in nanoseconds] instead.	6 years ago
Marko Mäkelä	97055e6b11	MDEV-14154: Remove ut_time_us() Use microsecond_interval_timer() or my_interval_timer() [in nanoseconds] instead.	6 years ago
Marko Mäkelä	8773bee9f7	MDEV-18582: Fix the 32-bit build	6 years ago
Marko Mäkelä	412533b4a7	MDEV-18582: Extend SHOW STATUS LIKE 'Innodb_%' Many InnoDB internal variables and counters were only exposed in an unstructured fashion via SHOW ENGINE INNODB STATUS. Expose more variables via SHOW STATUS. Many of these were exported in XtraDB. Also, introduce SHOW_SIZE_T and use the proper size for exporting the InnoDB variables. Remove some unnecessary indirection via export_vars, and bind some variables directly. dict_sys_t::rough_size(): Replaces dict_sys_get_size() and includes the hash table sizes. This is based on a contribution by Tony Liu from ServiceNow.	6 years ago
Thirunarayanan Balathandayuthapani	723a4b1d78	MDEV-17228 Encrypted temporary tables are not encrypted - Introduce a new variable called innodb_encrypt_temporary_tables which is a boolean variable. It decides whether to encrypt the temporary tablespace. - Encrypts the temporary tablespace based on full checksum format. - Introduced a new counter to track encrypted and decrypted temporary tablespace pages. - Warnings issued if temporary table creation has conflict value with innodb_encrypt_temporary_tables - Added a new test case which reads and writes the pages from/to temporary tablespace.	6 years ago
Marko Mäkelä	893472d005	MDEV-19570 Deprecate and ignore innodb_undo_logs, remove innodb_rollback_segments The option innodb_rollback_segments was deprecated already in MariaDB Server 10.0. Its misleadingly named replacement innodb_undo_logs is of very limited use. It makes sense to always create and use the maximum number of rollback segments. Let us remove the deprecated parameter innodb_rollback_segments and deprecate&ignore the parameter innodb_undo_logs (to be removed in a later major release). This work involves some cleanup of InnoDB startup. Similar to other write operations, DROP TABLE will no longer be allowed if innodb_force_recovery is set to a value larger than 3.	6 years ago
Marko Mäkelä	1a6f470464	MDEV-19544 Remove innodb_locks_unsafe_for_binlog The transaction isolation levels READ COMMITTED and READ UNCOMMITTED should behave similarly to the old deprecated setting innodb_locks_unsafe_for_binlog=1, that is, avoid acquiring gap locks. row_search_mvcc(): Reduce the scope of some variables, and clean up the initialization and use of the variable set_also_gap_locks.	6 years ago
Marko Mäkelä	198ed24cac	MDEV-19513: Rename dict_operation_lock to dict_sys.latch dict_sys.lock(), dict_sys_lock(): Acquire both mutex and latch. dict_sys.unlock(), dict_sys_unlock(): Release both mutex and latch. dict_sys.assert_locked(): Assert that both mutex and latch are held.	6 years ago
Marko Mäkelä	b93ecea65c	Remove unnecessary pointer indirection for rw_lock_t In MySQL 5.7.8 an extra level of pointer indirection was added to dict_operation_lock and some other rw_lock_t without solid justification, in mysql/mysql-server@52720f1772f9f424bf3dd62fa9c214dd608cd036. Let us revert that change and remove the rather useless rw_lock_t constructor and destructor and the magic_n field. In this way, some unnecessary pointer dereferences and heap allocation will be avoided and debugging might be a little easier.	6 years ago
Vicențiu Ciorbaru	c0ac0b8860	Update FSF address	7 years ago
Marko Mäkelä	3db94d2403	MDEV-19346: Remove dummy InnoDB log checkpoints log_checkpoint(), log_make_checkpoint_at(): Remove the parameter write_always. It seems that the primary purpose of this parameter was to ensure in the function recv_reset_logs() that both checkpoint header pages will be overwritten, when the function is called from the never-enabled function recv_recovery_from_archive_start(). create_log_files(): Merge recv_reset_logs() to its only caller. Debug instrumentation: Prefer to flush the redo log, instead of triggering a redo log checkpoint. page_header_set_field(): Disable a debug assertion that will always fail due to MDEV-19344, now that we no longer initiate a redo log checkpoint before an injected crash. In recv_reset_logs() there used to be two calls to log_make_checkpoint_at(). The apparent purpose of this was to ensure that both InnoDB redo log checkpoint header pages will be initialized or overwritten. The second call was removed (without any explanation) in MySQL 5.6.3: mysql/mysql-server@4ca37968da54ddc6b3b6628f41428ddba1c79bb8 In MySQL 5.6.8 WL#6494, starting with mysql/mysql-server@00a0ba8ad92569fcf08212b3b8cf046dc8a0ce10 the function recv_reset_logs() was not only invoked during InnoDB data file initialization, but also during a regular startup when the redo log is being resized. mysql/mysql-server@45e91679832219e2593c77185342f11f85232b58 in MySQL 5.7.2 removed the UNIV_LOG_ARCHIVE code, but still did not remove the parameter write_always.	7 years ago
Marko Mäkelä	b2dbc781c7	Implement --debug=d,ib_log_checkpoint_avoid Normally, the InnoDB master thread executes InnoDB log checkpoints so frequently that bugs in crash recovery or redo logging can be hard to reproduce. This is because crash recovery would start replaying the log only from the latest checkpoint. Because the InnoDB redo log format only allows saving information for at most 2 latest checkpoints, and because the log files are written in a circular fashion, it would be challenging to implement a debug option that would start the redo log apply from the very start of the redo log file.	7 years ago
Marko Mäkelä	0a1c3477bf	MDEV-18493 Remove page_size_t MySQL 5.7 introduced the class page_size_t and increased the size of buffer pool page descriptors by introducing this object to them. Maybe the intention of this exercise was to prepare for a future where the buffer pool could accommodate multiple page sizes. But that future never arrived, not even in MySQL 8.0. It is much easier to manage a pool of a single page size, and typically all storage devices of an InnoDB instance benefit from using the same page size. Let us remove page_size_t from MariaDB Server. This will make it easier to remove support for ROW_FORMAT=COMPRESSED (or make it a compile-time option) in the future, just by removing various occurrences of zip_size.	7 years ago
Marko Mäkelä	52d13036d8	MDEV-17933 slow server status - dict_sys_get_size() dict_sys_get_size(): Replace the time-consuming loop with a crude estimate that can be computed without holding any mutex. Even before dict_sys->size was removed in MDEV-13325, not all memory allocations by the InnoDB data dictionary cache were being accounted for. One example is foreign key constraints. Another example is virtual column metadata, starting with 10.2.	7 years ago
Brave Galera Crew	36a2a185fe	Galera4	7 years ago
Sergey Vojtovich	65083ba64c	MDEV-17441 - InnoDB transition to C++11 atomics Simplified mutex code. MutexDebug uses OSMutex instead of atomic operations now.	7 years ago
Marko Mäkelä	2465d3e00b	Remove unnecessary __attribute__((unused))	7 years ago
Michael Widenius	88cc78c9c4	Removed compiler warnings	7 years ago
Sergey Vojtovich	28d627392b	MDEV-17441 - InnoDB transition to C++11 atomics srv_sys_t::n_threads_active transition to Atomic_counter.	7 years ago
Sergey Vojtovich	830a7c67a4	MDEV-17441 - InnoDB transition to C++11 atomics Simplified worker threads completion waiting, made it use std::atomic.	7 years ago
Sergey Vojtovich	f401ba477c	MDEV-17441 - InnoDB transition to C++11 atomics purge_sys_t::m_paused transition to Atomic_counter. paused_latched() doesn't make much sense: in this particular case it is as fast as atomic load. The sole caller has to reload it's value anyway, due to rw_lock_x_lock(&purge_sys.latch) issuing acquire memory barrier.	7 years ago
Sergey Vojtovich	edde1f6e0d	MDEV-17441 - InnoDB transition to C++11 atomics trx_sys_t::rseg_history_len transition to Atomic_counter.	7 years ago
Sergey Vojtovich	4404ee2901	MDEV-17441 - InnoDB transition to C++11 atomics os_total_large_mem_allocated transition to Atomic_counter.	7 years ago
Sergey Vojtovich	5c657e9fa5	MDEV-17441 - InnoDB transition to C++11 atomics Trivial srv_running transition.	7 years ago
Marko Mäkelä	ab4bc84420	Follow-up to MDEV-14585: Remove dict_temp_file_num Now that InnoDB startup drops tables whose name starts with #sql, the table ID alone should be enough to create a unique table name. dict_temp_file_num, dict_mem_init(): Remove. dict_mem_create_temporary_tablename(): Use a combination of "#sql-ib" and the ID of the table that is being rebuilt in TRUNCATE TABLE, or discarded in the commit of table-rebuilding ALTER TABLE...ALGORITHM=INPLACE, or deferred to the DROP queue. Thanks to Sergey Vojtovich for pointing this out.	7 years ago
Marko Mäkelä	eb6364619f	Remove the redundant variable fil_n_file_opened	7 years ago
Marko Mäkelä	ff88e4bb8a	Remove many redundant #include from InnoDB	7 years ago
Marko Mäkelä	6319c0b541	MDEV-13564: Replace innodb_unsafe_truncate with innodb_safe_truncate Rename the 10.2-specific configuration option innodb_unsafe_truncate to innodb_safe_truncate, and invert its value. The default (for now) is innodb_safe_truncate=OFF, to avoid disrupting users with an undo and redo log format change within a Generally Available (GA) release series.	7 years ago
Marko Mäkelä	3448ceb02a	MDEV-13564: Implement innodb_unsafe_truncate=ON for compatibility While MariaDB Server 10.2 is not really guaranteed to be compatible with Percona XtraBackup 2.4 (for example, the MySQL 5.7 undo log format change that could be present in XtraBackup, but was reverted from MariaDB in MDEV-12289), we do not want to disrupt users who have deployed xtrabackup and MariaDB Server 10.2 in their environments. With this change, MariaDB 10.2 will continue to use the backup-unsafe TRUNCATE TABLE code, so that neither the undo log nor the redo log formats will change in an incompatible way. Undo tablespace truncation will keep using the redo log only. Recovery or backup with old code will fail to shrink the undo tablespace files, but the contents will be recovered just fine. In the MariaDB Server 10.2 series only, we introduce the configuration parameter innodb_unsafe_truncate and make it ON by default. To allow MariaDB Backup (mariabackup) to work properly with TRUNCATE TABLE operations, use loose_innodb_unsafe_truncate=OFF. MariaDB Server 10.3.10 and later releases will always use the backup-safe TRUNCATE TABLE, and this parameter will not be added there. recv_recovery_rollback_active(): Skip row_mysql_drop_garbage_tables() unless innodb_unsafe_truncate=OFF. It is too unsafe to drop orphan tables if RENAME operations are not transactional within InnoDB. LOG_HEADER_FORMAT_10_3: Replaces LOG_HEADER_FORMAT_CURRENT. log_init(), log_group_file_header_flush(), srv_prepare_to_delete_redo_log_files(), innobase_start_or_create_for_mysql(): Choose the redo log format and subformat based on the value of innodb_unsafe_truncate.	7 years ago
Marko Mäkelä	ac8c7a1c08	Fix -Wunused-variable	7 years ago
Marko Mäkelä	09af00cbde	MDEV-13564: Remove old crash-upgrade logic in 10.4 Stop supporting the additional trunc.log files that were introduced via MySQL 5.7 to MariaDB Server 10.2 and 10.3. DB_TABLESPACE_TRUNCATED: Remove. purge_sys.truncate: A new structure to track undo tablespace file truncation. srv_start(): Remove the call to buf_pool_invalidate(). It is no longer necessary, given that we no longer access things in ways that violate the ARIES protocol. This call was originally added for innodb_file_format, and it may later have been necessary for the proper function of the MySQL 5.7 TRUNCATE recovery, which we are now removing. trx_purge_cleanse_purge_queue(): Take the undo tablespace as a parameter. trx_purge_truncate_history(): Rewrite everything mostly in a single function, replacing references to undo::Truncate. recv_apply_hashed_log_recs(): If any redo log is to be applied, and if the log_sys.log.subformat indicates that separately logged truncate may have been used, refuse to proceed except if innodb_force_recovery is set. We will still refuse crash-upgrade if TRUNCATE TABLE was logged. Undo tablespace truncation would only be logged in undotrunc.log files, which we are no longer checking for.	7 years ago
Marko Mäkelä	59950df533	Remove some debug-only global status variables	7 years ago
Marko Mäkelä	45dbd47026	MDEV-17003 service_manager_extend_timeout() being called too often buf_dump(): Only generate the output when shutdown is in progress. log_write_up_to(): Only generate the output before actually writing to the redo log files. srv_purge_should_exit(): Rate-limit the output, and instead of displaying the work done, indicate the work that remains to be done until the completion of the slow shutdown.	7 years ago
Marko Mäkelä	13f7ac2269	MDEV-15705 Remove global status counter Innodb_pages0_read MDEV-9931 introduced a counter for keeping track of reads of the first page of InnoDB data files, because the original implementation of data-at-rest-encryption for InnoDB introduced new code paths for reading the pages. Ultimately, the extra reads of the first page were removed, and the encryption subsystem will be initialized whenever we first read the first page of each data file, in fil_node_open_file(). It should not be that interesting to observe how many times an InnoDB data file was opened for the first time.	7 years ago
Marko Mäkelä	1c8c6bcd6f	MDEV-13779 InnoDB fails to shut down purge, causing hang thd_destructor_proxy(): Ensure that purge actually exits, like the logic should have been ever since MDEV-14080. srv_purge_shutdown(): A new function to wait for the purge coordinator to exit. Before exiting, the purge coordinator will ensure that all purge workers have exited.	7 years ago
Marko Mäkelä	a4e7800701	MDEV-13779 InnoDB fails to shut down purge workers, causing hang srv_purge_coordinator_thread(): Wait for all purge worker threads to actually exit. An analysis of a core dump of a hung 10.3 server revealed that one srv_worker_thread did not exit, even though the purge coordinator had exited. This caused kill_server_thread and mysqld_main to wait indefinitely. The main InnoDB shutdown was never called, because unireg_end() was never called.	8 years ago
Marko Mäkelä	cd15e764a8	MDEV-16159 Use atomic memory access for purge_sys Thanks to Sergey Vojtovich for feedback and many ideas. purge_state_t: Remove. The states are replaced with purge_sys_t::enabled() and purge_sys_t::paused() as follows: PURGE_STATE_INIT, PURGE_STATE_EXIT, PURGE_STATE_DISABLED: !enabled(). PURGE_STATE_RUN, PURGE_STATE_STOP: paused() distinguishes these. purge_sys_t::m_paused: Renamed from purge_sys_t::n_stop. Protected by atomic memory access only, not purge_sys_t::latch. purge_sys_t::m_enabled: An atomically updated Boolean that replaces purge_sys_t::state. purge_sys_t:🏃 Remove, because it duplicates srv_sys.n_threads_active[SRV_PURGE]. purge_sys_t::running(): Accessor for srv_sys.n_threads_active[SRV_PURGE]. purge_sys_t::stop(): Renamed from trx_purge_stop(). purge_sys_t::resume(): Renamed from trx_purge_run(). Do not acquire latch; solely rely on atomics. purge_sys_t::is_initialised(), purge_sys_t::m_initialised: Remove. purge_sys_t::create(), purge_sys_t::close(): Instead of invoking is_initialised(), check whether event is NULL. purge_sys_t::event: Move before latch, so that fields that are protected by latch can reside on the same cache line with latch. srv_start_wait_for_purge_to_start(): Merge to the only caller srv_start().	8 years ago
Marko Mäkelä	fb5d579462	MDEV-13987 Hang in FLUSH TABLES...FOR EXPORT trx_purge_stop(): Release purge_sys->latch before attempting to wake up the purge threads, so that they can actually wake up. This is a regression of commit `a13a636c74` which attempted to fix MDEV-11802 by ensuring that srv_purge_wakeup() will actually wait until all purge threads wake up. Due to the purge_sys->latch, the threads might never wake up, because some purge threads could end up waiting for purge_sys->latch in trx_undo_prev_version_build() while holding dict_operation_lock in shared mode. This in turn would block any DDL operations, the InnoDB master thread, and InnoDB shutdown.	8 years ago
Thirunarayanan Balathandayuthapani	fe95cb2e40	MDEV-16125 Shutdown crash when innodb_force_recovery >= 2 Problem: ======= InnoDB master thread encounters the shutdown state as SRV_SHUTDOWN_FLUSH_PHASE when innodb_force_recovery >=2 and slow scheduling of master thread during shutdown. Fix: ==== There is no need for master thread itself if innodb_force_recovery >=2. Don't create the master thread if innodb_force_recovery >= 2	8 years ago
Marko Mäkelä	2b27ac8282	Fix many -Wunused-parameter Remove unused InnoDB function parameters and functions. i_s_sys_virtual_fill_table(): Do not allocate heap memory. mtr_is_block_fix(): Replace with mtr_memo_contains(). mtr_is_page_fix(): Replace with mtr_memo_contains_page().	8 years ago
Marko Mäkelä	8bbcc0d505	MDEV-12218: Put back mariabackup --innodb-flush-method Implement innodb_flush_method as an enum parameter in Mariabackup, instead of ignoring the option and hard-wiring it to a default value. xb0xb.h: Remove. Only xtrabackup.cc refers to the enum parameters. innodb_flush_method_names[], innodb_flush_method_typelib[]: Define as non-static, so that mariabackup can share the definitions. srv_file_flush_method: Change the type to ulong, to match the assignment in init_one_value() and handle_options() in mariabackup.	8 years ago

1 2 3 4 5 ...

319 Commits (5e62b6a5e06eb02cbde1e34e95e26f42d87fce02)