mariadb

Commit Graph

Author	SHA1	Message	Date
Marko Mäkelä	c091a0bc8d	MDEV-26826 Duplicated computations of buf_pool.page_hash addresses Since commit `bd5a6403ca` (MDEV-26033) we can actually calculate the buf_pool.page_hash cell and latch addresses while not holding buf_pool.mutex. buf_page_alloc_descriptor(): Remove the MEM_UNDEFINED. We now expect buf_page_t::hash to be zero-initialized. buf_pool_t::hash_chain: Dedicated data type for buf_pool.page_hash.array. buf_LRU_free_one_page(): Merged to the only caller buf_pool_t::corrupted_evict().	4 years ago
Marko Mäkelä	f1acd9f14b	MDEV-26819 SET GLOBAL innodb_max_dirty_pages_pct=0 occasionally fails to trigger writes innodb_max_dirty_pages_pct_update(), innodb_max_dirty_pages_pct_lwm_update(): Invoke buf_pool.page_cleaner_wakeup() in order to wake up buf_flush_page_cleaner. This allows the test innodb.page_cleaner to run without any occasional timeouts. The occasional hangs were introduced by commit `7b1252c03d` (MDEV-24278).	4 years ago
Marko Mäkelä	f5794e1dc6	MDEV-26445 innodb_undo_log_truncate is unnecessarily slow trx_purge_truncate_history(): Do not force a write of the undo tablespace that is being truncated. Instead, prevent page writes by acquiring an exclusive latch on all dirty pages of the tablespace. fseg_create(): Relax an assertion that could fail if a dirty undo page is being initialized during undo tablespace truncation (and trx_purge_truncate_history() already acquired an exclusive latch on it). fsp_page_create(): If we are truncating a tablespace, try to reuse a page that we may have already latched exclusively (because it was in buf_pool.flush_list). To some extent, this helps the test innodb.undo_truncate,16k to avoid running out of buffer pool. mtr_t::commit_shrink(): Mark as clean all pages that are outside the new bounds of the tablespace, and only add the newly reinitialized pages to the buf_pool.flush_list. buf_page_create(): Do not unnecessarily invoke change buffer merge on undo tablespaces. buf_page_t::clear_oldest_modification(bool temporary): Move some assertions to the caller buf_page_write_complete(). innodb.undo_truncate: Use a bigger innodb_buffer_pool_size=24M. On my system, it would otherwise hang 1 out of 1547 attempts (on the 40th repeat of innodb.undo_truncate,16k). Other page sizes were not affected.	4 years ago
Marko Mäkelä	15efb7ed48	MDEV-26626 fixup: Do not advance checkpoint during startup While the redo log is being resized in srv_start(), we must not write checkpoint information to the old log. Thanks to Matthias Leich for noticing this.	4 years ago
Marko Mäkelä	c430aa72ab	MDEV-26626 InnoDB fails to advance the log checkpoint buf_flush_page_cleaner(): Always try to advance the log checkpoint, even when no pages were flushed during the latest batch. Maybe, since the previous batch, there was an LRU flush that removed the last dirty pages. Failure to advance the log checkpoint will cause unnecessary work in Mariabackup and on crash recovery.	4 years ago
Eugene Kosov	748539837e	TSAN: unprotected global variable Write of size 1 at 0x0000067abe08 by thread T3 (mutexes: write M1372): #0 buf_flush_page_cleaner() /storage/innobase/buf/buf0flu.cc:2366:29 (mariadbd+0x20cea7c) #1 void std::__invoke_impl<void, void ()()>(std::__invoke_other, void (&&)()) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 (mariadbd+0x20c3a8a) #2 std::__invoke_result<void ()()>::type std::__invoke<void ()()>(void (&&)()) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 (mariadbd+0x20c399d) #3 void std:🧵:_Invoker<std::tuple<void ()()> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:253:13 (mariadbd+0x20c3945) #4 std:🧵:_Invoker<std::tuple<void ()()> >::operator()() /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:260:11 (mariadbd+0x20c38e5) #5 std:🧵:_State_impl<std:🧵:_Invoker<std::tuple<void ()()> > >::_M_run() /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:211:13 (mariadbd+0x20c37d9) #6 <null> <null> (libstdc++.so.6+0xd230f) Previous read of size 1 at 0x0000067abe08 by main thread: #0 logs_empty_and_mark_files_at_shutdown() /storage/innobase/log/log0log.cc:1094:6 (mariadbd+0x22eeff3) #1 innodb_shutdown() /storage/innobase/srv/srv0start.cc:1970:3 (mariadbd+0x256ffd6) #2 innobase_end(handlerton, ha_panic_function) /storage/innobase/handler/ha_innodb.cc:4265:3 (mariadbd+0x1ee3fc4) #3 ha_finalize_handlerton(st_plugin_int) /sql/handler.cc:595:5 (mariadbd+0xf7bac9) #4 plugin_deinitialize(st_plugin_int, bool) /sql/sql_plugin.cc:1266:9 (mariadbd+0x1611789) #5 reap_plugins() /sql/sql_plugin.cc:1342:7 (mariadbd+0x160e17d) #6 plugin_shutdown() /sql/sql_plugin.cc:2050:7 (mariadbd+0x1611f42) #7 clean_up(bool) /sql/mysqld.cc:1923:3 (mariadbd+0xd67a4c) #8 unireg_abort /sql/mysqld.cc:1835:3 (mariadbd+0xd67605) #9 mysqld_main(int, char*) /sql/mysqld.cc:5741:7 (mariadbd+0xd6b36a) #10 main /sql/main.cc:34:10 (mariadbd+0xd661a8) Location is global 'buf_page_cleaner_is_active' of size 1 at 0x0000067abe08 (mariadbd+0x67abe08)	4 years ago
Marko Mäkelä	bf435a3f4d	MDEV-26200 buf_pool.flush_list corrupted by buffer pool resizing or ROW_FORMAT=COMPRESSED The lazy deletion of clean blocks from buf_pool.flush_list that was introduced in commit `6441bc614a` (MDEV-25113) introduced a race condition around the function buf_flush_relocate_on_flush_list(). The test innodb_zip.wl5522_debug_zip as well as the buffer pool resizing tests would occasionally fail in debug builds due to buf_pool.flush_list.count disagreeing with the actual length of the doubly-linked list. The safe procedure for relocating a block in buf_pool.flush_list should be as follows, now that we implement lazy deletion from buf_pool.flush_list: 1. Acquire buf_pool.mutex. 2. Acquire the exclusive buf_pool.page_hash.latch. 3. Acquire buf_pool.flush_list_mutex. 4. Copy the block descriptor. 5. Invoke buf_flush_relocate_on_flush_list(). 6. Release buf_pool.flush_list_mutex. buf_flush_relocate_on_flush_list(): Assert that buf_pool.flush_list_mutex is being held. Invoke buf_page_t::oldest_modification() only once, using std::memory_order_relaxed, now that the mutex protects us. buf_LRU_free_page(), buf_LRU_block_remove_hashed(): Avoid an unlock-lock cycle on hash_lock. (We must not acquire hash_lock while already holding buf_pool.flush_list_mutex, because that could lead to a deadlock due to latching order violation.)	4 years ago
Marko Mäkelä	ae62f115be	MDEV-26166: non-functional changes	4 years ago
Marko Mäkelä	f0f47cbca1	MDEV-26017 fixup buf_flush_relocate_on_flush_list(): Use dpage->physical_size() because bpage->zip.ssize may already have been zeroed in page_zip_set_size() invoked by buf_pool_t::realloc(). This would cause occasional failures of the test innodb.innodb_buffer_pool_resize, which creates a ROW_FORMAT=COMPRESSED table.	4 years ago
Marko Mäkelä	30edd5549d	MDEV-26029: Sparse files are inefficient on thinly provisioned storage The MariaDB implementation of page_compressed tables for InnoDB used sparse files. In the worst case, in the data file, every data page will consist of some data followed by a hole. This may be extremely inefficient in some file systems. If the underlying storage device is thinly provisioned (can compress data on the fly), it would be good to write regular files (with sequences of NUL bytes at the end of each page_compressed block) and let the storage device take care of compressing the data. For reads, sparse file regions and regions containing NUL bytes will be indistinguishable. my_test_if_disable_punch_hole(): A new predicate for detecting thinly provisioned storage. (Not implemented yet.) innodb_atomic_writes: Correct the comment. buf_flush_page(): Support all values of fil_node_t::punch_hole. On a thinly provisioned storage device, we will always write NUL-padded innodb_page_size bytes also for page_compressed tables. buf_flush_freed_pages(): Remove a redundant condition. fil_space_t::atomic_write_supported: Remove. (This was duplicating fil_node_t::atomic_write.) fil_space_t::punch_hole: Remove. (Duplicated fil_node_t::punch_hole.) fil_node_t: Remove magic_n, and consolidate flags into bitfields. For punch_hole we introduce a third value that indicates a thinly provisioned storage device. fil_node_t::find_metadata(): Detect all attributes of the file.	4 years ago
Marko Mäkelä	fc2ff46469	MDEV-26017: Assertion stat.flush_list_bytes <= curr_pool_size buf_flush_relocate_on_flush_list(): If we are removing the block from buf_pool.flush_list, subtract its size from buf_pool.stat.flush_list_bytes. This fixes a regression that was introduced in commit `22b62edaed` (MDEV-25113).	4 years ago
Marko Mäkelä	759deaa0a2	MDEV-26010 fixup: Use acquire/release memory order In commit `5f22511e35` we depend on Total Store Ordering. For correct operation on ISAs that implement weaker memory ordering, we must explicitly use release/acquire stores and loads on buf_page_t::oldest_modification_ to prevent a race condition when buf_page_t::list does not happen to be on the same cache line. buf_page_t::clear_oldest_modification(): Assert that the block is not in buf_pool.flush_list, and use std::memory_order_release. buf_page_t::oldest_modification_acquire(): Read oldest_modification_ with std::memory_order_acquire. In this way, if the return value is 0, the caller may safely assume that it will not observe the buf_page_t as being in buf_pool.flush_list, even if it is not holding buf_pool.flush_list_mutex. buf_flush_relocate_on_flush_list(), buf_LRU_free_page(): Invoke buf_page_t::oldest_modification_acquire().	4 years ago
Marko Mäkelä	5f22511e35	MDEV-26010: Assertion lsn > 2 failed in buf_pool_t::get_oldest_modification In commit `22b62edaed` (MDEV-25113) we introduced a race condition. buf_LRU_free_page() would read buf_page_t::oldest_modification() as 0 and assume that buf_page_t::list can be used (for attaching the block to the buf_pool.free list). In the observed race condition, buf_pool_t::delete_from_flush_list() had cleared the field, and buf_pool_t::delete_from_flush_list_low() was executing concurrently with buf_LRU_block_free_non_file_page(), which resulted in buf_pool.flush_list.end becoming corrupted. buf_pool_t::delete_from_flush_list(), buf_flush_relocate_on_flush_list(): First remove the block from buf_pool.flush_list, and only then invoke buf_page_t::clear_oldest_modification(), to ensure that reading oldest_modification()==0 really implies that the block no longer is in buf_pool.flush_list.	4 years ago
Marko Mäkelä	60ed479711	MDEV-26004 Excessive wait times in buf_LRU_get_free_block() buf_LRU_get_free_block(): Initially wait for a single block to be freed, signaled by buf_pool.done_free. Only if that fails and no LRU eviction flushing batch is already running, we initiate a flushing batch that should serve all threads that are currently waiting in buf_LRU_get_free_block(). Note: In an extreme case, this may introduce a performance regression at larger numbers of connections. We observed this in sysbench oltp_update_index with 512MiB buffer pool, 4GiB of data on fast NVMe, and 1000 concurrent connections, on a 20-thread CPU. The contention point appears to be buf_pool.mutex, and the improvement would turn into a regression somewhere beyond 32 concurrent connections. On slower storage, such regression was not observed; instead, the throughput was improving and maximum latency was reduced. The excessive waits were pointed out by Vladislav Vaintroub.	4 years ago
Marko Mäkelä	6441bc614a	MDEV-25113: Introduce a page cleaner mode before 'furious flush' MDEV-23855 changed the way how the page cleaner is signaled by user threads. If a threshold is exceeded, a mini-transaction commit would invoke buf_flush_ahead() in order to initiate page flushing before all writers would eventually grind to halt in log_free_check(), waiting for the checkpoint age to reduce. However, buf_flush_ahead() would always initiate 'furious flushing', making the buf_flush_page_cleaner thread write innodb_io_capacity_max pages per batch, and sleeping no time between batches, until the limit LSN is reached. Because this could saturate the I/O subsystem, system throughput could significantly reduce during these 'furious flushing' spikes. With this change, we introduce a gentler version of flush-ahead, which would write innodb_io_capacity_max pages per second until the 'soft limit' is reached. buf_flush_ahead(): Add a parameter to specify whether furious flushing is requested. buf_flush_async_lsn: Similar to buf_flush_sync_lsn, a limit for the less intrusive flushing. buf_flush_page_cleaner(): Keep working until buf_flush_async_lsn has been reached. log_close(): Suppress a warning message in the event that a new log is being created during startup, when old logs did not exist. Return what type of page cleaning will be needed. mtr_t::finish_write(): Also when m_log.is_small(), invoke log_close(). Return what type of page cleaning will be needed. mtr_t::commit(): Invoke buf_flush_ahead() based on the return value of mtr_t::finish_write().	4 years ago
Marko Mäkelä	22b62edaed	MDEV-25113: Make page flushing faster buf_page_write_complete(): Reduce the buf_pool.mutex hold time, and do not acquire buf_pool.flush_list_mutex at all. Instead, mark blocks clean by setting oldest_modification to 1. Dirty pages of temporary tables will be identified by the special value 2 instead of the previous special value 1. (By design of the ib_logfile0 format, actual LSN values smaller than 2048 are not possible.) buf_LRU_free_page(), buf_pool_t::get_oldest_modification() and many other functions will remove the garbage (clean blocks) from buf_pool.flush_list while holding buf_pool.flush_list_mutex. buf_pool_t::n_flush_LRU, buf_pool_t::n_flush_list: Replaced with non-atomic variables, protected by buf_pool.mutex, to avoid unnecessary synchronization when modifying the counts. export_vars: Remove unnecessary indirection for innodb_pages_created, innodb_pages_read, innodb_pages_written.	4 years ago
Marko Mäkelä	8af538979b	MDEV-25801: buf_flush_dirty_pages() is very slow In commit `7cffb5f6e8` (MDEV-23399) the implementation of buf_flush_dirty_pages() was replaced with a slow one, which would perform excessive scans of the buf_pool.flush_list and make little progress. buf_flush_list(), buf_flush_LRU(): Split from buf_flush_lists(). Vladislav Vaintroub noticed that we will not need to invoke log_flush_task.wait() for the LRU eviction flushing. buf_flush_list_space(): Replaces buf_flush_dirty_pages(). This is like buf_flush_list(), but operating on a single tablespace at a time. Writes at most innodb_io_capacity pages. Returns whether some of the tablespace might remain in the buffer pool.	4 years ago
Marko Mäkelä	762bcb81b5	MDEV-25948 Remove log_flush_task Vladislav Vaintroub suggested that invoking log_flush_up_to() for every page could perform better than invoking a log write between buf_pool.flush_list batches, like we started doing in commit `3a9a3be1c6` (MDEV-23855). This could depend on the sequence in which pages are being modified. The buf_pool.flush_list is ordered by oldest_modification, while the FIL_PAGE_LSN of the pages is theoretically independent of that. In the pathological case, we will wait for a log write before writing each individual page. It turns out that we can defer the call to log_flush_up_to() until just before submitting the page write. If the doublewrite buffer is being used, we can submit a write batch of "future" pages to the doublewrite buffer, and only wait for the log write right before we are writing an already doublewritten page. The next doublewrite batch will not be initiated before the last page write from the current batch has completed. When a future version introduces asynchronous writes if the log, we could initiate a write at the start of a flushing batch, to reduce waiting further.	4 years ago
Marko Mäkelä	6dfd44c828	MDEV-25954: Trim os_aio_wait_until_no_pending_writes() It turns out that we had some unnecessary waits for no outstanding write requests to exist. They were basically working around a bug that was fixed in MDEV-25953. On write completion callback, blocks will be marked clean. So, it is sufficient to consult buf_pool.flush_list to determine which writes have not been completed yet. On FLUSH TABLES...FOR EXPORT we must still wait for all pending asynchronous writes to complete, because buf_flush_file_space() would merely guarantee that writes will have been initiated.	4 years ago
Marko Mäkelä	601eb41183	Cleanup: deduplicate code	5 years ago
Krunal Bauskar	3f4b7ed95a	MDEV-25093: Adaptive flushing fails to kick in even if innodb_adaptive_flushing_lwm is hit. (possible regression) adaptive flushing should kick in if a. dirty_pct (dirty pages in buffer pool) > innodb_max_dirty_pages_pct_lwm OR b. innodb_adaptive_flushing_lwm limit is reached (default to 10%) both conditions are mutually exclusive and whichever is first to evaluate true should kick-start the adaptive flushing. After recent changes to simplify the flushing algorithm logic, (b) got ignored that introduced the said regression.	5 years ago
Marko Mäkelä	3f8df01194	MDEV-25425 Useless message "If the mysqld execution user is authorized page cleaner thread priority can be changed." On startup, InnoDB tried to increase the priority of the page cleaner thread. This would usually fail, resulting in a server log message. When the page cleaner code was refactored in MDEV-23855, the throughput and latency were greatly improved even when the buf_flush_page_cleaner thread always ran the same priority with other threads in our benchmarks. So, this code should have no benefit in MariaDB Server 10.5.7 or later. Besides, starting with MariaDB Server 10.5, the executable is called mariadbd, not mysqld.	5 years ago
Marko Mäkelä	8c2e3259c1	MDEV-24302 follow-up: RESET MASTER hangs As pointed out by Andrei Elkin, the previous fix did not fix one race condition that may have caused the observed hang. innodb_log_flush_request(): If we are enqueueing the very first request at the same time the log write is being completed, we must ensure that a near-concurrent call to log_flush_notify() will not result in a missed notification. We guarantee this by release-acquire operations on log_requests.start and log_sys.flushed_to_disk_lsn. log_flush_notify_and_unlock(): Cleanup: Always release the mutex. log_sys_t::get_flushed_lsn(): Use acquire memory order. log_sys_t::set_flushed_lsn(): Use release memory order. log_sys_t::set_lsn(): Use release memory order. log_sys_t::get_lsn(): Use relaxed memory order by default, and allow the caller to specify acquire memory order explicitly. Whenever the log_sys.mutex is being held or when log writes are prohibited during startup, we can use a relaxed load. Likewise, in some assertions where reading a stale value of log_sys.lsn should not matter, we can use a relaxed load. This will cause some additional instructions to be emitted on architectures that do not implement Total Store Ordering (TSO), such as POWER, ARM, and RISC-V Weak Memory Ordering (RVWMO).	5 years ago
Eugene Kosov	dbe941e06f	cleanup: os_thread_create -> std::thread	5 years ago
Eugene Kosov	da3428805e	cleanup: os_thread_yield() -> std::this_thread::yield()	5 years ago
Eugene Kosov	62e4aaa240	cleanup: os_thread_sleep() -> std::this_thread::sleep_for() std version has an advantage of a more convenient units implementation from std::chrono. Now it's no need to multipy/divide to bring anything to micro seconds.	5 years ago
Marko Mäkelä	7a4fbb55b0	MDEV-25105 Remove innodb_checksum_algorithm values none,innodb,... Historically, InnoDB supported a buggy page checksum algorithm that did not compute a checksum over the full page. Later, well before MySQL 4.1 introduced .ibd files and the innodb_file_per_table option, the algorithm was corrected and the first 4 bytes of each page were redefined to be a checksum. The original checksum was so slow that an option to disable page checksum was introduced for benchmarketing purposes. The Intel Nehalem microarchitecture introduced the SSE4.2 instruction set extension, which includes instructions for faster computation of CRC-32C. In MySQL 5.6 (and MariaDB 10.0), innodb_checksum_algorithm=crc32 was implemented to make of that. As that option was changed to be the default in MySQL 5.7, a bug was found on big-endian platforms and some work-around code was added to weaken that checksum further. MariaDB disables that work-around by default since MDEV-17958. Later, SIMD-accelerated CRC-32C has been implemented in MariaDB for POWER and ARM and also for IA-32/AMD64, making use of carry-less multiplication where available. Long story short, innodb_checksum_algorithm=crc32 is faster and more secure than the pre-MySQL 5.6 checksum, called innodb_checksum_algorithm=innodb. It should have removed any need to use innodb_checksum_algorithm=none. The setting innodb_checksum_algorithm=crc32 is the default in MySQL 5.7 and MariaDB Server 10.2, 10.3, 10.4. In MariaDB 10.5, MDEV-19534 made innodb_checksum_algorithm=full_crc32 the default. It is even faster and more secure. The default settings in MariaDB do allow old data files to be read, no matter if a worse checksum algorithm had been used. (Unfortunately, before innodb_checksum_algorithm=full_crc32, the data files did not identify which checksum algorithm is being used.) The non-default settings innodb_checksum_algorithm=strict_crc32 or innodb_checksum_algorithm=strict_full_crc32 would only allow CRC-32C checksums. The incompatibility with old data files is why they are not the default. The newest server not to support innodb_checksum_algorithm=crc32 were MySQL 5.5 and MariaDB 5.5. Both have reached their end of life. A valid reason for using innodb_checksum_algorithm=innodb could have been the ability to downgrade. If it is really needed, data files can be converted with an older version of the innochecksum utility. Because there is no good reason to allow data files to be written with insecure checksums, we will reject those option values: innodb_checksum_algorithm=none innodb_checksum_algorithm=innodb innodb_checksum_algorithm=strict_none innodb_checksum_algorithm=strict_innodb Furthermore, the following innochecksum options will be removed, because only strict crc32 will be supported: innochecksum --strict-check=crc32 innochecksum -C crc32 innochecksum --write=crc32 innochecksum -w crc32 If a user wishes to convert a data file to use a different checksum (so that it might be used with the no-longer-supported MySQL 5.5 or MariaDB 5.5, which do not support IMPORT TABLESPACE nor system tablespace format changes that were made in MariaDB 10.3), then the innochecksum tool from MariaDB 10.2, 10.3, 10.4, 10.5 or MySQL 5.7 can be used. Reviewed by: Thirunarayanan Balathandayuthapani	5 years ago
Krunal Bauskar	f11b60879b	MDEV-24949: Enabling idle flushing (possible regression from MDEV-23855) - Currently page cleaner thread will stop flushing if dirty_pct < innodb_max_dirty_pages_pct_lwm. - If the server is not performing any activity then said resources/time could be used to flush the pending dirty pages and keep buffer pool clean for the next burst of the cycle. This flushing is called idle flushing. - flushing logic underwent a complete revamp in 10.5.7/8 and as part of the revamp idle flushing logic got removed. - New proposed logic of idle flushing is based on updated logic of the page cleaner that will enable idle flushing if - buf page cleaner is idle - there are dirty pages (< innodb_max_dirty_pages_pct_lwm) - server is not performing any activity Logic will kickstart the idle flushing bounded by innodb_io_capacity. (Thanks to Marko Makela for reviewing the patch and idea right from the its inception).	5 years ago
Marko Mäkelä	78284a4c11	MDEV-25085: Simplify instrumentation for LRU eviction Let us add the status variable innodb_buffer_pool_pages_LRU_freed to monitor the number of pages that were freed by a buffer pool LRU eviction scan, without flushing. Also, let us simplify the monitor interface: MONITOR_LRU_BATCH_FLUSH_COUNT, MONITOR_LRU_BATCH_FLUSH_PAGES, MONITOR_LRU_BATCH_EVICT_COUNT, MONITOR_LRU_BATCH_EVICT_PAGES: Remove. MONITOR_LRU_BATCH_FLUSH_TOTAL_PAGE: Track buf_lru_flush_page_count (innodb_buffer_pool_pages_LRU_flushed). MONITOR_LRU_BATCH_EVICT_TOTAL_PAGE: Track buf_lru_freed_page_count (buffer_pool_pages_LRU_freed). Reviewed by: Vladislav Vaintroub	5 years ago
Marko Mäkelä	93522bc9a9	MDEV-24917 Page cleaner wrongly remains idle commit `a993310593` (MDEV-24537) introduced the regression that the page cleaner will keep sleeping even if there is work to do. innodb_max_dirty_pages_pct_update(): Always wake up the page cleaner on any SET GLOBAL innodb_max_dirty_pages_pct= assignment. buf_flush_page_cleaner(): If innodb_max_dirty_pages_pct is nonzero, consult only that parameter when determining whether there is work to do. Else, consult innodb_max_dirty_pages.	5 years ago
Vlad Lesin	23833dce05	MDEV-24792 Assertion `!newest_lsn \|\| fil_page_get_type(page)' failed upon MariaBackup prepare in buf_flush_init_for_writing with innodb_log_optimize_ddl=off fsp_free_page() writes MLOG_INIT_FREE_PAGE, but does not update page type. But fil_crypt_rotate_page() checks the type to understand if the page is freshly initialized, and writes dummy record(updates space id) to force rotation during recovery. This dummy record causes assertion crash when the page is flushed after recovery, as it's supposed that pages LSN is 0 for freshly initialized pages. The bug is similiar to MDEV-24695, the difference is that in 10.5 the assertion crashes during log record applying, but in 10.4 it crashes during page flushing. The fix could be in marking page as freed and not writing dummy record during keys rotation procedure for such marked pages. But bpage->file_page_was_freed is not consistent enough for release builds in 10.4, and the issue is fixed in 10.5 and does not exist in 10.[23] as MLOG_INIT_FREE_PAGE was introduced since 10.4. So the better solution is just to relax the assertion and implement some additional property for freshly allocated pages, and check this property during pages flushing. The test is copied from MDEV-24695, the only change is in forcing pages flushing after each server start to cause crash in non-fixed code. There is no need to merge it to 10.5+, as the bug is already fixed by MDEV-24695.	5 years ago
Marko Mäkelä	4f4a4cf9eb	MDEV-23399 fixup: Use plain pthread_cond The condition variables that were introduced in commit `7cffb5f6e8` (MDEV-23399) are never instrumented with PERFORMANCE_SCHEMA. Let us avoid the storage overhead and dead code.	5 years ago
Marko Mäkelä	a993310593	MDEV-24537 innodb_max_dirty_pages_pct_lwm=0 lost its special meaning In commit `3a9a3be1c6` (MDEV-23855) some previous logic was replaced with the condition dirty_pct < srv_max_dirty_pages_pct_lwm, which caused the default value of the parameter innodb_max_dirty_pages_pct_lwm=0 to lose its special meaning: 'refer to innodb_max_dirty_pages_pct instead'. This implicit special meaning was visible in the function af_get_pct_for_dirty(), which was removed in commit `f0c295e2de` (MDEV-24369). page_cleaner_flush_pages_recommendation(): Restore the special meaning that was removed in MDEV-24369. buf_flush_page_cleaner(): If srv_max_dirty_pages_pct_lwm==0.0, refer to srv_max_buf_pool_modified_pct. This fixes the observed performance regression due to excessive page flushing. buf_pool_t::page_cleaner_wakeup(): Revise the wakeup condition. innodb_init(): Do initialize srv_max_io_capacity in Mariabackup. It was previously constantly 0, which caused mariadb-backup --prepare to hang in buf_flush_sync(), making no progress.	5 years ago
Marko Mäkelä	ff5d306e29	MDEV-21452: Replace ib_mutex_t with mysql_mutex_t SHOW ENGINE INNODB MUTEX functionality is completely removed, as are the InnoDB latching order checks. We will enforce innodb_fatal_semaphore_wait_threshold only for dict_sys.mutex and lock_sys.mutex. dict_sys_t::mutex_lock(): A single entry point for dict_sys.mutex. lock_sys_t::mutex_lock(): A single entry point for lock_sys.mutex. FIXME: srv_sys should be removed altogether; it is duplicating tpool functionality. fil_crypt_threads_init(): To prevent SAFE_MUTEX warnings, we must not hold fil_system.mutex. fil_close_all_files(): To prevent SAFE_MUTEX warnings for fil_space_destroy_crypt_data(), we must not hold fil_system.mutex while invoking fil_space_free_low() on a detached tablespace.	5 years ago
Marko Mäkelä	0c7c449267	Remove unused DBUG_EXECUTE_IF "ignore_punch_hole" Since commit `ea21d630be` we conditionally define a variable that only plays a role on systems that support hole-punching (explicit creation of sparse files). However, that broke debug builds on such systems. It turns out that the debug_dbug label "ignore_punch_hole" is not at all used in MariaDB server. It would be covered by the MySQL 5.7 test innodb.table_compress. (Note: MariaDB 10.1 implemented page_compressed tables before something comparable appeared in MySQL 5.7.)	5 years ago
Marko Mäkelä	5eb539555b	MDEV-12227 Defer writes to the InnoDB temporary tablespace The flushing of the InnoDB temporary tablespace is unnecessarily tied to the write-ahead redo logging and redo log checkpoints, which must be tied to the page writes of persistent tablespaces. Let us simply omit any pages of temporary tables from buf_pool.flush_list. In this way, log checkpoints will never incur any 'collateral damage' of writing out unmodified changes for temporary tables. After this change, pages of the temporary tablespace can only be written out by buf_flush_lists(n_pages,0) as part of LRU eviction. Hopefully, most of the time, that code will never be executed, and instead, the temporary pages will be evicted by buf_release_freed_page() without ever being written back to the temporary tablespace file. This should improve the efficiency of the checkpoint flushing and the buf_flush_page_cleaner thread. Reviewed by: Vladislav Vaintroub	5 years ago
Marko Mäkelä	ea21d630be	Fix -Wunused-but-set-variable	5 years ago
Marko Mäkelä	f0c295e2de	MDEV-24369 Page cleaner sleeps despite innodb_max_dirty_pages_pct_lwm being exceeded MDEV-24278 improved the page cleaner so that it will no longer wake up once per second on an idle server. However, with innodb_adaptive_flushing (the default) the function page_cleaner_flush_pages_recommendation() could initially return 0 even if there is work to do. af_get_pct_for_dirty(): Remove. Based on a comment here, it appears that an initial intention of innodb_max_dirty_pages_pct_lwm=0.0 (the default value) was to disable something. That ceased to hold in MDEV-23855: the value is a pure threshold; the page cleaner will not perform any work unless the threshold is exceeded. page_cleaner_flush_pages_recommendation(): Add the parameter dirty_blocks to ensure that buf_pool.flush_list will eventually be emptied.	5 years ago
Marko Mäkelä	aa0e380568	MDEV-24348 InnoDB shutdown hang with innodb_flush_sync=0 This hang was caused by MDEV-23855, and we failed to fix it in MDEV-24109 (commit `4cbfdeca84`). When buf_flush_ahead() is invoked soon before server shutdown and the non-default setting innodb_flush_sync=OFF is in effect and the buffer pool contains dirty pages of temporary tables, the page cleaner thread may remain in an infinite loop without completing its work, thus causing the shutdown to hang. buf_flush_page_cleaner(): If the buffer pool contains no unmodified persistent pages, ensure that buf_flush_sync_lsn= 0 will be assigned, so that shutdown will proceed. The test case is not deterministic. On my system, it reproduced the hang with 95% probability when running multiple instances of the test in parallel, and 4% when running single-threaded. Thanks to Eugene Kosov for debugging and testing this.	5 years ago
Marko Mäkelä	03ca6495df	MDEV-24142: Replace InnoDB rw_lock_t with sux_lock InnoDB buffer pool block and index tree latches depend on a special kind of read-update-write lock that allows reentrant (recursive) acquisition of the 'update' and 'write' locks as well as an upgrade from 'update' lock to 'write' lock. The 'update' lock allows any number of reader locks from other threads, but no concurrent 'update' or 'write' lock. If there were no requirement to support an upgrade from 'update' to 'write', we could compose the lock out of two srw_lock (implemented as any type of native rw-lock, such as SRWLOCK on Microsoft Windows). Removing this requirement is very difficult, so in commit `f7e7f487d4` we implemented an 'update' mode to our srw_lock. Re-entrant or recursive locking is mostly needed when writing or freeing BLOB pages, but also in crash recovery or when merging buffered changes to an index page. The re-entrancy allows us to attach a previously acquired page to a sub-mini-transaction that will be committed before whatever else is holding the page latch. The SUX lock supports Shared ('read'), Update, and eXclusive ('write') locking modes. The S latches are not re-entrant, but a single S latch may be acquired even if the thread already holds an U latch. The idea of the U latch is to allow a write of something that concurrent readers do not care about (such as the contents of BTR_SEG_LEAF, BTR_SEG_TOP and other page allocation metadata structures, or the MDEV-6076 PAGE_ROOT_AUTO_INC). (The PAGE_ROOT_AUTO_INC field is only updated when a dict_table_t for the table exists, and only read when a dict_table_t for the table is being added to dict_sys.) block_lock::u_lock_try(bool for_io=true) is used in buf_flush_page() to allow concurrent readers but no concurrent modifications while the page is being written to the data file. That latch will be released by buf_page_write_complete() in a different thread. Hence, we use the special lock owner value FOR_IO. The index_lock::u_lock() improves concurrency on operations that involve non-leaf index pages. The interface has been cleaned up a little. We will use x_lock_recursive() instead of x_lock() when we know that a lock is already held by the current thread. Similarly, a lock upgrade from U to X is only allowed via u_x_upgrade() or x_lock_upgraded() but not via x_lock(). We will disable the LatchDebug and sync_array interfaces to InnoDB rw-locks. The SEMAPHORES section of SHOW ENGINE INNODB STATUS output will no longer include any information about InnoDB rw-locks, only TTASEventMutex (cmake -DMUTEXTYPE=event) waits. This will make a part of the 'innotop' script dead code. The block_lock buf_block_t::lock will not be covered by any PERFORMANCE_SCHEMA instrumentation. SHOW ENGINE INNODB MUTEX and INFORMATION_SCHEMA.INNODB_MUTEXES will no longer output source code file names or line numbers. The dict_index_t::lock will be identified by index and table names, which should be much more useful. PERFORMANCE_SCHEMA is lumping information about all dict_index_t::lock together as event_name='wait/synch/sxlock/innodb/index_tree_rw_lock'. buf_page_free(): Remove the file,line parameters. The sux_lock will not store such diagnostic information. buf_block_dbg_add_level(): Define as empty macro, to be removed in a subsequent commit. Unless the build was configured with cmake -DPLUGIN_PERFSCHEMA=NO the index_lock dict_index_t::lock will be instrumented via PERFORMANCE_SCHEMA. Similar to commit `1669c8890c` we will distinguish lock waits by registering shared_lock,exclusive_lock events instead of try_shared_lock,try_exclusive_lock. Actual 'try' operations will not be instrumented at all. rw_lock_list: Remove. After MDEV-24167, this only covered buf_block_t::lock and dict_index_t::lock. We will output their information by traversing buf_pool or dict_sys.	5 years ago
Marko Mäkelä	8fa6e36375	MDEV-24308: Remove some os_thread_ functions os_thread_pf(): Remove. os_thread_eq(), os_thread_yield(), os_thread_get_curr_id(): Define as macros. ut_print_timestamp(), ut_sprintf_timestamp(): Simplify.	5 years ago
Marko Mäkelä	7b1252c03d	MDEV-24278 InnoDB page cleaner keeps waking up on idle server The purpose of the InnoDB page cleaner subsystem is to write out modified pages from the buffer pool to data files. When the innodb_max_dirty_pages_pct_lwm is not exceeded or innodb_adaptive_flushing=ON decides not to write out anything, the page cleaner should keep sleeping indefinitely until the state of the system changes: a dirty page is added to the buffer pool such that the page cleaner would no longer be idle. buf_flush_page_cleaner(): Explicitly note when the page cleaner is idle. When that happens, use mysql_cond_wait() instead of mysql_cond_timedwait(). buf_flush_insert_into_flush_list(): Wake up the page cleaner if needed. innodb_max_dirty_pages_pct_update(), innodb_max_dirty_pages_pct_lwm_update(): Wake up the page cleaner just in case. Note: buf_flush_ahead(), buf_flush_wait_flushed() and shutdown are already waking up the page cleaner thread.	5 years ago
Marko Mäkelä	4cbfdeca84	MDEV-24109 InnoDB hangs with innodb_flush_sync=OFF MDEV-23855 broke the handling of innodb_flush_sync=OFF. That parameter is supposed to limit the page write rate in case the log capacity is being exceeded and log checkpoints are needed. With this fix, the following should pass: ./mtr --mysqld=--loose-innodb-flush-sync=0 One of our best regression tests for page flushing is encryption.innochecksum. With innodb_page_size=16k and innodb_flush_sync=OFF it would likely hang without this fix. log_sys.last_checkpoint_lsn: Declare as Atomic_relaxed<lsn_t> so that we are allowed to read the value while not holding log_sys.mutex. buf_flush_wait_flushed(): Let the page cleaner perform the flushing also if innodb_flush_sync=OFF. After the page cleaner has completed, perform a checkpoint if it is needed, because buf_flush_sync_for_checkpoint() will not be run if innodb_flush_sync=OFF. buf_flush_ahead(): Simplify the condition. We do not really care whether buf_flush_page_cleaner() is running. buf_flush_page_cleaner(): Evaluate innodb_flush_sync at the low level. If innodb_flush_sync=OFF, rate-limit the batches to innodb_io_capacity_max pages per second. Reviewed by: Vladislav Vaintroub	5 years ago
Marko Mäkelä	b0ff791618	MDEV-24054 Assertion in_LRU_list failed in buf_flush_try_neighbors() buf_flush_try_neighbors(): Before invoking buf_page_t::ready_for_flush(), check that the freshly looked up buf_pool.page_hash entry actually is a buffer page and not a buf_pool.watch[] sentinel for purge buffering. This race condition was introduced in MDEV-15053 (commit `b1ab211dee`). It is rather hard to hit this bug, because buf_flush_check_neighbors() already checked the condition. The problem exists if buf_pool.watch_set() was invoked for a page in the range after the check in buf_flush_check_neighbor() had been finished.	5 years ago
Marko Mäkelä	c27e53f459	MDEV-23855: Use normal mutex for log_sys.mutex, log_sys.flush_order_mutex With an unreasonably small innodb_log_file_size, the page cleaner thread would frequently acquire log_sys.flush_order_mutex and spend a significant portion of CPU time spinning on that mutex when determining the checkpoint LSN.	5 years ago
Marko Mäkelä	a5a2ef079c	MDEV-23855: Implement asynchronous doublewrite Synchronous writes and calls to fdatasync(), fsync() or FlushFileBuffers() would ruin performance. So, let us submit asynchronous writes for the doublewrite buffer. We submit a single request for the likely case that the two doublewrite buffers are contiquous in the system tablespace. buf_dblwr_t::flush_buffered_writes_completed(): The completion callback of buf_dblwr_t::flush_buffered_writes(). os_aio_wait_until_no_pending_writes(): Also wait for doublewrite batches. buf_dblwr_t::element::space: Remove. We can simply use element::request.node->space instead. Reviewed by: Vladislav Vaintroub	5 years ago
Marko Mäkelä	8cb01c51fb	MDEV-16264 fixup: Clean up asynchronous I/O os_aio_userdata_t: Remove. It was basically duplicating IORequest. buf_page_write_complete(): Take only IORequest as a parameter. os_aio_func(), pfs_os_aio_func(): Replaced with os_aio() that has no redundant parameters. There is only one caller, so there is no point to pass __FILE__, __LINE__ as a parameter.	5 years ago
Marko Mäkelä	118e258aaa	MDEV-23855: Shrink fil_space_t Merge n_pending_ios, n_pending_ops to std::atomic<uint32_t> n_pending. Change some more fil_space_t members to uint32_t to reduce the memory footprint. fil_space_t::add(), fil_ibd_create(): Attach the already opened handle to the tablespace, and enforce the fil_system.n_open limit. dict_boot(): Initialize fil_system.max_assigned_id. srv_boot(): Call srv_thread_pool_init() before anything else, so that files should be opened in the correct mode on Windows. fil_ibd_create(): Create the file in OS_FILE_AIO mode, just like fil_node_open_file_low() does it. dict_table_t::is_accessible(): Replaces fil_table_accessible(). Reviewed by: Vladislav Vaintroub	5 years ago
Marko Mäkelä	45ed9dd957	MDEV-23855: Remove fil_system.LRU and reduce fil_system.mutex contention Also fixes MDEV-23929: innodb_flush_neighbors is not being ignored for system tablespace on SSD When the maximum configured number of file is exceeded, InnoDB will close data files. We used to maintain a fil_system.LRU list and a counter fil_node_t::n_pending to achieve this, at the huge cost of multiple fil_system.mutex operations per I/O operation. fil_node_open_file_low(): Implement a FIFO replacement policy: The last opened file will be moved to the end of fil_system.space_list, and files will be closed from the start of the list. However, we will not move tablespaces in fil_system.space_list while i_s_tablespaces_encryption_fill_table() is executing (producing output for INFORMATION_SCHEMA.INNODB_TABLESPACES_ENCRYPTION) because it may cause information of some tablespaces to go missing. We also avoid this in mariabackup --backup because datafiles_iter_next() assumes that the ordering is not changed. IORequest: Fold more parameters to IORequest::type. fil_space_t::io(): Replaces fil_io(). fil_space_t::flush(): Replaces fil_flush(). OS_AIO_IBUF: Remove. We will always issue synchronous reads of the change buffer pages in buf_read_page_low(). We will always ignore some errors for background reads. This should reduce fil_system.mutex contention a little. fil_node_t::complete_write(): Replaces fil_node_t::complete_io(). On both read and write completion, fil_space_t::release_for_io() will have to be called. fil_space_t::io(): Do not acquire fil_system.mutex in the normal code path. xb_delta_open_matching_space(): Do not try to open the system tablespace which was already opened. This fixes a file sharing violation in mariabackup --prepare --incremental. Reviewed by: Vladislav Vaintroub	5 years ago
Marko Mäkelä	3a9a3be1c6	MDEV-23855: Improve InnoDB log checkpoint performance After MDEV-15053, MDEV-22871, MDEV-23399 shifted the scalability bottleneck, log checkpoints became a new bottleneck. If innodb_io_capacity is set low or innodb_max_dirty_pct_lwm is set high and the workload fits in the buffer pool, the page cleaner thread will perform very little flushing. When we reach the capacity of the circular redo log file ib_logfile0 and must initiate a checkpoint, some 'furious flushing' will be necessary. (If innodb_flush_sync=OFF, then flushing would continue at the innodb_io_capacity rate, and writers would be throttled.) We have the best chance of advancing the checkpoint LSN immediately after a page flush batch has been completed. Hence, it is best to perform checkpoints after every batch in the page cleaner thread, attempting to run once per second. By initiating high-priority flushing in the page cleaner as early as possible, we aim to make the throughput more stable. The function buf_flush_wait_flushed() used to sleep for 10ms, hoping that the page cleaner thread would do something during that time. The observed end result was that a large number of threads that call log_free_check() would end up sleeping while nothing useful is happening. We will revise the design so that in the default innodb_flush_sync=ON mode, buf_flush_wait_flushed() will wake up the page cleaner thread to perform the necessary flushing, and it will wait for a signal from the page cleaner thread. If innodb_io_capacity is set to a low value (causing the page cleaner to throttle its work), a write workload would initially perform well, until the capacity of the circular ib_logfile0 is reached and log_free_check() will trigger checkpoints. At that point, the extra waiting in buf_flush_wait_flushed() will start reducing throughput. The page cleaner thread will also initiate log checkpoints after each buf_flush_lists() call, because that is the best point of time for the checkpoint LSN to advance by the maximum amount. Even in 'furious flushing' mode we invoke buf_flush_lists() with innodb_io_capacity_max pages at a time, and at the start of each batch (in the log_flush() callback function that runs in a separate task) we will invoke os_aio_wait_until_no_pending_writes(). This tweak allows the checkpoint to advance in smaller steps and significantly reduces the maximum latency. On an Intel Optane 960 NVMe SSD on Linux, it reduced from 4.6 seconds to 74 milliseconds. On Microsoft Windows with a slower SSD, it reduced from more than 180 seconds to 0.6 seconds. We will make innodb_adaptive_flushing=OFF simply flush innodb_io_capacity per second whenever the dirty proportion of buffer pool pages exceeds innodb_max_dirty_pages_pct_lwm. For innodb_adaptive_flushing=ON we try to make page_cleaner_flush_pages_recommendation() more consistent and predictable: if we are below innodb_adaptive_flushing_lwm, let us flush pages according to the return value of af_get_pct_for_dirty(). innodb_max_dirty_pages_pct_lwm: Revert the change of the default value that was made in MDEV-23399. The value innodb_max_dirty_pages_pct_lwm=0 guarantees that a shutdown of an idle server will be fast. Users might be surprised if normal shutdown suddenly became slower when upgrading within a GA release series. innodb_checkpoint_usec: Remove. The master task will no longer perform periodic log checkpoints. It is the duty of the page cleaner thread. log_sys.max_modified_age: Remove. The current span of the buf_pool.flush_list expressed in LSN only matters for adaptive flushing (outside the 'furious flushing' condition). For the correctness of checkpoints, the only thing that matters is the checkpoint age (log_sys.lsn - log_sys.last_checkpoint_lsn). This run-time constant was also reported as log_max_modified_age_sync. log_sys.max_checkpoint_age_async: Remove. This does not serve any purpose, because the checkpoints will now be triggered by the page cleaner thread. We will retain the log_sys.max_checkpoint_age limit for engaging 'furious flushing'. page_cleaner.slot: Remove. It turns out that page_cleaner_slot.flush_list_time was duplicating page_cleaner.slot.flush_time and page_cleaner.slot.flush_list_pass was duplicating page_cleaner.flush_pass. Likewise, there were some redundant monitor counters, because the page cleaner thread no longer performs any buf_pool.LRU flushing, and because there only is one buf_flush_page_cleaner thread. buf_flush_sync_lsn: Protect writes by buf_pool.flush_list_mutex. buf_pool_t::get_oldest_modification(): Add a parameter to specify the return value when no persistent data pages are dirty. Require the caller to hold buf_pool.flush_list_mutex. log_buf_pool_get_oldest_modification(): Take the fall-back LSN as a parameter. All callers will also invoke log_sys.get_lsn(). log_preflush_pool_modified_pages(): Replaced with buf_flush_wait_flushed(). buf_flush_wait_flushed(): Implement two limits. If not enough buffer pool has been flushed, signal the page cleaner (unless innodb_flush_sync=OFF) and wait for the page cleaner to complete. If the page cleaner thread is not running (which can be the case durign shutdown), initiate the flush and wait for it directly. buf_flush_ahead(): If innodb_flush_sync=ON (the default), submit a new buf_flush_sync_lsn target for the page cleaner but do not wait for the flushing to finish. log_get_capacity(), log_get_max_modified_age_async(): Remove, to make it easier to see that af_get_pct_for_lsn() is not acquiring any mutexes. page_cleaner_flush_pages_recommendation(): Protect all access to buf_pool.flush_list with buf_pool.flush_list_mutex. Previously there were some race conditions in the calculation. buf_flush_sync_for_checkpoint(): New function to process buf_flush_sync_lsn in the page cleaner thread. At the end of each batch, we try to wake up any blocked buf_flush_wait_flushed(). If everything up to buf_flush_sync_lsn has been flushed, we will reset buf_flush_sync_lsn=0. The page cleaner thread will keep 'furious flushing' until the limit is reached. Any threads that are waiting in buf_flush_wait_flushed() will be able to resume as soon as their own limit has been satisfied. buf_flush_page_cleaner: Prioritize buf_flush_sync_lsn and do not sleep as long as it is set. Do not update any page_cleaner statistics for this special mode of operation. In the normal mode (buf_flush_sync_lsn is not set for innodb_flush_sync=ON), try to wake up once per second. No longer check whether srv_inc_activity_count() has been called. After each batch, try to perform a log checkpoint, because the best chances for the checkpoint LSN to advance by the maximum amount are upon completing a flushing batch. log_t: Move buf_free, max_buf_free possibly to the same cache line with log_sys.mutex. log_margin_checkpoint_age(): Simplify the logic, and replace a 0.1-second sleep with a call to buf_flush_wait_flushed() to initiate flushing. Moved to the same compilation unit with the only caller. log_close(): Clean up the calculations. (Should be no functional change.) Return whether flush-ahead is needed. Moved to the same compilation unit with the only caller. mtr_t::finish_write(): Return whether flush-ahead is needed. mtr_t::commit(): Invoke buf_flush_ahead() when needed. Let us avoid external calls in mtr_t::commit() and make the logic easier to follow by having related code in a single compilation unit. Also, we will invoke srv_stats.log_write_requests.inc() only once per mini-transaction commit, while not holding mutexes. log_checkpoint_margin(): Only care about log_sys.max_checkpoint_age. Upon reaching log_sys.max_checkpoint_age where we must wait to prevent the log from getting corrupted, let us wait for at most 1MiB of LSN at a time, before rechecking the condition. This should allow writers to proceed even if the redo log capacity has been reached and 'furious flushing' is in progress. We no longer care about log_sys.max_modified_age_sync or log_sys.max_modified_age_async. The log_sys.max_modified_age_sync could be a relic from the time when there was a srv_master_thread that wrote dirty pages to data files. Also, we no longer have any log_sys.max_checkpoint_age_async limit, because log checkpoints will now be triggered by the page cleaner thread upon completing buf_flush_lists(). log_set_capacity(): Simplify the calculations of the limit (no functional change). log_checkpoint_low(): Split from log_checkpoint(). Moved to the same compilation unit with the caller. log_make_checkpoint(): Only wait for everything to be flushed until the current LSN. create_log_file(): After checkpoint, invoke log_write_up_to() to ensure that the FILE_CHECKPOINT record has been written. This avoids ut_ad(!srv_log_file_created) in create_log_file_rename(). srv_start(): Do not call recv_recovery_from_checkpoint_start() if the log has just been created. Set fil_system.space_id_reuse_warned before dict_boot() has been executed, and clear it after recovery has finished. dict_boot(): Initialize fil_system.max_assigned_id. srv_check_activity(): Remove. The activity count is counting transaction commits and therefore mostly interesting for the purge of history. BtrBulk::insert(): Do not explicitly wake up the page cleaner, but do invoke srv_inc_activity_count(), because that counter is still being used in buf_load_throttle_if_needed() for some heuristics. (It might be cleaner to execute buf_load() in the page cleaner thread!) Reviewed by: Vladislav Vaintroub	5 years ago

1 2 3 4 5 ...

275 Commits (c091a0bc8da87045f10bfc96618ed7194768fa2d)