mariadb

Commit Graph

Author	SHA1	Message	Date
Marko Mäkelä	8511f04fdb	Cleanup: Remove srv_start_lsn Most of the time, we can refer to recv_sys.recovered_lsn.	6 years ago
Vladislav Vaintroub	30ea63b7d2	MDEV-21534 - Improve innodb redo log group commit performance Introduce special synchronization primitive group_commit_lock for more efficient synchronization of redo log writing and flushing. The goal is to reduce CPU consumption on log_write_up_to, to reduce the spurious wakeups, and improve the throughput in write-intensive benchmarks.	6 years ago
Marko Mäkelä	96901d9545	Cleanup: Remove dict_ind_redundant There is no reason for the dummy index object dict_ind_redundant to exist any more. It was only being passed to btr_create(). btr_create(): If !index, assume that a ROW_FORMAT=REDUNDANT table is being created. We could pass ibuf.index, dict_sys.sys_tables->indexes.start and so on, if those objects had been initialized before the function btr_create() is called.	6 years ago
Eugene Kosov	9ef2d29ff4	MDEV-14425 deprecate and ignore innodb_log_files_in_group Now there can be only one log file instead of several which logically work as a single file. Possible names of redo log files: ib_logfile0, ib_logfile101 (for just created one) innodb_log_fiels_in_group: value of this variable is not used by InnoDB. Possible values are still 1..100, to not break upgrade LOG_FILE_NAME: add constant of value "ib_logfile0" LOG_FILE_NAME_PREFIX: add constant of value "ib_logfile" get_log_file_path(): convenience function that returns full path of a redo log file SRV_N_LOG_FILES_MAX: removed srv_n_log_files: we can't remove this for compatibility reasons, but now server doesn't use this variable log_sys_t::file::fd: now just one, not std::vector log_sys_t::log_capacity: removed word 'group' find_and_check_log_file(): part of logic from huge srv_start() moved here recv_sys_t::files: file descriptors of redo log files. There can be several of those in case we're upgrading from older MariaDB version. recv_sys_t::remove_extra_log_files: whether to remove ib_logfile{1,2,3...} after successfull upgrade. recv_sys_t::read(): open if needed and read from one of several log files recv_sys_t::files_size(): open if needed and return files count redo_file_sizes_are_correct(): check that redo log files sizes are equal. Just to log an error for a user. Corresponding check was moved from srv0start.cc namespace deprecated: put all deprecated variables here to prevent usage of it by us, developers	6 years ago
Marko Mäkelä	7ae21b18a6	MDEV-12353: Change the redo log encoding log_t::FORMAT_10_5: physical redo log format tag log_phys_t: Buffered records in the physical format. The log record bytes will follow the last data field, making use of alignment padding that would otherwise be wasted. If there are multiple records for the same page, also those may be appended to an existing log_phys_t object if the memory is available. In the physical format, the first byte of a record identifies the record and its length (up to 15 bytes). For longer records, the immediately following bytes will encode the remaining length in a variable-length encoding. Usually, a variable-length-encoded page identifier will follow, followed by optional payload, whose length is included in the initially encoded total record length. When a mini-transaction is updating multiple fields in a page, it can avoid repeating the tablespace identifier and page number by setting the same_page flag (most significant bit) in the first byte of the log record. The byte offset of the record will be relative to where the previous record for that page ended. Until MDEV-14425 introduces a separate file-level log for redo log checkpoints and file operations, we will write the file-level records in the page-level redo log file. The record FILE_CHECKPOINT (which replaces MLOG_CHECKPOINT) will be removed in MDEV-14425, and one sequential scan of the page recovery log will suffice. Compared to MLOG_FILE_CREATE2, FILE_CREATE will not include any flags. If the information is needed, it can be parsed from WRITE records that modify FSP_SPACE_FLAGS. MLOG_ZIP_WRITE_STRING: Remove. The record was only introduced temporarily as part of this work, before being replaced with WRITE (along with MLOG_WRITE_STRING, MLOG_1BYTE, MLOG_nBYTES). mtr_buf_t::empty(): Check if the buffer is empty. mtr_t::m_n_log_recs: Remove. It suffices to check if m_log is empty. mtr_t::m_last, mtr_t::m_last_offset: End of the latest m_log record, for the same_page encoding. page_recv_t::last_offset: Reflects mtr_t::m_last_offset. Valid values for last_offset during recovery should be 0 or above 8. (The first 8 bytes of a page are the checksum and the page number, and neither are ever updated directly by log records.) Internally, the special value 1 indicates that the same_page form will not be allowed for the subsequent record. mtr_t::page_create(): Take the block descriptor as parameter, so that it can be compared to mtr_t::m_last. The INIT_INDEX_PAGE record will always followed by a subtype byte, because same_page records must be longer than 1 byte. trx_undo_page_init(): Combine the writes in WRITE record. trx_undo_header_create(): Write 4 bytes using a special MEMSET record that includes 1 bytes of length and 2 bytes of payload. flst_write_addr(): Define as a static function. Combine the writes. flst_zero_both(): Replaces two flst_zero_addr() calls. flst_init(): Do not inline the function. fsp_free_seg_inode(): Zerofill the whole inode. fsp_apply_init_file_page(): Initialize FIL_PAGE_PREV,FIL_PAGE_NEXT to FIL_NULL when using the physical format. btr_create(): Assert !page_has_siblings() because fsp_apply_init_file_page() must have been invoked. fil_ibd_create(): Do not write FILE_MODIFY after FILE_CREATE. fil_names_dirty_and_write(): Remove the parameter mtr. Write the records using a separate mini-transaction object, because any FILE_ records must be at the start of a mini-transaction log. recv_recover_page(): Add a fil_space_t* parameter. After applying log to the a ROW_FORMAT=COMPRESSED page, invoke buf_zip_decompress() to restore the uncompressed page. buf_page_io_complete(): Remove the temporary hack to discard the uncompressed page of a ROW_FORMAT=COMPRESSED page. page_zip_write_header(): Remove. Use mtr_t::write() or mtr_t::memset() instead, and update the compressed page frame separately. trx_undo_header_add_space_for_xid(): Remove. trx_undo_seg_create(): Perform the changes that were previously made by trx_undo_header_add_space_for_xid(). btr_reset_instant(): New function: Reset the table to MariaDB 10.2 or 10.3 format when rolling back an instant ALTER TABLE operation. page_rec_find_owner_rec(): Merge with the only callers. page_cur_insert_rec_low(): Combine writes by using a local buffer. MEMMOVE data from the preceding record whenever feasible (copying at least 3 bytes). page_cur_insert_rec_zip(): Combine writes to page header fields. PageBulk::insertPage(): Issue MEMMOVE records to copy a matching part from the preceding record. PageBulk::finishPage(): Combine the writes to the page header and to the sparse page directory slots. mtr_t::write(): Only log the least significant (last) bytes of multi-byte fields that actually differ. For updating FSP_SIZE, we must always write all 4 bytes to the redo log, so that the fil_space_set_recv_size() logic in recv_sys_t::parse() will work. mtr_t::memcpy(), mtr_t::zmemcpy(): Take a pointer argument instead of a numeric offset to the page frame. Only log the last bytes of multi-byte fields that actually differ. In fil_space_crypt_t::write_page0(), we must log also any unchanged bytes, so that recovery will recognize the record and invoke fil_crypt_parse(). Future work: MDEV-21724 Optimize page_cur_insert_rec_low() redo logging MDEV-21725 Optimize btr_page_reorganize_low() redo logging MDEV-21727 Optimize redo logging for ROW_FORMAT=COMPRESSED	6 years ago
Marko Mäkelä	f37a29dd66	MDEV-12353: Write log by mtr_t member functions only mtr_t::log_write_low(): Replaces mlog_write_initial_log_record_low(). mtr_t::log_file_op(): Replaces fil_op_write_log(). mtr_t::free(): Write MLOG_INIT_FREE_PAGE. mtr_t::init(): Write MLOG_INIT_FILE_PAGE2. mtr_t::page_create(): Write record about the partial initialization of an index page. mlog_catenate_ulint(), mlog_catenate_string(), mlog_open(), mlog_close(): Remove.	6 years ago
Marko Mäkelä	1a6f708ec5	MDEV-15058: Deprecate and ignore innodb_buffer_pool_instances Our benchmarking efforts indicate that the reasons for splitting the buf_pool in commit `c18084f71b` have mostly gone away, possibly as a result of mysql/mysql-server@ce6109ebfdedfdf185e391a0c97dc6d33867ed78 or similar work. Only in one write-heavy benchmark where the working set size is ten times the buffer pool size, the buf_pool->mutex would be less contended with 4 buffer pool instances than with 1 instance, in buf_page_io_complete(). That contention could be alleviated further by making more use of std::atomic and by splitting buf_pool_t::mutex further (MDEV-15053). We will deprecate and ignore the following parameters: innodb_buffer_pool_instances innodb_page_cleaners There will be only one buffer pool and one page cleaner task. In a number of INFORMATION_SCHEMA views, columns that indicated the buffer pool instance will be removed: information_schema.innodb_buffer_page.pool_id information_schema.innodb_buffer_page_lru.pool_id information_schema.innodb_buffer_pool_stats.pool_id information_schema.innodb_cmpmem.buffer_pool_instance information_schema.innodb_cmpmem_reset.buffer_pool_instance	6 years ago
Marko Mäkelä	fc2f2fa853	MDEV-19747: Deprecate and ignore innodb_log_optimize_ddl During native table rebuild or index creation, InnoDB used to skip redo logging and write MLOG_INDEX_LOAD records to inform crash recovery and Mariabackup of the gaps in redo log. This is fragile and prohibits some optimizations, such as skipping the doublewrite buffer for newly (re)initialized pages (MDEV-19738). row_merge_write_redo(): Remove. We do not write MLOG_INDEX_LOAD records any more. Instead, we write full redo log. FlushObserver: Remove. fseg_free_page_func(): Remove the parameter log. Redo logging cannot be disabled. fil_space_t::redo_skipped_count: Remove. We cannot remove buf_block_t::skip_flush_check, because PageBulk will temporarily generate invalid B-tree pages in the buffer pool.	6 years ago
Eugene Kosov	691c691adc	clean up redo log main change: rename first redo log without file close second change: use os_offset_t to represent offset in a file third change: fix log texts	6 years ago
Eugene Kosov	b534a6675c	cleanup redo log class log_file_t: more or less sane RAII wrapper around redo log file descriptor and its path. This change is motivated by the need of using that log_file_t somewhere else.	6 years ago
Eugene Kosov	34dafb7e3a	redo log mics fixes os_file_flush_data_func(): fix builds on POSIX OSs where fdatasync() is not avaiable log_t::files::flush_data_only(): rename from fdatasync() log_t::files::fsync(): removed and replaced with flush_data_only(). It will flush everything we need for using redo log files.	6 years ago
Eugene Kosov	700e010309	fix aligned memcpy()-like functions usage I found that memcpy_aligned was used incorrectly at redo log and decided to put assertions in aligned functions. And found even more incorrect cases. Given the amount discovered of bugs, I left assertions to prevent future bugs. my_assume_aligned(): instead of MY_ASSUME_ALIGNED macro	6 years ago
Eugene Kosov	e9de6386ad	MDEV-18115 remove now unneeded constraint log_group_max_size: is not needed because redo log do not use fil_io() now	6 years ago
Eugene Kosov	562c037b48	MDEV-18115 Remove dummy tablespace for the redo log Redo log subsystem was decoupled from tablespace subsystem. It now manages file descriptors for redo log files by itself. FIL_TYPE_LOG: removed, code in various places was simplified SRV_LOG_SPACE_FIRST_ID: renamed to SRV_SPACE_ID_UPPER_BOUND to better match its purpose. Code in various places was simplified fil_n_log_flushes: replaced with log_sys::flushes fil_n_pending_log_flushes: replaced with log_sys::pending_flushes log_t::files::files: redo log file descriptors log_t::files::file_names: redo log file names log_t::files::set_file_names(): set file names without opening them log_t::files::open_files(): opens redo log files log_t::files::read(): treats several files as one big log_t::files::write(): treats several files as one big log_t::files::fsync(): flushes page cache to disk log_t::files::close_files(): closes redo log files fil_open_log_and_system_tablespace_files(): renamed to fil_open_system_tablespace_files() and obviously it now doesn't open redo log files global files[1000]: removed. Why it was needed at all?	6 years ago
Marko Mäkelä	fb4a897fd9	MDEV-12353 preparation: Remove UNIV_LOG_LSN_DEBUG The debug instrumentation with the MLOG_LSN pseudo-record has not been used for debugging for years. Let us remove this code now. It would have to be removed as part of MDEV-12353 or MDEV-14425 anyway, when implementing a new redo log file format.	6 years ago
Marko Mäkelä	504202bd7f	MDEV-21216: Remove fsp_header_get_space_id() The function fsp_header_get_space_id() returns ulint instead of uint32_t, only to be able to complain that the two adjacent tablespace ID fields in the page differ. Remove the function, and merge the check to the callers. Also, make some more use of aligned_malloc().	6 years ago
Marko Mäkelä	95e903261e	MDEV-21216 InnoDB does dirty read of TRX_SYS page before recovery InnoDB startup was discovering undo tablespaces in a dirty way. It was reading a possibly stale copy of the TRX_SYS page before processing any redo log records. srv_start(): Do not call buf_pool_invalidate(). Invoke trx_rseg_get_n_undo_tablespaces() after the recovery has been initiated. recv_recovery_from_checkpoint_start(): Assert that the buffer pool is empty. This used to be guaranteed by the buf_pool_invalidate() call. trx_rseg_get_n_undo_tablespaces(): Move to the calling compilation unit, and reimplement in a simpler way. srv_undo_tablespace_create(): Remove the constant parameter size=SRV_UNDO_TABLESPACE_SIZE_IN_PAGES. srv_undo_tablespace_open(): Reimplement in a cleaner way, with more robust error handling. srv_all_undo_tablespaces_open(): Split from srv_undo_tablespaces_init(). srv_undo_tablespaces_init(): Read all "undo001","undo002" tablespace files directly, without consulting the TRX_SYS page via calling trx_rseg_get_n_undo_tablespaces(). This is joint work with Thirunarayanan Balathandayuthapani.	6 years ago
Marko Mäkelä	56f6dab1d0	MDEV-21174: Replace mlog_write_ulint() with mtr_t::write() mtr_t::write(): Replaces mlog_write_ulint(), mlog_write_ull(). Optimize away writes if the page contents does not change, except when a dummy write has been explicitly requested. Because the member function template takes a block descriptor as a parameter, it is possible to introduce better consistency checks. Due to this, the code for handling file-based lists, undo logs and user transactions was refactored to pass around buf_block_t.	6 years ago
Marko Mäkelä	4beace3316	MDEV-21171 InnoDB is unnecessarily resetting FIL_PAGE_TYPE for full_crc32 files Before commit `c0f47a4a58` (MDEV-12026) introduced the innodb_checksum_algorithm=full_crc32 format, it was impossible to tell if InnoDB data files contained garbage in the FIL_PAGE_TYPE header field (and possibly other fields). This is because before commit `3926673ce7` in MySQL 5.1.48, InnoDB would write uninitialized data to some fields, and because there was no way to tell with which InnoDB version a data file was created. If fil_space_t::full_crc32() holds, the data file cannot contain uninitialized garbage or invalid FIL_PAGE_TYPE, and thus fil_block_check_type() should not be invoked to correct anything.	6 years ago
Marko Mäkelä	37f1ab2323	MDEV-21054 Crash on shutdown due to btr_search_latches=NULL innodb_shutdown(): Invoke os_aio_free() before btr_search_sys_free().	6 years ago
Vladislav Vaintroub	a808c18b73	Make .clang-format work with clang-8 Remove keywords that are too new.	6 years ago
Vladislav Vaintroub	5e62b6a5e0	MDEV-16264 Use threadpool for Innodb background work. Almost all threads have gone - the "ticking" threads, that sleep a while then do some work) (srv_monitor_thread, srv_error_monitor_thread, srv_master_thread) were replaced with timers. Some timers are periodic, e.g the "master" timer. - The btr_defragment_thread is also replaced by a timer , which reschedules it self when current defragment "item" needs throttling - the buf_resize_thread and buf_dump_threads are substitutes with tasks Ditto with page cleaner workers. - purge workers threads are not tasks as well, and purge cleaner coordinator is a combination of a task and timer. - All AIO is outsourced to tpool, Innodb just calls thread_pool::submit_io() and provides the callback. - The srv_slot_t was removed, and innodb_debug_sync used in purge is currently not working, and needs reimplementation.	6 years ago
Marko Mäkelä	0ccfdc8eff	Remove InnoDB wrappers of <string.h> functions ut_strcmp(), ut_strcpy(), ut_strlen(), ut_memcpy(), ut_memcmp(), ut_memmove(): Remove. Invoke the standard library functions directly.	6 years ago
Marko Mäkelä	d67ea8151f	Rename LOG_HEADER_FORMAT_ to log_t::FORMAT_	6 years ago
Marko Mäkelä	b42294bc64	MDEV-19514 Defer change buffer merge until pages are requested We will remove the InnoDB background operation of merging buffered changes to secondary index leaf pages. Changes will only be merged as a result of an operation that accesses a secondary index leaf page, such as a SQL statement that performs a lookup via that index, or is modifying the index. Also ROLLBACK and some background operations, such as purging the history of committed transactions, or computing index cardinality statistics, can cause change buffer merge. Encryption key rotation will not perform change buffer merge. The motivation of this change is to simplify the I/O logic and to allow crash recovery to happen in the background (MDEV-14481). We also hope that this will reduce the number of "mystery" crashes due to corrupted data. Because change buffer merge will typically take place as a result of executing SQL statements, there should be a clearer connection between the crash and the SQL statements that were executed when the server crashed. In many cases, a slight performance improvement was observed. This is joint work with Thirunarayanan Balathandayuthapani and was tested by Axel Schwenke and Matthias Leich. The InnoDB monitor counter innodb_ibuf_merge_usec will be removed. On slow shutdown (innodb_fast_shutdown=0), we will continue to merge all buffered changes (and purge all undo log history). Two InnoDB configuration parameters will be changed as follows: innodb_disable_background_merge: Removed. This parameter existed only in debug builds. All change buffer merges will use synchronous reads. innodb_force_recovery will be changed as follows: * innodb_force_recovery=4 will be the same as innodb_force_recovery=3 (the change buffer merge cannot be disabled; it can only happen as a result of an operation that accesses a secondary index leaf page). The option used to be capable of corrupting secondary index leaf pages. Now that capability is removed, and innodb_force_recovery=4 becomes 'safe'. * innodb_force_recovery=5 (which essentially hard-wires SET GLOBAL TRANSACTION ISOLATION LEVEL READ UNCOMMITTED) becomes safe to use. Bogus data can be returned to SQL, but persistent InnoDB data files will not be corrupted further. * innodb_force_recovery=6 (ignore the redo log files) will be the only option that can potentially cause persistent corruption of InnoDB data files. Code changes: buf_page_t::ibuf_exist: New flag, to indicate whether buffered changes exist for a buffer pool page. Pages with pending changes can be returned by buf_page_get_gen(). Previously, the changes were always merged inside buf_page_get_gen() if needed. ibuf_page_exists(const buf_page_t&): Check if a buffered changes exist for an X-latched or read-fixed page. buf_page_get_gen(): Add the parameter allow_ibuf_merge=false. All callers that know that they may be accessing a secondary index leaf page must pass this parameter as allow_ibuf_merge=true, unless it does not matter for that caller whether all buffered changes have been applied. Assert that whenever allow_ibuf_merge holds, the page actually is a leaf page. Attempt change buffer merge only to secondary B-tree index leaf pages. btr_block_get(): Add parameter 'bool merge'. All callers of btr_block_get() should know whether the page could be a secondary index leaf page. If it is not, we should avoid consulting the change buffer bitmap to even consider a merge. This is the main interface to requesting index pages from the buffer pool. ibuf_merge_or_delete_for_page(), recv_recover_page(): Replace buf_page_get_known_nowait() with much simpler logic, because it is now guaranteed that that the block is x-latched or read-fixed. mlog_init_t::mark_ibuf_exist(): Renamed from mlog_init_t::ibuf_merge(). On crash recovery, we will no longer merge any buffered changes for the pages that we read into the buffer pool during the last batch of applying log records. buf_page_get_gen_known_nowait(), BUF_MAKE_YOUNG, BUF_KEEP_OLD: Remove. btr_search_guess_on_hash(): Merge buf_page_get_gen_known_nowait() to its only remaining caller. buf_page_make_young_if_needed(): Define as an inline function. Add the parameter buf_pool. buf_page_peek_if_young(), buf_page_peek_if_too_old(): Add the parameter buf_pool. fil_space_validate_for_mtr_commit(): Remove a bogus comment about background merge of the change buffer. btr_cur_open_at_rnd_pos_func(), btr_cur_search_to_nth_level_func(), btr_cur_open_at_index_side_func(): Use narrower data types and scopes. ibuf_read_merge_pages(): Replaces buf_read_ibuf_merge_pages(). Merge the change buffer by invoking buf_page_get_gen().	6 years ago
Marko Mäkelä	6fde0073bf	Rename log_make_checkpoint_at() to log_make_checkpoint() The function was always called with lsn=LSN_MAX. Remove that redundant parameter. Spotted by Thirunarayanan Balathandayuthapani.	6 years ago
Marko Mäkelä	f13471c9fe	MDEV-20060: Remove bogus unit from error message	6 years ago
Vlad Lesin	d39d5dd2bc	MDEV-20060: Failing assertion: srv_log_file_size <= 512ULL << 30 while preparing backup The general reason why innodb redo log file is limited by 512G is that log_block_convert_lsn_to_no() returns value limited by 1G. But there is no need to have unique log block numbers in log group. The fix removes 512G limit and limits log group size by (uint32_t maximum value) * (minimum page size), which, in turns, can be removed if fil_io() is no longer used for innodb redo log io.	6 years ago
Marko Mäkelä	a7e9395f9d	fts_sync_table(), fts_sync() dead code removal fts_sync(): Remove the constant parameter has_dict=false. fts_sync_table(): Remove the constant parameter has_dict=false, and the redundant parameter unlock_cache = !wait. Make wait=true the default parameter.	6 years ago
Marko Mäkelä	f6ea0389a4	Replace ut_timer() with my_interval_timer() The function pointer ut_timer() was only used by the InnoDB defragmenting thread. Let InnoDB use a single monotonic high-precision timer, my_interval_timer() [in nanoseconds], occasionally wrapped by microsecond_interval_timer(). srv_defragment_interval: Change from "timer" units to nanoseconds. This concludes the InnoDB time function cleanup that was motivated by MDEV-14154. Only ut_time_ms() will remain for now, wrapping my_interval_timer().	6 years ago
Marko Mäkelä	61e26289fc	MDEV-18582 preparation: Allocate ibuf statically	6 years ago
Thirunarayanan Balathandayuthapani	dca9792a24	MDEV-17228 Encrypted temporary tables are not encrypted - log_crypt_init() should be called from create_log_files(). Because it should re-create the new random nonce compared to the old ones.	6 years ago
Thirunarayanan Balathandayuthapani	723a4b1d78	MDEV-17228 Encrypted temporary tables are not encrypted - Introduce a new variable called innodb_encrypt_temporary_tables which is a boolean variable. It decides whether to encrypt the temporary tablespace. - Encrypts the temporary tablespace based on full checksum format. - Introduced a new counter to track encrypted and decrypted temporary tablespace pages. - Warnings issued if temporary table creation has conflict value with innodb_encrypt_temporary_tables - Added a new test case which reads and writes the pages from/to temporary tablespace.	6 years ago
Marko Mäkelä	5d2619b693	MDEV-19584 Allocate recv_sys statically There is only one InnoDB crash recovery subsystem. Allocating recv_sys statically removes one level of pointer indirection and makes code more readable, and removes the awkward initialization of recv_sys->dblwr. recv_sys_t::create(): Replaces recv_sys_init(). recv_sys_t::debug_free(): Replaces recv_sys_debug_free(). recv_sys_t::close(): Replaces recv_sys_close(). recv_sys_t::add(): Replaces recv_add_to_hash_table(). recv_sys_t::empty(): Replaces recv_sys_empty_hash().	6 years ago
Sergey Vojtovich	5f5a0b3bb6	MDEV-16548 - Innodb fails to start on older kernels that don't support F_DUPFD_CLOEXEC InnoDB duplicates file descriptor returned by create_temp_file() to workaround further inconsistent use of this descriptor. Use mysys file descriptors consistently for innobase_mysql_tmpfile(NULL). Mostly close it by appropriate mysys wrappers.	6 years ago
Marko Mäkelä	893472d005	MDEV-19570 Deprecate and ignore innodb_undo_logs, remove innodb_rollback_segments The option innodb_rollback_segments was deprecated already in MariaDB Server 10.0. Its misleadingly named replacement innodb_undo_logs is of very limited use. It makes sense to always create and use the maximum number of rollback segments. Let us remove the deprecated parameter innodb_rollback_segments and deprecate&ignore the parameter innodb_undo_logs (to be removed in a later major release). This work involves some cleanup of InnoDB startup. Similar to other write operations, DROP TABLE will no longer be allowed if innodb_force_recovery is set to a value larger than 3.	6 years ago
Marko Mäkelä	5fd7502e77	MDEV-19513: Allocate dict_sys statically dict_sys_t::create(): Renamed from dict_init(). dict_sys_t::close(): Renamed from dict_close(). dict_sys_t::add(): Sliced from dict_table_t::add_to_cache(). dict_sys_t::remove(): Renamed from dict_table_remove_from_cache(). dict_sys_t::prevent_eviction(): Renamed from dict_table_move_from_lru_to_non_lru(). dict_sys_t::acquire(): Replaces dict_move_to_mru() and some more logic. dict_sys_t::resize(): Renamed from dict_resize(). dict_sys_t::find(): Replaces dict_lru_find_table() and dict_non_lru_find_table().	6 years ago
Vicențiu Ciorbaru	c0ac0b8860	Update FSF address	7 years ago
Marko Mäkelä	3db94d2403	MDEV-19346: Remove dummy InnoDB log checkpoints log_checkpoint(), log_make_checkpoint_at(): Remove the parameter write_always. It seems that the primary purpose of this parameter was to ensure in the function recv_reset_logs() that both checkpoint header pages will be overwritten, when the function is called from the never-enabled function recv_recovery_from_archive_start(). create_log_files(): Merge recv_reset_logs() to its only caller. Debug instrumentation: Prefer to flush the redo log, instead of triggering a redo log checkpoint. page_header_set_field(): Disable a debug assertion that will always fail due to MDEV-19344, now that we no longer initiate a redo log checkpoint before an injected crash. In recv_reset_logs() there used to be two calls to log_make_checkpoint_at(). The apparent purpose of this was to ensure that both InnoDB redo log checkpoint header pages will be initialized or overwritten. The second call was removed (without any explanation) in MySQL 5.6.3: mysql/mysql-server@4ca37968da54ddc6b3b6628f41428ddba1c79bb8 In MySQL 5.6.8 WL#6494, starting with mysql/mysql-server@00a0ba8ad92569fcf08212b3b8cf046dc8a0ce10 the function recv_reset_logs() was not only invoked during InnoDB data file initialization, but also during a regular startup when the redo log is being resized. mysql/mysql-server@45e91679832219e2593c77185342f11f85232b58 in MySQL 5.7.2 removed the UNIV_LOG_ARCHIVE code, but still did not remove the parameter write_always.	7 years ago
Marko Mäkelä	d315b4ff39	Remove IBUF_COUNT_DEBUG The compile-time option IBUF_COUNT_DEBUG has not been used for years. It would only work with up to 3 created .ibd files, with no buffered changes existing while InnoDB is started up.	7 years ago
Marko Mäkelä	169c00994b	MDEV-12699 Improve crash recovery of corrupted data pages InnoDB crash recovery used to read every data page for which redo log exists. This is unnecessary for those pages that are initialized by the redo log. If a newly created page is corrupted, recovery could unnecessarily fail. It would suffice to reinitialize the page based on the redo log records. To add insult to injury, InnoDB crash recovery could hang if it encountered a corrupted page. We will fix also that problem. InnoDB would normally refuse to start up if it encounters a corrupted page on recovery, but that can be overridden by setting innodb_force_recovery=1. Data pages are completely initialized by the records MLOG_INIT_FILE_PAGE2 and MLOG_ZIP_PAGE_COMPRESS. MariaDB 10.4 additionally recognizes MLOG_INIT_FREE_PAGE, which notifies that a page has been freed and its contents can be discarded (filled with zeroes). The record MLOG_INDEX_LOAD notifies that redo logging has been re-enabled after being disabled. We can avoid loading the page if all buffered redo log records predate the MLOG_INDEX_LOAD record. For the internal tables of FULLTEXT INDEX, no MLOG_INDEX_LOAD records were written before commit `aa3f7a107c`. Hence, we will skip these optimizations for tables whose name starts with FTS_. This is joint work with Thirunarayanan Balathandayuthapani. fil_space_t::enable_lsn, file_name_t::enable_lsn: The LSN of the latest recovered MLOG_INDEX_LOAD record for a tablespace. mlog_init: Page initialization operations discovered during redo log scanning. FIXME: This really belongs in recv_sys->addr_hash, and should be removed in MDEV-19176. recv_addr_state: Add the new state RECV_WILL_NOT_READ to indicate that according to mlog_init, the page will be initialized based on redo log record contents. recv_add_to_hash_table(): Set the RECV_WILL_NOT_READ state if appropriate. For now, we do not treat MLOG_ZIP_PAGE_COMPRESS as page initialization. This works around bugs in the crash recovery of ROW_FORMAT=COMPRESSED tables. recv_mark_log_index_load(): Process a MLOG_INDEX_LOAD record by resetting the state to RECV_NOT_PROCESSED and by updating the fil_name_t::enable_lsn. recv_init_crash_recovery_spaces(): Copy fil_name_t::enable_lsn to fil_space_t::enable_lsn. recv_recover_page(): Add the parameter init_lsn, to ignore any log records that precede the page initialization. Add DBUG output about skipped operations. buf_page_create(): Initialize FIL_PAGE_LSN, so that recv_recover_page() will not wrongly skip applying the page-initialization record due to the field containing some newer LSN as a leftover from a different page. Do not invoke ibuf_merge_or_delete_for_page() during crash recovery. recv_apply_hashed_log_recs(): Remove some unnecessary lookups. Note if a corrupted page was found during recovery. After invoking buf_page_create(), do invoke ibuf_merge_or_delete_for_page() via mlog_init.ibuf_merge() in the last recovery batch. ibuf_merge_or_delete_for_page(): Relax a debug assertion. innobase_start_or_create_for_mysql(): Abort startup if a corrupted page was found during recovery. Corrupted pages will not be flagged if innodb_force_recovery is set. However, the recv_sys->found_corrupt_fs flag can be set regardless of innodb_force_recovery if file names are found to be incorrect (for example, multiple files with the same tablespace ID).	7 years ago
Marko Mäkelä	e4c5551964	MDEV-13564: Temporarily restore a call to work around a bug srv_start(): Restore the call to buf_pool_invalidate() that was removed in commit `09af00cbde`. It turns out that the call is necessary to work around a bug that should be fixed in MDEV-19229.	7 years ago
Marko Mäkelä	cad56fbaba	MDEV-18733 MariaDB slow start after crash recovery If InnoDB crash recovery was needed, the InnoDB function srv_start() would invoke extra validation, reading something from every InnoDB data file. This should be unnecessary now that MDEV-14717 made RENAME operations crash-safe inside InnoDB (which can be disabled in MariaDB 10.2 by setting innodb_safe_truncate=OFF). dict_check_sys_tables(): Skip tables that would be dropped by row_mysql_drop_garbage_tables(). Perform extra validation only if innodb_safe_truncate=OFF, innodb_force_recovery=0 and crash recovery was needed. dict_load_table_one(): Validate the root page of the table. In this way, we can deny access to corrupted or mismatching tables not only after crash recovery, but also after a clean shutdown.	7 years ago
Marko Mäkelä	b761211685	MDEV-18659: Fix string truncation/overflow in InnoDB and XtraDB Fix the warnings issued by GCC 8 -Wstringop-truncation and -Wstringop-overflow in InnoDB and XtraDB. This work is motivated by Jan Lindström. The patch mainly differs from his original one as follows: (1) We remove explicit initialization of stack-allocated string buffers. The minimum amount of initialization that is needed is a terminating NUL character. (2) GCC issues a warning for invoking strncpy(dest, src, sizeof dest) because if strlen(src) >= sizeof dest, there would be no terminating NUL byte in dest. We avoid this problem by invoking strncpy() with a limit that is 1 less than the buffer size, and by always writing NUL to the last byte of the buffer. (3) We replace strncpy() with memcpy() or strcpy() in those cases when the result is functionally equivalent. Note: fts_fetch_index_words() never deals with len==UNIV_SQL_NULL. This was enforced by an assertion that limits the maximum length to FTS_MAX_WORD_LEN. Also, the encoding that InnoDB uses for the compressed fulltext index is not byte-order agnostic, that is, InnoDB data files that use FULLTEXT INDEX are not portable between big-endian and little-endian systems.	7 years ago
Thirunarayanan Balathandayuthapani	c0f47a4a58	MDEV-12026: Implement innodb_checksum_algorithm=full_crc32 MariaDB data-at-rest encryption (innodb_encrypt_tables) had repurposed the same unused data field that was repurposed in MySQL 5.7 (and MariaDB 10.2) for the Split Sequence Number (SSN) field of SPATIAL INDEX. Because of this, MariaDB was unable to support encryption on SPATIAL INDEX pages. Furthermore, InnoDB page checksums skipped some bytes, and there are multiple variations and checksum algorithms. By default, InnoDB accepts all variations of all algorithms that ever existed. This unnecessarily weakens the page checksums. We hereby introduce two more innodb_checksum_algorithm variants (full_crc32, strict_full_crc32) that are special in a way: When either setting is active, newly created data files will carry a flag (fil_space_t::full_crc32()) that indicates that all pages of the file will use a full CRC-32C checksum over the entire page contents (excluding the bytes where the checksum is stored, at the very end of the page). Such files will always use that checksum, no matter what the parameter innodb_checksum_algorithm is assigned to. For old files, the old checksum algorithms will continue to be used. The value strict_full_crc32 will be equivalent to strict_crc32 and the value full_crc32 will be equivalent to crc32. ROW_FORMAT=COMPRESSED tables will only use the old format. These tables do not support new features, such as larger innodb_page_size or instant ADD/DROP COLUMN. They may be deprecated in the future. We do not want an unnecessary file format change for them. The new full_crc32() format also cleans up the MariaDB tablespace flags. We will reserve flags to store the page_compressed compression algorithm, and to store the compressed payload length, so that checksum can be computed over the compressed (and possibly encrypted) stream and can be validated without decrypting or decompressing the page. In the full_crc32 format, there no longer are separate before-encryption and after-encryption checksums for pages. The single checksum is computed on the page contents that is written to the file. We do not make the new algorithm the default for two reasons. First, MariaDB 10.4.2 was a beta release, and the default values of parameters should not change after beta. Second, we did not yet implement the full_crc32 format for page_compressed pages. This will be fixed in MDEV-18644. This is joint work with Marko Mäkelä.	7 years ago
Aleksey Midenkov	a4cd91c526	MDEV-18603 Debug option to disable InnoDB monitor threads If --debug=d,innodb_skip_monitors is specified, skip the creation of srv_error_monitor_thread and srv_monitor_thread. In this way, interactive debugging sessions will not be interfered by messages about a mutex or rw_lock being held for a long time. Closes #1192	7 years ago
Marko Mäkelä	0a1c3477bf	MDEV-18493 Remove page_size_t MySQL 5.7 introduced the class page_size_t and increased the size of buffer pool page descriptors by introducing this object to them. Maybe the intention of this exercise was to prepare for a future where the buffer pool could accommodate multiple page sizes. But that future never arrived, not even in MySQL 8.0. It is much easier to manage a pool of a single page size, and typically all storage devices of an InnoDB instance benefit from using the same page size. Let us remove page_size_t from MariaDB Server. This will make it easier to remove support for ROW_FORMAT=COMPRESSED (or make it a compile-time option) in the future, just by removing various occurrences of zip_size.	7 years ago
Sergei Golubchik	2a0f1d6132	Bug#28867993: POSSIBLE ISSUE WITH MYSQL SERVER RESTART on startup innodb is checking whether files "ib_logfileN" (for N from 1 to 100) exist, and whether they're readable. A non-existent file aborted the scan. A directory instead of a file made InnoDB to fail. Now it treats "directory exists" as "file doesn't exist".	7 years ago
Brave Galera Crew	36a2a185fe	Galera4	7 years ago
Sergey Vojtovich	5c657e9fa5	MDEV-17441 - InnoDB transition to C++11 atomics Trivial srv_running transition.	7 years ago

1 2 3 4 5 ...

359 Commits (8511f04fdb4cc69a6c90b543624305ef715f9e53)