mariadb

Commit Graph

Author	SHA1	Message	Date
Eugene Kosov	700e010309	fix aligned memcpy()-like functions usage I found that memcpy_aligned was used incorrectly at redo log and decided to put assertions in aligned functions. And found even more incorrect cases. Given the amount discovered of bugs, I left assertions to prevent future bugs. my_assume_aligned(): instead of MY_ASSUME_ALIGNED macro	6 years ago
Eugene Kosov	496532b5c5	MDEV-20950: Fix 32-bit Windows build	6 years ago
Marko Mäkelä	745fd4b39f	MDEV-21174: Remove some mlog_write_initial_log_record_fast() Pass buf_block_t* to more functions that write redo log. page_zip_write_node_ptr(), page_zip_write_blob_ptr(), page_zip_compress_write_log_no_data(): Take buf_block_t* as parameter, and do not tolerate mtr=NULL. page_zip_compress(): Do not tolerate mtr=NULL. page_zip_dir_insert(): Take page_cur_t* as parameter. mlog_write_initial_log_record(): Remove. This function was unused. RecIterator::remove(): Remove the redundant page_zip parameter. PageConverter::m_page_zip_ptr: Remove.	6 years ago
Eugene Kosov	f0aa073f2b	MDEV-20950 Reduce size of record offsets offset_t: this is a type which represents one record offset. It's unsigned short int. a lot of functions: replace ulint with offset_t btr_pcur_restore_position_func(), page_validate(), row_ins_scan_sec_index_for_duplicate(), row_upd_clust_rec_by_insert_inherit_func(), row_vers_impl_x_locked_low(), trx_undo_prev_version_build(): allocate record offsets on the stack instead of waiting for rec_get_offsets() to allocate it from mem_heap_t. So, reducing memory allocations. RECORD_OFFSET, INDEX_OFFSET: now it's less convenient to store pointers in offset_t* array. One pointer occupies now several offset_t. And those constant are start indexes into array to places where to store pointer values REC_OFFS_HEADER_SIZE: adjusted for the new reality REC_OFFS_NORMAL_SIZE: increase size from 100 to 300 which means less heap allocations. And sizeof(offset_t[REC_OFFS_NORMAL_SIZE]) now is 600 bytes which is smaller than previous 800 bytes. REC_OFFS_SEC_INDEX_SIZE: adjusted for the new reality rem0rec.h, rem0rec.ic, rem0rec.cc: various arguments, return values and local variables types were changed to fix numerous integer conversions issues. enum field_type_t: offset types concept was introduces which replaces old offset flags stuff. Like in earlier version, 2 upper bits are used to store offset type. And this enum represents those types. REC_OFFS_SQL_NULL, REC_OFFS_MASK: removed get_type(), set_type(), get_value(), combine(): these are convenience functions to work with offsets and it's types rec_offs_base()[0]: still uses an old scheme with flags REC_OFFS_COMPACT and REC_OFFS_EXTERNAL rec_offs_base()[i]: these have type offset_t now. Two upper bits contains type.	6 years ago
Marko Mäkelä	42a4ae54c2	MDEV-21225 Remove ut_align() and use aligned_malloc() Before commit `90c52e5291` introduced aligned_malloc(), InnoDB always used a pattern of over-allocating memory and invoking ut_align() to guarantee the desired alignment. It is cleaner to invoke aligned_malloc() and aligned_free() directly. ut_align(): Remove. In assertions, ut_align_down() can be used instead.	6 years ago
Marko Mäkelä	caea64df18	Cleanup: Remove some page_get_page_no() calls Refer to buf_page_t::id instead of parsing the tablespace identifier or page number from the buffer pool page.	6 years ago
Marko Mäkelä	bf2cc46798	MDEV-21133: Remove buf_frame_copy()	6 years ago
Marko Mäkelä	326515878d	MDEV-21133: Alignment hints for ROW_FORMAT=COMPRESSED page_zip_compress(), page_zip_decompress_low(), page_zip_reorganize(): Make use of memcpy_aligned() and memset_aligned(), so that some operations can be translated more efficiently. No difference was observed for AMD64 on GCC 9.2.1. But, such hints could enable more efficient code on less common instruction set architectures, such as 32-bit ARM, POWER or MIPS.	6 years ago
Marko Mäkelä	51a4260f00	MDEV-21133: Introduce memmove_aligned() Both variants of the InnoDB page directory are aligned to the entry size (16 bits). Inform the compiler about it.	6 years ago
Marko Mäkelä	64a02e4fa2	MDEV-19586: Add const qualifiers Except for fil_name_process(), which invokes os_normalize_path(), the redo log record parser will not modify the redo log records. Add const qualifiers accordingly.	6 years ago
Eugene Kosov	ed0793e096	MDEV-19783: Add more REC_INFO_MIN_REC_FLAG checks btr_cur_pessimistic_delete(): code changed in a way that allows to put more REC_INFO_MIN_REC_FLAG assertions inside btr_set_min_rec_mark(). Without that change tests innodb.innodb-table-online, innodb.temp_table_savepoint and innodb_zip.prefix_index_liftedlimit fail. Removed basically duplicated page_zip_validate() calls which fails because of temporary(!) invariant violation. That fixed innodb_zip.wl5522_debug_zip and innodb_zip.prefix_index_liftedlimit	6 years ago
Marko Mäkelä	d480d28f4f	Add page_has_prev(), page_has_next(), page_has_siblings() Until now, InnoDB inefficiently compared the aligned fields FIL_PAGE_PREV, FIL_PAGE_NEXT to the byte-order-agnostic value FIL_NULL. This is a backport of `32170f8c6d` from MariaDB Server 10.3.	6 years ago
Marko Mäkelä	ab6dd77408	MDEV-14154: Remove ut_time_us() Use microsecond_interval_timer() or my_interval_timer() [in nanoseconds] instead.	6 years ago
Marko Mäkelä	97055e6b11	MDEV-14154: Remove ut_time_us() Use microsecond_interval_timer() or my_interval_timer() [in nanoseconds] instead.	6 years ago
Eugene Kosov	adb640e25a	MDEV-17441 InnoDB transition to C++11 atomics zip_pad_info_t::mutex_created: remove along with corresponding stuff zip_pad_info_t::mutex: make member value instead of a pointer	6 years ago
Marko Mäkelä	d46db415ce	MDEV-19738: Skip doublewrite on MLOG_ZIP_PAGE_COMPRESS page_zip_compress(), page_zip_compress_write_log(), page_zip_copy_recs(): Replace the parameters page,page_zip with block, and set buf_page_t::init_on_flush on success if innodb_log_optimize_ddl=OFF. page_zip_parse_compress_no_data(): Merge with the only caller recv_parse_or_apply_log_rec_body().	6 years ago
Marko Mäkelä	5d2619b693	MDEV-19584 Allocate recv_sys statically There is only one InnoDB crash recovery subsystem. Allocating recv_sys statically removes one level of pointer indirection and makes code more readable, and removes the awkward initialization of recv_sys->dblwr. recv_sys_t::create(): Replaces recv_sys_init(). recv_sys_t::debug_free(): Replaces recv_sys_debug_free(). recv_sys_t::close(): Replaces recv_sys_close(). recv_sys_t::add(): Replaces recv_add_to_hash_table(). recv_sys_t::empty(): Replaces recv_sys_empty_hash().	7 years ago
Vicențiu Ciorbaru	c0ac0b8860	Update FSF address	7 years ago
Marko Mäkelä	45d338dca8	MDEV-12699 preparation: Initialize the entire page on MLOG_ZIP_PAGE_COMPRESS The record MLOG_ZIP_PAGE_COMPRESS is similar to MLOG_INIT_FILE_PAGE2 that it contains all the information needed to initialize the page. Like for the other record, do initialize the entire page on recovery.	7 years ago
Marko Mäkelä	72b934e3f7	MDEV-14126: Detect unexpected emptying of B-tree pages If an index page becomes empty, btr_page_empty() should be called.	7 years ago
Marko Mäkelä	b59d484696	MDEV-14126: Remove page_is_root() The predicate page_is_root(), which was added in MariaDB Server 10.2.2, is based on a wrong assumption. Under some circumstances, InnoDB can transform B-trees into a degenerate state where a non-leaf page has no sibling pages. Because of this, we cannot assume that a page that has no siblings is the root page. This bug will be tracked as MDEV-19022. Because of the bug that may affect many InnoDB data files, we must remove and replace the wrong predicate. Using the wrong predicate can cause corruption. A leaf page is not allowed to be empty except if it is the root page, and the entire table is empty.	7 years ago
Marko Mäkelä	6b6fa3cdb1	MDEV-18644: Support full_crc32 for page_compressed This is a follow-up task to MDEV-12026, which introduced innodb_checksum_algorithm=full_crc32 and a simpler page format. MDEV-12026 did not enable full_crc32 for page_compressed tables, which we will be doing now. This is joint work with Thirunarayanan Balathandayuthapani. For innodb_checksum_algorithm=full_crc32 we change the page_compressed format as follows: FIL_PAGE_TYPE: The most significant bit will be set to indicate page_compressed format. The least significant bits will contain the compressed page size, rounded up to a multiple of 256 bytes. The checksum will be stored in the last 4 bytes of the page (whether it is the full page or a page_compressed page whose size is determined by FIL_PAGE_TYPE), covering all preceding bytes of the page. If encryption is used, then the page will be encrypted between compression and computing the checksum. For page_compressed, FIL_PAGE_LSN will not be repeated at the end of the page. FSP_SPACE_FLAGS (already implemented as part of MDEV-12026): We will store the innodb_compression_algorithm that may be used to compress pages. Previously, the choice of algorithm was written to each compressed data page separately, and one would be unable to know in advance which compression algorithm(s) are used. fil_space_t::full_crc32_page_compressed_len(): Determine if the page_compressed algorithm of the tablespace needs to know the exact length of the compressed data. If yes, we will reserve and write an extra byte for this right before the checksum. buf_page_is_compressed(): Determine if a page uses page_compressed (in any innodb_checksum_algorithm). fil_page_decompress(): Pass also fil_space_t::flags so that the format can be determined. buf_page_is_zeroes(): Check if a page is full of zero bytes. buf_page_full_crc32_is_corrupted(): Renamed from buf_encrypted_full_crc32_page_is_corrupted(). For full_crc32, we always simply validate the checksum to the page contents, while the physical page size is explicitly specified by an unencrypted part of the page header. buf_page_full_crc32_size(): Determine the size of a full_crc32 page. buf_dblwr_check_page_lsn(): Make this a debug-only function, because it involves potentially costly lookups of fil_space_t. create_table_info_t::check_table_options(), ha_innobase::check_if_supported_inplace_alter(): Do allow the creation of SPATIAL INDEX with full_crc32 also when page_compressed is used. commit_cache_norebuild(): Preserve the compression algorithm when updating the page_compression_level. dict_tf_to_fsp_flags(): Set the flags for page compression algorithm. FIXME: Maybe there should be a table option page_compression_algorithm and a session variable to back it?	7 years ago
Thirunarayanan Balathandayuthapani	c0f47a4a58	MDEV-12026: Implement innodb_checksum_algorithm=full_crc32 MariaDB data-at-rest encryption (innodb_encrypt_tables) had repurposed the same unused data field that was repurposed in MySQL 5.7 (and MariaDB 10.2) for the Split Sequence Number (SSN) field of SPATIAL INDEX. Because of this, MariaDB was unable to support encryption on SPATIAL INDEX pages. Furthermore, InnoDB page checksums skipped some bytes, and there are multiple variations and checksum algorithms. By default, InnoDB accepts all variations of all algorithms that ever existed. This unnecessarily weakens the page checksums. We hereby introduce two more innodb_checksum_algorithm variants (full_crc32, strict_full_crc32) that are special in a way: When either setting is active, newly created data files will carry a flag (fil_space_t::full_crc32()) that indicates that all pages of the file will use a full CRC-32C checksum over the entire page contents (excluding the bytes where the checksum is stored, at the very end of the page). Such files will always use that checksum, no matter what the parameter innodb_checksum_algorithm is assigned to. For old files, the old checksum algorithms will continue to be used. The value strict_full_crc32 will be equivalent to strict_crc32 and the value full_crc32 will be equivalent to crc32. ROW_FORMAT=COMPRESSED tables will only use the old format. These tables do not support new features, such as larger innodb_page_size or instant ADD/DROP COLUMN. They may be deprecated in the future. We do not want an unnecessary file format change for them. The new full_crc32() format also cleans up the MariaDB tablespace flags. We will reserve flags to store the page_compressed compression algorithm, and to store the compressed payload length, so that checksum can be computed over the compressed (and possibly encrypted) stream and can be validated without decrypting or decompressing the page. In the full_crc32 format, there no longer are separate before-encryption and after-encryption checksums for pages. The single checksum is computed on the page contents that is written to the file. We do not make the new algorithm the default for two reasons. First, MariaDB 10.4.2 was a beta release, and the default values of parameters should not change after beta. Second, we did not yet implement the full_crc32 format for page_compressed pages. This will be fixed in MDEV-18644. This is joint work with Marko Mäkelä.	7 years ago
Marko Mäkelä	0a1c3477bf	MDEV-18493 Remove page_size_t MySQL 5.7 introduced the class page_size_t and increased the size of buffer pool page descriptors by introducing this object to them. Maybe the intention of this exercise was to prepare for a future where the buffer pool could accommodate multiple page sizes. But that future never arrived, not even in MySQL 8.0. It is much easier to manage a pool of a single page size, and typically all storage devices of an InnoDB instance benefit from using the same page size. Let us remove page_size_t from MariaDB Server. This will make it easier to remove support for ROW_FORMAT=COMPRESSED (or make it a compile-time option) in the future, just by removing various occurrences of zip_size.	7 years ago
Marko Mäkelä	62d28f83b6	MDEV-17958: Remove WITH_INNODB_BUG_ENDIAN_CRC32 Remove the bug-compatible crc32 algorithm variant that was added to allow an upgrade from data files from big-endian systems where innodb_checksum_algorithm=crc32 was used on MySQL 5.6 or MariaDB 10.0 or 10.1. Affected users should be able to recompute page checksums using innochecksum.	7 years ago
Marko Mäkelä	1a780eefc9	MDEV-17958 Make bug-endian innodb_checksum_algorithm=crc32 optional In MySQL 5.7, it was noticed that files are not portable between big-endian and little-endian processor architectures (such as SPARC and x86), because the original implementation of innodb_checksum_algorithm=crc32 was not byte order agnostic. A byte order agnostic implementation of innodb_checksum_algorithm=crc32 was only added to MySQL 5.7, not backported to 5.6. Consequently, MariaDB Server versions 10.0 and 10.1 only contain the CRC-32C implementation that works incorrectly on big-endian architectures, and MariaDB Server 10.2.2 got the byte-order agnostic CRC-32C implementation from MySQL 5.7. MySQL 5.7 introduced a "legacy crc32" variant that is functionally equivalent to the big-endian version of the original crc32 implementation. Thanks to this variant, old data files can be transferred from big-endian systems to newer versions. Introducing new variants of checksum algorithms (without introducing new names for them, or something on the pages themselves to identify the algorithm) generally is a bad idea, because each checksum algorithm is like a lottery ticket. The more algorithms you try, the more likely it will be for the checksum to match on a corrupted page. So, essentially MySQL 5.7 weakened innodb_checksum_algorithm=crc32, and MariaDB 10.2.2 inherited this weakening. We introduce a build option that together with MDEV-17957 makes innodb_checksum_algorithm=strict_crc32 strict again by only allowing one variant of the checksum to match. WITH_INNODB_BUG_ENDIAN_CRC32: A new cmake option for enabling the bug-compatible "legacy crc32" checksum. This is only enabled on big-endian systems by default, to facilitate an upgrade from MariaDB 10.0 or 10.1. Checked by #ifdef INNODB_BUG_ENDIAN_CRC32. ut_crc32_byte_by_byte: Remove (unused function). legacy_big_endian_checksum: Remove. This variable seems to have unnecessarily complicated the logic. When the weakening is enabled, we must always fall back to the buggy checksum. buf_page_check_crc32(): A helper function to compute one or two CRC-32C variants.	7 years ago
Thirunarayanan Balathandayuthapani	5f5e73f1fe	MDEV-17957 Make Innodb_checksum_algorithm stricter for strict_* values Problem: Innodb_checksum_algorithm checks for all checksum algorithm to validate the page checksum even though the algorithm is specified as strict_crc32, strict_innodb, strict_none. Fix: Remove the checks for all checksum algorithm to validate the page checksum if the algo is specified as strict_* values.	7 years ago
Marko Mäkelä	17e371fffe	More InnoDB preprocessor cleanup Remove unnecessary #include. Remove references to UNIV_MATERIALIZE, UNIV_INLINE_ORIGINAL, UNIV_NONINL that are never defined.	7 years ago
Marko Mäkelä	f92d223fe2	MDEV-17750: Remove dict_index_get_sys_col_pos() dict_index_t::db_trx_id(): Return the position of DB_TRX_ID. Only valid for the clustered index. dict_index_t::db_roll_ptr(): Return the position of DB_ROLL_PTR. Only valid for the clustered index. dict_index_get_sys_col_pos(): Remove. This was performing unnecessarily complex computations, which only made sense for DB_ROW_ID, which would exist either as the first field in the clustered index or as the last field in a secondary index (only when a DB_ROW_ID column is materialised). row_sel_store_row_id_to_prebuilt(): Remove, and replace with simpler code. row_upd_index_entry_sys_field(): Remove. btr_cur_log_sys(): Replaces row_upd_write_sys_vals_to_log(). btr_cur_write_sys(): Write DB_TRX_ID,DB_ROLL_PTR to a data tuple.	7 years ago
Marko Mäkelä	0e5a4ac253	MDEV-15662 Instant DROP COLUMN or changing the order of columns Allow ADD COLUMN anywhere in a table, not only adding as the last column. Allow instant DROP COLUMN and instant changing the order of columns. The added columns will always be added last in clustered index records. In new records, instantly dropped columns will be stored as NULL or empty when possible. Information about dropped and reordered columns will be written in a metadata BLOB (mblob), which is stored before the first 'user' field in the hidden metadata record at the start of the clustered index. The presence of mblob is indicated by setting the delete-mark flag in the metadata record. The metadata BLOB stores the number of clustered index fields, followed by an array of column information for each field. For dropped columns, we store the NOT NULL flag, the fixed length, and for variable-length columns, whether the maximum length exceeded 255 bytes. For non-dropped columns, we store the column position. Unlike with MDEV-11369, when a table becomes empty, it cannot be converted back to the canonical format. The reason for this is that other threads may hold cached objects such as row_prebuilt_t::ins_node that could refer to dropped or reordered index fields. For instant DROP COLUMN and ROW_FORMAT=COMPACT or ROW_FORMAT=DYNAMIC, we must store the n_core_null_bytes in the root page, so that the chain of node pointer records can be followed in order to reach the leftmost leaf page where the metadata record is located. If the mblob is present, we will zero-initialize the strings "infimum" and "supremum" in the root page, and use the last byte of "supremum" for storing the number of null bytes (which are allocated but useless on node pointer pages). This is necessary for btr_cur_instant_init_metadata() to be able to navigate to the mblob. If the PRIMARY KEY contains any variable-length column and some nullable columns were instantly dropped, the dict_index_t::n_nullable in the data dictionary could be smaller than it actually is in the non-leaf pages. Because of this, the non-leaf pages could use more bytes for the null flags than the data dictionary expects, and we could be reading the lengths of the variable-length columns from the wrong offset, and thus reading the child page number from wrong place. This is the result of two design mistakes that involve unnecessary storage of data: First, it is nonsense to store any data fields for the leftmost node pointer records, because the comparisons would be resolved by the MIN_REC_FLAG alone. Second, there cannot be any null fields in the clustered index node pointer fields, but we nevertheless reserve space for all the null flags. Limitations (future work): MDEV-17459 Allow instant ALTER TABLE even if FULLTEXT INDEX exists MDEV-17468 Avoid table rebuild on operations on generated columns MDEV-17494 Refuse ALGORITHM=INSTANT when the row size is too large btr_page_reorganize_low(): Preserve any metadata in the root page. Call lock_move_reorganize_page() only after restoring the "infimum" and "supremum" records, to avoid a memcmp() assertion failure. dict_col_t::DROPPED: Magic value for dict_col_t::ind. dict_col_t::clear_instant(): Renamed from dict_col_t::remove_instant(). Do not assert that the column was instantly added, because we sometimes call this unconditionally for all columns. Convert an instantly added column to a "core column". The old name remove_instant() could be mistaken to refer to "instant DROP COLUMN". dict_col_t::is_added(): Rename from dict_col_t::is_instant(). dtype_t::metadata_blob_init(): Initialize the mblob data type. dtuple_t::is_metadata(), dtuple_t::is_alter_metadata(), upd_t::is_metadata(), upd_t::is_alter_metadata(): Check if info_bits refer to a metadata record. dict_table_t::instant: Metadata about dropped or reordered columns. dict_table_t::prepare_instant(): Prepare ha_innobase_inplace_ctx::instant_table for instant ALTER TABLE. innobase_instant_try() will pass this to dict_table_t::instant_column(). On rollback, dict_table_t::rollback_instant() will be called. dict_table_t::instant_column(): Renamed from instant_add_column(). Add the parameter col_map so that columns can be reordered. Copy and adjust v_cols[] as well. dict_table_t::find(): Find an old column based on a new column number. dict_table_t::serialise_columns(), dict_table_t::deserialise_columns(): Convert the mblob. dict_index_t::instant_metadata(): Create the metadata record for instant ALTER TABLE. Invoke dict_table_t::serialise_columns(). dict_index_t::reconstruct_fields(): Invoked by dict_table_t::deserialise_columns(). dict_index_t::clear_instant_alter(): Move the fields for the dropped columns to the end, and sort the surviving index fields in ascending order of column position. ha_innobase::check_if_supported_inplace_alter(): Do not allow adding a FTS_DOC_ID column if a hidden FTS_DOC_ID column exists due to FULLTEXT INDEX. (This always required ALGORITHM=COPY.) instant_alter_column_possible(): Add a parameter for InnoDB table, to check for additional conditions, such as the maximum number of index fields. ha_innobase_inplace_ctx::first_alter_pos: The first column whose position is affected by instant ADD, DROP, or changing the order of columns. innobase_build_col_map(): Skip added virtual columns. prepare_inplace_add_virtual(): Correctly compute num_to_add_vcol. Remove some unnecessary code. Note that the call to innodb_base_col_setup() should be executed later. commit_try_norebuild(): If ctx->is_instant(), let the virtual columns be added or dropped by innobase_instant_try(). innobase_instant_try(): Fill in a zero default value for the hidden column FTS_DOC_ID (to reduce the work needed in MDEV-17459). If any columns were dropped or reordered (or added not last), delete any SYS_COLUMNS records for the following columns, and insert SYS_COLUMNS records for all subsequent stored columns as well as for all virtual columns. If any virtual column is dropped, rewrite all virtual column metadata. Use a shortcut only for adding virtual columns. This is because innobase_drop_virtual_try() assumes that the dropped virtual columns still exist in ctx->old_table. innodb_update_cols(): Renamed from innodb_update_n_cols(). innobase_add_one_virtual(), innobase_insert_sys_virtual(): Change the return type to bool, and invoke my_error() when detecting an error. innodb_insert_sys_columns(): Insert a record into SYS_COLUMNS. Refactored from innobase_add_one_virtual() and innobase_instant_add_col(). innobase_instant_add_col(): Replace the parameter dfield with type. innobase_instant_drop_cols(): Drop matching columns from SYS_COLUMNS and all columns from SYS_VIRTUAL. innobase_add_virtual_try(), innobase_drop_virtual_try(): Let the caller invoke innodb_update_cols(). innobase_rename_column_try(): Skip dropped columns. commit_cache_norebuild(): Update table->fts->doc_col. dict_mem_table_col_rename_low(): Skip dropped columns. trx_undo_rec_get_partial_row(): Skip dropped columns. trx_undo_update_rec_get_update(): Handle the metadata BLOB correctly. trx_undo_page_report_modify(): Avoid out-of-bounds access to record fields. Log metadata records consistently. Apparently, the first fields of a clustered index may be updated in an update_undo vector when the index is ID_IND of SYS_FOREIGN, as part of renaming the table during ALTER TABLE. Normally, updates of the PRIMARY KEY should be logged as delete-mark and an insert. row_undo_mod_parse_undo_rec(), row_purge_parse_undo_rec(): Use trx_undo_metadata. row_undo_mod_clust_low(): On metadata rollback, roll back the root page too. row_undo_mod_clust(): Relax an assertion. The delete-mark flag was repurposed for ALTER TABLE metadata records. row_rec_to_index_entry_impl(): Add the template parameter mblob and the optional parameter info_bits for specifying the desired new info bits. For the metadata tuple, allow conversion between the original format (ADD COLUMN only) and the generic format (with hidden BLOB). Add the optional parameter "pad" to determine whether the tuple should be padded to the index fields (on ALTER TABLE it should), or whether it should remain at its original size (on rollback). row_build_index_entry_low(): Clean up the code, removing redundant variables and conditions. For instantly dropped columns, generate a dummy value that is NULL, the empty string, or a fixed length of NUL bytes, depending on the type of the dropped column. row_upd_clust_rec_by_insert_inherit_func(): On the update of PRIMARY KEY of a record that contained a dropped column whose value was stored externally, we will be inserting a dummy NULL or empty string value to the field of the dropped column. The externally stored column would eventually be dropped when purge removes the delete-marked record for the old PRIMARY KEY value. btr_index_rec_validate(): Recognize the metadata record. btr_discard_only_page_on_level(): Preserve the generic instant ALTER TABLE metadata. btr_set_instant(): Replaces page_set_instant(). This sets a clustered index root page to the appropriate format, or upgrades from the MDEV-11369 instant ADD COLUMN to generic ALTER TABLE format. btr_cur_instant_init_low(): Read and validate the metadata BLOB page before reconstructing the dictionary information based on it. btr_cur_instant_init_metadata(): Do not read any lengths from the metadata record header before reading the BLOB. At this point, we would not actually know how many nullable fields the metadata record contains. btr_cur_instant_root_init(): Initialize n_core_null_bytes in one of two possible ways. btr_cur_trim(): Handle the mblob record. row_metadata_to_tuple(): Convert a metadata record to a data tuple, based on the new info_bits of the metadata record. btr_cur_pessimistic_update(): Invoke row_metadata_to_tuple() if needed. Invoke dtuple_convert_big_rec() for metadata records if the record is too large, or if the mblob is not yet marked as externally stored. btr_cur_optimistic_delete_func(), btr_cur_pessimistic_delete(): When the last user record is deleted, do not delete the generic instant ALTER TABLE metadata record. Only delete MDEV-11369 instant ADD COLUMN metadata records. btr_cur_optimistic_insert(): Avoid unnecessary computation of rec_size. btr_pcur_store_position(): Allow a logically empty page to contain a metadata record for generic ALTER TABLE. REC_INFO_DEFAULT_ROW_ADD: Renamed from REC_INFO_DEFAULT_ROW. This is for the old instant ADD COLUMN (MDEV-11369) only. REC_INFO_DEFAULT_ROW_ALTER: The more generic metadata record, with additional information for dropped or reordered columns. rec_info_bits_valid(): Remove. The only case when this would fail is when the record is the generic ALTER TABLE metadata record. rec_is_alter_metadata(): Check if a record is the metadata record for instant ALTER TABLE (other than ADD COLUMN). NOTE: This function must not be invoked on node pointer records, because the delete-mark flag in those records may be set (it is garbage), and then a debug assertion could fail because index->is_instant() does not necessarily hold. rec_is_add_metadata(): Check if a record is MDEV-11369 ADD COLUMN metadata record (not more generic instant ALTER TABLE). rec_get_converted_size_comp_prefix_low(): Assume that the metadata field will be stored externally. In dtuple_convert_big_rec() during the rec_get_converted_size() call, it would not be there yet. rec_get_converted_size_comp(): Replace status,fields,n_fields with tuple. rec_init_offsets_comp_ordinary(), rec_get_converted_size_comp_prefix_low(), rec_convert_dtuple_to_rec_comp(): Add template<bool mblob = false>. With mblob=true, process a record with a metadata BLOB. rec_copy_prefix_to_buf(): Assert that no fields beyond the key and system columns are being copied. Exclude the metadata BLOB field. rec_convert_dtuple_to_metadata_comp(): Convert an alter metadata tuple into a record. row_upd_index_replace_metadata(): Apply an update vector to an alter_metadata tuple. row_log_allocate(): Replace dict_index_t::is_instant() with a more appropriate condition that ignores dict_table_t::instant. Only a table on which the MDEV-11369 ADD COLUMN was performed can "lose its instantness" when it becomes empty. After instant DROP COLUMN or reordering columns, we cannot simply convert the table to the canonical format, because the data dictionary cache and all possibly existing references to it from other client connection threads would have to be adjusted. row_quiesce_write_index_fields(): Do not crash when the table contains an instantly dropped column. Thanks to Thirunarayanan Balathandayuthapani for discussing the design and implementing an initial prototype of this. Thanks to Matthias Leich for testing.	7 years ago
Marko Mäkelä	09af00cbde	MDEV-13564: Remove old crash-upgrade logic in 10.4 Stop supporting the additional trunc.log files that were introduced via MySQL 5.7 to MariaDB Server 10.2 and 10.3. DB_TABLESPACE_TRUNCATED: Remove. purge_sys.truncate: A new structure to track undo tablespace file truncation. srv_start(): Remove the call to buf_pool_invalidate(). It is no longer necessary, given that we no longer access things in ways that violate the ARIES protocol. This call was originally added for innodb_file_format, and it may later have been necessary for the proper function of the MySQL 5.7 TRUNCATE recovery, which we are now removing. trx_purge_cleanse_purge_queue(): Take the undo tablespace as a parameter. trx_purge_truncate_history(): Rewrite everything mostly in a single function, replacing references to undo::Truncate. recv_apply_hashed_log_recs(): If any redo log is to be applied, and if the log_sys.log.subformat indicates that separately logged truncate may have been used, refuse to proceed except if innodb_force_recovery is set. We will still refuse crash-upgrade if TRUNCATE TABLE was logged. Undo tablespace truncation would only be logged in undotrunc.log files, which we are no longer checking for.	7 years ago
Marko Mäkelä	ba43914ec4	Replace dict_table_is_temporary(table) with table->is_temporary()	8 years ago
Marko Mäkelä	9801715cb0	Use compile_time_assert() in InnoDB Replace most use of #error. Some checks were impossible to evaluate in the preprocessor due to the use of named integer constants or enumerations.	8 years ago
Marko Mäkelä	8cd55ae2a9	Fix WITH_INNODB_EXTRA_DEBUG The checks that used to be enabled by the flags UNIV_AHI_DEBUG, UNIV_DDL_DEBUG, UNIV_DEBUG_FILE_ACCESSES were already enabled in debug builds. So, there is no point in setting these. Only UNIV_ZIP_DEBUG is set independently of the debug build. Allow WITH_INNODB_EXTRA_DEBUG to be set for non-debug builds as well. Currently it only implies UNIV_ZIP_DEBUG, that is, extra validation for operations on ROW_FORMAT=COMPRESSED tables. page_zip_validate_low(): Allow the code to be built on non-debug server. buf_LRU_block_remove_hashed(): Allow the code to be built without WITH_INNODB_AHI.	8 years ago
Marko Mäkelä	9ed2b2b2b8	Do not divide or multiply by srv_page_size Instead, shift by srv_page_size_shift.	8 years ago
Marko Mäkelä	a90100d756	Replace univ_page_size and UNIV_PAGE_SIZE Try to use one variable (srv_page_size) for innodb_page_size. Also, replace UNIV_PAGE_SIZE_SHIFT with srv_page_size_shift.	8 years ago
Marko Mäkelä	ba19764209	Fix most -Wsign-conversion in InnoDB Change innodb_buffer_pool_size, innodb_fill_factor to unsigned.	8 years ago
Marko Mäkelä	4cad42392a	MDEV-12266: Change dict_table_t::space to fil_space_t* InnoDB always keeps all tablespaces in the fil_system cache. The fil_system.LRU is only for closing file handles; the fil_space_t and fil_node_t for all data files will remain in main memory. Between startup to shutdown, they can only be created and removed by DDL statements. Therefore, we can let dict_table_t::space point directly to the fil_space_t. dict_table_t::space_id: A numeric tablespace ID for the corner cases where we do not have a tablespace. The most prominent examples are ALTER TABLE...DISCARD TABLESPACE or a missing or corrupted file. There are a few functional differences; most notably: (1) DROP TABLE will delete matching .ibd and .cfg files, even if they were not attached to the data dictionary. (2) Some error messages will report file names instead of numeric IDs. There still are many functions that use numeric tablespace IDs instead of fil_space_t, and many functions could be converted to fil_space_t member functions. Also, Tablespace and Datafile should be merged with fil_space_t and fil_node_t. page_id_t and buf_page_get_gen() could use fil_space_t& instead of a numeric ID, and after moving to a single buffer pool (MDEV-15058), buf_pool_t::page_hash could be moved to fil_space_t::page_hash. FilSpace: Remove. Only few calls to fil_space_acquire() will remain, and gradually they should be removed. mtr_t::set_named_space_id(ulint): Renamed from set_named_space(), to prevent accidental calls to this slower function. Very few callers remain. fseg_create(), fsp_reserve_free_extents(): Take fil_space_t as a parameter instead of a space_id. fil_space_t::rename(): Wrapper for fil_rename_tablespace_check(), fil_name_write_rename(), fil_rename_tablespace(). Mariabackup passes the parameter log=false; InnoDB passes log=true. dict_mem_table_create(): Take fil_space_t* instead of space_id as parameter. dict_process_sys_tables_rec_and_mtr_commit(): Replace the parameter 'status' with 'bool cached'. dict_get_and_save_data_dir_path(): Avoid copying the fil_node_t::name. fil_ibd_open(): Return the tablespace. fil_space_t::set_imported(): Replaces fil_space_set_imported(). truncate_t: Change many member function parameters to fil_space_t*, and remove page_size parameters. row_truncate_prepare(): Merge to its only caller. row_drop_table_from_cache(): Assert that the table is persistent. dict_create_sys_indexes_tuple(): Write SYS_INDEXES.SPACE=FIL_NULL if the tablespace has been discarded. row_import_update_discarded_flag(): Remove a constant parameter.	8 years ago
Marko Mäkelä	428e02895b	MDEV-12266: Remove dict_index_t::table_name Replace index->table_name with index->table->name.	8 years ago
Marko Mäkelä	604fea1ad6	MDEV-12266: Remove dict_index_t::space We can rely on the dict_table_t::space. All indexes of a table object are always in the same tablespace. (For fulltext indexes, the data is located in auxiliary tables, and these will continue to have their own table objects, separate from the main table.)	8 years ago
Marko Mäkelä	bd7ed1b923	MDEV-13935 INSERT stuck at state Unlocking tables Revert the dead code for MySQL 5.7 multi-master replication (GCS), also known as WL#6835: InnoDB: GCS Replication: Deterministic Deadlock Handling (High Prio Transactions in InnoDB). Also, make innodb_lock_schedule_algorithm=vats skip SPATIAL INDEX, because the code does not seem to be compatible with them. Add FIXME comments to some SPATIAL INDEX locking code. It looks like Galera write-set replication might not work with SPATIAL INDEX.	8 years ago
Marko Mäkelä	e4da20d438	MDEV-14238 Bogus assertion on row_get_rec_trx_id() page_zip_write_rec(): Do not attempt to access a non-existing DB_TRX_ID column when writing a record to a non-leaf page.	8 years ago
Marko Mäkelä	32170f8c6d	Add page_has_prev(), page_has_next(), page_has_siblings() Until now, InnoDB inefficiently compared the aligned fields FIL_PAGE_PREV, FIL_PAGE_NEXT to the byte-order-agnostic value FIL_NULL.	8 years ago
Marko Mäkelä	c19ef508b8	InnoDB: Remove ut_snprintf() and the use of my_snprintf(); use snprintf()	8 years ago
Marko Mäkelä	a4948dafcd	MDEV-11369 Instant ADD COLUMN for InnoDB For InnoDB tables, adding, dropping and reordering columns has required a rebuild of the table and all its indexes. Since MySQL 5.6 (and MariaDB 10.0) this has been supported online (LOCK=NONE), allowing concurrent modification of the tables. This work revises the InnoDB ROW_FORMAT=REDUNDANT, ROW_FORMAT=COMPACT and ROW_FORMAT=DYNAMIC so that columns can be appended instantaneously, with only minor changes performed to the table structure. The counter innodb_instant_alter_column in INFORMATION_SCHEMA.GLOBAL_STATUS is incremented whenever a table rebuild operation is converted into an instant ADD COLUMN operation. ROW_FORMAT=COMPRESSED tables will not support instant ADD COLUMN. Some usability limitations will be addressed in subsequent work: MDEV-13134 Introduce ALTER TABLE attributes ALGORITHM=NOCOPY and ALGORITHM=INSTANT MDEV-14016 Allow instant ADD COLUMN, ADD INDEX, LOCK=NONE The format of the clustered index (PRIMARY KEY) is changed as follows: (1) The FIL_PAGE_TYPE of the root page will be FIL_PAGE_TYPE_INSTANT, and a new field PAGE_INSTANT will contain the original number of fields in the clustered index ('core' fields). If instant ADD COLUMN has not been used or the table becomes empty, or the very first instant ADD COLUMN operation is rolled back, the fields PAGE_INSTANT and FIL_PAGE_TYPE will be reset to 0 and FIL_PAGE_INDEX. (2) A special 'default row' record is inserted into the leftmost leaf, between the page infimum and the first user record. This record is distinguished by the REC_INFO_MIN_REC_FLAG, and it is otherwise in the same format as records that contain values for the instantly added columns. This 'default row' always has the same number of fields as the clustered index according to the table definition. The values of 'core' fields are to be ignored. For other fields, the 'default row' will contain the default values as they were during the ALTER TABLE statement. (If the column default values are changed later, those values will only be stored in the .frm file. The 'default row' will contain the original evaluated values, which must be the same for every row.) The 'default row' must be completely hidden from higher-level access routines. Assertions have been added to ensure that no 'default row' is ever present in the adaptive hash index or in locked records. The 'default row' is never delete-marked. (3) In clustered index leaf page records, the number of fields must reside between the number of 'core' fields (dict_index_t::n_core_fields introduced in this work) and dict_index_t::n_fields. If the number of fields is less than dict_index_t::n_fields, the missing fields are replaced with the column value of the 'default row'. Note: The number of fields in the record may shrink if some of the last instantly added columns are updated to the value that is in the 'default row'. The function btr_cur_trim() implements this 'compression' on update and rollback; dtuple::trim() implements it on insert. (4) In ROW_FORMAT=COMPACT and ROW_FORMAT=DYNAMIC records, the new status value REC_STATUS_COLUMNS_ADDED will indicate the presence of a new record header that will encode n_fields-n_core_fields-1 in 1 or 2 bytes. (In ROW_FORMAT=REDUNDANT records, the record header always explicitly encodes the number of fields.) We introduce the undo log record type TRX_UNDO_INSERT_DEFAULT for covering the insert of the 'default row' record when instant ADD COLUMN is used for the first time. Subsequent instant ADD COLUMN can use TRX_UNDO_UPD_EXIST_REC. This is joint work with Vin Chen (陈福荣) from Tencent. The design that was discussed in April 2017 would not have allowed import or export of data files, because instead of the 'default row' it would have introduced a data dictionary table. The test rpl.rpl_alter_instant is exactly as contributed in pull request #408. The test innodb.instant_alter is based on a contributed test. The redo log record format changes for ROW_FORMAT=DYNAMIC and ROW_FORMAT=COMPACT are as contributed. (With this change present, crash recovery from MariaDB 10.3.1 will fail in spectacular ways!) Also the semantics of higher-level redo log records that modify the PAGE_INSTANT field is changed. The redo log format version identifier was already changed to LOG_HEADER_FORMAT_CURRENT=103 in MariaDB 10.3.1. Everything else has been rewritten by me. Thanks to Elena Stepanova, the code has been tested extensively. When rolling back an instant ADD COLUMN operation, we must empty the PAGE_FREE list after deleting or shortening the 'default row' record, by calling either btr_page_empty() or btr_page_reorganize(). We must know the size of each entry in the PAGE_FREE list. If rollback left a freed copy of the 'default row' in the PAGE_FREE list, we would be unable to determine its size (if it is in ROW_FORMAT=COMPACT or ROW_FORMAT=DYNAMIC) because it would contain more fields than the rolled-back definition of the clustered index. UNIV_SQL_DEFAULT: A new special constant that designates an instantly added column that is not present in the clustered index record. len_is_stored(): Check if a length is an actual length. There are two magic length values: UNIV_SQL_DEFAULT, UNIV_SQL_NULL. dict_col_t::def_val: The 'default row' value of the column. If the column is not added instantly, def_val.len will be UNIV_SQL_DEFAULT. dict_col_t: Add the accessors is_virtual(), is_nullable(), is_instant(), instant_value(). dict_col_t::remove_instant(): Remove the 'instant ADD' status of a column. dict_col_t::name(const dict_table_t& table): Replaces dict_table_get_col_name(). dict_index_t::n_core_fields: The original number of fields. For secondary indexes and if instant ADD COLUMN has not been used, this will be equal to dict_index_t::n_fields. dict_index_t::n_core_null_bytes: Number of bytes needed to represent the null flags; usually equal to UT_BITS_IN_BYTES(n_nullable). dict_index_t::NO_CORE_NULL_BYTES: Magic value signalling that n_core_null_bytes was not initialized yet from the clustered index root page. dict_index_t: Add the accessors is_instant(), is_clust(), get_n_nullable(), instant_field_value(). dict_index_t::instant_add_field(): Adjust clustered index metadata for instant ADD COLUMN. dict_index_t::remove_instant(): Remove the 'instant ADD' status of a clustered index when the table becomes empty, or the very first instant ADD COLUMN operation is rolled back. dict_table_t: Add the accessors is_instant(), is_temporary(), supports_instant(). dict_table_t::instant_add_column(): Adjust metadata for instant ADD COLUMN. dict_table_t::rollback_instant(): Adjust metadata on the rollback of instant ADD COLUMN. prepare_inplace_alter_table_dict(): First create the ctx->new_table, and only then decide if the table really needs to be rebuilt. We must split the creation of table or index metadata from the creation of the dictionary table records and the creation of the data. In this way, we can transform a table-rebuilding operation into an instant ADD COLUMN operation. Dictionary objects will only be added to cache when table rebuilding or index creation is needed. The ctx->instant_table will never be added to cache. dict_table_t::add_to_cache(): Modified and renamed from dict_table_add_to_cache(). Do not modify the table metadata. Let the callers invoke dict_table_add_system_columns() and if needed, set can_be_evicted. dict_create_sys_tables_tuple(), dict_create_table_step(): Omit the system columns (which will now exist in the dict_table_t object already at this point). dict_create_table_step(): Expect the callers to invoke dict_table_add_system_columns(). pars_create_table(): Before creating the table creation execution graph, invoke dict_table_add_system_columns(). row_create_table_for_mysql(): Expect all callers to invoke dict_table_add_system_columns(). create_index_dict(): Replaces row_merge_create_index_graph(). innodb_update_n_cols(): Renamed from innobase_update_n_virtual(). Call my_error() if an error occurs. btr_cur_instant_init(), btr_cur_instant_init_low(), btr_cur_instant_root_init(): Load additional metadata from the clustered index and set dict_index_t::n_core_null_bytes. This is invoked when table metadata is first loaded into the data dictionary. dict_boot(): Initialize n_core_null_bytes for the four hard-coded dictionary tables. dict_create_index_step(): Initialize n_core_null_bytes. This is executed as part of CREATE TABLE. dict_index_build_internal_clust(): Initialize n_core_null_bytes to NO_CORE_NULL_BYTES if table->supports_instant(). row_create_index_for_mysql(): Initialize n_core_null_bytes for CREATE TEMPORARY TABLE. commit_cache_norebuild(): Call the code to rename or enlarge columns in the cache only if instant ADD COLUMN is not being used. (Instant ADD COLUMN would copy all column metadata from instant_table to old_table, including the names and lengths.) PAGE_INSTANT: A new 13-bit field for storing dict_index_t::n_core_fields. This is repurposing the 16-bit field PAGE_DIRECTION, of which only the least significant 3 bits were used. The original byte containing PAGE_DIRECTION will be accessible via the new constant PAGE_DIRECTION_B. page_get_instant(), page_set_instant(): Accessors for the PAGE_INSTANT. page_ptr_get_direction(), page_get_direction(), page_ptr_set_direction(): Accessors for PAGE_DIRECTION. page_direction_reset(): Reset PAGE_DIRECTION, PAGE_N_DIRECTION. page_direction_increment(): Increment PAGE_N_DIRECTION and set PAGE_DIRECTION. rec_get_offsets(): Use the 'leaf' parameter for non-debug purposes, and assume that heap_no is always set. Initialize all dict_index_t::n_fields for ROW_FORMAT=REDUNDANT records, even if the record contains fewer fields. rec_offs_make_valid(): Add the parameter 'leaf'. rec_copy_prefix_to_dtuple(): Assert that the tuple is only built on the core fields. Instant ADD COLUMN only applies to the clustered index, and we should never build a search key that has more than the PRIMARY KEY and possibly DB_TRX_ID,DB_ROLL_PTR. All these columns are always present. dict_index_build_data_tuple(): Remove assertions that would be duplicated in rec_copy_prefix_to_dtuple(). rec_init_offsets(): Support ROW_FORMAT=REDUNDANT records whose number of fields is between n_core_fields and n_fields. cmp_rec_rec_with_match(): Implement the comparison between two MIN_REC_FLAG records. trx_t::in_rollback: Make the field available in non-debug builds. trx_start_for_ddl_low(): Remove dangerous error-tolerance. A dictionary transaction must be flagged as such before it has generated any undo log records. This is because trx_undo_assign_undo() will mark the transaction as a dictionary transaction in the undo log header right before the very first undo log record is being written. btr_index_rec_validate(): Account for instant ADD COLUMN row_undo_ins_remove_clust_rec(): On the rollback of an insert into SYS_COLUMNS, revert instant ADD COLUMN in the cache by removing the last column from the table and the clustered index. row_search_on_row_ref(), row_undo_mod_parse_undo_rec(), row_undo_mod(), trx_undo_update_rec_get_update(): Handle the 'default row' as a special case. dtuple_t::trim(index): Omit a redundant suffix of an index tuple right before insert or update. After instant ADD COLUMN, if the last fields of a clustered index tuple match the 'default row', there is no need to store them. While trimming the entry, we must hold a page latch, so that the table cannot be emptied and the 'default row' be deleted. btr_cur_optimistic_update(), btr_cur_pessimistic_update(), row_upd_clust_rec_by_insert(), row_ins_clust_index_entry_low(): Invoke dtuple_t::trim() if needed. row_ins_clust_index_entry(): Restore dtuple_t::n_fields after calling row_ins_clust_index_entry_low(). rec_get_converted_size(), rec_get_converted_size_comp(): Allow the number of fields to be between n_core_fields and n_fields. Do not support infimum,supremum. They are never supposed to be stored in dtuple_t, because page creation nowadays uses a lower-level method for initializing them. rec_convert_dtuple_to_rec_comp(): Assign the status bits based on the number of fields. btr_cur_trim(): In an update, trim the index entry as needed. For the 'default row', handle rollback specially. For user records, omit fields that match the 'default row'. btr_cur_optimistic_delete_func(), btr_cur_pessimistic_delete(): Skip locking and adaptive hash index for the 'default row'. row_log_table_apply_convert_mrec(): Replace 'default row' values if needed. In the temporary file that is applied by row_log_table_apply(), we must identify whether the records contain the extra header for instantly added columns. For now, we will allocate an additional byte for this for ROW_T_INSERT and ROW_T_UPDATE records when the source table has been subject to instant ADD COLUMN. The ROW_T_DELETE records are fine, as they will be converted and will only contain 'core' columns (PRIMARY KEY and some system columns) that are converted from dtuple_t. rec_get_converted_size_temp(), rec_init_offsets_temp(), rec_convert_dtuple_to_temp(): Add the parameter 'status'. REC_INFO_DEFAULT_ROW = REC_INFO_MIN_REC_FLAG \| REC_STATUS_COLUMNS_ADDED: An info_bits constant for distinguishing the 'default row' record. rec_comp_status_t: An enum of the status bit values. rec_leaf_format: An enum that replaces the bool parameter of rec_init_offsets_comp_ordinary().	8 years ago
Marko Mäkelä	48192f963a	Add the parameter bool leaf to rec_get_offsets() This should affect debug builds only. Debug builds will check that the status bits of ROW_FORMAT!=REDUNDANT records match the is_leaf parameter. The only observable change to non-debug should be the addition of the is_leaf parameter to the function rec_copy_prefix_to_dtuple(), and the removal of some calls to update the adaptive hash index (it is only built for the leaf pages). This change should have been made in MySQL 5.0.3, instead of introducing the status flags in the ROW_FORMAT=COMPACT record header.	8 years ago
Jan Lindström	34eef269eb	MDEV-11939: innochecksum mistakes a file for an encrypted one (page 0 invalid) Always read full page 0 to determine does tablespace contain encryption metadata. Tablespaces that are page compressed or page compressed and encrypted do not compare checksum as it does not exists. For encrypted tables use checksum verification written for encrypted tables and normal tables use normal method. buf_page_is_checksum_valid_crc32 buf_page_is_checksum_valid_innodb buf_page_is_checksum_valid_none Modify Innochecksum logging to file to avoid compilation warnings. fil0crypt.cc fil0crypt.h Modify to be able to use in innochecksum compilation and move fil_space_verify_crypt_checksum to end of the file. Add innochecksum logging to file. univ.i Add innochecksum strict_verify, log_file and cur_page_num variables as extern. page_zip_verify_checksum Add innochecksum logging to file and remove unnecessary code. innochecksum.cc Lot of changes most notable able to read encryption metadata from page 0 of the tablespace. Added test case where we corrupt intentionally FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION (encryption key version) FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION+4 (post encryption checksum) FIL_DATA+10 (data)	8 years ago
Jan Lindström	2ef7a5a13a	MDEV-13443: Port innochecksum tests from 10.2 innodb_zip suite to 10.1 This is basically port of WL6045:Improve Innochecksum with some code refactoring on innochecksum. Added page0size.h include from 10.2 to make 10.1 vrs 10.2 innochecksum as identical as possible. Added page 0 checksum checking and if that fails whole test fails.	8 years ago
Jan Lindström	8b019f87dd	MDEV-11939: innochecksum mistakes a file for an encrypted one (page 0 invalid) Always read full page 0 to determine does tablespace contain encryption metadata. Tablespaces that are page compressed or page compressed and encrypted do not compare checksum as it does not exists. For encrypted tables use checksum verification written for encrypted tables and normal tables use normal method. buf_page_is_checksum_valid_crc32 buf_page_is_checksum_valid_innodb buf_page_is_checksum_valid_none Add Innochecksum logging to file buf_page_is_corrupted Remove ib_logf and page_warn_strict_checksum calls in innochecksum compilation. Add innochecksum logging to file. fil0crypt.cc fil0crypt.h Modify to be able to use in innochecksum compilation and move fil_space_verify_crypt_checksum to end of the file. Add innochecksum logging to file. univ.i Add innochecksum strict_verify, log_file and cur_page_num variables as extern. page_zip_verify_checksum Add innochecksum logging to file. innochecksum.cc Lot of changes most notable able to read encryption metadata from page 0 of the tablespace. Added test case where we corrupt intentionally FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION (encryption key version) FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION+4 (post encryption checksum) FIL_DATA+10 (data)	8 years ago
Marko Mäkelä	3c09f148f3	MDEV-12288 Reset DB_TRX_ID when the history is removed, to speed up MVCC Let InnoDB purge reset DB_TRX_ID,DB_ROLL_PTR when the history is removed. [TODO: It appears that the resetting is not taking place as often as it could be. We should test that a simple INSERT should eventually cause row_purge_reset_trx_id() to be invoked unless DROP TABLE is invoked soon enough.] The InnoDB clustered index record system columns DB_TRX_ID,DB_ROLL_PTR are used by multi-versioning. After the history is no longer needed, these columns can safely be reset to 0 and 1<<55 (to indicate a fresh insert). When a reader sees 0 in the DB_TRX_ID column, it can instantly determine that the record is present the read view. There is no need to acquire the transaction system mutex to check if the transaction exists, because writes can never be conducted by a transaction whose ID is 0. The persistent InnoDB undo log used to be split into two parts: insert_undo and update_undo. The insert_undo log was discarded at transaction commit or rollback, and the update_undo log was processed by the purge subsystem. As part of this change, we will only generate a single undo log for new transactions, and the purge subsystem will reset the DB_TRX_ID whenever a clustered index record is touched. That is, all persistent undo log will be preserved at transaction commit or rollback, to be removed by purge. The InnoDB redo log format is changed in two ways: We remove the redo log record type MLOG_UNDO_HDR_REUSE, and we introduce the MLOG_ZIP_WRITE_TRX_ID record for updating the DB_TRX_ID,DB_ROLL_PTR in a ROW_FORMAT=COMPRESSED table. This is also changing the format of persistent InnoDB data files: undo log and clustered index leaf page records. It will still be possible via import and export to exchange data files with earlier versions of MariaDB. The change to clustered index leaf page records is simple: we allow DB_TRX_ID to be 0. When it comes to the undo log, we must be able to upgrade from earlier MariaDB versions after a clean shutdown (no redo log to apply). While it would be nice to perform a slow shutdown (innodb_fast_shutdown=0) before an upgrade, to empty the undo logs, we cannot assume that this has been done. So, separate insert_undo log may exist for recovered uncommitted transactions. These transactions may be automatically rolled back, or they may be in XA PREPARE state, in which case InnoDB will preserve the transaction until an explicit XA COMMIT or XA ROLLBACK. Upgrade has been tested by starting up MariaDB 10.2 with ./mysql-test-run --manual-gdb innodb.read_only_recovery and then starting up this patched server with and without --innodb-read-only. trx_undo_ptr_t::undo: Renamed from update_undo. trx_undo_ptr_t::old_insert: Renamed from insert_undo. trx_rseg_t::undo_list: Renamed from update_undo_list. trx_rseg_t::undo_cached: Merged from update_undo_cached and insert_undo_cached. trx_rseg_t::old_insert_list: Renamed from insert_undo_list. row_purge_reset_trx_id(): New function to reset the columns. This will be called for all undo processing in purge that does not remove the clustered index record. trx_undo_update_rec_get_update(): Allow trx_id=0 when copying the old DB_TRX_ID of the record to the undo log. ReadView::changes_visible(): Allow id==0. (Return true for it. This is what speeds up the MVCC.) row_vers_impl_x_locked_low(), row_vers_build_for_semi_consistent_read(): Implement a fast path for DB_TRX_ID=0. Always initialize the TRX_UNDO_PAGE_TYPE to 0. Remove undo->type. MLOG_UNDO_HDR_REUSE: Remove. This changes the redo log format! innobase_start_or_create_for_mysql(): Set srv_undo_sources before starting any transactions. The parsing of the MLOG_ZIP_WRITE_TRX_ID record was successfully tested by running the following: ./mtr --parallel=auto --mysqld=--debug=d,ib_log innodb_zip.bug56680 grep MLOG_ZIP_WRITE_TRX_ID var/*/log/mysqld.1.err	8 years ago

1 2 3

128 Commits (700e010309ea9d0f7c32d0973ec88c3140c67956)