mariadb

Commit Graph

Author	SHA1	Message	Date
Marko Mäkelä	312569e2fd	MDEV-21132 Remove buf_page_t::newest_modification At each mini-transaction commit, the log sequence number of the mini-transaction must be written to each modified page, so that it will be available in the FIL_PAGE_LSN field when the page is being read in crash recovery. InnoDB was unnecessarily allocating redundant storage for the field, in buf_page_t::newest_modification. Let us access FIL_PAGE_LSN directly. Furthermore, on ALTER TABLE...IMPORT TABLESPACE, let us write 0 to FIL_PAGE_LSN instead of using log_sys.lsn. buf_flush_init_for_writing(), buf_flush_update_zip_checksum(), fil_encrypt_buf_for_full_crc32(), fil_encrypt_buf(), fil_space_encrypt(): Remove the parameter lsn. buf_page_get_newest_modification(): Merge with the only caller. buf_tmp_reserve_compression_buf(), buf_tmp_page_encrypt(), buf_page_encrypt(): Define static in the same compilation unit with the only caller. PageConverter::m_current_lsn: Remove. Write 0 to FIL_PAGE_LSN on ALTER TABLE...IMPORT TABLESPACE.	6 years ago
Marko Mäkelä	5d596064d6	Cleanup: Define FIL_PAGE_TYPE constants as constexpr	6 years ago
Marko Mäkelä	d09aec7a15	MDEV-19940 Clean up INFORMATION_SCHEMA.INNODB_ tables Shorten some VARCHAR attributes to a more reasonable length. INNODB_METRICS: Rename the column STATUS to ENABLED, and make it Boolean. Replace with INT(1) many Boolean attributes that were declared as VARCHAR containing 'NO','YES','disabled','enabled','Uninitialized','Initialized'. Replace some VARCHAR attributes with ENUM. Replace some BIGINT with INT when 32 bits are sufficient. Remove INNODB_SYS_TABLESPACES.SPACE_TYPE. The type of a tablespace can be derived from the tablespace ID. A fixed number is used for the system tablespace and the temporary tablespace. All other tablespaces are single-table or single-partition tablespaces. i_s_locks_row_t::lock_type, lock_get_type_str(): Remove. This is a redundant field. Table and record locks can be distinguished by whether i_s_locks_row_t::lock_index is NULL. fill_trx_row(): Do not unnecessarily copy the constant strings that trx->op_info is pointing to. i_s_locks_row_t::lock_mode: Replace string with integer. lock_get_mode_str(), lock_get_trx_id(), lock_get_trx(): Remove. field_store_ulint(): Remove.	6 years ago
Eugene Kosov	52a5097764	MDEV-17441 InnoDB transition to C++11 atomics dict_table_t::stats_latch_created: remove along with related stuff dict_table_t::stats_latch: make value member, not pointer. And always lock this for simplicity, even to stats cloned table. based on the work of Sergey Vojtovich	6 years ago
Alexander Barkov	6f3612fa4d	MDEV-19861 Add intfastructure to have ENUM columns in INFORMATION_SCHEMA	6 years ago
Alexander Barkov	9561b0b47e	MDEV-19810 Reuse new I_S table definition helper classes for InnoDB	6 years ago
Marko Mäkelä	c874040729	Declare INFORMATION_SCHEMA.INNODB_SYS_VIRTUAL stable The INFORMATION_SCHEMA plugin INNODB_SYS_VIRTUAL, which was introduced in MariaDB 10.2.2 along with the dictionary table SYS_VIRTUAL, is similar to other, much older and already stable plugins that provide access to InnoDB dictionary tables.	6 years ago
Marko Mäkelä	198ed24cac	MDEV-19513: Rename dict_operation_lock to dict_sys.latch dict_sys.lock(), dict_sys_lock(): Acquire both mutex and latch. dict_sys.unlock(), dict_sys_unlock(): Release both mutex and latch. dict_sys.assert_locked(): Assert that both mutex and latch are held.	6 years ago
Marko Mäkelä	5fd7502e77	MDEV-19513: Allocate dict_sys statically dict_sys_t::create(): Renamed from dict_init(). dict_sys_t::close(): Renamed from dict_close(). dict_sys_t::add(): Sliced from dict_table_t::add_to_cache(). dict_sys_t::remove(): Renamed from dict_table_remove_from_cache(). dict_sys_t::prevent_eviction(): Renamed from dict_table_move_from_lru_to_non_lru(). dict_sys_t::acquire(): Replaces dict_move_to_mru() and some more logic. dict_sys_t::resize(): Renamed from dict_resize(). dict_sys_t::find(): Replaces dict_lru_find_table() and dict_non_lru_find_table().	6 years ago
Marko Mäkelä	4937339705	MDEV-19445: After-merge fix Pass a correct string pointer to RETURN_IF_INNODB_NOT_STARTED. This was caught on Windows. The error was introduced in the merge commit `874f8f30f2`.	6 years ago
Marko Mäkelä	b93ecea65c	Remove unnecessary pointer indirection for rw_lock_t In MySQL 5.7.8 an extra level of pointer indirection was added to dict_operation_lock and some other rw_lock_t without solid justification, in mysql/mysql-server@52720f1772f9f424bf3dd62fa9c214dd608cd036. Let us revert that change and remove the rather useless rw_lock_t constructor and destructor and the magic_n field. In this way, some unnecessary pointer dereferences and heap allocation will be avoided and debugging might be a little easier.	6 years ago
Marko Mäkelä	2647fd101d	MDEV-19445 heap-use-after-free related to innodb_ft_aux_table Try to fix the race conditions between SET GLOBAL innodb_ft_aux_table = ...; and access to the INFORMATION_SCHEMA tables that depend on this variable. innodb_ft_aux_table: Replaces fts_internal_tbl_name,fts_internal_tbl_name2. Just store the user-specified parameter as is. innodb_ft_aux_table_id: The table_id corresponding to SET GLOBAL innodb_ft_aux_table, or 0 if the table does not exist or does not contain FULLTEXT INDEX. If the table is renamed later, the INFORMATION_SCHEMA tables will continue to refer to the table. If the table is dropped or rebuilt, the INFORMATION_SCHEMA tables will not find the table.	6 years ago
Vicențiu Ciorbaru	5543b75550	Update FSF Address * Update wrong zip-code	7 years ago
Vicențiu Ciorbaru	c0ac0b8860	Update FSF address	7 years ago
Marko Mäkelä	f3718a112a	MDEV-18220: Backport some code from MariaDB 10.2 fts_get_table_name(): Output to a caller-allocated buffer. fts_get_table_name_prefix(): Use the lower-overhead allocation ut_malloc() instead of mem_alloc(). This is based on mysql/mysql-server@d1584b9f38ff0bcf609d181db35f74108e022168 in MySQL 5.7.4.	7 years ago
Marko Mäkelä	10dd290b4b	MDEV-17380 innodb_flush_neighbors=ON should be ignored on SSD For tablespaces that do not reside on spinning storage, it does not make sense to attempt to write nearby pages when writing out dirty pages from the InnoDB buffer pool. It is actually detrimental to performance and to the life span of flash ROM storage. With this change, MariaDB will detect whether an InnoDB file resides on solid-state storage. The detection has been implemented for Linux and Microsoft Windows. For other systems, we will err on the safe side and assume that files reside on SSD. As part of this change, we will reduce the number of fstat() calls when opening data files on POSIX systems and slightly clean up some file I/O code. FIXME: os_is_sparse_file_supported() on POSIX works in a destructive manner. Thus, we can only invoke it when creating files, not when opening them. For diagnostics, we introduce the column ON_SSD to the table INFORMATION_SCHEMA.INNODB_TABLESPACES_SCRUBBING. The table INNODB_SYS_TABLESPACES might seem more appropriate, but its purpose is to reflect the contents of the InnoDB system table SYS_TABLESPACES, which we would like to remove at some point. On Microsoft Windows, querying StorageDeviceSeekPenaltyProperty sometimes returns ERROR_GEN_FAILURE instead of ERROR_INVALID_FUNCTION or ERROR_NOT_SUPPORTED. We will silently ignore also this error, and assume that the file does not reside on SSD. On Linux, the detection will be based on the files /sys/block//queue/rotational and /sys/block//dev. Especially for USB storage, it is possible that /sys/block//queue/rotational will wrongly report 1 instead of 0. fil_node_t::on_ssd: Whether the InnoDB data file resides on solid-state storage. fil_system_t::ssd: Collection of Linux block devices that reside on non-rotational storage. fil_system_t::create(): Detect ssd on Linux based on the contents of /sys/block//queue/rotational and /sys/block/*/dev. fil_system_t::is_ssd(dev_t): Determine if a Linux block device is non-rotational. Partitions will be identified with the containing block device by assuming that the least significant 4 bits of the minor number identify a partition, and that the "partition number" of the entire device is 0.	7 years ago
Marko Mäkelä	23eeecd68f	MDEV-19111 Unused field INFORMATION_SCHEMA.INNODB_TABLESPACES_SCRUBBING.ROTATING_OR_FLUSHING The MDEV-11738/MDEV-11581 fix was supposed to add the column ROTATING_OR_FLUSHING to the INFORMATION_SCHEMA table INNODB_TABLESPACES_ENCRYPTION, but it also added that column to INNODB_TABLESPACES_SCRUBBING in InnoDB (not XtraDB). The extra column was never initialized. We will remove it, because key rotation has nothing to do with the scrubbing of tablespace data.	7 years ago
Thirunarayanan Balathandayuthapani	c0f47a4a58	MDEV-12026: Implement innodb_checksum_algorithm=full_crc32 MariaDB data-at-rest encryption (innodb_encrypt_tables) had repurposed the same unused data field that was repurposed in MySQL 5.7 (and MariaDB 10.2) for the Split Sequence Number (SSN) field of SPATIAL INDEX. Because of this, MariaDB was unable to support encryption on SPATIAL INDEX pages. Furthermore, InnoDB page checksums skipped some bytes, and there are multiple variations and checksum algorithms. By default, InnoDB accepts all variations of all algorithms that ever existed. This unnecessarily weakens the page checksums. We hereby introduce two more innodb_checksum_algorithm variants (full_crc32, strict_full_crc32) that are special in a way: When either setting is active, newly created data files will carry a flag (fil_space_t::full_crc32()) that indicates that all pages of the file will use a full CRC-32C checksum over the entire page contents (excluding the bytes where the checksum is stored, at the very end of the page). Such files will always use that checksum, no matter what the parameter innodb_checksum_algorithm is assigned to. For old files, the old checksum algorithms will continue to be used. The value strict_full_crc32 will be equivalent to strict_crc32 and the value full_crc32 will be equivalent to crc32. ROW_FORMAT=COMPRESSED tables will only use the old format. These tables do not support new features, such as larger innodb_page_size or instant ADD/DROP COLUMN. They may be deprecated in the future. We do not want an unnecessary file format change for them. The new full_crc32() format also cleans up the MariaDB tablespace flags. We will reserve flags to store the page_compressed compression algorithm, and to store the compressed payload length, so that checksum can be computed over the compressed (and possibly encrypted) stream and can be validated without decrypting or decompressing the page. In the full_crc32 format, there no longer are separate before-encryption and after-encryption checksums for pages. The single checksum is computed on the page contents that is written to the file. We do not make the new algorithm the default for two reasons. First, MariaDB 10.4.2 was a beta release, and the default values of parameters should not change after beta. Second, we did not yet implement the full_crc32 format for page_compressed pages. This will be fixed in MDEV-18644. This is joint work with Marko Mäkelä.	7 years ago
Marko Mäkelä	0a1c3477bf	MDEV-18493 Remove page_size_t MySQL 5.7 introduced the class page_size_t and increased the size of buffer pool page descriptors by introducing this object to them. Maybe the intention of this exercise was to prepare for a future where the buffer pool could accommodate multiple page sizes. But that future never arrived, not even in MySQL 8.0. It is much easier to manage a pool of a single page size, and typically all storage devices of an InnoDB instance benefit from using the same page size. Let us remove page_size_t from MariaDB Server. This will make it easier to remove support for ROW_FORMAT=COMPRESSED (or make it a compile-time option) in the future, just by removing various occurrences of zip_size.	7 years ago
Marko Mäkelä	5eb3e4d83c	MDEV-15798 Mutex leak on accessing INFORMATION_SCHEMA.INNODB_MUTEXES i_s_innodb_mutexes_fill_table(): Use the C++ RAII pattern to ensure that the mutexes are released if an OK() macro returns from the function prematurely.	7 years ago
Marko Mäkelä	ff88e4bb8a	Remove many redundant #include from InnoDB	7 years ago
Marko Mäkelä	d17e9a02c4	Fix InnoDB/XtraDB warnings by GCC 8.2.0	7 years ago
Marko Mäkelä	a61724a3ca	MDEV-16267 Wrong INFORMATION_SCHEMA.INNODB_BUFFER_PAGE.TABLE_NAME i_s_innodb_buffer_page_fill(), i_s_innodb_buf_page_lru_fill(): Only invoke Field::set_notnull() if the index was found.	7 years ago
Marko Mäkelä	52df804026	MDEV-16267 Wrong INFORMATION_SCHEMA.INNODB_BUFFER_PAGE.TABLE_NAME This is the MariaDB 10.2 version of the patch. field_store_string(): Simplify the code. field_store_index_name(): Remove, and use field_store_string() instead. Starting with MariaDB 10.2.2, there is the predicate dict_index_t::is_committed(), and dict_index_t::name never contains the magic byte 0xff. Correct some comments to refer to TEMP_INDEX_PREFIX_STR. i_s_cmp_per_index_fill_low(): Use the appropriate value NULL to identify that an index was not found. Check that storing each column value succeeded. i_s_innodb_buffer_page_fill(), i_s_innodb_buf_page_lru_fill(): Only invoke Field::set_notnull() if the index was found. (This fixes the bug.) i_s_dict_fill_sys_indexes(): Adjust the index->name that was directly loaded from SYS_INDEXES.NAME (which can start with the 0xff byte). This was the only function that depended on the translation in field_store_index_name().	7 years ago
Marko Mäkelä	93b69825ad	MDEV-16169 InnoDB: Failing assertion: !space->referenced() Before invoking fil_space_t::acquire(), check space->is_stopping() to ensure that the tablespace actually is accessible. This fixes a regression introduced by MDEV-15983. fil_space_next(): Remove some duplicated code for prev_space==NULL, and check is_stopping() also on the first tablespace. i_s_tablespaces_encryption_fill_table(), i_s_tablespaces_scrubbing_fill_table(): Check is_stopping().	8 years ago
Marko Mäkelä	c57e9835ff	Replace dict_col_is_virtual(col) with col->is_virtual()	8 years ago
Marko Mäkelä	2b27ac8282	Fix many -Wunused-parameter Remove unused InnoDB function parameters and functions. i_s_sys_virtual_fill_table(): Do not allocate heap memory. mtr_is_block_fix(): Replace with mtr_memo_contains(). mtr_is_page_fix(): Replace with mtr_memo_contains_page().	8 years ago
Marko Mäkelä	9801715cb0	Use compile_time_assert() in InnoDB Replace most use of #error. Some checks were impossible to evaluate in the preprocessor due to the use of named integer constants or enumerations.	8 years ago
Marko Mäkelä	ba19764209	Fix most -Wsign-conversion in InnoDB Change innodb_buffer_pool_size, innodb_fill_factor to unsigned.	8 years ago
Marko Mäkelä	de942c9f61	MDEV-15983 Reduce fil_system.mutex contention further fil_space_t::n_pending_ops, n_pending_ios: Use a combination of fil_system.mutex and atomic memory access for protection. fil_space_t::release(): Replaces fil_space_release(). Does not acquire fil_system.mutex. fil_space_t::release_for_io(): Replaces fil_space_release_for_io(). Does not acquire fil_system.mutex.	8 years ago
Sergey Vojtovich	3d5f7ad23a	MDEV-15773 - Simplified away trx_free_for_(mysql\|background)	8 years ago
Sergey Vojtovich	d6d58836bb	MDEV-15773 - trx_allocate_for_background() -> trx_create() trx_free_resurrected(): Remove, unused function	8 years ago
Marko Mäkelä	4cad42392a	MDEV-12266: Change dict_table_t::space to fil_space_t* InnoDB always keeps all tablespaces in the fil_system cache. The fil_system.LRU is only for closing file handles; the fil_space_t and fil_node_t for all data files will remain in main memory. Between startup to shutdown, they can only be created and removed by DDL statements. Therefore, we can let dict_table_t::space point directly to the fil_space_t. dict_table_t::space_id: A numeric tablespace ID for the corner cases where we do not have a tablespace. The most prominent examples are ALTER TABLE...DISCARD TABLESPACE or a missing or corrupted file. There are a few functional differences; most notably: (1) DROP TABLE will delete matching .ibd and .cfg files, even if they were not attached to the data dictionary. (2) Some error messages will report file names instead of numeric IDs. There still are many functions that use numeric tablespace IDs instead of fil_space_t, and many functions could be converted to fil_space_t member functions. Also, Tablespace and Datafile should be merged with fil_space_t and fil_node_t. page_id_t and buf_page_get_gen() could use fil_space_t& instead of a numeric ID, and after moving to a single buffer pool (MDEV-15058), buf_pool_t::page_hash could be moved to fil_space_t::page_hash. FilSpace: Remove. Only few calls to fil_space_acquire() will remain, and gradually they should be removed. mtr_t::set_named_space_id(ulint): Renamed from set_named_space(), to prevent accidental calls to this slower function. Very few callers remain. fseg_create(), fsp_reserve_free_extents(): Take fil_space_t as a parameter instead of a space_id. fil_space_t::rename(): Wrapper for fil_rename_tablespace_check(), fil_name_write_rename(), fil_rename_tablespace(). Mariabackup passes the parameter log=false; InnoDB passes log=true. dict_mem_table_create(): Take fil_space_t* instead of space_id as parameter. dict_process_sys_tables_rec_and_mtr_commit(): Replace the parameter 'status' with 'bool cached'. dict_get_and_save_data_dir_path(): Avoid copying the fil_node_t::name. fil_ibd_open(): Return the tablespace. fil_space_t::set_imported(): Replaces fil_space_set_imported(). truncate_t: Change many member function parameters to fil_space_t*, and remove page_size parameters. row_truncate_prepare(): Merge to its only caller. row_drop_table_from_cache(): Assert that the table is persistent. dict_create_sys_indexes_tuple(): Write SYS_INDEXES.SPACE=FIL_NULL if the tablespace has been discarded. row_import_update_discarded_flag(): Remove a constant parameter.	8 years ago
Marko Mäkelä	428e02895b	MDEV-12266: Remove dict_index_t::table_name Replace index->table_name with index->table->name.	8 years ago
Marko Mäkelä	604fea1ad6	MDEV-12266: Remove dict_index_t::space We can rely on the dict_table_t::space. All indexes of a table object are always in the same tablespace. (For fulltext indexes, the data is located in auxiliary tables, and these will continue to have their own table objects, separate from the main table.)	8 years ago
Marko Mäkelä	600c85e85a	Allocate the singleton fil_system statically fil_system_t::create(): Replaces fil_init(), fsp_init(). fil_system_t::close(): Replaces fil_close(). fil_system_t::max_n_open: Remove. Use srv_max_n_open_files directly.	8 years ago
Monty	e2664ee836	MDEV-14275 Improving memory utilization for information schema SELECT queries on information_schema table will now not store data in the temporary table for columns that are not used in the query. This can drastically reduce memory when there are many rows involved in a query, like using SELECT table_name FROM information_schema.table on a system with many tables. The difference for the above query is about 14K per existing table. The code works this way: - Create a bitmap for all information_schema fields used in the query - For BLOB and VARCHAR fields that are not use, create a Field_null field.	8 years ago
Marko Mäkelä	2fb31821de	MDEV-11984 Avoid accessing SYS_TABLESPACES unnecessarily The following INFORMATION_SCHEMA views were unnecessarily retrieving the data from the SYS_TABLESPACES table instead of directly fetching it from the fil_system cache: information_schema.innodb_tablespaces_encryption information_schema.innodb_tablespaces_scrubbing InnoDB always loads all tablespace metadata into memory at startup and never evicts it while the tablespace exists. With this fix, accessing these views will be much faster and use less memory, and include data about all tablespaces, including undo tablespaces. The view information_schema.innodb_sys_tablespaces will still reflect the contents of the SYS_TABLESPACES table.	8 years ago
Monty	a7e352b54d	Changed database, tablename and alias to be LEX_CSTRING This was done in, among other things: - thd->db and thd->db_length - TABLE_LIST tablename, db, alias and schema_name - Audit plugin database name - lex->db - All db and table names in Alter_table_ctx - st_select_lex db Other things: - Changed a lot of functions to take const LEX_CSTRING* as argument for db, table_name and alias. See init_one_table() as an example. - Changed some function arguments from LEX_CSTRING to const LEX_CSTRING - Changed some lists from LEX_STRING to LEX_CSTRING - threads_mysql.result changed because process list_db wasn't always correctly updated - New append_identifier() function that takes LEX_CSTRING* as arguments - Added new element tmp_buff to Alter_table_ctx to separate temp name handling from temporary space - Ensure we store the length after my_casedn_str() of table/db names - Removed not used version of rename_table_in_stat_tables() - Changed Natural_join_column::table_name and db_name() to never return NULL (used for print) - thd->get_db() now returns db as a printable string (thd->db.str or "")	8 years ago
Eugene Kosov	22f2b39c14	fix data races in rwlock	8 years ago
Eugene Kosov	90c809a61d	IB: VTQ cleanups [#305 ] * btr_pcur_move_to_prev_user_rec() removed * usage of ut_usectime() removed * other VTQ-related leftovers	8 years ago
Aleksey Midenkov	33085349e9	IB, SQL: removed VTQ, added TRT on SQL layer [closes #305 ]	8 years ago
Eugene Kosov	4a662df945	IB: do not use long and ullong types [#306 ]	8 years ago
Marko Mäkelä	c19ef508b8	InnoDB: Remove ut_snprintf() and the use of my_snprintf(); use snprintf()	8 years ago
Marko Mäkelä	a9faad5f78	Remove an unused variable	8 years ago
Marko Mäkelä	24bffb080b	MDEV-13918 Race condition between INFORMATION_SCHEMA.INNODB_SYS_TABLESTATS and ALTER/DROP/TRUNCATE TABLE MDEV-14021 Failing assertion: table->get_ref_count() == 0 on ALTER TABLE with implicit ALGORITHM=INPLACE Post-merge fix: Remove the now-useless use of table reference count. The dict_operation_lock S-latch whose use was introduced in the 10.0 fix will prevent the table from disappearing due to concurrent DDL operations (DROP, table-rebuilding ALTER, TRUNCATE).	8 years ago
Marko Mäkelä	2eb3c5e542	MDEV-13918 Race condition between INFORMATION_SCHEMA.INNODB_SYS_TABLESTATS and ALTER/DROP/TRUNCATE TABLE i_s_sys_tables_fill_table_stats(): Acquire dict_operation_lock S-latch before acquiring dict_sys->mutex, to prevent the table from being removed from the data dictionary cache and from being freed while i_s_dict_fill_sys_tablestats() is accessing the table handle.	8 years ago
Marko Mäkelä	d11af09865	MDEV-14076 InnoDB: Failing assertion when accessing INFORMATION_SCHEMA.INNODB_SYS_TABLESPACES upon upgrade from 10.1.0 to 10.1.20 i_s_dict_fill_sys_tablespaces(): Adjust the tablespace flags if needed.	8 years ago
Marko Mäkelä	a4948dafcd	MDEV-11369 Instant ADD COLUMN for InnoDB For InnoDB tables, adding, dropping and reordering columns has required a rebuild of the table and all its indexes. Since MySQL 5.6 (and MariaDB 10.0) this has been supported online (LOCK=NONE), allowing concurrent modification of the tables. This work revises the InnoDB ROW_FORMAT=REDUNDANT, ROW_FORMAT=COMPACT and ROW_FORMAT=DYNAMIC so that columns can be appended instantaneously, with only minor changes performed to the table structure. The counter innodb_instant_alter_column in INFORMATION_SCHEMA.GLOBAL_STATUS is incremented whenever a table rebuild operation is converted into an instant ADD COLUMN operation. ROW_FORMAT=COMPRESSED tables will not support instant ADD COLUMN. Some usability limitations will be addressed in subsequent work: MDEV-13134 Introduce ALTER TABLE attributes ALGORITHM=NOCOPY and ALGORITHM=INSTANT MDEV-14016 Allow instant ADD COLUMN, ADD INDEX, LOCK=NONE The format of the clustered index (PRIMARY KEY) is changed as follows: (1) The FIL_PAGE_TYPE of the root page will be FIL_PAGE_TYPE_INSTANT, and a new field PAGE_INSTANT will contain the original number of fields in the clustered index ('core' fields). If instant ADD COLUMN has not been used or the table becomes empty, or the very first instant ADD COLUMN operation is rolled back, the fields PAGE_INSTANT and FIL_PAGE_TYPE will be reset to 0 and FIL_PAGE_INDEX. (2) A special 'default row' record is inserted into the leftmost leaf, between the page infimum and the first user record. This record is distinguished by the REC_INFO_MIN_REC_FLAG, and it is otherwise in the same format as records that contain values for the instantly added columns. This 'default row' always has the same number of fields as the clustered index according to the table definition. The values of 'core' fields are to be ignored. For other fields, the 'default row' will contain the default values as they were during the ALTER TABLE statement. (If the column default values are changed later, those values will only be stored in the .frm file. The 'default row' will contain the original evaluated values, which must be the same for every row.) The 'default row' must be completely hidden from higher-level access routines. Assertions have been added to ensure that no 'default row' is ever present in the adaptive hash index or in locked records. The 'default row' is never delete-marked. (3) In clustered index leaf page records, the number of fields must reside between the number of 'core' fields (dict_index_t::n_core_fields introduced in this work) and dict_index_t::n_fields. If the number of fields is less than dict_index_t::n_fields, the missing fields are replaced with the column value of the 'default row'. Note: The number of fields in the record may shrink if some of the last instantly added columns are updated to the value that is in the 'default row'. The function btr_cur_trim() implements this 'compression' on update and rollback; dtuple::trim() implements it on insert. (4) In ROW_FORMAT=COMPACT and ROW_FORMAT=DYNAMIC records, the new status value REC_STATUS_COLUMNS_ADDED will indicate the presence of a new record header that will encode n_fields-n_core_fields-1 in 1 or 2 bytes. (In ROW_FORMAT=REDUNDANT records, the record header always explicitly encodes the number of fields.) We introduce the undo log record type TRX_UNDO_INSERT_DEFAULT for covering the insert of the 'default row' record when instant ADD COLUMN is used for the first time. Subsequent instant ADD COLUMN can use TRX_UNDO_UPD_EXIST_REC. This is joint work with Vin Chen (陈福荣) from Tencent. The design that was discussed in April 2017 would not have allowed import or export of data files, because instead of the 'default row' it would have introduced a data dictionary table. The test rpl.rpl_alter_instant is exactly as contributed in pull request #408. The test innodb.instant_alter is based on a contributed test. The redo log record format changes for ROW_FORMAT=DYNAMIC and ROW_FORMAT=COMPACT are as contributed. (With this change present, crash recovery from MariaDB 10.3.1 will fail in spectacular ways!) Also the semantics of higher-level redo log records that modify the PAGE_INSTANT field is changed. The redo log format version identifier was already changed to LOG_HEADER_FORMAT_CURRENT=103 in MariaDB 10.3.1. Everything else has been rewritten by me. Thanks to Elena Stepanova, the code has been tested extensively. When rolling back an instant ADD COLUMN operation, we must empty the PAGE_FREE list after deleting or shortening the 'default row' record, by calling either btr_page_empty() or btr_page_reorganize(). We must know the size of each entry in the PAGE_FREE list. If rollback left a freed copy of the 'default row' in the PAGE_FREE list, we would be unable to determine its size (if it is in ROW_FORMAT=COMPACT or ROW_FORMAT=DYNAMIC) because it would contain more fields than the rolled-back definition of the clustered index. UNIV_SQL_DEFAULT: A new special constant that designates an instantly added column that is not present in the clustered index record. len_is_stored(): Check if a length is an actual length. There are two magic length values: UNIV_SQL_DEFAULT, UNIV_SQL_NULL. dict_col_t::def_val: The 'default row' value of the column. If the column is not added instantly, def_val.len will be UNIV_SQL_DEFAULT. dict_col_t: Add the accessors is_virtual(), is_nullable(), is_instant(), instant_value(). dict_col_t::remove_instant(): Remove the 'instant ADD' status of a column. dict_col_t::name(const dict_table_t& table): Replaces dict_table_get_col_name(). dict_index_t::n_core_fields: The original number of fields. For secondary indexes and if instant ADD COLUMN has not been used, this will be equal to dict_index_t::n_fields. dict_index_t::n_core_null_bytes: Number of bytes needed to represent the null flags; usually equal to UT_BITS_IN_BYTES(n_nullable). dict_index_t::NO_CORE_NULL_BYTES: Magic value signalling that n_core_null_bytes was not initialized yet from the clustered index root page. dict_index_t: Add the accessors is_instant(), is_clust(), get_n_nullable(), instant_field_value(). dict_index_t::instant_add_field(): Adjust clustered index metadata for instant ADD COLUMN. dict_index_t::remove_instant(): Remove the 'instant ADD' status of a clustered index when the table becomes empty, or the very first instant ADD COLUMN operation is rolled back. dict_table_t: Add the accessors is_instant(), is_temporary(), supports_instant(). dict_table_t::instant_add_column(): Adjust metadata for instant ADD COLUMN. dict_table_t::rollback_instant(): Adjust metadata on the rollback of instant ADD COLUMN. prepare_inplace_alter_table_dict(): First create the ctx->new_table, and only then decide if the table really needs to be rebuilt. We must split the creation of table or index metadata from the creation of the dictionary table records and the creation of the data. In this way, we can transform a table-rebuilding operation into an instant ADD COLUMN operation. Dictionary objects will only be added to cache when table rebuilding or index creation is needed. The ctx->instant_table will never be added to cache. dict_table_t::add_to_cache(): Modified and renamed from dict_table_add_to_cache(). Do not modify the table metadata. Let the callers invoke dict_table_add_system_columns() and if needed, set can_be_evicted. dict_create_sys_tables_tuple(), dict_create_table_step(): Omit the system columns (which will now exist in the dict_table_t object already at this point). dict_create_table_step(): Expect the callers to invoke dict_table_add_system_columns(). pars_create_table(): Before creating the table creation execution graph, invoke dict_table_add_system_columns(). row_create_table_for_mysql(): Expect all callers to invoke dict_table_add_system_columns(). create_index_dict(): Replaces row_merge_create_index_graph(). innodb_update_n_cols(): Renamed from innobase_update_n_virtual(). Call my_error() if an error occurs. btr_cur_instant_init(), btr_cur_instant_init_low(), btr_cur_instant_root_init(): Load additional metadata from the clustered index and set dict_index_t::n_core_null_bytes. This is invoked when table metadata is first loaded into the data dictionary. dict_boot(): Initialize n_core_null_bytes for the four hard-coded dictionary tables. dict_create_index_step(): Initialize n_core_null_bytes. This is executed as part of CREATE TABLE. dict_index_build_internal_clust(): Initialize n_core_null_bytes to NO_CORE_NULL_BYTES if table->supports_instant(). row_create_index_for_mysql(): Initialize n_core_null_bytes for CREATE TEMPORARY TABLE. commit_cache_norebuild(): Call the code to rename or enlarge columns in the cache only if instant ADD COLUMN is not being used. (Instant ADD COLUMN would copy all column metadata from instant_table to old_table, including the names and lengths.) PAGE_INSTANT: A new 13-bit field for storing dict_index_t::n_core_fields. This is repurposing the 16-bit field PAGE_DIRECTION, of which only the least significant 3 bits were used. The original byte containing PAGE_DIRECTION will be accessible via the new constant PAGE_DIRECTION_B. page_get_instant(), page_set_instant(): Accessors for the PAGE_INSTANT. page_ptr_get_direction(), page_get_direction(), page_ptr_set_direction(): Accessors for PAGE_DIRECTION. page_direction_reset(): Reset PAGE_DIRECTION, PAGE_N_DIRECTION. page_direction_increment(): Increment PAGE_N_DIRECTION and set PAGE_DIRECTION. rec_get_offsets(): Use the 'leaf' parameter for non-debug purposes, and assume that heap_no is always set. Initialize all dict_index_t::n_fields for ROW_FORMAT=REDUNDANT records, even if the record contains fewer fields. rec_offs_make_valid(): Add the parameter 'leaf'. rec_copy_prefix_to_dtuple(): Assert that the tuple is only built on the core fields. Instant ADD COLUMN only applies to the clustered index, and we should never build a search key that has more than the PRIMARY KEY and possibly DB_TRX_ID,DB_ROLL_PTR. All these columns are always present. dict_index_build_data_tuple(): Remove assertions that would be duplicated in rec_copy_prefix_to_dtuple(). rec_init_offsets(): Support ROW_FORMAT=REDUNDANT records whose number of fields is between n_core_fields and n_fields. cmp_rec_rec_with_match(): Implement the comparison between two MIN_REC_FLAG records. trx_t::in_rollback: Make the field available in non-debug builds. trx_start_for_ddl_low(): Remove dangerous error-tolerance. A dictionary transaction must be flagged as such before it has generated any undo log records. This is because trx_undo_assign_undo() will mark the transaction as a dictionary transaction in the undo log header right before the very first undo log record is being written. btr_index_rec_validate(): Account for instant ADD COLUMN row_undo_ins_remove_clust_rec(): On the rollback of an insert into SYS_COLUMNS, revert instant ADD COLUMN in the cache by removing the last column from the table and the clustered index. row_search_on_row_ref(), row_undo_mod_parse_undo_rec(), row_undo_mod(), trx_undo_update_rec_get_update(): Handle the 'default row' as a special case. dtuple_t::trim(index): Omit a redundant suffix of an index tuple right before insert or update. After instant ADD COLUMN, if the last fields of a clustered index tuple match the 'default row', there is no need to store them. While trimming the entry, we must hold a page latch, so that the table cannot be emptied and the 'default row' be deleted. btr_cur_optimistic_update(), btr_cur_pessimistic_update(), row_upd_clust_rec_by_insert(), row_ins_clust_index_entry_low(): Invoke dtuple_t::trim() if needed. row_ins_clust_index_entry(): Restore dtuple_t::n_fields after calling row_ins_clust_index_entry_low(). rec_get_converted_size(), rec_get_converted_size_comp(): Allow the number of fields to be between n_core_fields and n_fields. Do not support infimum,supremum. They are never supposed to be stored in dtuple_t, because page creation nowadays uses a lower-level method for initializing them. rec_convert_dtuple_to_rec_comp(): Assign the status bits based on the number of fields. btr_cur_trim(): In an update, trim the index entry as needed. For the 'default row', handle rollback specially. For user records, omit fields that match the 'default row'. btr_cur_optimistic_delete_func(), btr_cur_pessimistic_delete(): Skip locking and adaptive hash index for the 'default row'. row_log_table_apply_convert_mrec(): Replace 'default row' values if needed. In the temporary file that is applied by row_log_table_apply(), we must identify whether the records contain the extra header for instantly added columns. For now, we will allocate an additional byte for this for ROW_T_INSERT and ROW_T_UPDATE records when the source table has been subject to instant ADD COLUMN. The ROW_T_DELETE records are fine, as they will be converted and will only contain 'core' columns (PRIMARY KEY and some system columns) that are converted from dtuple_t. rec_get_converted_size_temp(), rec_init_offsets_temp(), rec_convert_dtuple_to_temp(): Add the parameter 'status'. REC_INFO_DEFAULT_ROW = REC_INFO_MIN_REC_FLAG \| REC_STATUS_COLUMNS_ADDED: An info_bits constant for distinguishing the 'default row' record. rec_comp_status_t: An enum of the status bit values. rec_leaf_format: An enum that replaces the bool parameter of rec_init_offsets_comp_ordinary().	8 years ago
Jan Lindström	eca238aea7	MDEV-13557: Startup failure, unable to decrypt ibdata1 Fixes also MDEV-13488: InnoDB writes CRYPT_INFO even though encryption is not enabled. Fixes also MDEV-13093: Leak of Datafile::m_crypt_info on shutdown after failed startup. Problem was that we created encryption metadata (crypt_data) for system tablespace even when no encryption was enabled and too early. System tablespace can be encrypted only using key rotation. Test innodb-key-rotation-disable, innodb_encryption, innodb_lotoftables require adjustment because INFORMATION_SCHEMA INNODB_TABLESPACES_ENCRYPTION contain row only if tablespace really has encryption metadata. xb_load_single_table_tablespace(): Do not call fil_space_destroy_crypt_data() any more, because Datafile::m_crypt_data has been removed. fil_crypt_realloc_iops(): Avoid divide by zero. fil_crypt_set_thread_cnt(): Set fil_crypt_threads_event if encryption threads exist. This is required to find tablespaces requiring key rotation if no other changes happen. fil_crypt_find_space_to_rotate(): Decrease the amount of time waiting when nothing happens to better enable key rotation on startup. fil_ibd_open(), fil_ibd_load(): Load possible crypt_data from first page. class Datafile, class SysTablespace : remove m_crypt_info field. Datafile::get_first_page(): Return a pointer to first page buffer. fsp_header_init(): Write encryption metadata to page 0 only if tablespace is encrypted or encryption is disabled by table option. i_s_dict_fill_tablespaces_encryption(): Skip tablespaces that do not contain encryption metadata. This is required to avoid too early wait condition trigger in encrypted -> unencrypted state transfer.	8 years ago

1 2 3 4

171 Commits (312569e2fd79a6582161900e726847ad688ee3b0)