mariadb

Commit Graph

Author	SHA1	Message	Date
Sergey Vojtovich	131d9a5d0c	Allocate lock_sys statically There is only one lock_sys. Allocate it statically in order to avoid dereferencing a pointer whenever accessing it. Also, align some members to their own cache line in order to avoid false sharing. lock_sys_t::create(): The deferred constructor. lock_sys_t::close(): The early destructor.	8 years ago
Marko Mäkelä	03400d974d	Dead code removal: sess_t The session object is not really needed for anything. We can directly create and free the dummy purge_sys->query->trx.	8 years ago
Marko Mäkelä	27ea2963fc	Dead code removal: sess_t The session object is not really needed for anything. We can directly create and free the dummy purge_sys->query->trx.	8 years ago
Sergey Vojtovich	c0d5d7c0ef	MDEV-15104 - Remove trx_sys_t::serialisation_list serialisation_list was supposed to instantly give minimum registered transaction serialisation number. However maintaining and accessing this list requires global mutex protection. Since we already take MVCC snapshot by iterating trx_sys_t::rw_trx_hash, it is cheap to integrate minimum registered transaction lookup into this iteration.	8 years ago
Marko Mäkelä	041a32abcd	Remove trx_mod_tables_t::vers_by_trx Only invoke set_versioned() on trx_id versioned tables. dict_table_t::versioned_by_id(): New accessor, to determine if a table is system versioned by transaction ID.	8 years ago
Howard Su	6fe953cb71	Fix build on OSX with 10.13 SDK	8 years ago
Sergey Vojtovich	4dc30f3c17	MDEV-15019 - InnoDB: store ReadView on trx This will allow us to reduce critical section protected by trx_sys.mutex: - no need to maintain global m_free list - eliminate if (trx->read_view == NULL) condition. On x86_64 sizeof(Readview) is 144 mostly due to padding, sizeof(trx_t) with ReadView is 1200. Also don't close ReadView for read-write transactions, just mark it closed similarly to read-only. Clean-up: removed n_prepared_recovered_trx and n_prepared_trx, which accidentally re-appeared after some rebase.	8 years ago
Marko Mäkelä	4f8555f1f6	MDEV-14941 Timeouts on persistent statistics tables caused by MDEV-14511 MDEV-14511 tried to avoid some consistency problems related to InnoDB persistent statistics. The persistent statistics are being written by an InnoDB internal SQL interpreter that requires the InnoDB data dictionary cache to be locked. Before MDEV-14511, the statistics were written during DDL in separate transactions, which could unnecessarily reduce performance (each commit would require a redo log flush) and break atomicity, because the statistics would be updated separately from the dictionary transaction. However, because it is unacceptable to hold the InnoDB data dictionary cache locked while suspending the execution for waiting for a transactional lock (in the mysql.innodb_index_stats or mysql.innodb_table_stats tables) to be released, any lock conflict was immediately be reported as "lock wait timeout". To fix MDEV-14941, an attempt to reduce these lock conflicts by acquiring transactional locks on the user tables in both the statistics and DDL operations was made, but it would still not entirely prevent lock conflicts on the mysql.innodb_index_stats and mysql.innodb_table_stats tables. Fixing the remaining problems would require a change that is too intrusive for a GA release series, such as MariaDB 10.2. Thefefore, we revert the change MDEV-14511. To silence the MDEV-13201 assertion, we use the pre-existing flag trx_t::internal.	8 years ago
Sergey Vojtovich	ec32c05072	Get rid of trx->read_view pointer juggling trx->read_view\|= 1 was done in a silly attempt to fix race condition where trx->read_view was closed without trx_sys.mutex lock by read-only trasnactions. This just made the problem less likely to happen. In fact there was race condition in const version of trx_get_read_view(): pointer may change to garbage any moment after MVCC::is_view_active(trx->read_view) check and before this function returns. This patch doesn't fix this race condition, but rather makes it's consequences less destructive.	8 years ago
Marko Mäkelä	f8882cce93	Replace trx_sys_t* trx_sys with trx_sys_t trx_sys There is only one transaction system object in InnoDB. Allocate the storage for it at link time, not at runtime. lock_rec_fetch_page(): Use the correct fetch mode BUF_GET. Pages may never be deallocated from a tablespace while record locks are pointing to them.	8 years ago
Sergey Vojtovich	c6d2842d9a	MDEV-14756 - Remove trx_sys_t::rw_trx_list Remove rw_trx_list.	8 years ago
Sergey Vojtovich	a447980ff3	MDEV-14756 - Remove trx_sys_t::rw_trx_list Let lock_print_info_all_transactions() iterate rw_trx_hash instead of rw_trx_list. When printing info of locks for transactions, InnoDB monitor doesn't attempt to read relevant page from disk anymore. The code was prone to race conditions. Note that TrxListIterator didn't work as advertised: it iterated rw_trx_list only.	8 years ago
Sergey Vojtovich	886af392d3	MDEV-14756 - Remove trx_sys_t::rw_trx_list Let trx_rollback_recovered() iterate rw_trx_hash instead of rw_trx_list.	8 years ago
Sergey Vojtovich	02270b44d0	MDEV-14756 - Remove trx_sys_t::rw_trx_list Let lock_validate_table_locks(), lock_rec_other_trx_holds_expl(), lock_table_locks_lookup(), trx_recover_for_mysql(), trx_get_trx_by_xid(), trx_roll_must_shutdown(), fetch_data_into_cache() iterate rw_trx_hash instead of rw_trx_list.	8 years ago
Sergey Vojtovich	d09f146934	MDEV-14756 - Remove trx_sys_t::rw_trx_list Reduce divergence between trx_sys_t::rw_trx_hash and trx_sys_t::rw_trx_list by not adding recovered COMMITTED transactions to trx_sys_t::rw_trx_list. Such transactions are discarded immediately without creating trx object. This also required to split rollback and cleanup phases of recovery. To reflect these updates the following renames happened: trx_rollback_or_clean_all_recovered() -> trx_rollback_all_recovered() trx_rollback_or_clean_is_active -> trx_rollback_is_active trx_rollback_or_clean_recovered() -> trx_rollback_recovered() trx_cleanup_at_db_startup() -> trx_cleanup_recovered() Also removed a hack from lock_trx_release_locks(). Instead let recovery rollback thread to skip committed XA transactions.	8 years ago
Sergey Vojtovich	0ca2ea1a65	MDEV-14638 - Replace trx_sys_t::rw_trx_set with LF_HASH trx reference counter was updated under mutex and read without any protection. This is both slow and unsafe. Use atomic operations for reference counter accesses.	8 years ago
Sergey Vojtovich	380069c235	MDEV-14638 - Replace trx_sys_t::rw_trx_set with LF_HASH trx_sys_t::rw_trx_set is implemented as std::set, which does a few quite expensive operations under trx_sys_t::mutex protection: e.g. malloc/free when adding/removing elements. Traversing b-tree is not that cheap either. This has negative scalability impact, which is especially visible when running oltp_update_index.lua benchmark on a ramdisk. To reduce trx_sys_t::mutex contention std::set is replaced with LF_HASH. None of LF_HASH operations require trx_sys_t::mutex (nor any other global mutex) protection. Another interesting issue observed with std::set is reproducible ~2% performance decline after benchmark is ran for ~60 seconds. With LF_HASH results are stable. All in all this patch optimises away one of three trx_sys->mutex locks per oltp_update_index.lua query. The other two critical sections became smaller. Relevant clean-ups: Replaced rw_trx_set iteration at startup with local set. The latter is needed because values inserted to rw_trx_list must be ordered by trx->id. Removed redundant conditions from trx_reference(): it is (and even was) never called with transactions that have trx->state == TRX_STATE_COMMITTED_IN_MEMORY. do_ref_count doesn't (and probably even didn't) make any sense: now it is called only when reference counter increment is actually requested. Moved condition out of mutex in trx_erase_lists(). trx_rw_is_active(), trx_rw_is_active_low() and trx_get_rw_trx_by_id() were greatly simplified and replaced by appropriate trx_rw_hash_t methods. Compared to rw_trx_set, rw_trx_hash holds transactions only in PREPARED or ACTIVE states. Transactions in COMMITTED state were required to be found at InnoDB startup only. They are now looked up in the local set. Removed unused trx_assert_recovered(). Removed unused innobase_get_trx() declaration. Removed rather semantically incorrect trx_sys_rw_trx_add(). Moved information printout from trx_sys_init_at_db_start() to trx_lists_init_at_db_start().	8 years ago
Aleksey Midenkov	b55a149194	Timestamp-based versioning for InnoDB [closes #209 ] * Removed integer_fields check * Reworked Vers_parse_info::check_sys_fields() * Misc renames * versioned as vers_sys_type_t * Removed versioned_by_sql(), versioned_by_engine() versioned() works as before; versioned(VERS_TIMESTAMP) is versioned_by_sql(); versioned(VERS_TRX_ID) is versioned_by_engine(). * create_tmp_table() fix * Foreign constraints for timestamp-based * Range auto-specifier fix * SQL: 1-row partition rotation fix [fixes #260] * Fix 'drop system versioning, algorithm=inplace'	8 years ago
Marko Mäkelä	77fb7ccba4	Follow-up fix to MDEV-13201 Assertion `srv_undo_sources \|\| ...` failed on shutdown during DDL operation Introduce the debug flag trx_t::persistent_stats to suppress the assertion for the updates of persistent statistics during fast shutdown. dict_stats_exec_sql(): Do execute the statement even though shutdown has been initiated.	8 years ago
Marko Mäkelä	0c6ff122fa	Follow-up fix to MDEV-14477: Use proper std::map allocator	8 years ago
Marko Mäkelä	0b89a42ffc	Remove the flag vers_update_trt THD::vers_update_trt, trx_t::vers_update_trt, trx_savept_t::vers_update_trt: Remove. Instead, determine from trx_t::mod_tables whether versioned columns were affected by the transaction. handlerton::prepare_commit_versioned: Replaces vers_get_trt_data. Return the transaction start ID and also the commit ID, in case the transaction modified any system-versioned columns (0 if not). TR_table::store_data(): Remove (merge with update() below). TR_table::update(): Add the parameters start_id, end_id. ha_commit_trans(): Remove a condition on SQLCOM_ALTER_TABLE. If we need something special for ALTER TABLE...ALGORITHM=INPLACE, that can be done inside InnoDB by modifying trx_t::mod_tables. innodb_prepare_commit_versioned(): Renamed from innodb_get_trt_data(). Check trx_t::mod_tables to see if any changes to versioned columns are present. trx_mod_table_time_t: A pair of logical timestamps, replacing the undo_no_t in trx_mod_tables_t. Keep track of not only the first modification to a persistent table in each transaction, but also the first modification of a versioned column in a table. dtype_t, dict_col_t: Add the accessor is_any_versioned(), to check if the type refers to a system-versioned user or system column. upd_t::affects_versioned(): Check if an update affects a versioned column. trx_undo_report_row_operation(): If a versioned column is affected by the update, invoke trx_mod_table_time_t::set_versioned(). trx_rollback_to_savepoint_low(): If all changes to versioned columns were rolled back, invoke trx_mod_table_time_t::rollback_versioned(), so that trx_mod_table_time_t::is_versioned() will no longer hold.	8 years ago
Marko Mäkelä	d8ada08152	MDEV-14477 InnoDB update_time is wrongly updated after partial rollback or internal COMMIT The non-persistent UPDATE_TIME for InnoDB tables was not being updated consistently at transaction commit. If a transaction is partly rolled back so that in the end it will not modify a table that it intended to modify, the update_time will be updated nevertheless. This will also happen when InnoDB fails to write an undo log record for the intended modification. If a transaction is committed internally in InnoDB, instead of being committed from the SQL interface, then the trx_t::mod_tables will not be applied to the update_time of the tables. trx_t::mod_tables: Replace the std::set<dict_table_t> with std::map<dict_table_t,undo_no_t>, so that the very first modification within the transaction is identified. trx_undo_report_row_operation(): Update mod_tables for every operation after the undo log record was successfully written. trx_rollback_to_savepoint_low(): After partial rollback, erase from trx_t::mod_tables any tables for which all changes were rolled back. trx_commit_in_memory(): Tighten some assertions and simplify conditions. Invoke trx_update_mod_tables_timestamp() if persistent tables were affected. trx_commit_for_mysql(): Remove the call to trx_update_mod_tables_timestamp(), as it is now invoked at the lower level, in trx_commit_in_memory(). trx_rollback_finish(): Clear mod_tables before invoking trx_commit(), because the trx_commit_in_memory() would otherwise wrongly process mod_tables after a full ROLLBACK.	8 years ago
Eugene Kosov	90c809a61d	IB: VTQ cleanups [#305 ] * btr_pcur_move_to_prev_user_rec() removed * usage of ut_usectime() removed * other VTQ-related leftovers	8 years ago
Aleksey Midenkov	33085349e9	IB, SQL: removed VTQ, added TRT on SQL layer [closes #305 ]	8 years ago
Eugene Kosov	fa79f6ac86	IB: style changes [closes #306 ]	8 years ago
Aleksey Midenkov	669fd1d962	IB: ulint compilation fixes	8 years ago
Marko Mäkelä	a4948dafcd	MDEV-11369 Instant ADD COLUMN for InnoDB For InnoDB tables, adding, dropping and reordering columns has required a rebuild of the table and all its indexes. Since MySQL 5.6 (and MariaDB 10.0) this has been supported online (LOCK=NONE), allowing concurrent modification of the tables. This work revises the InnoDB ROW_FORMAT=REDUNDANT, ROW_FORMAT=COMPACT and ROW_FORMAT=DYNAMIC so that columns can be appended instantaneously, with only minor changes performed to the table structure. The counter innodb_instant_alter_column in INFORMATION_SCHEMA.GLOBAL_STATUS is incremented whenever a table rebuild operation is converted into an instant ADD COLUMN operation. ROW_FORMAT=COMPRESSED tables will not support instant ADD COLUMN. Some usability limitations will be addressed in subsequent work: MDEV-13134 Introduce ALTER TABLE attributes ALGORITHM=NOCOPY and ALGORITHM=INSTANT MDEV-14016 Allow instant ADD COLUMN, ADD INDEX, LOCK=NONE The format of the clustered index (PRIMARY KEY) is changed as follows: (1) The FIL_PAGE_TYPE of the root page will be FIL_PAGE_TYPE_INSTANT, and a new field PAGE_INSTANT will contain the original number of fields in the clustered index ('core' fields). If instant ADD COLUMN has not been used or the table becomes empty, or the very first instant ADD COLUMN operation is rolled back, the fields PAGE_INSTANT and FIL_PAGE_TYPE will be reset to 0 and FIL_PAGE_INDEX. (2) A special 'default row' record is inserted into the leftmost leaf, between the page infimum and the first user record. This record is distinguished by the REC_INFO_MIN_REC_FLAG, and it is otherwise in the same format as records that contain values for the instantly added columns. This 'default row' always has the same number of fields as the clustered index according to the table definition. The values of 'core' fields are to be ignored. For other fields, the 'default row' will contain the default values as they were during the ALTER TABLE statement. (If the column default values are changed later, those values will only be stored in the .frm file. The 'default row' will contain the original evaluated values, which must be the same for every row.) The 'default row' must be completely hidden from higher-level access routines. Assertions have been added to ensure that no 'default row' is ever present in the adaptive hash index or in locked records. The 'default row' is never delete-marked. (3) In clustered index leaf page records, the number of fields must reside between the number of 'core' fields (dict_index_t::n_core_fields introduced in this work) and dict_index_t::n_fields. If the number of fields is less than dict_index_t::n_fields, the missing fields are replaced with the column value of the 'default row'. Note: The number of fields in the record may shrink if some of the last instantly added columns are updated to the value that is in the 'default row'. The function btr_cur_trim() implements this 'compression' on update and rollback; dtuple::trim() implements it on insert. (4) In ROW_FORMAT=COMPACT and ROW_FORMAT=DYNAMIC records, the new status value REC_STATUS_COLUMNS_ADDED will indicate the presence of a new record header that will encode n_fields-n_core_fields-1 in 1 or 2 bytes. (In ROW_FORMAT=REDUNDANT records, the record header always explicitly encodes the number of fields.) We introduce the undo log record type TRX_UNDO_INSERT_DEFAULT for covering the insert of the 'default row' record when instant ADD COLUMN is used for the first time. Subsequent instant ADD COLUMN can use TRX_UNDO_UPD_EXIST_REC. This is joint work with Vin Chen (陈福荣) from Tencent. The design that was discussed in April 2017 would not have allowed import or export of data files, because instead of the 'default row' it would have introduced a data dictionary table. The test rpl.rpl_alter_instant is exactly as contributed in pull request #408. The test innodb.instant_alter is based on a contributed test. The redo log record format changes for ROW_FORMAT=DYNAMIC and ROW_FORMAT=COMPACT are as contributed. (With this change present, crash recovery from MariaDB 10.3.1 will fail in spectacular ways!) Also the semantics of higher-level redo log records that modify the PAGE_INSTANT field is changed. The redo log format version identifier was already changed to LOG_HEADER_FORMAT_CURRENT=103 in MariaDB 10.3.1. Everything else has been rewritten by me. Thanks to Elena Stepanova, the code has been tested extensively. When rolling back an instant ADD COLUMN operation, we must empty the PAGE_FREE list after deleting or shortening the 'default row' record, by calling either btr_page_empty() or btr_page_reorganize(). We must know the size of each entry in the PAGE_FREE list. If rollback left a freed copy of the 'default row' in the PAGE_FREE list, we would be unable to determine its size (if it is in ROW_FORMAT=COMPACT or ROW_FORMAT=DYNAMIC) because it would contain more fields than the rolled-back definition of the clustered index. UNIV_SQL_DEFAULT: A new special constant that designates an instantly added column that is not present in the clustered index record. len_is_stored(): Check if a length is an actual length. There are two magic length values: UNIV_SQL_DEFAULT, UNIV_SQL_NULL. dict_col_t::def_val: The 'default row' value of the column. If the column is not added instantly, def_val.len will be UNIV_SQL_DEFAULT. dict_col_t: Add the accessors is_virtual(), is_nullable(), is_instant(), instant_value(). dict_col_t::remove_instant(): Remove the 'instant ADD' status of a column. dict_col_t::name(const dict_table_t& table): Replaces dict_table_get_col_name(). dict_index_t::n_core_fields: The original number of fields. For secondary indexes and if instant ADD COLUMN has not been used, this will be equal to dict_index_t::n_fields. dict_index_t::n_core_null_bytes: Number of bytes needed to represent the null flags; usually equal to UT_BITS_IN_BYTES(n_nullable). dict_index_t::NO_CORE_NULL_BYTES: Magic value signalling that n_core_null_bytes was not initialized yet from the clustered index root page. dict_index_t: Add the accessors is_instant(), is_clust(), get_n_nullable(), instant_field_value(). dict_index_t::instant_add_field(): Adjust clustered index metadata for instant ADD COLUMN. dict_index_t::remove_instant(): Remove the 'instant ADD' status of a clustered index when the table becomes empty, or the very first instant ADD COLUMN operation is rolled back. dict_table_t: Add the accessors is_instant(), is_temporary(), supports_instant(). dict_table_t::instant_add_column(): Adjust metadata for instant ADD COLUMN. dict_table_t::rollback_instant(): Adjust metadata on the rollback of instant ADD COLUMN. prepare_inplace_alter_table_dict(): First create the ctx->new_table, and only then decide if the table really needs to be rebuilt. We must split the creation of table or index metadata from the creation of the dictionary table records and the creation of the data. In this way, we can transform a table-rebuilding operation into an instant ADD COLUMN operation. Dictionary objects will only be added to cache when table rebuilding or index creation is needed. The ctx->instant_table will never be added to cache. dict_table_t::add_to_cache(): Modified and renamed from dict_table_add_to_cache(). Do not modify the table metadata. Let the callers invoke dict_table_add_system_columns() and if needed, set can_be_evicted. dict_create_sys_tables_tuple(), dict_create_table_step(): Omit the system columns (which will now exist in the dict_table_t object already at this point). dict_create_table_step(): Expect the callers to invoke dict_table_add_system_columns(). pars_create_table(): Before creating the table creation execution graph, invoke dict_table_add_system_columns(). row_create_table_for_mysql(): Expect all callers to invoke dict_table_add_system_columns(). create_index_dict(): Replaces row_merge_create_index_graph(). innodb_update_n_cols(): Renamed from innobase_update_n_virtual(). Call my_error() if an error occurs. btr_cur_instant_init(), btr_cur_instant_init_low(), btr_cur_instant_root_init(): Load additional metadata from the clustered index and set dict_index_t::n_core_null_bytes. This is invoked when table metadata is first loaded into the data dictionary. dict_boot(): Initialize n_core_null_bytes for the four hard-coded dictionary tables. dict_create_index_step(): Initialize n_core_null_bytes. This is executed as part of CREATE TABLE. dict_index_build_internal_clust(): Initialize n_core_null_bytes to NO_CORE_NULL_BYTES if table->supports_instant(). row_create_index_for_mysql(): Initialize n_core_null_bytes for CREATE TEMPORARY TABLE. commit_cache_norebuild(): Call the code to rename or enlarge columns in the cache only if instant ADD COLUMN is not being used. (Instant ADD COLUMN would copy all column metadata from instant_table to old_table, including the names and lengths.) PAGE_INSTANT: A new 13-bit field for storing dict_index_t::n_core_fields. This is repurposing the 16-bit field PAGE_DIRECTION, of which only the least significant 3 bits were used. The original byte containing PAGE_DIRECTION will be accessible via the new constant PAGE_DIRECTION_B. page_get_instant(), page_set_instant(): Accessors for the PAGE_INSTANT. page_ptr_get_direction(), page_get_direction(), page_ptr_set_direction(): Accessors for PAGE_DIRECTION. page_direction_reset(): Reset PAGE_DIRECTION, PAGE_N_DIRECTION. page_direction_increment(): Increment PAGE_N_DIRECTION and set PAGE_DIRECTION. rec_get_offsets(): Use the 'leaf' parameter for non-debug purposes, and assume that heap_no is always set. Initialize all dict_index_t::n_fields for ROW_FORMAT=REDUNDANT records, even if the record contains fewer fields. rec_offs_make_valid(): Add the parameter 'leaf'. rec_copy_prefix_to_dtuple(): Assert that the tuple is only built on the core fields. Instant ADD COLUMN only applies to the clustered index, and we should never build a search key that has more than the PRIMARY KEY and possibly DB_TRX_ID,DB_ROLL_PTR. All these columns are always present. dict_index_build_data_tuple(): Remove assertions that would be duplicated in rec_copy_prefix_to_dtuple(). rec_init_offsets(): Support ROW_FORMAT=REDUNDANT records whose number of fields is between n_core_fields and n_fields. cmp_rec_rec_with_match(): Implement the comparison between two MIN_REC_FLAG records. trx_t::in_rollback: Make the field available in non-debug builds. trx_start_for_ddl_low(): Remove dangerous error-tolerance. A dictionary transaction must be flagged as such before it has generated any undo log records. This is because trx_undo_assign_undo() will mark the transaction as a dictionary transaction in the undo log header right before the very first undo log record is being written. btr_index_rec_validate(): Account for instant ADD COLUMN row_undo_ins_remove_clust_rec(): On the rollback of an insert into SYS_COLUMNS, revert instant ADD COLUMN in the cache by removing the last column from the table and the clustered index. row_search_on_row_ref(), row_undo_mod_parse_undo_rec(), row_undo_mod(), trx_undo_update_rec_get_update(): Handle the 'default row' as a special case. dtuple_t::trim(index): Omit a redundant suffix of an index tuple right before insert or update. After instant ADD COLUMN, if the last fields of a clustered index tuple match the 'default row', there is no need to store them. While trimming the entry, we must hold a page latch, so that the table cannot be emptied and the 'default row' be deleted. btr_cur_optimistic_update(), btr_cur_pessimistic_update(), row_upd_clust_rec_by_insert(), row_ins_clust_index_entry_low(): Invoke dtuple_t::trim() if needed. row_ins_clust_index_entry(): Restore dtuple_t::n_fields after calling row_ins_clust_index_entry_low(). rec_get_converted_size(), rec_get_converted_size_comp(): Allow the number of fields to be between n_core_fields and n_fields. Do not support infimum,supremum. They are never supposed to be stored in dtuple_t, because page creation nowadays uses a lower-level method for initializing them. rec_convert_dtuple_to_rec_comp(): Assign the status bits based on the number of fields. btr_cur_trim(): In an update, trim the index entry as needed. For the 'default row', handle rollback specially. For user records, omit fields that match the 'default row'. btr_cur_optimistic_delete_func(), btr_cur_pessimistic_delete(): Skip locking and adaptive hash index for the 'default row'. row_log_table_apply_convert_mrec(): Replace 'default row' values if needed. In the temporary file that is applied by row_log_table_apply(), we must identify whether the records contain the extra header for instantly added columns. For now, we will allocate an additional byte for this for ROW_T_INSERT and ROW_T_UPDATE records when the source table has been subject to instant ADD COLUMN. The ROW_T_DELETE records are fine, as they will be converted and will only contain 'core' columns (PRIMARY KEY and some system columns) that are converted from dtuple_t. rec_get_converted_size_temp(), rec_init_offsets_temp(), rec_convert_dtuple_to_temp(): Add the parameter 'status'. REC_INFO_DEFAULT_ROW = REC_INFO_MIN_REC_FLAG \| REC_STATUS_COLUMNS_ADDED: An info_bits constant for distinguishing the 'default row' record. rec_comp_status_t: An enum of the status bit values. rec_leaf_format: An enum that replaces the bool parameter of rec_init_offsets_comp_ordinary().	8 years ago
Marko Mäkelä	3c09f148f3	MDEV-12288 Reset DB_TRX_ID when the history is removed, to speed up MVCC Let InnoDB purge reset DB_TRX_ID,DB_ROLL_PTR when the history is removed. [TODO: It appears that the resetting is not taking place as often as it could be. We should test that a simple INSERT should eventually cause row_purge_reset_trx_id() to be invoked unless DROP TABLE is invoked soon enough.] The InnoDB clustered index record system columns DB_TRX_ID,DB_ROLL_PTR are used by multi-versioning. After the history is no longer needed, these columns can safely be reset to 0 and 1<<55 (to indicate a fresh insert). When a reader sees 0 in the DB_TRX_ID column, it can instantly determine that the record is present the read view. There is no need to acquire the transaction system mutex to check if the transaction exists, because writes can never be conducted by a transaction whose ID is 0. The persistent InnoDB undo log used to be split into two parts: insert_undo and update_undo. The insert_undo log was discarded at transaction commit or rollback, and the update_undo log was processed by the purge subsystem. As part of this change, we will only generate a single undo log for new transactions, and the purge subsystem will reset the DB_TRX_ID whenever a clustered index record is touched. That is, all persistent undo log will be preserved at transaction commit or rollback, to be removed by purge. The InnoDB redo log format is changed in two ways: We remove the redo log record type MLOG_UNDO_HDR_REUSE, and we introduce the MLOG_ZIP_WRITE_TRX_ID record for updating the DB_TRX_ID,DB_ROLL_PTR in a ROW_FORMAT=COMPRESSED table. This is also changing the format of persistent InnoDB data files: undo log and clustered index leaf page records. It will still be possible via import and export to exchange data files with earlier versions of MariaDB. The change to clustered index leaf page records is simple: we allow DB_TRX_ID to be 0. When it comes to the undo log, we must be able to upgrade from earlier MariaDB versions after a clean shutdown (no redo log to apply). While it would be nice to perform a slow shutdown (innodb_fast_shutdown=0) before an upgrade, to empty the undo logs, we cannot assume that this has been done. So, separate insert_undo log may exist for recovered uncommitted transactions. These transactions may be automatically rolled back, or they may be in XA PREPARE state, in which case InnoDB will preserve the transaction until an explicit XA COMMIT or XA ROLLBACK. Upgrade has been tested by starting up MariaDB 10.2 with ./mysql-test-run --manual-gdb innodb.read_only_recovery and then starting up this patched server with and without --innodb-read-only. trx_undo_ptr_t::undo: Renamed from update_undo. trx_undo_ptr_t::old_insert: Renamed from insert_undo. trx_rseg_t::undo_list: Renamed from update_undo_list. trx_rseg_t::undo_cached: Merged from update_undo_cached and insert_undo_cached. trx_rseg_t::old_insert_list: Renamed from insert_undo_list. row_purge_reset_trx_id(): New function to reset the columns. This will be called for all undo processing in purge that does not remove the clustered index record. trx_undo_update_rec_get_update(): Allow trx_id=0 when copying the old DB_TRX_ID of the record to the undo log. ReadView::changes_visible(): Allow id==0. (Return true for it. This is what speeds up the MVCC.) row_vers_impl_x_locked_low(), row_vers_build_for_semi_consistent_read(): Implement a fast path for DB_TRX_ID=0. Always initialize the TRX_UNDO_PAGE_TYPE to 0. Remove undo->type. MLOG_UNDO_HDR_REUSE: Remove. This changes the redo log format! innobase_start_or_create_for_mysql(): Set srv_undo_sources before starting any transactions. The parsing of the MLOG_ZIP_WRITE_TRX_ID record was successfully tested by running the following: ./mtr --parallel=auto --mysqld=--debug=d,ib_log innodb_zip.bug56680 grep MLOG_ZIP_WRITE_TRX_ID var/*/log/mysqld.1.err	8 years ago
Marko Mäkelä	50faeda4d6	Remove trx_t::has_search_latch and simplify debug code When the btr_search_latch was split into an array of latches in MySQL 5.7.8 as part of the Oracle Bug#20985298 fix, the "caching" of the latch across storage engine API calls was removed, and the field trx->has_search_latch would only be set during a short time frame in the execution of row_search_mvcc(), which was formerly called row_search_for_mysql(). This means that the column INFORMATION_SCHEMA.INNODB_TRX.TRX_ADAPTIVE_HASH_LATCHED will always report 0. That column cannot be removed in MariaDB 10.2, but it can be removed in future releases. trx_t::has_search_latch: Remove. trx_assert_no_search_latch(): Remove. row_sel_try_search_shortcut_for_mysql(): Remove a redundant condition on trx->has_search_latch (it was always true). sync_check_iterate(): Make the parameter const. sync_check_functor_t: Make the operator() const, and remove result() and the virtual destructor. There is no need to have mutable state in the functors. sync_checker<bool>: Replaces dict_sync_check and btrsea_sync_check. sync_check: Replaces btrsea_sync_check. dict_sync_check: Instantiated from sync_checker. sync_allowed_latches: Use std::find() directly on the array. Remove the std::vector. TrxInInnoDB::enter(), TrxInInnoDB::exit(): Remove obviously redundant debug assertions on trx->in_depth, and use equality comparison against 0 because it could be more efficient on some architectures.	8 years ago
Jan Lindström	f4df8c00c9	MDEV-11197: TrxInInnoDB::is_aborted(const trx_t*): Assertion `srv_read_only_mode \|\| trx->in_depth > 0' failed TrxInInnoDB should be constructed to track if a transaction is executing inside InnoDB code i.e. it is like a gate between Server and InnoDB ::rnd_next() is called from Server and thus construct TrxInInnoDB also there. Applied suggested clean-up to TrxInInnoDB class functions enter() and exit(). Note that exactly original did not work for enter().	9 years ago
Aleksey Midenkov	d54d36c45e	IB, SQL: (0.4) COMMIT_ID-based ordering of transactions IB: * removed CONCURR_TRX from VTQ; * new fields in VTQ: COMMIT_ID, ISO_LEVEL. SQL: * renamed BEGIN_TS, COMMIT_TS to VTQ_BEGIN_TS, VTQ_COMMIT_TS; * new functions: VTQ_COMMIT_ID, VTQ_ISO_LEVEL, VTQ_TRX_ID, VTQ_TRX_SEES, VTQ_TRX_SEES_EQ; * versioned SELECT for IB uses VTQ_TRX_SEES, VTQ_TRX_SEES_EQ. Closes #71	9 years ago
Aleksey Midenkov	53a892fcfd	IB: 0.2 part IV * BEGIN_TS(), COMMIT_TS() SQL functions; * VTQ instead of packed stores secs + usecs like my_timestamp_to_binary() does; * versioned SELECT to IB is translated with COMMIT_TS(); * SQL fixes: - FOR_SYSTEM_TIME_UNSPECIFIED condition compares to TIMESTAMP_MAX_VALUE; - segfault fix #36: multiple execute of prepared stmt; - different tables to same stored procedure fix (#39) * Fixes of previous parts: ON DUPLICATE KEY, other misc fixes.	9 years ago
Aleksey Midenkov	bdb12d1499	IB: 0.2 part II * moved vers_notify_vtq() to commit phase; * low_level insert (load test passed); * rest of SYS_VTQ columns filled: COMMIT_TS, CONCURR_TRX; * savepoints support; * I_S.INNODB_SYS_VTQ adjustments: - limit to I_S_SYS_VTQ_LIMIT(10000) of most recent records; - CONCURR_TRX limit to I_S_MAX_CONCURR_TRX(100) with '...' truncation marker; - TIMESTAMP fields show fractions of seconds.	9 years ago
Aleksey Midenkov	84e1971128	IB: 0.2 part I * SYS_VTQ internal InnoDB table; * I_S.INNODB_SYS_VTQ table; * vers_notify_vtq(): add record to SYS_VTQ on versioned DML; * SYS_VTQ columns filled: TRX_ID, BEGIN_TS.	9 years ago
Debarun Banerjee	bf5be32376	BUG#25032066 PREPARED TRANSACTION SHOULD NOT BE ROLLED BACK BY HIGH PRIORITY TRANSACTION Problem : --------- 1. delete_all_rows() and rnd_init() are not returning error after async rollback in 5.7. This results in assert in innodb in next call. 2. High priority transaction is rolling back prepared transaction. This is because TRX_FORCE_ROLLBACK_DISABLE is getting set only for first entry [TrxInInnoDB]. Solution : ---------- 1. return DB_FORCED_ABORT error after rollback. 2. check and disable rollback in TrxInInnodb::enter always. Reviewed-by: Sunny Bains <sunny.bains@oracle.com> RB: 13777	9 years ago
Marko Mäkelä	858d8f0c6e	Remove innodb_support_xa (deprecated in 5.7.10 and 10.2.)	9 years ago
Marko Mäkelä	85da56bf2d	Remove the unused variable trx_t::support_xa. Also, merge some test changes from MySQL 5.7.10, for deprecating innodb_support_xa.	9 years ago
Sachin Setiya	cdd1dc829b	MW-28, codership/mysql-wsrep#28 Fix sync_thread_levels debug assert Introduced a new wsrep_trx_print_locking() which may be called under lock_sys->mutex if the trx has locks. Signed-off-by: Sachin Setiya <sachin.setiya@mariadb.com>	9 years ago
Marko Mäkelä	124bae082b	MDEV-12289 Keep 128 persistent rollback segments for compatibility and performance InnoDB divides the allocation of undo logs into rollback segments. The DB_ROLL_PTR system column of clustered indexes can address up to 128 rollback segments (TRX_SYS_N_RSEGS). Originally, InnoDB only created one rollback segment. In MySQL 5.5 or in the InnoDB Plugin for MySQL 5.1, all 128 rollback segments were created. MySQL 5.7 hard-codes the rollback segment IDs 1..32 for temporary undo logs. On upgrade, unless a slow shutdown (innodb_fast_shutdown=0) was performed on the old server instance, these rollback segments could be in use by transactions that are in XA PREPARE state or transactions that were left behind by a server kill followed by a normal shutdown immediately after restart. Persistent tables cannot refer to temporary undo logs or vice versa. Therefore, we should keep two distinct sets of rollback segments: one for persistent tables and another for temporary tables. In this way, all 128 rollback segments will be available for both types of tables, which could improve performance. Also, MariaDB 10.2 will remain more compatible than MySQL 5.7 with data files from earlier versions of MySQL or MariaDB. trx_sys_t::temp_rsegs[TRX_SYS_N_RSEGS]: A new array of temporary rollback segments. The trx_sys_t::rseg_array[TRX_SYS_N_RSEGS] will be solely for persistent undo logs. srv_tmp_undo_logs. Remove. Use the constant TRX_SYS_N_RSEGS. srv_available_undo_logs: Change the type to ulong. trx_rseg_get_on_id(): Remove. Instead, let the callers refer to trx_sys directly. trx_rseg_create(), trx_sysf_rseg_find_free(): Remove unneeded parameters. These functions only deal with persistent undo logs. trx_temp_rseg_create(): New function, to create all temporary rollback segments at server startup. trx_rseg_t::is_persistent(): Determine if the rollback segment is for persistent tables. trx_sys_is_noredo_rseg_slot(): Remove. The callers must know based on context (such as table handle) whether the DB_ROLL_PTR is referring to a persistent undo log. trx_sys_create_rsegs(): Remove all parameters, which were always passed as global variables. Instead, modify the global variables directly. enum trx_rseg_type_t: Remove. trx_t::get_temp_rseg(): A method to ensure that a temporary rollback segment has been assigned for the transaction. trx_t::assign_temp_rseg(): Replaces trx_assign_rseg(). trx_purge_free_segment(), trx_purge_truncate_rseg_history(): Remove the redundant variable noredo=false. Temporary undo logs are discarded immediately at transaction commit or rollback, not lazily by purge. trx_purge_mark_undo_for_truncate(): Remove references to the temporary rollback segments. trx_purge_mark_undo_for_truncate(): Remove a check for temporary rollback segments. Only the dedicated persistent undo log tablespaces can be truncated. trx_undo_get_undo_rec_low(), trx_undo_get_undo_rec(): Add the parameter is_temp. trx_rseg_mem_restore(): Split from trx_rseg_mem_create(). Initialize the undo log and the rollback segment from the file data structures. trx_sysf_get_n_rseg_slots(): Renamed from trx_sysf_used_slots_for_redo_rseg(). Count the persistent rollback segment headers that have been initialized. trx_sys_close(): Also free trx_sys->temp_rsegs[]. get_next_redo_rseg(): Merged to trx_assign_rseg_low(). trx_assign_rseg_low(): Remove the parameters and access the global variables directly. Revert to simple round-robin, now that the whole trx_sys->rseg_array[] is for persistent undo log again. get_next_noredo_rseg(): Moved to trx_t::assign_temp_rseg(). srv_undo_tablespaces_init(): Remove some parameters and use the global variables directly. Clarify some error messages. Adjust the test innodb.log_file. Apparently, before these changes, InnoDB somehow ignored missing dedicated undo tablespace files that are pointed by the TRX_SYS header page, possibly losing part of essential transaction system state.	9 years ago
Marko Mäkelä	a0c79bcf00	Rename InnoDB transaction undo logging predicates. trx::has_logged_persistent(): Renamed from trx_is_redo_rseg_updated(). Determines if a transaction has generated any persistent undo log. trx::has_logged(): Renamed from trx_is_rseg_updated(). Determines if a transaction has generated any undo log.	9 years ago
Vladislav Vaintroub	e5b67a46bc	MDEV-12345 Performance : replace calls to clock() inside trx_start_low() by THD::start_utime	9 years ago
Jan Lindström	53c6195eed	Fixed test failure on galere_wsrep_log_conflicts on XtraDB. Problem was that trx_sys->mutex was acquired to print trx info even when we already hold trx_sys->mutex. Fixed similarly as in InnoDB, i.e. with wsrep_trx_print_locking() function that does not acquire trx_sys->mutex.	9 years ago
Marko Mäkelä	97acc4a1c3	MDEV-12270 Port MySQL 8.0 Bug#21141390 REMOVE UNUSED FUNCTIONS AND CONVERT GLOBAL SYMBOLS TO STATIC InnoDB defines some functions that are not called at all. Other functions are called, but only from the same compilation unit. Remove some function declarations and definitions, and add 'static' keywords. Some symbols must be kept for separately compiled tools, such as innochecksum.	11 years ago
Marko Mäkelä	4e1116b2c6	MDEV-12271 Port MySQL 8.0 Bug#23150562 REMOVE UNIV_MUST_NOT_INLINE AND UNIV_NONINL Also, remove empty .ic files that were not removed by my MySQL commit. Problem: InnoDB used to support a compilation mode that allowed to choose whether the function definitions in .ic files are to be inlined or not. This stopped making sense when InnoDB moved to C++ in MySQL 5.6 (and ha_innodb.cc started to #include .ic files), and more so in MySQL 5.7 when inline methods and functions were introduced in .h files. Solution: Remove all references to UNIV_NONINL and UNIV_MUST_NOT_INLINE from all files, assuming that the symbols are never defined. Remove the files fut0fut.cc and ut0byte.cc which only mattered when UNIV_NONINL was defined.	10 years ago
Sachin Setiya	0e0ae0bb06	MW-28, codership/mysql-wsrep#28 Fix sync_thread_levels debug assert Introduced a new wsrep_trx_print_locking() which may be called under lock_sys->mutex if the trx has locks. Signed-off-by: Sachin Setiya <sachinsetia1001@gmail.com>	9 years ago
Marko Mäkelä	13e5c9de80	MDEV-12219 Discard temporary undo logs at transaction commit Starting with MySQL 5.7, temporary tables in InnoDB are handled differently from persistent tables. Because temporary tables are private to a connection, concurrency control and multi-versioning (MVCC) are not applicable. For performance reasons, purge is disabled as well. Rollback is supported for temporary tables; that is why we have the temporary undo logs in the first place. Because MVCC and purge are disabled for temporary tables, we should discard all temporary undo logs already at transaction commit, just like we discard the persistent insert_undo logs. Before this change, update_undo logs were being preserved. trx_temp_undo_t: A wrapper for temporary undo logs, comprising a rollback segment and a single temporary undo log. trx_rsegs_t::m_noredo: Use trx_temp_undo_t. (Instead of insert_undo, update_undo, there will be a single undo.) trx_is_noredo_rseg_updated(), trx_is_rseg_assigned(): Remove. trx_undo_add_page(): Remove the parameter undo_ptr. Acquire and release the rollback segment mutex inside the function. trx_undo_free_last_page(): Remove the parameter trx. trx_undo_truncate_end(): Remove the parameter trx, and add the parameter is_temp. Clean up the code a bit. trx_undo_assign_undo(): Split the parameter undo_ptr into rseg, undo. trx_undo_commit_cleanup(): Renamed from trx_undo_insert_cleanup(). Replace the parameter undo_ptr with undo. This will discard the temporary undo or insert_undo log at commit/rollback. trx_purge_add_update_undo_to_history(), trx_undo_update_cleanup(): Remove 3 parameters. Always operate on the persistent update_undo. trx_serialise(): Renamed from trx_serialisation_number_get(). trx_write_serialisation_history(): Simplify the code flow. If there are no persistent changes, do not update MONITOR_TRX_COMMIT_UNDO. trx_commit_in_memory(): Simplify the logic, and add assertions. trx_undo_page_report_modify(): Keep a direct reference to the persistent update_undo log. trx_undo_report_row_operation(): Simplify some code. Always assign TRX_UNDO_INSERT for temporary undo logs. trx_prepare_low(): Keep only one parameter. Prepare all 3 undo logs. trx_roll_try_truncate(): Remove the parameter undo_ptr. Try to truncate all 3 undo logs of the transaction. trx_roll_pop_top_rec_of_trx_low(): Remove. trx_roll_pop_top_rec_of_trx(): Remove the redundant parameter trx->roll_limit. Clear roll_limit when exhausting the undo logs. Consider all 3 undo logs at once, prioritizing the persistent undo logs. row_undo(): Minor cleanup. Let trx_roll_pop_top_rec_of_trx() reset the trx->roll_limit.	9 years ago
Marko Mäkelä	7a30d86e9d	Simplify InnoDB startup. InnoDB needs to collect transactions from the persistent data files in trx_rseg_array_init() before trx_lists_init_at_db_start() is executed. But there is no need to create purge_sys->purge_queue separately from purge_sys. trx_sys_init_at_db_start(): Change the return type to void. Remove the direct call to trx_rseg_array_init(). It will be called by trx_lists_init_at_db_start(), which we are calling. Initialize the purge system read view. trx_lists_init_at_db_start(): Call trx_purge_sys_create(), which will invoke trx_rseg_array_init() to read the undo log segments. trx_purge_sys_create(): Remove the parameters. Do not initialize the purge system read view, because trx_sys->rw_trx_list has not been recovered yet. The purge_sys->view will be initialized at the end of trx_sys_init_at_db_start(). trx_rseg_array_init(): Remove the parameter. Use purge_sys->purge_queue directly. innobase_start_or_create_for_mysql(): Remove the local variable purge_queue. Do not call trx_purge_sys_create(), because it will be called by trx_sys_init_at_db_start().	9 years ago
Marko Mäkelä	89d80c1b0b	Fix many -Wconversion warnings. Define my_thread_id as an unsigned type, to avoid mismatch with ulonglong. Change some parameters to this type. Use size_t in a few more places. Declare many flag constants as unsigned to avoid sign mismatch when shifting bits or applying the unary ~ operator. When applying the unary ~ operator to enum constants, explictly cast the result to an unsigned type, because enum constants can be treated as signed. In InnoDB, change the source code line number parameters from ulint to unsigned type. Also, make some InnoDB functions return a narrower type (unsigned or uint32_t instead of ulint; bool instead of ibool).	9 years ago
Marko Mäkelä	27b9989d31	MDEV-12121 Introduce build option WITH_INNODB_AHI to disable innodb_adaptive_hash_index The InnoDB adaptive hash index is sometimes degrading the performance of InnoDB, and it is sometimes disabled to get more consistent performance. We should have a compile-time option to disable the adaptive hash index. Let us introduce two options: OPTION(WITH_INNODB_AHI "Include innodb_adaptive_hash_index" ON) OPTION(WITH_INNODB_ROOT_GUESS "Cache index root block descriptors" ON) where WITH_INNODB_AHI always implies WITH_INNODB_ROOT_GUESS. As part of this change, the misleadingly named function trx_search_latch_release_if_reserved(trx) will be replaced with the macro trx_assert_no_search_latch(trx) that will be empty unless BTR_CUR_HASH_ADAPT is defined (cmake -DWITH_INNODB_AHI=ON). We will also remove the unused column INFORMATION_SCHEMA.INNODB_TRX.TRX_ADAPTIVE_HASH_TIMEOUT. In MariaDB Server 10.1, it used to reflect the value of trx_t::search_latch_timeout which could be adjusted during row_search_for_mysql(). In 10.2, there is no such field. Other than the removal of the unused column TRX_ADAPTIVE_HASH_TIMEOUT, this is an almost non-functional change to the server when using the default build options. Some tests are adjusted so that they will work with both -DWITH_INNODB_AHI=ON and -DWITH_INNODB_AHI=OFF. The test innodb.innodb_monitor has been renamed to innodb.monitor in order to track MySQL 5.7, and the duplicate tests sys_vars.innodb_monitor_* are removed.	9 years ago
Marko Mäkelä	588eca31e3	Post-fix for MDEV-12050 Remove unused InnoDB Memcached hooks Remove also the field trx_t::read_write that was only used by the Memcached hooks.	9 years ago

1 2 3

131 Commits (131d9a5d0cb51fa88a0461c939dfd2a7c68d5664)