You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

1496 lines
52 KiB

17 years ago
9 years ago
17 years ago
17 years ago
16 years ago
15 years ago
15 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
Merge Google encryption commit 195158e9889365dc3298f8c1f3bcaa745992f27f Author: Minli Zhu <minliz@google.com> Date: Mon Nov 25 11:05:55 2013 -0800 Innodb redo log encryption/decryption. Use start lsn of a log block as part of AES CTR counter. Record key version with each checkpoint. Internally key version 0 means no encryption. Tests done (see test_innodb_log_encryption.sh for detail): - Verify flag innodb_encrypt_log on or off, combined with various key versions passed through CLI, and dynamically set after startup, will not corrupt database. This includes tests from being unencrypted to encrypted, and encrypted to unencrypted. - Verify start-up with no redo logs succeeds. - Verify fresh start-up succeeds. Change-Id: I4ce4c2afdf3076be2fce90ebbc2a7ce01184b612 commit c1b97273659f07866758c25f4a56f680a1fbad24 Author: Jonas Oreland <jonaso@google.com> Date: Tue Dec 3 18:47:27 2013 +0100 encryption of aria data&index files this patch implements encryption of aria data & index files. this is implemented as 1) add read/write hooks (renamed from callbacks) that does encrypt/decrypt (also add pre_read and post_write hooks) 2) modify page headers for data/index to contain key version (making the data-page header size different for with/without encryption) 3) modify index page 0 to contain IV (and crypt header) 4) AES CRT crypt functions 5) counter block is implemented using combination of page no, lsn and table specific id NOTE: 1) log files are not encrypted, this is not needed for if aria is only used for internal temporary tables and they are not transactional (i.e not logged) 2) all encrypted tables are using PAGE_CHECKSUM (crc) normal internal temporary tables are (currently) not CHECKSUM:ed 3) This patch adds insert-order semantics to aria block_format. The default behaviour of aria block-format is best-fit, meaning that rows gets allocated to page trying to fill the pages as much as possible. However, certain sql constructs materialize temporary result in tmp-tables, and expect that a table scan will later return the rows in the same order they were inserted. This implementation of insert-order is only enabled when explicitly requested by sql-layer. CHANGES: 1) found bug in ma_write that made code try to abort a record that was never written unsure why this is not exposed Change-Id: Ia82bbaa92e2c0629c08693c5add2f56b815c0509 commit 89dc1ab651fe0205d55b4eb588f62df550aa65fc Author: Jonas Oreland <jonaso@google.com> Date: Mon Feb 17 08:04:50 2014 -0800 Implement encryption of innodb datafiles. Pages are encrypted before written to disk and decrypted when read from disk. Each page except first page (page 0) in tablespace is encrypted. Page 0 is unencrypted and contains IV for the tablespace. FIL_PAGE_FILE_FLUSH_LSN on each page (except page 0) is used to store a 32-bit key-version, so that multiple keys can be active in a tablespace simultaneous. The other 32-bit of the FIL_PAGE_FILE_FLUSH_LSN field contains a checksum that is computed after encryption. This checksum is used by innochecksum and when restoring from double-write-buffer. The encryption is performed using AES CRT. Monitoring of encryption is enabled using new IS-table INNODB_TABLESPACES_ENCRYPTION. In addition to that new status variables innodb_encryption_rotation_{ pages_read_from_cache, pages_read_from_disk, pages_modified,pages_flushed } has been added. The following tunables are introduces - innodb_encrypt_tables - innodb_encryption_threads - innodb_encryption_rotate_key_age - innodb_encryption_rotation_iops Change-Id: I8f651795a30b52e71b16d6bc9cb7559be349d0b2 commit a17eef2f6948e58219c9e26fc35633d6fd4de1de Author: Andrew Ford <andrewford@google.com> Date: Thu Jan 2 15:43:09 2014 -0800 Key management skeleton with debug hooks. Change-Id: Ifd6aa3743d7ea291c70083f433a059c439aed866 commit 68a399838ad72264fd61b3dc67fecd29bbdb0af1 Author: Andrew Ford <andrewford@google.com> Date: Mon Oct 28 16:27:44 2013 -0700 Add AES-128 CTR and GCM encryption classes. Change-Id: I116305eced2a233db15306bc2ef5b9d398d1a3a2
11 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
MDEV-11738: Mariadb uses 100% of several of my 8 cpus doing nothing MDEV-11581: Mariadb starts InnoDB encryption threads when key has not changed or data scrubbing turned off Background: Key rotation is based on background threads (innodb-encryption-threads) periodically going through all tablespaces on fil_system. For each tablespace current used key version is compared to max key age (innodb-encryption-rotate-key-age). This process naturally takes CPU. Similarly, in same time need for scrubbing is investigated. Currently, key rotation is fully supported on Amazon AWS key management plugin only but InnoDB does not have knowledge what key management plugin is used. This patch re-purposes innodb-encryption-rotate-key-age=0 to disable key rotation and background data scrubbing. All new tables are added to special list for key rotation and key rotation is based on sending a event to background encryption threads instead of using periodic checking (i.e. timeout). fil0fil.cc: Added functions fil_space_acquire_low() to acquire a tablespace when it could be dropped concurrently. This function is used from fil_space_acquire() or fil_space_acquire_silent() that will not print any messages if we try to acquire space that does not exist. fil_space_release() to release a acquired tablespace. fil_space_next() to iterate tablespaces in fil_system using fil_space_acquire() and fil_space_release(). Similarly, fil_space_keyrotation_next() to iterate new list fil_system->rotation_list where new tables. are added if key rotation is disabled. Removed unnecessary functions fil_get_first_space_safe() fil_get_next_space_safe() fil_node_open_file(): After page 0 is read read also crypt_info if it is not yet read. btr_scrub_lock_dict_func() buf_page_check_corrupt() buf_page_encrypt_before_write() buf_merge_or_delete_for_page() lock_print_info_all_transactions() row_fts_psort_info_init() row_truncate_table_for_mysql() row_drop_table_for_mysql() Use fil_space_acquire()/release() to access fil_space_t. buf_page_decrypt_after_read(): Use fil_space_get_crypt_data() because at this point we might not yet have read page 0. fil0crypt.cc/fil0fil.h: Lot of changes. Pass fil_space_t* directly to functions needing it and store fil_space_t* to rotation state. Use fil_space_acquire()/release() when iterating tablespaces and removed unnecessary is_closing from fil_crypt_t. Use fil_space_t::is_stopping() to detect when access to tablespace should be stopped. Removed unnecessary fil_space_get_crypt_data(). fil_space_create(): Inform key rotation that there could be something to do if key rotation is disabled and new table with encryption enabled is created. Remove unnecessary functions fil_get_first_space_safe() and fil_get_next_space_safe(). fil_space_acquire() and fil_space_release() are used instead. Moved fil_space_get_crypt_data() and fil_space_set_crypt_data() to fil0crypt.cc. fsp_header_init(): Acquire fil_space_t*, write crypt_data and release space. check_table_options() Renamed FIL_SPACE_ENCRYPTION_* TO FIL_ENCRYPTION_* i_s.cc: Added ROTATING_OR_FLUSHING field to information_schema.innodb_tablespace_encryption to show current status of key rotation.
9 years ago
16 years ago
16 years ago
16 years ago
Merge Google encryption commit 195158e9889365dc3298f8c1f3bcaa745992f27f Author: Minli Zhu <minliz@google.com> Date: Mon Nov 25 11:05:55 2013 -0800 Innodb redo log encryption/decryption. Use start lsn of a log block as part of AES CTR counter. Record key version with each checkpoint. Internally key version 0 means no encryption. Tests done (see test_innodb_log_encryption.sh for detail): - Verify flag innodb_encrypt_log on or off, combined with various key versions passed through CLI, and dynamically set after startup, will not corrupt database. This includes tests from being unencrypted to encrypted, and encrypted to unencrypted. - Verify start-up with no redo logs succeeds. - Verify fresh start-up succeeds. Change-Id: I4ce4c2afdf3076be2fce90ebbc2a7ce01184b612 commit c1b97273659f07866758c25f4a56f680a1fbad24 Author: Jonas Oreland <jonaso@google.com> Date: Tue Dec 3 18:47:27 2013 +0100 encryption of aria data&index files this patch implements encryption of aria data & index files. this is implemented as 1) add read/write hooks (renamed from callbacks) that does encrypt/decrypt (also add pre_read and post_write hooks) 2) modify page headers for data/index to contain key version (making the data-page header size different for with/without encryption) 3) modify index page 0 to contain IV (and crypt header) 4) AES CRT crypt functions 5) counter block is implemented using combination of page no, lsn and table specific id NOTE: 1) log files are not encrypted, this is not needed for if aria is only used for internal temporary tables and they are not transactional (i.e not logged) 2) all encrypted tables are using PAGE_CHECKSUM (crc) normal internal temporary tables are (currently) not CHECKSUM:ed 3) This patch adds insert-order semantics to aria block_format. The default behaviour of aria block-format is best-fit, meaning that rows gets allocated to page trying to fill the pages as much as possible. However, certain sql constructs materialize temporary result in tmp-tables, and expect that a table scan will later return the rows in the same order they were inserted. This implementation of insert-order is only enabled when explicitly requested by sql-layer. CHANGES: 1) found bug in ma_write that made code try to abort a record that was never written unsure why this is not exposed Change-Id: Ia82bbaa92e2c0629c08693c5add2f56b815c0509 commit 89dc1ab651fe0205d55b4eb588f62df550aa65fc Author: Jonas Oreland <jonaso@google.com> Date: Mon Feb 17 08:04:50 2014 -0800 Implement encryption of innodb datafiles. Pages are encrypted before written to disk and decrypted when read from disk. Each page except first page (page 0) in tablespace is encrypted. Page 0 is unencrypted and contains IV for the tablespace. FIL_PAGE_FILE_FLUSH_LSN on each page (except page 0) is used to store a 32-bit key-version, so that multiple keys can be active in a tablespace simultaneous. The other 32-bit of the FIL_PAGE_FILE_FLUSH_LSN field contains a checksum that is computed after encryption. This checksum is used by innochecksum and when restoring from double-write-buffer. The encryption is performed using AES CRT. Monitoring of encryption is enabled using new IS-table INNODB_TABLESPACES_ENCRYPTION. In addition to that new status variables innodb_encryption_rotation_{ pages_read_from_cache, pages_read_from_disk, pages_modified,pages_flushed } has been added. The following tunables are introduces - innodb_encrypt_tables - innodb_encryption_threads - innodb_encryption_rotate_key_age - innodb_encryption_rotation_iops Change-Id: I8f651795a30b52e71b16d6bc9cb7559be349d0b2 commit a17eef2f6948e58219c9e26fc35633d6fd4de1de Author: Andrew Ford <andrewford@google.com> Date: Thu Jan 2 15:43:09 2014 -0800 Key management skeleton with debug hooks. Change-Id: Ifd6aa3743d7ea291c70083f433a059c439aed866 commit 68a399838ad72264fd61b3dc67fecd29bbdb0af1 Author: Andrew Ford <andrewford@google.com> Date: Mon Oct 28 16:27:44 2013 -0700 Add AES-128 CTR and GCM encryption classes. Change-Id: I116305eced2a233db15306bc2ef5b9d398d1a3a2
11 years ago
MDEV-11738: Mariadb uses 100% of several of my 8 cpus doing nothing MDEV-11581: Mariadb starts InnoDB encryption threads when key has not changed or data scrubbing turned off Background: Key rotation is based on background threads (innodb-encryption-threads) periodically going through all tablespaces on fil_system. For each tablespace current used key version is compared to max key age (innodb-encryption-rotate-key-age). This process naturally takes CPU. Similarly, in same time need for scrubbing is investigated. Currently, key rotation is fully supported on Amazon AWS key management plugin only but InnoDB does not have knowledge what key management plugin is used. This patch re-purposes innodb-encryption-rotate-key-age=0 to disable key rotation and background data scrubbing. All new tables are added to special list for key rotation and key rotation is based on sending a event to background encryption threads instead of using periodic checking (i.e. timeout). fil0fil.cc: Added functions fil_space_acquire_low() to acquire a tablespace when it could be dropped concurrently. This function is used from fil_space_acquire() or fil_space_acquire_silent() that will not print any messages if we try to acquire space that does not exist. fil_space_release() to release a acquired tablespace. fil_space_next() to iterate tablespaces in fil_system using fil_space_acquire() and fil_space_release(). Similarly, fil_space_keyrotation_next() to iterate new list fil_system->rotation_list where new tables. are added if key rotation is disabled. Removed unnecessary functions fil_get_first_space_safe() fil_get_next_space_safe() fil_node_open_file(): After page 0 is read read also crypt_info if it is not yet read. btr_scrub_lock_dict_func() buf_page_check_corrupt() buf_page_encrypt_before_write() buf_merge_or_delete_for_page() lock_print_info_all_transactions() row_fts_psort_info_init() row_truncate_table_for_mysql() row_drop_table_for_mysql() Use fil_space_acquire()/release() to access fil_space_t. buf_page_decrypt_after_read(): Use fil_space_get_crypt_data() because at this point we might not yet have read page 0. fil0crypt.cc/fil0fil.h: Lot of changes. Pass fil_space_t* directly to functions needing it and store fil_space_t* to rotation state. Use fil_space_acquire()/release() when iterating tablespaces and removed unnecessary is_closing from fil_crypt_t. Use fil_space_t::is_stopping() to detect when access to tablespace should be stopped. Removed unnecessary fil_space_get_crypt_data(). fil_space_create(): Inform key rotation that there could be something to do if key rotation is disabled and new table with encryption enabled is created. Remove unnecessary functions fil_get_first_space_safe() and fil_get_next_space_safe(). fil_space_acquire() and fil_space_release() are used instead. Moved fil_space_get_crypt_data() and fil_space_set_crypt_data() to fil0crypt.cc. fsp_header_init(): Acquire fil_space_t*, write crypt_data and release space. check_table_options() Renamed FIL_SPACE_ENCRYPTION_* TO FIL_ENCRYPTION_* i_s.cc: Added ROTATING_OR_FLUSHING field to information_schema.innodb_tablespace_encryption to show current status of key rotation.
9 years ago
MDEV-11556 InnoDB redo log apply fails to adjust data file sizes fil_space_t::recv_size: New member: recovered tablespace size in pages; 0 if no size change was read from the redo log, or if the size change was implemented. fil_space_set_recv_size(): New function for setting space->recv_size. innodb_data_file_size_debug: A debug parameter for setting the system tablespace size in recovery even when the redo log does not contain any size changes. It is hard to write a small test case that would cause the system tablespace to be extended at the critical moment. recv_parse_log_rec(): Note those tablespaces whose size is being changed by the redo log, by invoking fil_space_set_recv_size(). innobase_init(): Correct an error message, and do not require a larger innodb_buffer_pool_size when starting up with a smaller innodb_page_size. innobase_start_or_create_for_mysql(): Allow startup with any initial size of the ibdata1 file if the autoextend attribute is set. Require the minimum size of fixed-size system tablespaces to be 640 pages, not 10 megabytes. Implement innodb_data_file_size_debug. open_or_create_data_files(): Round the system tablespace size down to pages, not to full megabytes, (Our test truncates the system tablespace to more than 800 pages with innodb_page_size=4k. InnoDB should not imagine that it was truncated to 768 pages and then overwrite good pages in the tablespace.) fil_flush_low(): Refactored from fil_flush(). fil_space_extend_must_retry(): Refactored from fil_extend_space_to_desired_size(). fil_mutex_enter_and_prepare_for_io(): Extend the tablespace if fil_space_set_recv_size() was called. The test case has been successfully run with all the innodb_page_size values 4k, 8k, 16k, 32k, 64k.
9 years ago
MDEV-11623 MariaDB 10.1 fails to start datadir created with MariaDB 10.0/MySQL 5.6 using innodb-page-size!=16K The storage format of FSP_SPACE_FLAGS was accidentally broken already in MariaDB 10.1.0. This fix is bringing the format in line with other MySQL and MariaDB release series. Please refer to the comments that were added to fsp0fsp.h for details. This is an INCOMPATIBLE CHANGE that affects users of page_compression and non-default innodb_page_size. Upgrading to this release will correct the flags in the data files. If you want to downgrade to earlier MariaDB 10.1.x, please refer to the test innodb.101_compatibility how to reset the FSP_SPACE_FLAGS in the files. NOTE: MariaDB 10.1.0 to 10.1.20 can misinterpret uncompressed data files with innodb_page_size=4k or 64k as compressed innodb_page_size=16k files, and then probably fail when trying to access the pages. See the comments in the function fsp_flags_convert_from_101() for detailed analysis. Move PAGE_COMPRESSION to FSP_SPACE_FLAGS bit position 16. In this way, compressed innodb_page_size=16k tablespaces will not be mistaken for uncompressed ones by MariaDB 10.1.0 to 10.1.20. Derive PAGE_COMPRESSION_LEVEL, ATOMIC_WRITES and DATA_DIR from the dict_table_t::flags when the table is available, in fil_space_for_table_exists_in_mem() or fil_open_single_table_tablespace(). During crash recovery, fil_load_single_table_tablespace() will use innodb_compression_level for the PAGE_COMPRESSION_LEVEL. FSP_FLAGS_MEM_MASK: A bitmap of the memory-only fil_space_t::flags that are not to be written to FSP_SPACE_FLAGS. Currently, these will include PAGE_COMPRESSION_LEVEL, ATOMIC_WRITES and DATA_DIR. Introduce the macro FSP_FLAGS_PAGE_SSIZE(). We only support one innodb_page_size for the whole instance. When creating a dummy tablespace for the redo log, use fil_space_t::flags=0. The flags are never written to the redo log files. Remove many FSP_FLAGS_SET_ macros. dict_tf_verify_flags(): Remove. This is basically only duplicating the logic of dict_tf_to_fsp_flags(), used in a debug assertion. fil_space_t::mark: Remove. This flag was not used for anything. fil_space_for_table_exists_in_mem(): Remove the unnecessary parameter mark_space, and add a parameter for table flags. Check that fil_space_t::flags match the table flags, and adjust the (memory-only) flags based on the table flags. fil_node_open_file(): Remove some redundant or unreachable conditions, do not use stderr for output, and avoid unnecessary server aborts. fil_user_tablespace_restore_page(): Convert the flags, so that the correct page_size will be used when restoring a page from the doublewrite buffer. fil_space_get_page_compressed(), fsp_flags_is_page_compressed(): Remove. It suffices to have fil_space_is_page_compressed(). FSP_FLAGS_WIDTH_DATA_DIR, FSP_FLAGS_WIDTH_PAGE_COMPRESSION_LEVEL, FSP_FLAGS_WIDTH_ATOMIC_WRITES: Remove, because these flags do not exist in the FSP_SPACE_FLAGS but only in memory. fsp_flags_try_adjust(): New function, to adjust the FSP_SPACE_FLAGS in page 0. Called by fil_open_single_table_tablespace(), fil_space_for_table_exists_in_mem(), innobase_start_or_create_for_mysql() except if --innodb-read-only is active. fsp_flags_is_valid(ulint): Reimplement from the scratch, with accurate comments. Do not display any details of detected inconsistencies, because the output could be confusing when dealing with MariaDB 10.1.x data files. fsp_flags_convert_from_101(ulint): Convert flags from buggy MariaDB 10.1.x format, or return ULINT_UNDEFINED if the flags cannot be in MariaDB 10.1.x format. fsp_flags_match(): Check the flags when probing files. Implemented based on fsp_flags_is_valid() and fsp_flags_convert_from_101(). dict_check_tablespaces_and_store_max_id(): Do not access the page after committing the mini-transaction. IMPORT TABLESPACE fixes: AbstractCallback::init(): Convert the flags. FetchIndexRootPages::operator(): Check that the tablespace flags match the table flags. Do not attempt to convert tablespace flags to table flags, because the conversion would necessarily be lossy. PageConverter::update_header(): Write back the correct flags. This takes care of the flags in IMPORT TABLESPACE.
9 years ago
MDEV-11738: Mariadb uses 100% of several of my 8 cpus doing nothing MDEV-11581: Mariadb starts InnoDB encryption threads when key has not changed or data scrubbing turned off Background: Key rotation is based on background threads (innodb-encryption-threads) periodically going through all tablespaces on fil_system. For each tablespace current used key version is compared to max key age (innodb-encryption-rotate-key-age). This process naturally takes CPU. Similarly, in same time need for scrubbing is investigated. Currently, key rotation is fully supported on Amazon AWS key management plugin only but InnoDB does not have knowledge what key management plugin is used. This patch re-purposes innodb-encryption-rotate-key-age=0 to disable key rotation and background data scrubbing. All new tables are added to special list for key rotation and key rotation is based on sending a event to background encryption threads instead of using periodic checking (i.e. timeout). fil0fil.cc: Added functions fil_space_acquire_low() to acquire a tablespace when it could be dropped concurrently. This function is used from fil_space_acquire() or fil_space_acquire_silent() that will not print any messages if we try to acquire space that does not exist. fil_space_release() to release a acquired tablespace. fil_space_next() to iterate tablespaces in fil_system using fil_space_acquire() and fil_space_release(). Similarly, fil_space_keyrotation_next() to iterate new list fil_system->rotation_list where new tables. are added if key rotation is disabled. Removed unnecessary functions fil_get_first_space_safe() fil_get_next_space_safe() fil_node_open_file(): After page 0 is read read also crypt_info if it is not yet read. btr_scrub_lock_dict_func() buf_page_check_corrupt() buf_page_encrypt_before_write() buf_merge_or_delete_for_page() lock_print_info_all_transactions() row_fts_psort_info_init() row_truncate_table_for_mysql() row_drop_table_for_mysql() Use fil_space_acquire()/release() to access fil_space_t. buf_page_decrypt_after_read(): Use fil_space_get_crypt_data() because at this point we might not yet have read page 0. fil0crypt.cc/fil0fil.h: Lot of changes. Pass fil_space_t* directly to functions needing it and store fil_space_t* to rotation state. Use fil_space_acquire()/release() when iterating tablespaces and removed unnecessary is_closing from fil_crypt_t. Use fil_space_t::is_stopping() to detect when access to tablespace should be stopped. Removed unnecessary fil_space_get_crypt_data(). fil_space_create(): Inform key rotation that there could be something to do if key rotation is disabled and new table with encryption enabled is created. Remove unnecessary functions fil_get_first_space_safe() and fil_get_next_space_safe(). fil_space_acquire() and fil_space_release() are used instead. Moved fil_space_get_crypt_data() and fil_space_set_crypt_data() to fil0crypt.cc. fsp_header_init(): Acquire fil_space_t*, write crypt_data and release space. check_table_options() Renamed FIL_SPACE_ENCRYPTION_* TO FIL_ENCRYPTION_* i_s.cc: Added ROTATING_OR_FLUSHING field to information_schema.innodb_tablespace_encryption to show current status of key rotation.
9 years ago
MDEV-11738: Mariadb uses 100% of several of my 8 cpus doing nothing MDEV-11581: Mariadb starts InnoDB encryption threads when key has not changed or data scrubbing turned off Background: Key rotation is based on background threads (innodb-encryption-threads) periodically going through all tablespaces on fil_system. For each tablespace current used key version is compared to max key age (innodb-encryption-rotate-key-age). This process naturally takes CPU. Similarly, in same time need for scrubbing is investigated. Currently, key rotation is fully supported on Amazon AWS key management plugin only but InnoDB does not have knowledge what key management plugin is used. This patch re-purposes innodb-encryption-rotate-key-age=0 to disable key rotation and background data scrubbing. All new tables are added to special list for key rotation and key rotation is based on sending a event to background encryption threads instead of using periodic checking (i.e. timeout). fil0fil.cc: Added functions fil_space_acquire_low() to acquire a tablespace when it could be dropped concurrently. This function is used from fil_space_acquire() or fil_space_acquire_silent() that will not print any messages if we try to acquire space that does not exist. fil_space_release() to release a acquired tablespace. fil_space_next() to iterate tablespaces in fil_system using fil_space_acquire() and fil_space_release(). Similarly, fil_space_keyrotation_next() to iterate new list fil_system->rotation_list where new tables. are added if key rotation is disabled. Removed unnecessary functions fil_get_first_space_safe() fil_get_next_space_safe() fil_node_open_file(): After page 0 is read read also crypt_info if it is not yet read. btr_scrub_lock_dict_func() buf_page_check_corrupt() buf_page_encrypt_before_write() buf_merge_or_delete_for_page() lock_print_info_all_transactions() row_fts_psort_info_init() row_truncate_table_for_mysql() row_drop_table_for_mysql() Use fil_space_acquire()/release() to access fil_space_t. buf_page_decrypt_after_read(): Use fil_space_get_crypt_data() because at this point we might not yet have read page 0. fil0crypt.cc/fil0fil.h: Lot of changes. Pass fil_space_t* directly to functions needing it and store fil_space_t* to rotation state. Use fil_space_acquire()/release() when iterating tablespaces and removed unnecessary is_closing from fil_crypt_t. Use fil_space_t::is_stopping() to detect when access to tablespace should be stopped. Removed unnecessary fil_space_get_crypt_data(). fil_space_create(): Inform key rotation that there could be something to do if key rotation is disabled and new table with encryption enabled is created. Remove unnecessary functions fil_get_first_space_safe() and fil_get_next_space_safe(). fil_space_acquire() and fil_space_release() are used instead. Moved fil_space_get_crypt_data() and fil_space_set_crypt_data() to fil0crypt.cc. fsp_header_init(): Acquire fil_space_t*, write crypt_data and release space. check_table_options() Renamed FIL_SPACE_ENCRYPTION_* TO FIL_ENCRYPTION_* i_s.cc: Added ROTATING_OR_FLUSHING field to information_schema.innodb_tablespace_encryption to show current status of key rotation.
9 years ago
MDEV-11738: Mariadb uses 100% of several of my 8 cpus doing nothing MDEV-11581: Mariadb starts InnoDB encryption threads when key has not changed or data scrubbing turned off Background: Key rotation is based on background threads (innodb-encryption-threads) periodically going through all tablespaces on fil_system. For each tablespace current used key version is compared to max key age (innodb-encryption-rotate-key-age). This process naturally takes CPU. Similarly, in same time need for scrubbing is investigated. Currently, key rotation is fully supported on Amazon AWS key management plugin only but InnoDB does not have knowledge what key management plugin is used. This patch re-purposes innodb-encryption-rotate-key-age=0 to disable key rotation and background data scrubbing. All new tables are added to special list for key rotation and key rotation is based on sending a event to background encryption threads instead of using periodic checking (i.e. timeout). fil0fil.cc: Added functions fil_space_acquire_low() to acquire a tablespace when it could be dropped concurrently. This function is used from fil_space_acquire() or fil_space_acquire_silent() that will not print any messages if we try to acquire space that does not exist. fil_space_release() to release a acquired tablespace. fil_space_next() to iterate tablespaces in fil_system using fil_space_acquire() and fil_space_release(). Similarly, fil_space_keyrotation_next() to iterate new list fil_system->rotation_list where new tables. are added if key rotation is disabled. Removed unnecessary functions fil_get_first_space_safe() fil_get_next_space_safe() fil_node_open_file(): After page 0 is read read also crypt_info if it is not yet read. btr_scrub_lock_dict_func() buf_page_check_corrupt() buf_page_encrypt_before_write() buf_merge_or_delete_for_page() lock_print_info_all_transactions() row_fts_psort_info_init() row_truncate_table_for_mysql() row_drop_table_for_mysql() Use fil_space_acquire()/release() to access fil_space_t. buf_page_decrypt_after_read(): Use fil_space_get_crypt_data() because at this point we might not yet have read page 0. fil0crypt.cc/fil0fil.h: Lot of changes. Pass fil_space_t* directly to functions needing it and store fil_space_t* to rotation state. Use fil_space_acquire()/release() when iterating tablespaces and removed unnecessary is_closing from fil_crypt_t. Use fil_space_t::is_stopping() to detect when access to tablespace should be stopped. Removed unnecessary fil_space_get_crypt_data(). fil_space_create(): Inform key rotation that there could be something to do if key rotation is disabled and new table with encryption enabled is created. Remove unnecessary functions fil_get_first_space_safe() and fil_get_next_space_safe(). fil_space_acquire() and fil_space_release() are used instead. Moved fil_space_get_crypt_data() and fil_space_set_crypt_data() to fil0crypt.cc. fsp_header_init(): Acquire fil_space_t*, write crypt_data and release space. check_table_options() Renamed FIL_SPACE_ENCRYPTION_* TO FIL_ENCRYPTION_* i_s.cc: Added ROTATING_OR_FLUSHING field to information_schema.innodb_tablespace_encryption to show current status of key rotation.
9 years ago
MDEV-11738: Mariadb uses 100% of several of my 8 cpus doing nothing MDEV-11581: Mariadb starts InnoDB encryption threads when key has not changed or data scrubbing turned off Background: Key rotation is based on background threads (innodb-encryption-threads) periodically going through all tablespaces on fil_system. For each tablespace current used key version is compared to max key age (innodb-encryption-rotate-key-age). This process naturally takes CPU. Similarly, in same time need for scrubbing is investigated. Currently, key rotation is fully supported on Amazon AWS key management plugin only but InnoDB does not have knowledge what key management plugin is used. This patch re-purposes innodb-encryption-rotate-key-age=0 to disable key rotation and background data scrubbing. All new tables are added to special list for key rotation and key rotation is based on sending a event to background encryption threads instead of using periodic checking (i.e. timeout). fil0fil.cc: Added functions fil_space_acquire_low() to acquire a tablespace when it could be dropped concurrently. This function is used from fil_space_acquire() or fil_space_acquire_silent() that will not print any messages if we try to acquire space that does not exist. fil_space_release() to release a acquired tablespace. fil_space_next() to iterate tablespaces in fil_system using fil_space_acquire() and fil_space_release(). Similarly, fil_space_keyrotation_next() to iterate new list fil_system->rotation_list where new tables. are added if key rotation is disabled. Removed unnecessary functions fil_get_first_space_safe() fil_get_next_space_safe() fil_node_open_file(): After page 0 is read read also crypt_info if it is not yet read. btr_scrub_lock_dict_func() buf_page_check_corrupt() buf_page_encrypt_before_write() buf_merge_or_delete_for_page() lock_print_info_all_transactions() row_fts_psort_info_init() row_truncate_table_for_mysql() row_drop_table_for_mysql() Use fil_space_acquire()/release() to access fil_space_t. buf_page_decrypt_after_read(): Use fil_space_get_crypt_data() because at this point we might not yet have read page 0. fil0crypt.cc/fil0fil.h: Lot of changes. Pass fil_space_t* directly to functions needing it and store fil_space_t* to rotation state. Use fil_space_acquire()/release() when iterating tablespaces and removed unnecessary is_closing from fil_crypt_t. Use fil_space_t::is_stopping() to detect when access to tablespace should be stopped. Removed unnecessary fil_space_get_crypt_data(). fil_space_create(): Inform key rotation that there could be something to do if key rotation is disabled and new table with encryption enabled is created. Remove unnecessary functions fil_get_first_space_safe() and fil_get_next_space_safe(). fil_space_acquire() and fil_space_release() are used instead. Moved fil_space_get_crypt_data() and fil_space_set_crypt_data() to fil0crypt.cc. fsp_header_init(): Acquire fil_space_t*, write crypt_data and release space. check_table_options() Renamed FIL_SPACE_ENCRYPTION_* TO FIL_ENCRYPTION_* i_s.cc: Added ROTATING_OR_FLUSHING field to information_schema.innodb_tablespace_encryption to show current status of key rotation.
9 years ago
MDEV-11738: Mariadb uses 100% of several of my 8 cpus doing nothing MDEV-11581: Mariadb starts InnoDB encryption threads when key has not changed or data scrubbing turned off Background: Key rotation is based on background threads (innodb-encryption-threads) periodically going through all tablespaces on fil_system. For each tablespace current used key version is compared to max key age (innodb-encryption-rotate-key-age). This process naturally takes CPU. Similarly, in same time need for scrubbing is investigated. Currently, key rotation is fully supported on Amazon AWS key management plugin only but InnoDB does not have knowledge what key management plugin is used. This patch re-purposes innodb-encryption-rotate-key-age=0 to disable key rotation and background data scrubbing. All new tables are added to special list for key rotation and key rotation is based on sending a event to background encryption threads instead of using periodic checking (i.e. timeout). fil0fil.cc: Added functions fil_space_acquire_low() to acquire a tablespace when it could be dropped concurrently. This function is used from fil_space_acquire() or fil_space_acquire_silent() that will not print any messages if we try to acquire space that does not exist. fil_space_release() to release a acquired tablespace. fil_space_next() to iterate tablespaces in fil_system using fil_space_acquire() and fil_space_release(). Similarly, fil_space_keyrotation_next() to iterate new list fil_system->rotation_list where new tables. are added if key rotation is disabled. Removed unnecessary functions fil_get_first_space_safe() fil_get_next_space_safe() fil_node_open_file(): After page 0 is read read also crypt_info if it is not yet read. btr_scrub_lock_dict_func() buf_page_check_corrupt() buf_page_encrypt_before_write() buf_merge_or_delete_for_page() lock_print_info_all_transactions() row_fts_psort_info_init() row_truncate_table_for_mysql() row_drop_table_for_mysql() Use fil_space_acquire()/release() to access fil_space_t. buf_page_decrypt_after_read(): Use fil_space_get_crypt_data() because at this point we might not yet have read page 0. fil0crypt.cc/fil0fil.h: Lot of changes. Pass fil_space_t* directly to functions needing it and store fil_space_t* to rotation state. Use fil_space_acquire()/release() when iterating tablespaces and removed unnecessary is_closing from fil_crypt_t. Use fil_space_t::is_stopping() to detect when access to tablespace should be stopped. Removed unnecessary fil_space_get_crypt_data(). fil_space_create(): Inform key rotation that there could be something to do if key rotation is disabled and new table with encryption enabled is created. Remove unnecessary functions fil_get_first_space_safe() and fil_get_next_space_safe(). fil_space_acquire() and fil_space_release() are used instead. Moved fil_space_get_crypt_data() and fil_space_set_crypt_data() to fil0crypt.cc. fsp_header_init(): Acquire fil_space_t*, write crypt_data and release space. check_table_options() Renamed FIL_SPACE_ENCRYPTION_* TO FIL_ENCRYPTION_* i_s.cc: Added ROTATING_OR_FLUSHING field to information_schema.innodb_tablespace_encryption to show current status of key rotation.
9 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
9 years ago
MDEV-11623 MariaDB 10.1 fails to start datadir created with MariaDB 10.0/MySQL 5.6 using innodb-page-size!=16K The storage format of FSP_SPACE_FLAGS was accidentally broken already in MariaDB 10.1.0. This fix is bringing the format in line with other MySQL and MariaDB release series. Please refer to the comments that were added to fsp0fsp.h for details. This is an INCOMPATIBLE CHANGE that affects users of page_compression and non-default innodb_page_size. Upgrading to this release will correct the flags in the data files. If you want to downgrade to earlier MariaDB 10.1.x, please refer to the test innodb.101_compatibility how to reset the FSP_SPACE_FLAGS in the files. NOTE: MariaDB 10.1.0 to 10.1.20 can misinterpret uncompressed data files with innodb_page_size=4k or 64k as compressed innodb_page_size=16k files, and then probably fail when trying to access the pages. See the comments in the function fsp_flags_convert_from_101() for detailed analysis. Move PAGE_COMPRESSION to FSP_SPACE_FLAGS bit position 16. In this way, compressed innodb_page_size=16k tablespaces will not be mistaken for uncompressed ones by MariaDB 10.1.0 to 10.1.20. Derive PAGE_COMPRESSION_LEVEL, ATOMIC_WRITES and DATA_DIR from the dict_table_t::flags when the table is available, in fil_space_for_table_exists_in_mem() or fil_open_single_table_tablespace(). During crash recovery, fil_load_single_table_tablespace() will use innodb_compression_level for the PAGE_COMPRESSION_LEVEL. FSP_FLAGS_MEM_MASK: A bitmap of the memory-only fil_space_t::flags that are not to be written to FSP_SPACE_FLAGS. Currently, these will include PAGE_COMPRESSION_LEVEL, ATOMIC_WRITES and DATA_DIR. Introduce the macro FSP_FLAGS_PAGE_SSIZE(). We only support one innodb_page_size for the whole instance. When creating a dummy tablespace for the redo log, use fil_space_t::flags=0. The flags are never written to the redo log files. Remove many FSP_FLAGS_SET_ macros. dict_tf_verify_flags(): Remove. This is basically only duplicating the logic of dict_tf_to_fsp_flags(), used in a debug assertion. fil_space_t::mark: Remove. This flag was not used for anything. fil_space_for_table_exists_in_mem(): Remove the unnecessary parameter mark_space, and add a parameter for table flags. Check that fil_space_t::flags match the table flags, and adjust the (memory-only) flags based on the table flags. fil_node_open_file(): Remove some redundant or unreachable conditions, do not use stderr for output, and avoid unnecessary server aborts. fil_user_tablespace_restore_page(): Convert the flags, so that the correct page_size will be used when restoring a page from the doublewrite buffer. fil_space_get_page_compressed(), fsp_flags_is_page_compressed(): Remove. It suffices to have fil_space_is_page_compressed(). FSP_FLAGS_WIDTH_DATA_DIR, FSP_FLAGS_WIDTH_PAGE_COMPRESSION_LEVEL, FSP_FLAGS_WIDTH_ATOMIC_WRITES: Remove, because these flags do not exist in the FSP_SPACE_FLAGS but only in memory. fsp_flags_try_adjust(): New function, to adjust the FSP_SPACE_FLAGS in page 0. Called by fil_open_single_table_tablespace(), fil_space_for_table_exists_in_mem(), innobase_start_or_create_for_mysql() except if --innodb-read-only is active. fsp_flags_is_valid(ulint): Reimplement from the scratch, with accurate comments. Do not display any details of detected inconsistencies, because the output could be confusing when dealing with MariaDB 10.1.x data files. fsp_flags_convert_from_101(ulint): Convert flags from buggy MariaDB 10.1.x format, or return ULINT_UNDEFINED if the flags cannot be in MariaDB 10.1.x format. fsp_flags_match(): Check the flags when probing files. Implemented based on fsp_flags_is_valid() and fsp_flags_convert_from_101(). dict_check_tablespaces_and_store_max_id(): Do not access the page after committing the mini-transaction. IMPORT TABLESPACE fixes: AbstractCallback::init(): Convert the flags. FetchIndexRootPages::operator(): Check that the tablespace flags match the table flags. Do not attempt to convert tablespace flags to table flags, because the conversion would necessarily be lossy. PageConverter::update_header(): Write back the correct flags. This takes care of the flags in IMPORT TABLESPACE.
9 years ago
16 years ago
16 years ago
16 years ago
MDEV-11738: Mariadb uses 100% of several of my 8 cpus doing nothing MDEV-11581: Mariadb starts InnoDB encryption threads when key has not changed or data scrubbing turned off Background: Key rotation is based on background threads (innodb-encryption-threads) periodically going through all tablespaces on fil_system. For each tablespace current used key version is compared to max key age (innodb-encryption-rotate-key-age). This process naturally takes CPU. Similarly, in same time need for scrubbing is investigated. Currently, key rotation is fully supported on Amazon AWS key management plugin only but InnoDB does not have knowledge what key management plugin is used. This patch re-purposes innodb-encryption-rotate-key-age=0 to disable key rotation and background data scrubbing. All new tables are added to special list for key rotation and key rotation is based on sending a event to background encryption threads instead of using periodic checking (i.e. timeout). fil0fil.cc: Added functions fil_space_acquire_low() to acquire a tablespace when it could be dropped concurrently. This function is used from fil_space_acquire() or fil_space_acquire_silent() that will not print any messages if we try to acquire space that does not exist. fil_space_release() to release a acquired tablespace. fil_space_next() to iterate tablespaces in fil_system using fil_space_acquire() and fil_space_release(). Similarly, fil_space_keyrotation_next() to iterate new list fil_system->rotation_list where new tables. are added if key rotation is disabled. Removed unnecessary functions fil_get_first_space_safe() fil_get_next_space_safe() fil_node_open_file(): After page 0 is read read also crypt_info if it is not yet read. btr_scrub_lock_dict_func() buf_page_check_corrupt() buf_page_encrypt_before_write() buf_merge_or_delete_for_page() lock_print_info_all_transactions() row_fts_psort_info_init() row_truncate_table_for_mysql() row_drop_table_for_mysql() Use fil_space_acquire()/release() to access fil_space_t. buf_page_decrypt_after_read(): Use fil_space_get_crypt_data() because at this point we might not yet have read page 0. fil0crypt.cc/fil0fil.h: Lot of changes. Pass fil_space_t* directly to functions needing it and store fil_space_t* to rotation state. Use fil_space_acquire()/release() when iterating tablespaces and removed unnecessary is_closing from fil_crypt_t. Use fil_space_t::is_stopping() to detect when access to tablespace should be stopped. Removed unnecessary fil_space_get_crypt_data(). fil_space_create(): Inform key rotation that there could be something to do if key rotation is disabled and new table with encryption enabled is created. Remove unnecessary functions fil_get_first_space_safe() and fil_get_next_space_safe(). fil_space_acquire() and fil_space_release() are used instead. Moved fil_space_get_crypt_data() and fil_space_set_crypt_data() to fil0crypt.cc. fsp_header_init(): Acquire fil_space_t*, write crypt_data and release space. check_table_options() Renamed FIL_SPACE_ENCRYPTION_* TO FIL_ENCRYPTION_* i_s.cc: Added ROTATING_OR_FLUSHING field to information_schema.innodb_tablespace_encryption to show current status of key rotation.
9 years ago
16 years ago
MDEV-11738: Mariadb uses 100% of several of my 8 cpus doing nothing MDEV-11581: Mariadb starts InnoDB encryption threads when key has not changed or data scrubbing turned off Background: Key rotation is based on background threads (innodb-encryption-threads) periodically going through all tablespaces on fil_system. For each tablespace current used key version is compared to max key age (innodb-encryption-rotate-key-age). This process naturally takes CPU. Similarly, in same time need for scrubbing is investigated. Currently, key rotation is fully supported on Amazon AWS key management plugin only but InnoDB does not have knowledge what key management plugin is used. This patch re-purposes innodb-encryption-rotate-key-age=0 to disable key rotation and background data scrubbing. All new tables are added to special list for key rotation and key rotation is based on sending a event to background encryption threads instead of using periodic checking (i.e. timeout). fil0fil.cc: Added functions fil_space_acquire_low() to acquire a tablespace when it could be dropped concurrently. This function is used from fil_space_acquire() or fil_space_acquire_silent() that will not print any messages if we try to acquire space that does not exist. fil_space_release() to release a acquired tablespace. fil_space_next() to iterate tablespaces in fil_system using fil_space_acquire() and fil_space_release(). Similarly, fil_space_keyrotation_next() to iterate new list fil_system->rotation_list where new tables. are added if key rotation is disabled. Removed unnecessary functions fil_get_first_space_safe() fil_get_next_space_safe() fil_node_open_file(): After page 0 is read read also crypt_info if it is not yet read. btr_scrub_lock_dict_func() buf_page_check_corrupt() buf_page_encrypt_before_write() buf_merge_or_delete_for_page() lock_print_info_all_transactions() row_fts_psort_info_init() row_truncate_table_for_mysql() row_drop_table_for_mysql() Use fil_space_acquire()/release() to access fil_space_t. buf_page_decrypt_after_read(): Use fil_space_get_crypt_data() because at this point we might not yet have read page 0. fil0crypt.cc/fil0fil.h: Lot of changes. Pass fil_space_t* directly to functions needing it and store fil_space_t* to rotation state. Use fil_space_acquire()/release() when iterating tablespaces and removed unnecessary is_closing from fil_crypt_t. Use fil_space_t::is_stopping() to detect when access to tablespace should be stopped. Removed unnecessary fil_space_get_crypt_data(). fil_space_create(): Inform key rotation that there could be something to do if key rotation is disabled and new table with encryption enabled is created. Remove unnecessary functions fil_get_first_space_safe() and fil_get_next_space_safe(). fil_space_acquire() and fil_space_release() are used instead. Moved fil_space_get_crypt_data() and fil_space_set_crypt_data() to fil0crypt.cc. fsp_header_init(): Acquire fil_space_t*, write crypt_data and release space. check_table_options() Renamed FIL_SPACE_ENCRYPTION_* TO FIL_ENCRYPTION_* i_s.cc: Added ROTATING_OR_FLUSHING field to information_schema.innodb_tablespace_encryption to show current status of key rotation.
9 years ago
MDEV-11738: Mariadb uses 100% of several of my 8 cpus doing nothing MDEV-11581: Mariadb starts InnoDB encryption threads when key has not changed or data scrubbing turned off Background: Key rotation is based on background threads (innodb-encryption-threads) periodically going through all tablespaces on fil_system. For each tablespace current used key version is compared to max key age (innodb-encryption-rotate-key-age). This process naturally takes CPU. Similarly, in same time need for scrubbing is investigated. Currently, key rotation is fully supported on Amazon AWS key management plugin only but InnoDB does not have knowledge what key management plugin is used. This patch re-purposes innodb-encryption-rotate-key-age=0 to disable key rotation and background data scrubbing. All new tables are added to special list for key rotation and key rotation is based on sending a event to background encryption threads instead of using periodic checking (i.e. timeout). fil0fil.cc: Added functions fil_space_acquire_low() to acquire a tablespace when it could be dropped concurrently. This function is used from fil_space_acquire() or fil_space_acquire_silent() that will not print any messages if we try to acquire space that does not exist. fil_space_release() to release a acquired tablespace. fil_space_next() to iterate tablespaces in fil_system using fil_space_acquire() and fil_space_release(). Similarly, fil_space_keyrotation_next() to iterate new list fil_system->rotation_list where new tables. are added if key rotation is disabled. Removed unnecessary functions fil_get_first_space_safe() fil_get_next_space_safe() fil_node_open_file(): After page 0 is read read also crypt_info if it is not yet read. btr_scrub_lock_dict_func() buf_page_check_corrupt() buf_page_encrypt_before_write() buf_merge_or_delete_for_page() lock_print_info_all_transactions() row_fts_psort_info_init() row_truncate_table_for_mysql() row_drop_table_for_mysql() Use fil_space_acquire()/release() to access fil_space_t. buf_page_decrypt_after_read(): Use fil_space_get_crypt_data() because at this point we might not yet have read page 0. fil0crypt.cc/fil0fil.h: Lot of changes. Pass fil_space_t* directly to functions needing it and store fil_space_t* to rotation state. Use fil_space_acquire()/release() when iterating tablespaces and removed unnecessary is_closing from fil_crypt_t. Use fil_space_t::is_stopping() to detect when access to tablespace should be stopped. Removed unnecessary fil_space_get_crypt_data(). fil_space_create(): Inform key rotation that there could be something to do if key rotation is disabled and new table with encryption enabled is created. Remove unnecessary functions fil_get_first_space_safe() and fil_get_next_space_safe(). fil_space_acquire() and fil_space_release() are used instead. Moved fil_space_get_crypt_data() and fil_space_set_crypt_data() to fil0crypt.cc. fsp_header_init(): Acquire fil_space_t*, write crypt_data and release space. check_table_options() Renamed FIL_SPACE_ENCRYPTION_* TO FIL_ENCRYPTION_* i_s.cc: Added ROTATING_OR_FLUSHING field to information_schema.innodb_tablespace_encryption to show current status of key rotation.
9 years ago
Merge Google encryption commit 195158e9889365dc3298f8c1f3bcaa745992f27f Author: Minli Zhu <minliz@google.com> Date: Mon Nov 25 11:05:55 2013 -0800 Innodb redo log encryption/decryption. Use start lsn of a log block as part of AES CTR counter. Record key version with each checkpoint. Internally key version 0 means no encryption. Tests done (see test_innodb_log_encryption.sh for detail): - Verify flag innodb_encrypt_log on or off, combined with various key versions passed through CLI, and dynamically set after startup, will not corrupt database. This includes tests from being unencrypted to encrypted, and encrypted to unencrypted. - Verify start-up with no redo logs succeeds. - Verify fresh start-up succeeds. Change-Id: I4ce4c2afdf3076be2fce90ebbc2a7ce01184b612 commit c1b97273659f07866758c25f4a56f680a1fbad24 Author: Jonas Oreland <jonaso@google.com> Date: Tue Dec 3 18:47:27 2013 +0100 encryption of aria data&index files this patch implements encryption of aria data & index files. this is implemented as 1) add read/write hooks (renamed from callbacks) that does encrypt/decrypt (also add pre_read and post_write hooks) 2) modify page headers for data/index to contain key version (making the data-page header size different for with/without encryption) 3) modify index page 0 to contain IV (and crypt header) 4) AES CRT crypt functions 5) counter block is implemented using combination of page no, lsn and table specific id NOTE: 1) log files are not encrypted, this is not needed for if aria is only used for internal temporary tables and they are not transactional (i.e not logged) 2) all encrypted tables are using PAGE_CHECKSUM (crc) normal internal temporary tables are (currently) not CHECKSUM:ed 3) This patch adds insert-order semantics to aria block_format. The default behaviour of aria block-format is best-fit, meaning that rows gets allocated to page trying to fill the pages as much as possible. However, certain sql constructs materialize temporary result in tmp-tables, and expect that a table scan will later return the rows in the same order they were inserted. This implementation of insert-order is only enabled when explicitly requested by sql-layer. CHANGES: 1) found bug in ma_write that made code try to abort a record that was never written unsure why this is not exposed Change-Id: Ia82bbaa92e2c0629c08693c5add2f56b815c0509 commit 89dc1ab651fe0205d55b4eb588f62df550aa65fc Author: Jonas Oreland <jonaso@google.com> Date: Mon Feb 17 08:04:50 2014 -0800 Implement encryption of innodb datafiles. Pages are encrypted before written to disk and decrypted when read from disk. Each page except first page (page 0) in tablespace is encrypted. Page 0 is unencrypted and contains IV for the tablespace. FIL_PAGE_FILE_FLUSH_LSN on each page (except page 0) is used to store a 32-bit key-version, so that multiple keys can be active in a tablespace simultaneous. The other 32-bit of the FIL_PAGE_FILE_FLUSH_LSN field contains a checksum that is computed after encryption. This checksum is used by innochecksum and when restoring from double-write-buffer. The encryption is performed using AES CRT. Monitoring of encryption is enabled using new IS-table INNODB_TABLESPACES_ENCRYPTION. In addition to that new status variables innodb_encryption_rotation_{ pages_read_from_cache, pages_read_from_disk, pages_modified,pages_flushed } has been added. The following tunables are introduces - innodb_encrypt_tables - innodb_encryption_threads - innodb_encryption_rotate_key_age - innodb_encryption_rotation_iops Change-Id: I8f651795a30b52e71b16d6bc9cb7559be349d0b2 commit a17eef2f6948e58219c9e26fc35633d6fd4de1de Author: Andrew Ford <andrewford@google.com> Date: Thu Jan 2 15:43:09 2014 -0800 Key management skeleton with debug hooks. Change-Id: Ifd6aa3743d7ea291c70083f433a059c439aed866 commit 68a399838ad72264fd61b3dc67fecd29bbdb0af1 Author: Andrew Ford <andrewford@google.com> Date: Mon Oct 28 16:27:44 2013 -0700 Add AES-128 CTR and GCM encryption classes. Change-Id: I116305eced2a233db15306bc2ef5b9d398d1a3a2
11 years ago
16 years ago
MDEV-11556 InnoDB redo log apply fails to adjust data file sizes fil_space_t::recv_size: New member: recovered tablespace size in pages; 0 if no size change was read from the redo log, or if the size change was implemented. fil_space_set_recv_size(): New function for setting space->recv_size. innodb_data_file_size_debug: A debug parameter for setting the system tablespace size in recovery even when the redo log does not contain any size changes. It is hard to write a small test case that would cause the system tablespace to be extended at the critical moment. recv_parse_log_rec(): Note those tablespaces whose size is being changed by the redo log, by invoking fil_space_set_recv_size(). innobase_init(): Correct an error message, and do not require a larger innodb_buffer_pool_size when starting up with a smaller innodb_page_size. innobase_start_or_create_for_mysql(): Allow startup with any initial size of the ibdata1 file if the autoextend attribute is set. Require the minimum size of fixed-size system tablespaces to be 640 pages, not 10 megabytes. Implement innodb_data_file_size_debug. open_or_create_data_files(): Round the system tablespace size down to pages, not to full megabytes, (Our test truncates the system tablespace to more than 800 pages with innodb_page_size=4k. InnoDB should not imagine that it was truncated to 768 pages and then overwrite good pages in the tablespace.) fil_flush_low(): Refactored from fil_flush(). fil_space_extend_must_retry(): Refactored from fil_extend_space_to_desired_size(). fil_mutex_enter_and_prepare_for_io(): Extend the tablespace if fil_space_set_recv_size() was called. The test case has been successfully run with all the innodb_page_size values 4k, 8k, 16k, 32k, 64k.
9 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
MDEV-11738: Mariadb uses 100% of several of my 8 cpus doing nothing MDEV-11581: Mariadb starts InnoDB encryption threads when key has not changed or data scrubbing turned off Background: Key rotation is based on background threads (innodb-encryption-threads) periodically going through all tablespaces on fil_system. For each tablespace current used key version is compared to max key age (innodb-encryption-rotate-key-age). This process naturally takes CPU. Similarly, in same time need for scrubbing is investigated. Currently, key rotation is fully supported on Amazon AWS key management plugin only but InnoDB does not have knowledge what key management plugin is used. This patch re-purposes innodb-encryption-rotate-key-age=0 to disable key rotation and background data scrubbing. All new tables are added to special list for key rotation and key rotation is based on sending a event to background encryption threads instead of using periodic checking (i.e. timeout). fil0fil.cc: Added functions fil_space_acquire_low() to acquire a tablespace when it could be dropped concurrently. This function is used from fil_space_acquire() or fil_space_acquire_silent() that will not print any messages if we try to acquire space that does not exist. fil_space_release() to release a acquired tablespace. fil_space_next() to iterate tablespaces in fil_system using fil_space_acquire() and fil_space_release(). Similarly, fil_space_keyrotation_next() to iterate new list fil_system->rotation_list where new tables. are added if key rotation is disabled. Removed unnecessary functions fil_get_first_space_safe() fil_get_next_space_safe() fil_node_open_file(): After page 0 is read read also crypt_info if it is not yet read. btr_scrub_lock_dict_func() buf_page_check_corrupt() buf_page_encrypt_before_write() buf_merge_or_delete_for_page() lock_print_info_all_transactions() row_fts_psort_info_init() row_truncate_table_for_mysql() row_drop_table_for_mysql() Use fil_space_acquire()/release() to access fil_space_t. buf_page_decrypt_after_read(): Use fil_space_get_crypt_data() because at this point we might not yet have read page 0. fil0crypt.cc/fil0fil.h: Lot of changes. Pass fil_space_t* directly to functions needing it and store fil_space_t* to rotation state. Use fil_space_acquire()/release() when iterating tablespaces and removed unnecessary is_closing from fil_crypt_t. Use fil_space_t::is_stopping() to detect when access to tablespace should be stopped. Removed unnecessary fil_space_get_crypt_data(). fil_space_create(): Inform key rotation that there could be something to do if key rotation is disabled and new table with encryption enabled is created. Remove unnecessary functions fil_get_first_space_safe() and fil_get_next_space_safe(). fil_space_acquire() and fil_space_release() are used instead. Moved fil_space_get_crypt_data() and fil_space_set_crypt_data() to fil0crypt.cc. fsp_header_init(): Acquire fil_space_t*, write crypt_data and release space. check_table_options() Renamed FIL_SPACE_ENCRYPTION_* TO FIL_ENCRYPTION_* i_s.cc: Added ROTATING_OR_FLUSHING field to information_schema.innodb_tablespace_encryption to show current status of key rotation.
9 years ago
MDEV-11738: Mariadb uses 100% of several of my 8 cpus doing nothing MDEV-11581: Mariadb starts InnoDB encryption threads when key has not changed or data scrubbing turned off Background: Key rotation is based on background threads (innodb-encryption-threads) periodically going through all tablespaces on fil_system. For each tablespace current used key version is compared to max key age (innodb-encryption-rotate-key-age). This process naturally takes CPU. Similarly, in same time need for scrubbing is investigated. Currently, key rotation is fully supported on Amazon AWS key management plugin only but InnoDB does not have knowledge what key management plugin is used. This patch re-purposes innodb-encryption-rotate-key-age=0 to disable key rotation and background data scrubbing. All new tables are added to special list for key rotation and key rotation is based on sending a event to background encryption threads instead of using periodic checking (i.e. timeout). fil0fil.cc: Added functions fil_space_acquire_low() to acquire a tablespace when it could be dropped concurrently. This function is used from fil_space_acquire() or fil_space_acquire_silent() that will not print any messages if we try to acquire space that does not exist. fil_space_release() to release a acquired tablespace. fil_space_next() to iterate tablespaces in fil_system using fil_space_acquire() and fil_space_release(). Similarly, fil_space_keyrotation_next() to iterate new list fil_system->rotation_list where new tables. are added if key rotation is disabled. Removed unnecessary functions fil_get_first_space_safe() fil_get_next_space_safe() fil_node_open_file(): After page 0 is read read also crypt_info if it is not yet read. btr_scrub_lock_dict_func() buf_page_check_corrupt() buf_page_encrypt_before_write() buf_merge_or_delete_for_page() lock_print_info_all_transactions() row_fts_psort_info_init() row_truncate_table_for_mysql() row_drop_table_for_mysql() Use fil_space_acquire()/release() to access fil_space_t. buf_page_decrypt_after_read(): Use fil_space_get_crypt_data() because at this point we might not yet have read page 0. fil0crypt.cc/fil0fil.h: Lot of changes. Pass fil_space_t* directly to functions needing it and store fil_space_t* to rotation state. Use fil_space_acquire()/release() when iterating tablespaces and removed unnecessary is_closing from fil_crypt_t. Use fil_space_t::is_stopping() to detect when access to tablespace should be stopped. Removed unnecessary fil_space_get_crypt_data(). fil_space_create(): Inform key rotation that there could be something to do if key rotation is disabled and new table with encryption enabled is created. Remove unnecessary functions fil_get_first_space_safe() and fil_get_next_space_safe(). fil_space_acquire() and fil_space_release() are used instead. Moved fil_space_get_crypt_data() and fil_space_set_crypt_data() to fil0crypt.cc. fsp_header_init(): Acquire fil_space_t*, write crypt_data and release space. check_table_options() Renamed FIL_SPACE_ENCRYPTION_* TO FIL_ENCRYPTION_* i_s.cc: Added ROTATING_OR_FLUSHING field to information_schema.innodb_tablespace_encryption to show current status of key rotation.
9 years ago
MDEV-11738: Mariadb uses 100% of several of my 8 cpus doing nothing MDEV-11581: Mariadb starts InnoDB encryption threads when key has not changed or data scrubbing turned off Background: Key rotation is based on background threads (innodb-encryption-threads) periodically going through all tablespaces on fil_system. For each tablespace current used key version is compared to max key age (innodb-encryption-rotate-key-age). This process naturally takes CPU. Similarly, in same time need for scrubbing is investigated. Currently, key rotation is fully supported on Amazon AWS key management plugin only but InnoDB does not have knowledge what key management plugin is used. This patch re-purposes innodb-encryption-rotate-key-age=0 to disable key rotation and background data scrubbing. All new tables are added to special list for key rotation and key rotation is based on sending a event to background encryption threads instead of using periodic checking (i.e. timeout). fil0fil.cc: Added functions fil_space_acquire_low() to acquire a tablespace when it could be dropped concurrently. This function is used from fil_space_acquire() or fil_space_acquire_silent() that will not print any messages if we try to acquire space that does not exist. fil_space_release() to release a acquired tablespace. fil_space_next() to iterate tablespaces in fil_system using fil_space_acquire() and fil_space_release(). Similarly, fil_space_keyrotation_next() to iterate new list fil_system->rotation_list where new tables. are added if key rotation is disabled. Removed unnecessary functions fil_get_first_space_safe() fil_get_next_space_safe() fil_node_open_file(): After page 0 is read read also crypt_info if it is not yet read. btr_scrub_lock_dict_func() buf_page_check_corrupt() buf_page_encrypt_before_write() buf_merge_or_delete_for_page() lock_print_info_all_transactions() row_fts_psort_info_init() row_truncate_table_for_mysql() row_drop_table_for_mysql() Use fil_space_acquire()/release() to access fil_space_t. buf_page_decrypt_after_read(): Use fil_space_get_crypt_data() because at this point we might not yet have read page 0. fil0crypt.cc/fil0fil.h: Lot of changes. Pass fil_space_t* directly to functions needing it and store fil_space_t* to rotation state. Use fil_space_acquire()/release() when iterating tablespaces and removed unnecessary is_closing from fil_crypt_t. Use fil_space_t::is_stopping() to detect when access to tablespace should be stopped. Removed unnecessary fil_space_get_crypt_data(). fil_space_create(): Inform key rotation that there could be something to do if key rotation is disabled and new table with encryption enabled is created. Remove unnecessary functions fil_get_first_space_safe() and fil_get_next_space_safe(). fil_space_acquire() and fil_space_release() are used instead. Moved fil_space_get_crypt_data() and fil_space_set_crypt_data() to fil0crypt.cc. fsp_header_init(): Acquire fil_space_t*, write crypt_data and release space. check_table_options() Renamed FIL_SPACE_ENCRYPTION_* TO FIL_ENCRYPTION_* i_s.cc: Added ROTATING_OR_FLUSHING field to information_schema.innodb_tablespace_encryption to show current status of key rotation.
9 years ago
MDEV-11738: Mariadb uses 100% of several of my 8 cpus doing nothing MDEV-11581: Mariadb starts InnoDB encryption threads when key has not changed or data scrubbing turned off Background: Key rotation is based on background threads (innodb-encryption-threads) periodically going through all tablespaces on fil_system. For each tablespace current used key version is compared to max key age (innodb-encryption-rotate-key-age). This process naturally takes CPU. Similarly, in same time need for scrubbing is investigated. Currently, key rotation is fully supported on Amazon AWS key management plugin only but InnoDB does not have knowledge what key management plugin is used. This patch re-purposes innodb-encryption-rotate-key-age=0 to disable key rotation and background data scrubbing. All new tables are added to special list for key rotation and key rotation is based on sending a event to background encryption threads instead of using periodic checking (i.e. timeout). fil0fil.cc: Added functions fil_space_acquire_low() to acquire a tablespace when it could be dropped concurrently. This function is used from fil_space_acquire() or fil_space_acquire_silent() that will not print any messages if we try to acquire space that does not exist. fil_space_release() to release a acquired tablespace. fil_space_next() to iterate tablespaces in fil_system using fil_space_acquire() and fil_space_release(). Similarly, fil_space_keyrotation_next() to iterate new list fil_system->rotation_list where new tables. are added if key rotation is disabled. Removed unnecessary functions fil_get_first_space_safe() fil_get_next_space_safe() fil_node_open_file(): After page 0 is read read also crypt_info if it is not yet read. btr_scrub_lock_dict_func() buf_page_check_corrupt() buf_page_encrypt_before_write() buf_merge_or_delete_for_page() lock_print_info_all_transactions() row_fts_psort_info_init() row_truncate_table_for_mysql() row_drop_table_for_mysql() Use fil_space_acquire()/release() to access fil_space_t. buf_page_decrypt_after_read(): Use fil_space_get_crypt_data() because at this point we might not yet have read page 0. fil0crypt.cc/fil0fil.h: Lot of changes. Pass fil_space_t* directly to functions needing it and store fil_space_t* to rotation state. Use fil_space_acquire()/release() when iterating tablespaces and removed unnecessary is_closing from fil_crypt_t. Use fil_space_t::is_stopping() to detect when access to tablespace should be stopped. Removed unnecessary fil_space_get_crypt_data(). fil_space_create(): Inform key rotation that there could be something to do if key rotation is disabled and new table with encryption enabled is created. Remove unnecessary functions fil_get_first_space_safe() and fil_get_next_space_safe(). fil_space_acquire() and fil_space_release() are used instead. Moved fil_space_get_crypt_data() and fil_space_set_crypt_data() to fil0crypt.cc. fsp_header_init(): Acquire fil_space_t*, write crypt_data and release space. check_table_options() Renamed FIL_SPACE_ENCRYPTION_* TO FIL_ENCRYPTION_* i_s.cc: Added ROTATING_OR_FLUSHING field to information_schema.innodb_tablespace_encryption to show current status of key rotation.
9 years ago
MDEV-11738: Mariadb uses 100% of several of my 8 cpus doing nothing MDEV-11581: Mariadb starts InnoDB encryption threads when key has not changed or data scrubbing turned off Background: Key rotation is based on background threads (innodb-encryption-threads) periodically going through all tablespaces on fil_system. For each tablespace current used key version is compared to max key age (innodb-encryption-rotate-key-age). This process naturally takes CPU. Similarly, in same time need for scrubbing is investigated. Currently, key rotation is fully supported on Amazon AWS key management plugin only but InnoDB does not have knowledge what key management plugin is used. This patch re-purposes innodb-encryption-rotate-key-age=0 to disable key rotation and background data scrubbing. All new tables are added to special list for key rotation and key rotation is based on sending a event to background encryption threads instead of using periodic checking (i.e. timeout). fil0fil.cc: Added functions fil_space_acquire_low() to acquire a tablespace when it could be dropped concurrently. This function is used from fil_space_acquire() or fil_space_acquire_silent() that will not print any messages if we try to acquire space that does not exist. fil_space_release() to release a acquired tablespace. fil_space_next() to iterate tablespaces in fil_system using fil_space_acquire() and fil_space_release(). Similarly, fil_space_keyrotation_next() to iterate new list fil_system->rotation_list where new tables. are added if key rotation is disabled. Removed unnecessary functions fil_get_first_space_safe() fil_get_next_space_safe() fil_node_open_file(): After page 0 is read read also crypt_info if it is not yet read. btr_scrub_lock_dict_func() buf_page_check_corrupt() buf_page_encrypt_before_write() buf_merge_or_delete_for_page() lock_print_info_all_transactions() row_fts_psort_info_init() row_truncate_table_for_mysql() row_drop_table_for_mysql() Use fil_space_acquire()/release() to access fil_space_t. buf_page_decrypt_after_read(): Use fil_space_get_crypt_data() because at this point we might not yet have read page 0. fil0crypt.cc/fil0fil.h: Lot of changes. Pass fil_space_t* directly to functions needing it and store fil_space_t* to rotation state. Use fil_space_acquire()/release() when iterating tablespaces and removed unnecessary is_closing from fil_crypt_t. Use fil_space_t::is_stopping() to detect when access to tablespace should be stopped. Removed unnecessary fil_space_get_crypt_data(). fil_space_create(): Inform key rotation that there could be something to do if key rotation is disabled and new table with encryption enabled is created. Remove unnecessary functions fil_get_first_space_safe() and fil_get_next_space_safe(). fil_space_acquire() and fil_space_release() are used instead. Moved fil_space_get_crypt_data() and fil_space_set_crypt_data() to fil0crypt.cc. fsp_header_init(): Acquire fil_space_t*, write crypt_data and release space. check_table_options() Renamed FIL_SPACE_ENCRYPTION_* TO FIL_ENCRYPTION_* i_s.cc: Added ROTATING_OR_FLUSHING field to information_schema.innodb_tablespace_encryption to show current status of key rotation.
9 years ago
MDEV-11738: Mariadb uses 100% of several of my 8 cpus doing nothing MDEV-11581: Mariadb starts InnoDB encryption threads when key has not changed or data scrubbing turned off Background: Key rotation is based on background threads (innodb-encryption-threads) periodically going through all tablespaces on fil_system. For each tablespace current used key version is compared to max key age (innodb-encryption-rotate-key-age). This process naturally takes CPU. Similarly, in same time need for scrubbing is investigated. Currently, key rotation is fully supported on Amazon AWS key management plugin only but InnoDB does not have knowledge what key management plugin is used. This patch re-purposes innodb-encryption-rotate-key-age=0 to disable key rotation and background data scrubbing. All new tables are added to special list for key rotation and key rotation is based on sending a event to background encryption threads instead of using periodic checking (i.e. timeout). fil0fil.cc: Added functions fil_space_acquire_low() to acquire a tablespace when it could be dropped concurrently. This function is used from fil_space_acquire() or fil_space_acquire_silent() that will not print any messages if we try to acquire space that does not exist. fil_space_release() to release a acquired tablespace. fil_space_next() to iterate tablespaces in fil_system using fil_space_acquire() and fil_space_release(). Similarly, fil_space_keyrotation_next() to iterate new list fil_system->rotation_list where new tables. are added if key rotation is disabled. Removed unnecessary functions fil_get_first_space_safe() fil_get_next_space_safe() fil_node_open_file(): After page 0 is read read also crypt_info if it is not yet read. btr_scrub_lock_dict_func() buf_page_check_corrupt() buf_page_encrypt_before_write() buf_merge_or_delete_for_page() lock_print_info_all_transactions() row_fts_psort_info_init() row_truncate_table_for_mysql() row_drop_table_for_mysql() Use fil_space_acquire()/release() to access fil_space_t. buf_page_decrypt_after_read(): Use fil_space_get_crypt_data() because at this point we might not yet have read page 0. fil0crypt.cc/fil0fil.h: Lot of changes. Pass fil_space_t* directly to functions needing it and store fil_space_t* to rotation state. Use fil_space_acquire()/release() when iterating tablespaces and removed unnecessary is_closing from fil_crypt_t. Use fil_space_t::is_stopping() to detect when access to tablespace should be stopped. Removed unnecessary fil_space_get_crypt_data(). fil_space_create(): Inform key rotation that there could be something to do if key rotation is disabled and new table with encryption enabled is created. Remove unnecessary functions fil_get_first_space_safe() and fil_get_next_space_safe(). fil_space_acquire() and fil_space_release() are used instead. Moved fil_space_get_crypt_data() and fil_space_set_crypt_data() to fil0crypt.cc. fsp_header_init(): Acquire fil_space_t*, write crypt_data and release space. check_table_options() Renamed FIL_SPACE_ENCRYPTION_* TO FIL_ENCRYPTION_* i_s.cc: Added ROTATING_OR_FLUSHING field to information_schema.innodb_tablespace_encryption to show current status of key rotation.
9 years ago
16 years ago
16 years ago
MDEV-11623 MariaDB 10.1 fails to start datadir created with MariaDB 10.0/MySQL 5.6 using innodb-page-size!=16K The storage format of FSP_SPACE_FLAGS was accidentally broken already in MariaDB 10.1.0. This fix is bringing the format in line with other MySQL and MariaDB release series. Please refer to the comments that were added to fsp0fsp.h for details. This is an INCOMPATIBLE CHANGE that affects users of page_compression and non-default innodb_page_size. Upgrading to this release will correct the flags in the data files. If you want to downgrade to earlier MariaDB 10.1.x, please refer to the test innodb.101_compatibility how to reset the FSP_SPACE_FLAGS in the files. NOTE: MariaDB 10.1.0 to 10.1.20 can misinterpret uncompressed data files with innodb_page_size=4k or 64k as compressed innodb_page_size=16k files, and then probably fail when trying to access the pages. See the comments in the function fsp_flags_convert_from_101() for detailed analysis. Move PAGE_COMPRESSION to FSP_SPACE_FLAGS bit position 16. In this way, compressed innodb_page_size=16k tablespaces will not be mistaken for uncompressed ones by MariaDB 10.1.0 to 10.1.20. Derive PAGE_COMPRESSION_LEVEL, ATOMIC_WRITES and DATA_DIR from the dict_table_t::flags when the table is available, in fil_space_for_table_exists_in_mem() or fil_open_single_table_tablespace(). During crash recovery, fil_load_single_table_tablespace() will use innodb_compression_level for the PAGE_COMPRESSION_LEVEL. FSP_FLAGS_MEM_MASK: A bitmap of the memory-only fil_space_t::flags that are not to be written to FSP_SPACE_FLAGS. Currently, these will include PAGE_COMPRESSION_LEVEL, ATOMIC_WRITES and DATA_DIR. Introduce the macro FSP_FLAGS_PAGE_SSIZE(). We only support one innodb_page_size for the whole instance. When creating a dummy tablespace for the redo log, use fil_space_t::flags=0. The flags are never written to the redo log files. Remove many FSP_FLAGS_SET_ macros. dict_tf_verify_flags(): Remove. This is basically only duplicating the logic of dict_tf_to_fsp_flags(), used in a debug assertion. fil_space_t::mark: Remove. This flag was not used for anything. fil_space_for_table_exists_in_mem(): Remove the unnecessary parameter mark_space, and add a parameter for table flags. Check that fil_space_t::flags match the table flags, and adjust the (memory-only) flags based on the table flags. fil_node_open_file(): Remove some redundant or unreachable conditions, do not use stderr for output, and avoid unnecessary server aborts. fil_user_tablespace_restore_page(): Convert the flags, so that the correct page_size will be used when restoring a page from the doublewrite buffer. fil_space_get_page_compressed(), fsp_flags_is_page_compressed(): Remove. It suffices to have fil_space_is_page_compressed(). FSP_FLAGS_WIDTH_DATA_DIR, FSP_FLAGS_WIDTH_PAGE_COMPRESSION_LEVEL, FSP_FLAGS_WIDTH_ATOMIC_WRITES: Remove, because these flags do not exist in the FSP_SPACE_FLAGS but only in memory. fsp_flags_try_adjust(): New function, to adjust the FSP_SPACE_FLAGS in page 0. Called by fil_open_single_table_tablespace(), fil_space_for_table_exists_in_mem(), innobase_start_or_create_for_mysql() except if --innodb-read-only is active. fsp_flags_is_valid(ulint): Reimplement from the scratch, with accurate comments. Do not display any details of detected inconsistencies, because the output could be confusing when dealing with MariaDB 10.1.x data files. fsp_flags_convert_from_101(ulint): Convert flags from buggy MariaDB 10.1.x format, or return ULINT_UNDEFINED if the flags cannot be in MariaDB 10.1.x format. fsp_flags_match(): Check the flags when probing files. Implemented based on fsp_flags_is_valid() and fsp_flags_convert_from_101(). dict_check_tablespaces_and_store_max_id(): Do not access the page after committing the mini-transaction. IMPORT TABLESPACE fixes: AbstractCallback::init(): Convert the flags. FetchIndexRootPages::operator(): Check that the tablespace flags match the table flags. Do not attempt to convert tablespace flags to table flags, because the conversion would necessarily be lossy. PageConverter::update_header(): Write back the correct flags. This takes care of the flags in IMPORT TABLESPACE.
9 years ago
Merge Google encryption commit 195158e9889365dc3298f8c1f3bcaa745992f27f Author: Minli Zhu <minliz@google.com> Date: Mon Nov 25 11:05:55 2013 -0800 Innodb redo log encryption/decryption. Use start lsn of a log block as part of AES CTR counter. Record key version with each checkpoint. Internally key version 0 means no encryption. Tests done (see test_innodb_log_encryption.sh for detail): - Verify flag innodb_encrypt_log on or off, combined with various key versions passed through CLI, and dynamically set after startup, will not corrupt database. This includes tests from being unencrypted to encrypted, and encrypted to unencrypted. - Verify start-up with no redo logs succeeds. - Verify fresh start-up succeeds. Change-Id: I4ce4c2afdf3076be2fce90ebbc2a7ce01184b612 commit c1b97273659f07866758c25f4a56f680a1fbad24 Author: Jonas Oreland <jonaso@google.com> Date: Tue Dec 3 18:47:27 2013 +0100 encryption of aria data&index files this patch implements encryption of aria data & index files. this is implemented as 1) add read/write hooks (renamed from callbacks) that does encrypt/decrypt (also add pre_read and post_write hooks) 2) modify page headers for data/index to contain key version (making the data-page header size different for with/without encryption) 3) modify index page 0 to contain IV (and crypt header) 4) AES CRT crypt functions 5) counter block is implemented using combination of page no, lsn and table specific id NOTE: 1) log files are not encrypted, this is not needed for if aria is only used for internal temporary tables and they are not transactional (i.e not logged) 2) all encrypted tables are using PAGE_CHECKSUM (crc) normal internal temporary tables are (currently) not CHECKSUM:ed 3) This patch adds insert-order semantics to aria block_format. The default behaviour of aria block-format is best-fit, meaning that rows gets allocated to page trying to fill the pages as much as possible. However, certain sql constructs materialize temporary result in tmp-tables, and expect that a table scan will later return the rows in the same order they were inserted. This implementation of insert-order is only enabled when explicitly requested by sql-layer. CHANGES: 1) found bug in ma_write that made code try to abort a record that was never written unsure why this is not exposed Change-Id: Ia82bbaa92e2c0629c08693c5add2f56b815c0509 commit 89dc1ab651fe0205d55b4eb588f62df550aa65fc Author: Jonas Oreland <jonaso@google.com> Date: Mon Feb 17 08:04:50 2014 -0800 Implement encryption of innodb datafiles. Pages are encrypted before written to disk and decrypted when read from disk. Each page except first page (page 0) in tablespace is encrypted. Page 0 is unencrypted and contains IV for the tablespace. FIL_PAGE_FILE_FLUSH_LSN on each page (except page 0) is used to store a 32-bit key-version, so that multiple keys can be active in a tablespace simultaneous. The other 32-bit of the FIL_PAGE_FILE_FLUSH_LSN field contains a checksum that is computed after encryption. This checksum is used by innochecksum and when restoring from double-write-buffer. The encryption is performed using AES CRT. Monitoring of encryption is enabled using new IS-table INNODB_TABLESPACES_ENCRYPTION. In addition to that new status variables innodb_encryption_rotation_{ pages_read_from_cache, pages_read_from_disk, pages_modified,pages_flushed } has been added. The following tunables are introduces - innodb_encrypt_tables - innodb_encryption_threads - innodb_encryption_rotate_key_age - innodb_encryption_rotation_iops Change-Id: I8f651795a30b52e71b16d6bc9cb7559be349d0b2 commit a17eef2f6948e58219c9e26fc35633d6fd4de1de Author: Andrew Ford <andrewford@google.com> Date: Thu Jan 2 15:43:09 2014 -0800 Key management skeleton with debug hooks. Change-Id: Ifd6aa3743d7ea291c70083f433a059c439aed866 commit 68a399838ad72264fd61b3dc67fecd29bbdb0af1 Author: Andrew Ford <andrewford@google.com> Date: Mon Oct 28 16:27:44 2013 -0700 Add AES-128 CTR and GCM encryption classes. Change-Id: I116305eced2a233db15306bc2ef5b9d398d1a3a2
11 years ago
16 years ago
11 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
9 years ago
16 years ago
10 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
MDEV-11623 MariaDB 10.1 fails to start datadir created with MariaDB 10.0/MySQL 5.6 using innodb-page-size!=16K The storage format of FSP_SPACE_FLAGS was accidentally broken already in MariaDB 10.1.0. This fix is bringing the format in line with other MySQL and MariaDB release series. Please refer to the comments that were added to fsp0fsp.h for details. This is an INCOMPATIBLE CHANGE that affects users of page_compression and non-default innodb_page_size. Upgrading to this release will correct the flags in the data files. If you want to downgrade to earlier MariaDB 10.1.x, please refer to the test innodb.101_compatibility how to reset the FSP_SPACE_FLAGS in the files. NOTE: MariaDB 10.1.0 to 10.1.20 can misinterpret uncompressed data files with innodb_page_size=4k or 64k as compressed innodb_page_size=16k files, and then probably fail when trying to access the pages. See the comments in the function fsp_flags_convert_from_101() for detailed analysis. Move PAGE_COMPRESSION to FSP_SPACE_FLAGS bit position 16. In this way, compressed innodb_page_size=16k tablespaces will not be mistaken for uncompressed ones by MariaDB 10.1.0 to 10.1.20. Derive PAGE_COMPRESSION_LEVEL, ATOMIC_WRITES and DATA_DIR from the dict_table_t::flags when the table is available, in fil_space_for_table_exists_in_mem() or fil_open_single_table_tablespace(). During crash recovery, fil_load_single_table_tablespace() will use innodb_compression_level for the PAGE_COMPRESSION_LEVEL. FSP_FLAGS_MEM_MASK: A bitmap of the memory-only fil_space_t::flags that are not to be written to FSP_SPACE_FLAGS. Currently, these will include PAGE_COMPRESSION_LEVEL, ATOMIC_WRITES and DATA_DIR. Introduce the macro FSP_FLAGS_PAGE_SSIZE(). We only support one innodb_page_size for the whole instance. When creating a dummy tablespace for the redo log, use fil_space_t::flags=0. The flags are never written to the redo log files. Remove many FSP_FLAGS_SET_ macros. dict_tf_verify_flags(): Remove. This is basically only duplicating the logic of dict_tf_to_fsp_flags(), used in a debug assertion. fil_space_t::mark: Remove. This flag was not used for anything. fil_space_for_table_exists_in_mem(): Remove the unnecessary parameter mark_space, and add a parameter for table flags. Check that fil_space_t::flags match the table flags, and adjust the (memory-only) flags based on the table flags. fil_node_open_file(): Remove some redundant or unreachable conditions, do not use stderr for output, and avoid unnecessary server aborts. fil_user_tablespace_restore_page(): Convert the flags, so that the correct page_size will be used when restoring a page from the doublewrite buffer. fil_space_get_page_compressed(), fsp_flags_is_page_compressed(): Remove. It suffices to have fil_space_is_page_compressed(). FSP_FLAGS_WIDTH_DATA_DIR, FSP_FLAGS_WIDTH_PAGE_COMPRESSION_LEVEL, FSP_FLAGS_WIDTH_ATOMIC_WRITES: Remove, because these flags do not exist in the FSP_SPACE_FLAGS but only in memory. fsp_flags_try_adjust(): New function, to adjust the FSP_SPACE_FLAGS in page 0. Called by fil_open_single_table_tablespace(), fil_space_for_table_exists_in_mem(), innobase_start_or_create_for_mysql() except if --innodb-read-only is active. fsp_flags_is_valid(ulint): Reimplement from the scratch, with accurate comments. Do not display any details of detected inconsistencies, because the output could be confusing when dealing with MariaDB 10.1.x data files. fsp_flags_convert_from_101(ulint): Convert flags from buggy MariaDB 10.1.x format, or return ULINT_UNDEFINED if the flags cannot be in MariaDB 10.1.x format. fsp_flags_match(): Check the flags when probing files. Implemented based on fsp_flags_is_valid() and fsp_flags_convert_from_101(). dict_check_tablespaces_and_store_max_id(): Do not access the page after committing the mini-transaction. IMPORT TABLESPACE fixes: AbstractCallback::init(): Convert the flags. FetchIndexRootPages::operator(): Check that the tablespace flags match the table flags. Do not attempt to convert tablespace flags to table flags, because the conversion would necessarily be lossy. PageConverter::update_header(): Write back the correct flags. This takes care of the flags in IMPORT TABLESPACE.
9 years ago
16 years ago
16 years ago
16 years ago
MDEV-11623 MariaDB 10.1 fails to start datadir created with MariaDB 10.0/MySQL 5.6 using innodb-page-size!=16K The storage format of FSP_SPACE_FLAGS was accidentally broken already in MariaDB 10.1.0. This fix is bringing the format in line with other MySQL and MariaDB release series. Please refer to the comments that were added to fsp0fsp.h for details. This is an INCOMPATIBLE CHANGE that affects users of page_compression and non-default innodb_page_size. Upgrading to this release will correct the flags in the data files. If you want to downgrade to earlier MariaDB 10.1.x, please refer to the test innodb.101_compatibility how to reset the FSP_SPACE_FLAGS in the files. NOTE: MariaDB 10.1.0 to 10.1.20 can misinterpret uncompressed data files with innodb_page_size=4k or 64k as compressed innodb_page_size=16k files, and then probably fail when trying to access the pages. See the comments in the function fsp_flags_convert_from_101() for detailed analysis. Move PAGE_COMPRESSION to FSP_SPACE_FLAGS bit position 16. In this way, compressed innodb_page_size=16k tablespaces will not be mistaken for uncompressed ones by MariaDB 10.1.0 to 10.1.20. Derive PAGE_COMPRESSION_LEVEL, ATOMIC_WRITES and DATA_DIR from the dict_table_t::flags when the table is available, in fil_space_for_table_exists_in_mem() or fil_open_single_table_tablespace(). During crash recovery, fil_load_single_table_tablespace() will use innodb_compression_level for the PAGE_COMPRESSION_LEVEL. FSP_FLAGS_MEM_MASK: A bitmap of the memory-only fil_space_t::flags that are not to be written to FSP_SPACE_FLAGS. Currently, these will include PAGE_COMPRESSION_LEVEL, ATOMIC_WRITES and DATA_DIR. Introduce the macro FSP_FLAGS_PAGE_SSIZE(). We only support one innodb_page_size for the whole instance. When creating a dummy tablespace for the redo log, use fil_space_t::flags=0. The flags are never written to the redo log files. Remove many FSP_FLAGS_SET_ macros. dict_tf_verify_flags(): Remove. This is basically only duplicating the logic of dict_tf_to_fsp_flags(), used in a debug assertion. fil_space_t::mark: Remove. This flag was not used for anything. fil_space_for_table_exists_in_mem(): Remove the unnecessary parameter mark_space, and add a parameter for table flags. Check that fil_space_t::flags match the table flags, and adjust the (memory-only) flags based on the table flags. fil_node_open_file(): Remove some redundant or unreachable conditions, do not use stderr for output, and avoid unnecessary server aborts. fil_user_tablespace_restore_page(): Convert the flags, so that the correct page_size will be used when restoring a page from the doublewrite buffer. fil_space_get_page_compressed(), fsp_flags_is_page_compressed(): Remove. It suffices to have fil_space_is_page_compressed(). FSP_FLAGS_WIDTH_DATA_DIR, FSP_FLAGS_WIDTH_PAGE_COMPRESSION_LEVEL, FSP_FLAGS_WIDTH_ATOMIC_WRITES: Remove, because these flags do not exist in the FSP_SPACE_FLAGS but only in memory. fsp_flags_try_adjust(): New function, to adjust the FSP_SPACE_FLAGS in page 0. Called by fil_open_single_table_tablespace(), fil_space_for_table_exists_in_mem(), innobase_start_or_create_for_mysql() except if --innodb-read-only is active. fsp_flags_is_valid(ulint): Reimplement from the scratch, with accurate comments. Do not display any details of detected inconsistencies, because the output could be confusing when dealing with MariaDB 10.1.x data files. fsp_flags_convert_from_101(ulint): Convert flags from buggy MariaDB 10.1.x format, or return ULINT_UNDEFINED if the flags cannot be in MariaDB 10.1.x format. fsp_flags_match(): Check the flags when probing files. Implemented based on fsp_flags_is_valid() and fsp_flags_convert_from_101(). dict_check_tablespaces_and_store_max_id(): Do not access the page after committing the mini-transaction. IMPORT TABLESPACE fixes: AbstractCallback::init(): Convert the flags. FetchIndexRootPages::operator(): Check that the tablespace flags match the table flags. Do not attempt to convert tablespace flags to table flags, because the conversion would necessarily be lossy. PageConverter::update_header(): Write back the correct flags. This takes care of the flags in IMPORT TABLESPACE.
9 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
MDEV-11623 MariaDB 10.1 fails to start datadir created with MariaDB 10.0/MySQL 5.6 using innodb-page-size!=16K The storage format of FSP_SPACE_FLAGS was accidentally broken already in MariaDB 10.1.0. This fix is bringing the format in line with other MySQL and MariaDB release series. Please refer to the comments that were added to fsp0fsp.h for details. This is an INCOMPATIBLE CHANGE that affects users of page_compression and non-default innodb_page_size. Upgrading to this release will correct the flags in the data files. If you want to downgrade to earlier MariaDB 10.1.x, please refer to the test innodb.101_compatibility how to reset the FSP_SPACE_FLAGS in the files. NOTE: MariaDB 10.1.0 to 10.1.20 can misinterpret uncompressed data files with innodb_page_size=4k or 64k as compressed innodb_page_size=16k files, and then probably fail when trying to access the pages. See the comments in the function fsp_flags_convert_from_101() for detailed analysis. Move PAGE_COMPRESSION to FSP_SPACE_FLAGS bit position 16. In this way, compressed innodb_page_size=16k tablespaces will not be mistaken for uncompressed ones by MariaDB 10.1.0 to 10.1.20. Derive PAGE_COMPRESSION_LEVEL, ATOMIC_WRITES and DATA_DIR from the dict_table_t::flags when the table is available, in fil_space_for_table_exists_in_mem() or fil_open_single_table_tablespace(). During crash recovery, fil_load_single_table_tablespace() will use innodb_compression_level for the PAGE_COMPRESSION_LEVEL. FSP_FLAGS_MEM_MASK: A bitmap of the memory-only fil_space_t::flags that are not to be written to FSP_SPACE_FLAGS. Currently, these will include PAGE_COMPRESSION_LEVEL, ATOMIC_WRITES and DATA_DIR. Introduce the macro FSP_FLAGS_PAGE_SSIZE(). We only support one innodb_page_size for the whole instance. When creating a dummy tablespace for the redo log, use fil_space_t::flags=0. The flags are never written to the redo log files. Remove many FSP_FLAGS_SET_ macros. dict_tf_verify_flags(): Remove. This is basically only duplicating the logic of dict_tf_to_fsp_flags(), used in a debug assertion. fil_space_t::mark: Remove. This flag was not used for anything. fil_space_for_table_exists_in_mem(): Remove the unnecessary parameter mark_space, and add a parameter for table flags. Check that fil_space_t::flags match the table flags, and adjust the (memory-only) flags based on the table flags. fil_node_open_file(): Remove some redundant or unreachable conditions, do not use stderr for output, and avoid unnecessary server aborts. fil_user_tablespace_restore_page(): Convert the flags, so that the correct page_size will be used when restoring a page from the doublewrite buffer. fil_space_get_page_compressed(), fsp_flags_is_page_compressed(): Remove. It suffices to have fil_space_is_page_compressed(). FSP_FLAGS_WIDTH_DATA_DIR, FSP_FLAGS_WIDTH_PAGE_COMPRESSION_LEVEL, FSP_FLAGS_WIDTH_ATOMIC_WRITES: Remove, because these flags do not exist in the FSP_SPACE_FLAGS but only in memory. fsp_flags_try_adjust(): New function, to adjust the FSP_SPACE_FLAGS in page 0. Called by fil_open_single_table_tablespace(), fil_space_for_table_exists_in_mem(), innobase_start_or_create_for_mysql() except if --innodb-read-only is active. fsp_flags_is_valid(ulint): Reimplement from the scratch, with accurate comments. Do not display any details of detected inconsistencies, because the output could be confusing when dealing with MariaDB 10.1.x data files. fsp_flags_convert_from_101(ulint): Convert flags from buggy MariaDB 10.1.x format, or return ULINT_UNDEFINED if the flags cannot be in MariaDB 10.1.x format. fsp_flags_match(): Check the flags when probing files. Implemented based on fsp_flags_is_valid() and fsp_flags_convert_from_101(). dict_check_tablespaces_and_store_max_id(): Do not access the page after committing the mini-transaction. IMPORT TABLESPACE fixes: AbstractCallback::init(): Convert the flags. FetchIndexRootPages::operator(): Check that the tablespace flags match the table flags. Do not attempt to convert tablespace flags to table flags, because the conversion would necessarily be lossy. PageConverter::update_header(): Write back the correct flags. This takes care of the flags in IMPORT TABLESPACE.
9 years ago
16 years ago
MDEV-11623 MariaDB 10.1 fails to start datadir created with MariaDB 10.0/MySQL 5.6 using innodb-page-size!=16K The storage format of FSP_SPACE_FLAGS was accidentally broken already in MariaDB 10.1.0. This fix is bringing the format in line with other MySQL and MariaDB release series. Please refer to the comments that were added to fsp0fsp.h for details. This is an INCOMPATIBLE CHANGE that affects users of page_compression and non-default innodb_page_size. Upgrading to this release will correct the flags in the data files. If you want to downgrade to earlier MariaDB 10.1.x, please refer to the test innodb.101_compatibility how to reset the FSP_SPACE_FLAGS in the files. NOTE: MariaDB 10.1.0 to 10.1.20 can misinterpret uncompressed data files with innodb_page_size=4k or 64k as compressed innodb_page_size=16k files, and then probably fail when trying to access the pages. See the comments in the function fsp_flags_convert_from_101() for detailed analysis. Move PAGE_COMPRESSION to FSP_SPACE_FLAGS bit position 16. In this way, compressed innodb_page_size=16k tablespaces will not be mistaken for uncompressed ones by MariaDB 10.1.0 to 10.1.20. Derive PAGE_COMPRESSION_LEVEL, ATOMIC_WRITES and DATA_DIR from the dict_table_t::flags when the table is available, in fil_space_for_table_exists_in_mem() or fil_open_single_table_tablespace(). During crash recovery, fil_load_single_table_tablespace() will use innodb_compression_level for the PAGE_COMPRESSION_LEVEL. FSP_FLAGS_MEM_MASK: A bitmap of the memory-only fil_space_t::flags that are not to be written to FSP_SPACE_FLAGS. Currently, these will include PAGE_COMPRESSION_LEVEL, ATOMIC_WRITES and DATA_DIR. Introduce the macro FSP_FLAGS_PAGE_SSIZE(). We only support one innodb_page_size for the whole instance. When creating a dummy tablespace for the redo log, use fil_space_t::flags=0. The flags are never written to the redo log files. Remove many FSP_FLAGS_SET_ macros. dict_tf_verify_flags(): Remove. This is basically only duplicating the logic of dict_tf_to_fsp_flags(), used in a debug assertion. fil_space_t::mark: Remove. This flag was not used for anything. fil_space_for_table_exists_in_mem(): Remove the unnecessary parameter mark_space, and add a parameter for table flags. Check that fil_space_t::flags match the table flags, and adjust the (memory-only) flags based on the table flags. fil_node_open_file(): Remove some redundant or unreachable conditions, do not use stderr for output, and avoid unnecessary server aborts. fil_user_tablespace_restore_page(): Convert the flags, so that the correct page_size will be used when restoring a page from the doublewrite buffer. fil_space_get_page_compressed(), fsp_flags_is_page_compressed(): Remove. It suffices to have fil_space_is_page_compressed(). FSP_FLAGS_WIDTH_DATA_DIR, FSP_FLAGS_WIDTH_PAGE_COMPRESSION_LEVEL, FSP_FLAGS_WIDTH_ATOMIC_WRITES: Remove, because these flags do not exist in the FSP_SPACE_FLAGS but only in memory. fsp_flags_try_adjust(): New function, to adjust the FSP_SPACE_FLAGS in page 0. Called by fil_open_single_table_tablespace(), fil_space_for_table_exists_in_mem(), innobase_start_or_create_for_mysql() except if --innodb-read-only is active. fsp_flags_is_valid(ulint): Reimplement from the scratch, with accurate comments. Do not display any details of detected inconsistencies, because the output could be confusing when dealing with MariaDB 10.1.x data files. fsp_flags_convert_from_101(ulint): Convert flags from buggy MariaDB 10.1.x format, or return ULINT_UNDEFINED if the flags cannot be in MariaDB 10.1.x format. fsp_flags_match(): Check the flags when probing files. Implemented based on fsp_flags_is_valid() and fsp_flags_convert_from_101(). dict_check_tablespaces_and_store_max_id(): Do not access the page after committing the mini-transaction. IMPORT TABLESPACE fixes: AbstractCallback::init(): Convert the flags. FetchIndexRootPages::operator(): Check that the tablespace flags match the table flags. Do not attempt to convert tablespace flags to table flags, because the conversion would necessarily be lossy. PageConverter::update_header(): Write back the correct flags. This takes care of the flags in IMPORT TABLESPACE.
9 years ago
MDEV-11623 MariaDB 10.1 fails to start datadir created with MariaDB 10.0/MySQL 5.6 using innodb-page-size!=16K The storage format of FSP_SPACE_FLAGS was accidentally broken already in MariaDB 10.1.0. This fix is bringing the format in line with other MySQL and MariaDB release series. Please refer to the comments that were added to fsp0fsp.h for details. This is an INCOMPATIBLE CHANGE that affects users of page_compression and non-default innodb_page_size. Upgrading to this release will correct the flags in the data files. If you want to downgrade to earlier MariaDB 10.1.x, please refer to the test innodb.101_compatibility how to reset the FSP_SPACE_FLAGS in the files. NOTE: MariaDB 10.1.0 to 10.1.20 can misinterpret uncompressed data files with innodb_page_size=4k or 64k as compressed innodb_page_size=16k files, and then probably fail when trying to access the pages. See the comments in the function fsp_flags_convert_from_101() for detailed analysis. Move PAGE_COMPRESSION to FSP_SPACE_FLAGS bit position 16. In this way, compressed innodb_page_size=16k tablespaces will not be mistaken for uncompressed ones by MariaDB 10.1.0 to 10.1.20. Derive PAGE_COMPRESSION_LEVEL, ATOMIC_WRITES and DATA_DIR from the dict_table_t::flags when the table is available, in fil_space_for_table_exists_in_mem() or fil_open_single_table_tablespace(). During crash recovery, fil_load_single_table_tablespace() will use innodb_compression_level for the PAGE_COMPRESSION_LEVEL. FSP_FLAGS_MEM_MASK: A bitmap of the memory-only fil_space_t::flags that are not to be written to FSP_SPACE_FLAGS. Currently, these will include PAGE_COMPRESSION_LEVEL, ATOMIC_WRITES and DATA_DIR. Introduce the macro FSP_FLAGS_PAGE_SSIZE(). We only support one innodb_page_size for the whole instance. When creating a dummy tablespace for the redo log, use fil_space_t::flags=0. The flags are never written to the redo log files. Remove many FSP_FLAGS_SET_ macros. dict_tf_verify_flags(): Remove. This is basically only duplicating the logic of dict_tf_to_fsp_flags(), used in a debug assertion. fil_space_t::mark: Remove. This flag was not used for anything. fil_space_for_table_exists_in_mem(): Remove the unnecessary parameter mark_space, and add a parameter for table flags. Check that fil_space_t::flags match the table flags, and adjust the (memory-only) flags based on the table flags. fil_node_open_file(): Remove some redundant or unreachable conditions, do not use stderr for output, and avoid unnecessary server aborts. fil_user_tablespace_restore_page(): Convert the flags, so that the correct page_size will be used when restoring a page from the doublewrite buffer. fil_space_get_page_compressed(), fsp_flags_is_page_compressed(): Remove. It suffices to have fil_space_is_page_compressed(). FSP_FLAGS_WIDTH_DATA_DIR, FSP_FLAGS_WIDTH_PAGE_COMPRESSION_LEVEL, FSP_FLAGS_WIDTH_ATOMIC_WRITES: Remove, because these flags do not exist in the FSP_SPACE_FLAGS but only in memory. fsp_flags_try_adjust(): New function, to adjust the FSP_SPACE_FLAGS in page 0. Called by fil_open_single_table_tablespace(), fil_space_for_table_exists_in_mem(), innobase_start_or_create_for_mysql() except if --innodb-read-only is active. fsp_flags_is_valid(ulint): Reimplement from the scratch, with accurate comments. Do not display any details of detected inconsistencies, because the output could be confusing when dealing with MariaDB 10.1.x data files. fsp_flags_convert_from_101(ulint): Convert flags from buggy MariaDB 10.1.x format, or return ULINT_UNDEFINED if the flags cannot be in MariaDB 10.1.x format. fsp_flags_match(): Check the flags when probing files. Implemented based on fsp_flags_is_valid() and fsp_flags_convert_from_101(). dict_check_tablespaces_and_store_max_id(): Do not access the page after committing the mini-transaction. IMPORT TABLESPACE fixes: AbstractCallback::init(): Convert the flags. FetchIndexRootPages::operator(): Check that the tablespace flags match the table flags. Do not attempt to convert tablespace flags to table flags, because the conversion would necessarily be lossy. PageConverter::update_header(): Write back the correct flags. This takes care of the flags in IMPORT TABLESPACE.
9 years ago
16 years ago
MDEV-11623 MariaDB 10.1 fails to start datadir created with MariaDB 10.0/MySQL 5.6 using innodb-page-size!=16K The storage format of FSP_SPACE_FLAGS was accidentally broken already in MariaDB 10.1.0. This fix is bringing the format in line with other MySQL and MariaDB release series. Please refer to the comments that were added to fsp0fsp.h for details. This is an INCOMPATIBLE CHANGE that affects users of page_compression and non-default innodb_page_size. Upgrading to this release will correct the flags in the data files. If you want to downgrade to earlier MariaDB 10.1.x, please refer to the test innodb.101_compatibility how to reset the FSP_SPACE_FLAGS in the files. NOTE: MariaDB 10.1.0 to 10.1.20 can misinterpret uncompressed data files with innodb_page_size=4k or 64k as compressed innodb_page_size=16k files, and then probably fail when trying to access the pages. See the comments in the function fsp_flags_convert_from_101() for detailed analysis. Move PAGE_COMPRESSION to FSP_SPACE_FLAGS bit position 16. In this way, compressed innodb_page_size=16k tablespaces will not be mistaken for uncompressed ones by MariaDB 10.1.0 to 10.1.20. Derive PAGE_COMPRESSION_LEVEL, ATOMIC_WRITES and DATA_DIR from the dict_table_t::flags when the table is available, in fil_space_for_table_exists_in_mem() or fil_open_single_table_tablespace(). During crash recovery, fil_load_single_table_tablespace() will use innodb_compression_level for the PAGE_COMPRESSION_LEVEL. FSP_FLAGS_MEM_MASK: A bitmap of the memory-only fil_space_t::flags that are not to be written to FSP_SPACE_FLAGS. Currently, these will include PAGE_COMPRESSION_LEVEL, ATOMIC_WRITES and DATA_DIR. Introduce the macro FSP_FLAGS_PAGE_SSIZE(). We only support one innodb_page_size for the whole instance. When creating a dummy tablespace for the redo log, use fil_space_t::flags=0. The flags are never written to the redo log files. Remove many FSP_FLAGS_SET_ macros. dict_tf_verify_flags(): Remove. This is basically only duplicating the logic of dict_tf_to_fsp_flags(), used in a debug assertion. fil_space_t::mark: Remove. This flag was not used for anything. fil_space_for_table_exists_in_mem(): Remove the unnecessary parameter mark_space, and add a parameter for table flags. Check that fil_space_t::flags match the table flags, and adjust the (memory-only) flags based on the table flags. fil_node_open_file(): Remove some redundant or unreachable conditions, do not use stderr for output, and avoid unnecessary server aborts. fil_user_tablespace_restore_page(): Convert the flags, so that the correct page_size will be used when restoring a page from the doublewrite buffer. fil_space_get_page_compressed(), fsp_flags_is_page_compressed(): Remove. It suffices to have fil_space_is_page_compressed(). FSP_FLAGS_WIDTH_DATA_DIR, FSP_FLAGS_WIDTH_PAGE_COMPRESSION_LEVEL, FSP_FLAGS_WIDTH_ATOMIC_WRITES: Remove, because these flags do not exist in the FSP_SPACE_FLAGS but only in memory. fsp_flags_try_adjust(): New function, to adjust the FSP_SPACE_FLAGS in page 0. Called by fil_open_single_table_tablespace(), fil_space_for_table_exists_in_mem(), innobase_start_or_create_for_mysql() except if --innodb-read-only is active. fsp_flags_is_valid(ulint): Reimplement from the scratch, with accurate comments. Do not display any details of detected inconsistencies, because the output could be confusing when dealing with MariaDB 10.1.x data files. fsp_flags_convert_from_101(ulint): Convert flags from buggy MariaDB 10.1.x format, or return ULINT_UNDEFINED if the flags cannot be in MariaDB 10.1.x format. fsp_flags_match(): Check the flags when probing files. Implemented based on fsp_flags_is_valid() and fsp_flags_convert_from_101(). dict_check_tablespaces_and_store_max_id(): Do not access the page after committing the mini-transaction. IMPORT TABLESPACE fixes: AbstractCallback::init(): Convert the flags. FetchIndexRootPages::operator(): Check that the tablespace flags match the table flags. Do not attempt to convert tablespace flags to table flags, because the conversion would necessarily be lossy. PageConverter::update_header(): Write back the correct flags. This takes care of the flags in IMPORT TABLESPACE.
9 years ago
16 years ago
MDEV-11623 MariaDB 10.1 fails to start datadir created with MariaDB 10.0/MySQL 5.6 using innodb-page-size!=16K The storage format of FSP_SPACE_FLAGS was accidentally broken already in MariaDB 10.1.0. This fix is bringing the format in line with other MySQL and MariaDB release series. Please refer to the comments that were added to fsp0fsp.h for details. This is an INCOMPATIBLE CHANGE that affects users of page_compression and non-default innodb_page_size. Upgrading to this release will correct the flags in the data files. If you want to downgrade to earlier MariaDB 10.1.x, please refer to the test innodb.101_compatibility how to reset the FSP_SPACE_FLAGS in the files. NOTE: MariaDB 10.1.0 to 10.1.20 can misinterpret uncompressed data files with innodb_page_size=4k or 64k as compressed innodb_page_size=16k files, and then probably fail when trying to access the pages. See the comments in the function fsp_flags_convert_from_101() for detailed analysis. Move PAGE_COMPRESSION to FSP_SPACE_FLAGS bit position 16. In this way, compressed innodb_page_size=16k tablespaces will not be mistaken for uncompressed ones by MariaDB 10.1.0 to 10.1.20. Derive PAGE_COMPRESSION_LEVEL, ATOMIC_WRITES and DATA_DIR from the dict_table_t::flags when the table is available, in fil_space_for_table_exists_in_mem() or fil_open_single_table_tablespace(). During crash recovery, fil_load_single_table_tablespace() will use innodb_compression_level for the PAGE_COMPRESSION_LEVEL. FSP_FLAGS_MEM_MASK: A bitmap of the memory-only fil_space_t::flags that are not to be written to FSP_SPACE_FLAGS. Currently, these will include PAGE_COMPRESSION_LEVEL, ATOMIC_WRITES and DATA_DIR. Introduce the macro FSP_FLAGS_PAGE_SSIZE(). We only support one innodb_page_size for the whole instance. When creating a dummy tablespace for the redo log, use fil_space_t::flags=0. The flags are never written to the redo log files. Remove many FSP_FLAGS_SET_ macros. dict_tf_verify_flags(): Remove. This is basically only duplicating the logic of dict_tf_to_fsp_flags(), used in a debug assertion. fil_space_t::mark: Remove. This flag was not used for anything. fil_space_for_table_exists_in_mem(): Remove the unnecessary parameter mark_space, and add a parameter for table flags. Check that fil_space_t::flags match the table flags, and adjust the (memory-only) flags based on the table flags. fil_node_open_file(): Remove some redundant or unreachable conditions, do not use stderr for output, and avoid unnecessary server aborts. fil_user_tablespace_restore_page(): Convert the flags, so that the correct page_size will be used when restoring a page from the doublewrite buffer. fil_space_get_page_compressed(), fsp_flags_is_page_compressed(): Remove. It suffices to have fil_space_is_page_compressed(). FSP_FLAGS_WIDTH_DATA_DIR, FSP_FLAGS_WIDTH_PAGE_COMPRESSION_LEVEL, FSP_FLAGS_WIDTH_ATOMIC_WRITES: Remove, because these flags do not exist in the FSP_SPACE_FLAGS but only in memory. fsp_flags_try_adjust(): New function, to adjust the FSP_SPACE_FLAGS in page 0. Called by fil_open_single_table_tablespace(), fil_space_for_table_exists_in_mem(), innobase_start_or_create_for_mysql() except if --innodb-read-only is active. fsp_flags_is_valid(ulint): Reimplement from the scratch, with accurate comments. Do not display any details of detected inconsistencies, because the output could be confusing when dealing with MariaDB 10.1.x data files. fsp_flags_convert_from_101(ulint): Convert flags from buggy MariaDB 10.1.x format, or return ULINT_UNDEFINED if the flags cannot be in MariaDB 10.1.x format. fsp_flags_match(): Check the flags when probing files. Implemented based on fsp_flags_is_valid() and fsp_flags_convert_from_101(). dict_check_tablespaces_and_store_max_id(): Do not access the page after committing the mini-transaction. IMPORT TABLESPACE fixes: AbstractCallback::init(): Convert the flags. FetchIndexRootPages::operator(): Check that the tablespace flags match the table flags. Do not attempt to convert tablespace flags to table flags, because the conversion would necessarily be lossy. PageConverter::update_header(): Write back the correct flags. This takes care of the flags in IMPORT TABLESPACE.
9 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
16 years ago
9 years ago
9 years ago
10 years ago
10 years ago
10 years ago
9 years ago
Merge Google encryption commit 195158e9889365dc3298f8c1f3bcaa745992f27f Author: Minli Zhu <minliz@google.com> Date: Mon Nov 25 11:05:55 2013 -0800 Innodb redo log encryption/decryption. Use start lsn of a log block as part of AES CTR counter. Record key version with each checkpoint. Internally key version 0 means no encryption. Tests done (see test_innodb_log_encryption.sh for detail): - Verify flag innodb_encrypt_log on or off, combined with various key versions passed through CLI, and dynamically set after startup, will not corrupt database. This includes tests from being unencrypted to encrypted, and encrypted to unencrypted. - Verify start-up with no redo logs succeeds. - Verify fresh start-up succeeds. Change-Id: I4ce4c2afdf3076be2fce90ebbc2a7ce01184b612 commit c1b97273659f07866758c25f4a56f680a1fbad24 Author: Jonas Oreland <jonaso@google.com> Date: Tue Dec 3 18:47:27 2013 +0100 encryption of aria data&index files this patch implements encryption of aria data & index files. this is implemented as 1) add read/write hooks (renamed from callbacks) that does encrypt/decrypt (also add pre_read and post_write hooks) 2) modify page headers for data/index to contain key version (making the data-page header size different for with/without encryption) 3) modify index page 0 to contain IV (and crypt header) 4) AES CRT crypt functions 5) counter block is implemented using combination of page no, lsn and table specific id NOTE: 1) log files are not encrypted, this is not needed for if aria is only used for internal temporary tables and they are not transactional (i.e not logged) 2) all encrypted tables are using PAGE_CHECKSUM (crc) normal internal temporary tables are (currently) not CHECKSUM:ed 3) This patch adds insert-order semantics to aria block_format. The default behaviour of aria block-format is best-fit, meaning that rows gets allocated to page trying to fill the pages as much as possible. However, certain sql constructs materialize temporary result in tmp-tables, and expect that a table scan will later return the rows in the same order they were inserted. This implementation of insert-order is only enabled when explicitly requested by sql-layer. CHANGES: 1) found bug in ma_write that made code try to abort a record that was never written unsure why this is not exposed Change-Id: Ia82bbaa92e2c0629c08693c5add2f56b815c0509 commit 89dc1ab651fe0205d55b4eb588f62df550aa65fc Author: Jonas Oreland <jonaso@google.com> Date: Mon Feb 17 08:04:50 2014 -0800 Implement encryption of innodb datafiles. Pages are encrypted before written to disk and decrypted when read from disk. Each page except first page (page 0) in tablespace is encrypted. Page 0 is unencrypted and contains IV for the tablespace. FIL_PAGE_FILE_FLUSH_LSN on each page (except page 0) is used to store a 32-bit key-version, so that multiple keys can be active in a tablespace simultaneous. The other 32-bit of the FIL_PAGE_FILE_FLUSH_LSN field contains a checksum that is computed after encryption. This checksum is used by innochecksum and when restoring from double-write-buffer. The encryption is performed using AES CRT. Monitoring of encryption is enabled using new IS-table INNODB_TABLESPACES_ENCRYPTION. In addition to that new status variables innodb_encryption_rotation_{ pages_read_from_cache, pages_read_from_disk, pages_modified,pages_flushed } has been added. The following tunables are introduces - innodb_encrypt_tables - innodb_encryption_threads - innodb_encryption_rotate_key_age - innodb_encryption_rotation_iops Change-Id: I8f651795a30b52e71b16d6bc9cb7559be349d0b2 commit a17eef2f6948e58219c9e26fc35633d6fd4de1de Author: Andrew Ford <andrewford@google.com> Date: Thu Jan 2 15:43:09 2014 -0800 Key management skeleton with debug hooks. Change-Id: Ifd6aa3743d7ea291c70083f433a059c439aed866 commit 68a399838ad72264fd61b3dc67fecd29bbdb0af1 Author: Andrew Ford <andrewford@google.com> Date: Mon Oct 28 16:27:44 2013 -0700 Add AES-128 CTR and GCM encryption classes. Change-Id: I116305eced2a233db15306bc2ef5b9d398d1a3a2
11 years ago
Merge Google encryption commit 195158e9889365dc3298f8c1f3bcaa745992f27f Author: Minli Zhu <minliz@google.com> Date: Mon Nov 25 11:05:55 2013 -0800 Innodb redo log encryption/decryption. Use start lsn of a log block as part of AES CTR counter. Record key version with each checkpoint. Internally key version 0 means no encryption. Tests done (see test_innodb_log_encryption.sh for detail): - Verify flag innodb_encrypt_log on or off, combined with various key versions passed through CLI, and dynamically set after startup, will not corrupt database. This includes tests from being unencrypted to encrypted, and encrypted to unencrypted. - Verify start-up with no redo logs succeeds. - Verify fresh start-up succeeds. Change-Id: I4ce4c2afdf3076be2fce90ebbc2a7ce01184b612 commit c1b97273659f07866758c25f4a56f680a1fbad24 Author: Jonas Oreland <jonaso@google.com> Date: Tue Dec 3 18:47:27 2013 +0100 encryption of aria data&index files this patch implements encryption of aria data & index files. this is implemented as 1) add read/write hooks (renamed from callbacks) that does encrypt/decrypt (also add pre_read and post_write hooks) 2) modify page headers for data/index to contain key version (making the data-page header size different for with/without encryption) 3) modify index page 0 to contain IV (and crypt header) 4) AES CRT crypt functions 5) counter block is implemented using combination of page no, lsn and table specific id NOTE: 1) log files are not encrypted, this is not needed for if aria is only used for internal temporary tables and they are not transactional (i.e not logged) 2) all encrypted tables are using PAGE_CHECKSUM (crc) normal internal temporary tables are (currently) not CHECKSUM:ed 3) This patch adds insert-order semantics to aria block_format. The default behaviour of aria block-format is best-fit, meaning that rows gets allocated to page trying to fill the pages as much as possible. However, certain sql constructs materialize temporary result in tmp-tables, and expect that a table scan will later return the rows in the same order they were inserted. This implementation of insert-order is only enabled when explicitly requested by sql-layer. CHANGES: 1) found bug in ma_write that made code try to abort a record that was never written unsure why this is not exposed Change-Id: Ia82bbaa92e2c0629c08693c5add2f56b815c0509 commit 89dc1ab651fe0205d55b4eb588f62df550aa65fc Author: Jonas Oreland <jonaso@google.com> Date: Mon Feb 17 08:04:50 2014 -0800 Implement encryption of innodb datafiles. Pages are encrypted before written to disk and decrypted when read from disk. Each page except first page (page 0) in tablespace is encrypted. Page 0 is unencrypted and contains IV for the tablespace. FIL_PAGE_FILE_FLUSH_LSN on each page (except page 0) is used to store a 32-bit key-version, so that multiple keys can be active in a tablespace simultaneous. The other 32-bit of the FIL_PAGE_FILE_FLUSH_LSN field contains a checksum that is computed after encryption. This checksum is used by innochecksum and when restoring from double-write-buffer. The encryption is performed using AES CRT. Monitoring of encryption is enabled using new IS-table INNODB_TABLESPACES_ENCRYPTION. In addition to that new status variables innodb_encryption_rotation_{ pages_read_from_cache, pages_read_from_disk, pages_modified,pages_flushed } has been added. The following tunables are introduces - innodb_encrypt_tables - innodb_encryption_threads - innodb_encryption_rotate_key_age - innodb_encryption_rotation_iops Change-Id: I8f651795a30b52e71b16d6bc9cb7559be349d0b2 commit a17eef2f6948e58219c9e26fc35633d6fd4de1de Author: Andrew Ford <andrewford@google.com> Date: Thu Jan 2 15:43:09 2014 -0800 Key management skeleton with debug hooks. Change-Id: Ifd6aa3743d7ea291c70083f433a059c439aed866 commit 68a399838ad72264fd61b3dc67fecd29bbdb0af1 Author: Andrew Ford <andrewford@google.com> Date: Mon Oct 28 16:27:44 2013 -0700 Add AES-128 CTR and GCM encryption classes. Change-Id: I116305eced2a233db15306bc2ef5b9d398d1a3a2
11 years ago
Merge Google encryption commit 195158e9889365dc3298f8c1f3bcaa745992f27f Author: Minli Zhu <minliz@google.com> Date: Mon Nov 25 11:05:55 2013 -0800 Innodb redo log encryption/decryption. Use start lsn of a log block as part of AES CTR counter. Record key version with each checkpoint. Internally key version 0 means no encryption. Tests done (see test_innodb_log_encryption.sh for detail): - Verify flag innodb_encrypt_log on or off, combined with various key versions passed through CLI, and dynamically set after startup, will not corrupt database. This includes tests from being unencrypted to encrypted, and encrypted to unencrypted. - Verify start-up with no redo logs succeeds. - Verify fresh start-up succeeds. Change-Id: I4ce4c2afdf3076be2fce90ebbc2a7ce01184b612 commit c1b97273659f07866758c25f4a56f680a1fbad24 Author: Jonas Oreland <jonaso@google.com> Date: Tue Dec 3 18:47:27 2013 +0100 encryption of aria data&index files this patch implements encryption of aria data & index files. this is implemented as 1) add read/write hooks (renamed from callbacks) that does encrypt/decrypt (also add pre_read and post_write hooks) 2) modify page headers for data/index to contain key version (making the data-page header size different for with/without encryption) 3) modify index page 0 to contain IV (and crypt header) 4) AES CRT crypt functions 5) counter block is implemented using combination of page no, lsn and table specific id NOTE: 1) log files are not encrypted, this is not needed for if aria is only used for internal temporary tables and they are not transactional (i.e not logged) 2) all encrypted tables are using PAGE_CHECKSUM (crc) normal internal temporary tables are (currently) not CHECKSUM:ed 3) This patch adds insert-order semantics to aria block_format. The default behaviour of aria block-format is best-fit, meaning that rows gets allocated to page trying to fill the pages as much as possible. However, certain sql constructs materialize temporary result in tmp-tables, and expect that a table scan will later return the rows in the same order they were inserted. This implementation of insert-order is only enabled when explicitly requested by sql-layer. CHANGES: 1) found bug in ma_write that made code try to abort a record that was never written unsure why this is not exposed Change-Id: Ia82bbaa92e2c0629c08693c5add2f56b815c0509 commit 89dc1ab651fe0205d55b4eb588f62df550aa65fc Author: Jonas Oreland <jonaso@google.com> Date: Mon Feb 17 08:04:50 2014 -0800 Implement encryption of innodb datafiles. Pages are encrypted before written to disk and decrypted when read from disk. Each page except first page (page 0) in tablespace is encrypted. Page 0 is unencrypted and contains IV for the tablespace. FIL_PAGE_FILE_FLUSH_LSN on each page (except page 0) is used to store a 32-bit key-version, so that multiple keys can be active in a tablespace simultaneous. The other 32-bit of the FIL_PAGE_FILE_FLUSH_LSN field contains a checksum that is computed after encryption. This checksum is used by innochecksum and when restoring from double-write-buffer. The encryption is performed using AES CRT. Monitoring of encryption is enabled using new IS-table INNODB_TABLESPACES_ENCRYPTION. In addition to that new status variables innodb_encryption_rotation_{ pages_read_from_cache, pages_read_from_disk, pages_modified,pages_flushed } has been added. The following tunables are introduces - innodb_encrypt_tables - innodb_encryption_threads - innodb_encryption_rotate_key_age - innodb_encryption_rotation_iops Change-Id: I8f651795a30b52e71b16d6bc9cb7559be349d0b2 commit a17eef2f6948e58219c9e26fc35633d6fd4de1de Author: Andrew Ford <andrewford@google.com> Date: Thu Jan 2 15:43:09 2014 -0800 Key management skeleton with debug hooks. Change-Id: Ifd6aa3743d7ea291c70083f433a059c439aed866 commit 68a399838ad72264fd61b3dc67fecd29bbdb0af1 Author: Andrew Ford <andrewford@google.com> Date: Mon Oct 28 16:27:44 2013 -0700 Add AES-128 CTR and GCM encryption classes. Change-Id: I116305eced2a233db15306bc2ef5b9d398d1a3a2
11 years ago
  1. /*****************************************************************************
  2. Copyright (c) 1995, 2016, Oracle and/or its affiliates. All Rights Reserved.
  3. Copyright (c) 2013, 2017, MariaDB Corporation. All Rights Reserved.
  4. This program is free software; you can redistribute it and/or modify it under
  5. the terms of the GNU General Public License as published by the Free Software
  6. Foundation; version 2 of the License.
  7. This program is distributed in the hope that it will be useful, but WITHOUT
  8. ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
  9. FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
  10. You should have received a copy of the GNU General Public License along with
  11. this program; if not, write to the Free Software Foundation, Inc.,
  12. 51 Franklin Street, Suite 500, Boston, MA 02110-1335 USA
  13. *****************************************************************************/
  14. /**************************************************//**
  15. @file include/fil0fil.h
  16. The low-level file system
  17. Created 10/25/1995 Heikki Tuuri
  18. *******************************************************/
  19. #ifndef fil0fil_h
  20. #define fil0fil_h
  21. #include "univ.i"
  22. #ifndef UNIV_INNOCHECKSUM
  23. #include "dict0types.h"
  24. #include "ut0byte.h"
  25. #include "os0file.h"
  26. #include "hash0hash.h"
  27. #ifndef UNIV_HOTBACKUP
  28. #include "sync0rw.h"
  29. #include "ibuf0types.h"
  30. #include "log0log.h"
  31. #endif /* !UNIV_HOTBACKUP */
  32. #include "trx0types.h"
  33. #include <list>
  34. // Forward declaration
  35. struct trx_t;
  36. typedef std::list<const char*> space_name_list_t;
  37. /** When mysqld is run, the default directory "." is the mysqld datadir,
  38. but in the MySQL Embedded Server Library and mysqlbackup it is not the default
  39. directory, and we must set the base file path explicitly */
  40. extern const char* fil_path_to_mysql_datadir;
  41. /** Initial size of a single-table tablespace in pages */
  42. #define FIL_IBD_FILE_INITIAL_SIZE 4
  43. /** 'null' (undefined) page offset in the context of file spaces */
  44. #define FIL_NULL ULINT32_UNDEFINED
  45. /* Space address data type; this is intended to be used when
  46. addresses accurate to a byte are stored in file pages. If the page part
  47. of the address is FIL_NULL, the address is considered undefined. */
  48. typedef byte fil_faddr_t; /*!< 'type' definition in C: an address
  49. stored in a file page is a string of bytes */
  50. #define FIL_ADDR_PAGE 0 /* first in address is the page offset */
  51. #define FIL_ADDR_BYTE 4 /* then comes 2-byte byte offset within page*/
  52. #define FIL_ADDR_SIZE 6 /* address size is 6 bytes */
  53. /** File space address */
  54. struct fil_addr_t{
  55. ulint page; /*!< page number within a space */
  56. ulint boffset; /*!< byte offset within the page */
  57. };
  58. /** The null file address */
  59. extern fil_addr_t fil_addr_null;
  60. #endif /* !UNIV_INNOCHECKSUM */
  61. /** The byte offsets on a file page for various variables @{ */
  62. #define FIL_PAGE_SPACE_OR_CHKSUM 0 /*!< in < MySQL-4.0.14 space id the
  63. page belongs to (== 0) but in later
  64. versions the 'new' checksum of the
  65. page */
  66. #define FIL_PAGE_OFFSET 4 /*!< page offset inside space */
  67. #define FIL_PAGE_PREV 8 /*!< if there is a 'natural'
  68. predecessor of the page, its
  69. offset. Otherwise FIL_NULL.
  70. This field is not set on BLOB
  71. pages, which are stored as a
  72. singly-linked list. See also
  73. FIL_PAGE_NEXT. */
  74. #define FIL_PAGE_NEXT 12 /*!< if there is a 'natural' successor
  75. of the page, its offset.
  76. Otherwise FIL_NULL.
  77. B-tree index pages
  78. (FIL_PAGE_TYPE contains FIL_PAGE_INDEX)
  79. on the same PAGE_LEVEL are maintained
  80. as a doubly linked list via
  81. FIL_PAGE_PREV and FIL_PAGE_NEXT
  82. in the collation order of the
  83. smallest user record on each page. */
  84. #define FIL_PAGE_LSN 16 /*!< lsn of the end of the newest
  85. modification log record to the page */
  86. #define FIL_PAGE_TYPE 24 /*!< file page type: FIL_PAGE_INDEX,...,
  87. 2 bytes.
  88. The contents of this field can only
  89. be trusted in the following case:
  90. if the page is an uncompressed
  91. B-tree index page, then it is
  92. guaranteed that the value is
  93. FIL_PAGE_INDEX.
  94. The opposite does not hold.
  95. In tablespaces created by
  96. MySQL/InnoDB 5.1.7 or later, the
  97. contents of this field is valid
  98. for all uncompressed pages. */
  99. #define FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION 26 /*!< for the first page
  100. in a system tablespace data file
  101. (ibdata*, not *.ibd): the file has
  102. been flushed to disk at least up
  103. to this lsn
  104. for other pages: a 32-bit key version
  105. used to encrypt the page + 32-bit checksum
  106. or 64 bits of zero if no encryption
  107. */
  108. #define FIL_PAGE_ARCH_LOG_NO_OR_SPACE_ID 34 /*!< starting from 4.1.x this
  109. contains the space id of the page */
  110. #define FIL_PAGE_SPACE_ID FIL_PAGE_ARCH_LOG_NO_OR_SPACE_ID
  111. #define FIL_PAGE_DATA 38 /*!< start of the data on the page */
  112. /* Following are used when page compression is used */
  113. #define FIL_PAGE_COMPRESSED_SIZE 2 /*!< Number of bytes used to store
  114. actual payload data size on
  115. compressed pages. */
  116. #define FIL_PAGE_COMPRESSION_METHOD_SIZE 2
  117. /*!< Number of bytes used to store
  118. actual compression method. */
  119. /* @} */
  120. /** File page trailer @{ */
  121. #define FIL_PAGE_END_LSN_OLD_CHKSUM 8 /*!< the low 4 bytes of this are used
  122. to store the page checksum, the
  123. last 4 bytes should be identical
  124. to the last 4 bytes of FIL_PAGE_LSN */
  125. #define FIL_PAGE_DATA_END 8 /*!< size of the page trailer */
  126. /* @} */
  127. /** File page types (values of FIL_PAGE_TYPE) @{ */
  128. #define FIL_PAGE_PAGE_COMPRESSED_ENCRYPTED 37401 /*!< Page is compressed and
  129. then encrypted */
  130. #define FIL_PAGE_PAGE_COMPRESSED 34354 /*!< Page compressed page */
  131. #define FIL_PAGE_INDEX 17855 /*!< B-tree node */
  132. #define FIL_PAGE_UNDO_LOG 2 /*!< Undo log page */
  133. #define FIL_PAGE_INODE 3 /*!< Index node */
  134. #define FIL_PAGE_IBUF_FREE_LIST 4 /*!< Insert buffer free list */
  135. /* File page types introduced in MySQL/InnoDB 5.1.7 */
  136. #define FIL_PAGE_TYPE_ALLOCATED 0 /*!< Freshly allocated page */
  137. #define FIL_PAGE_IBUF_BITMAP 5 /*!< Insert buffer bitmap */
  138. #define FIL_PAGE_TYPE_SYS 6 /*!< System page */
  139. #define FIL_PAGE_TYPE_TRX_SYS 7 /*!< Transaction system data */
  140. #define FIL_PAGE_TYPE_FSP_HDR 8 /*!< File space header */
  141. #define FIL_PAGE_TYPE_XDES 9 /*!< Extent descriptor page */
  142. #define FIL_PAGE_TYPE_BLOB 10 /*!< Uncompressed BLOB page */
  143. #define FIL_PAGE_TYPE_ZBLOB 11 /*!< First compressed BLOB page */
  144. #define FIL_PAGE_TYPE_ZBLOB2 12 /*!< Subsequent compressed BLOB page */
  145. #define FIL_PAGE_TYPE_COMPRESSED 13 /*!< Compressed page */
  146. #define FIL_PAGE_TYPE_LAST FIL_PAGE_TYPE_COMPRESSED
  147. /*!< Last page type */
  148. /* @} */
  149. #ifndef UNIV_INNOCHECKSUM
  150. /** Space types @{ */
  151. #define FIL_TABLESPACE 501 /*!< tablespace */
  152. #define FIL_LOG 502 /*!< redo log */
  153. /* @} */
  154. /** Structure containing encryption specification */
  155. struct fil_space_crypt_t;
  156. /** Enum values for encryption table option */
  157. enum fil_encryption_t {
  158. /** Encrypted if innodb_encrypt_tables=ON (srv_encrypt_tables) */
  159. FIL_ENCRYPTION_DEFAULT,
  160. /** Encrypted */
  161. FIL_ENCRYPTION_ON,
  162. /** Not encrypted */
  163. FIL_ENCRYPTION_OFF
  164. };
  165. /** The number of fsyncs done to the log */
  166. extern ulint fil_n_log_flushes;
  167. /** Number of pending redo log flushes */
  168. extern ulint fil_n_pending_log_flushes;
  169. /** Number of pending tablespace flushes */
  170. extern ulint fil_n_pending_tablespace_flushes;
  171. /** Number of files currently open */
  172. extern ulint fil_n_file_opened;
  173. struct fsp_open_info {
  174. ibool success; /*!< Has the tablespace been opened? */
  175. const char* check_msg; /*!< fil_check_first_page() message */
  176. ibool valid; /*!< Is the tablespace valid? */
  177. os_file_t file; /*!< File handle */
  178. char* filepath; /*!< File path to open */
  179. lsn_t lsn; /*!< Flushed LSN from header page */
  180. ulint id; /*!< Space ID */
  181. ulint flags; /*!< Tablespace flags */
  182. ulint encryption_error; /*!< if an encryption error occurs */
  183. fil_space_crypt_t* crypt_data; /*!< crypt data */
  184. dict_table_t* table; /*!< table */
  185. };
  186. struct fil_space_t;
  187. /** File node of a tablespace or the log data space */
  188. struct fil_node_t {
  189. fil_space_t* space; /*!< backpointer to the space where this node
  190. belongs */
  191. char* name; /*!< path to the file */
  192. ibool open; /*!< TRUE if file open */
  193. os_file_t handle; /*!< OS handle to the file, if file open */
  194. os_event_t sync_event;/*!< Condition event to group and
  195. serialize calls to fsync;
  196. os_event_set() and os_event_reset()
  197. are protected by fil_system_t::mutex */
  198. ibool is_raw_disk;/*!< TRUE if the 'file' is actually a raw
  199. device or a raw disk partition */
  200. ulint size; /*!< size of the file in database pages, 0 if
  201. not known yet; the possible last incomplete
  202. megabyte may be ignored if space == 0 */
  203. ulint n_pending;
  204. /*!< count of pending i/o's on this file;
  205. closing of the file is not allowed if
  206. this is > 0 */
  207. ulint n_pending_flushes;
  208. /*!< count of pending flushes on this file;
  209. closing of the file is not allowed if
  210. this is > 0 */
  211. ibool being_extended;
  212. /*!< TRUE if the node is currently
  213. being extended. */
  214. ib_int64_t modification_counter;/*!< when we write to the file we
  215. increment this by one */
  216. ib_int64_t flush_counter;/*!< up to what
  217. modification_counter value we have
  218. flushed the modifications to disk */
  219. ulint file_block_size;/*!< file system block size */
  220. UT_LIST_NODE_T(fil_node_t) chain;
  221. /*!< link field for the file chain */
  222. UT_LIST_NODE_T(fil_node_t) LRU;
  223. /*!< link field for the LRU list */
  224. ulint magic_n;/*!< FIL_NODE_MAGIC_N */
  225. };
  226. /** Value of fil_node_t::magic_n */
  227. #define FIL_NODE_MAGIC_N 89389
  228. /** Tablespace or log data space: let us call them by a common name space */
  229. struct fil_space_t {
  230. char* name; /*!< space name = the path to the first file in
  231. it */
  232. ulint id; /*!< space id */
  233. ib_int64_t tablespace_version;
  234. /*!< in DISCARD/IMPORT this timestamp
  235. is used to check if we should ignore
  236. an insert buffer merge request for a
  237. page because it actually was for the
  238. previous incarnation of the space */
  239. ibool stop_ios;/*!< TRUE if we want to rename the
  240. .ibd file of tablespace and want to
  241. stop temporarily posting of new i/o
  242. requests on the file */
  243. bool stop_new_ops;
  244. /*!< we set this true when we start
  245. deleting a single-table tablespace.
  246. When this is set following new ops
  247. are not allowed:
  248. * read IO request
  249. * ibuf merge
  250. * file flush
  251. Note that we can still possibly have
  252. new write operations because we don't
  253. check this flag when doing flush
  254. batches. */
  255. ulint purpose;/*!< FIL_TABLESPACE, FIL_LOG, or
  256. FIL_ARCH_LOG */
  257. UT_LIST_BASE_NODE_T(fil_node_t) chain;
  258. /*!< base node for the file chain */
  259. ulint size; /*!< space size in pages; 0 if a single-table
  260. tablespace whose size we do not know yet;
  261. last incomplete megabytes in data files may be
  262. ignored if space == 0 */
  263. ulint recv_size;
  264. /*!< recovered tablespace size in pages;
  265. 0 if no size change was read from the redo log,
  266. or if the size change was implemented */
  267. ulint flags; /*!< FSP_SPACE_FLAGS and FSP_FLAGS_MEM_ flags;
  268. see fsp0fsp.h,
  269. fsp_flags_is_valid(),
  270. fsp_flags_get_zip_size() */
  271. ulint n_reserved_extents;
  272. /*!< number of reserved free extents for
  273. ongoing operations like B-tree page split */
  274. ulint n_pending_flushes; /*!< this is positive when flushing
  275. the tablespace to disk; dropping of the
  276. tablespace is forbidden if this is positive */
  277. ulint n_pending_ops;/*!< this is positive when we
  278. have pending operations against this
  279. tablespace. The pending operations can
  280. be ibuf merges or lock validation code
  281. trying to read a block.
  282. Dropping of the tablespace is forbidden
  283. if this is positive */
  284. hash_node_t hash; /*!< hash chain node */
  285. hash_node_t name_hash;/*!< hash chain the name_hash table */
  286. #ifndef UNIV_HOTBACKUP
  287. prio_rw_lock_t latch; /*!< latch protecting the file space storage
  288. allocation */
  289. #endif /* !UNIV_HOTBACKUP */
  290. UT_LIST_NODE_T(fil_space_t) unflushed_spaces;
  291. /*!< list of spaces with at least one unflushed
  292. file we have written to */
  293. bool is_in_unflushed_spaces;
  294. /*!< true if this space is currently in
  295. unflushed_spaces */
  296. /** True if srv_pass_corrupt_table=true and tablespace contains
  297. corrupted page. */
  298. bool is_corrupt;
  299. /*!< true if tablespace corrupted */
  300. bool printed_compression_failure;
  301. /*!< true if we have already printed
  302. compression failure */
  303. fil_space_crypt_t* crypt_data;
  304. /*!< tablespace crypt data or NULL */
  305. bool page_0_crypt_read;
  306. /*!< tablespace crypt data has been
  307. read */
  308. ulint file_block_size;
  309. /*!< file system block size */
  310. UT_LIST_NODE_T(fil_space_t) space_list;
  311. /*!< list of all spaces */
  312. /*!< Protected by fil_system */
  313. UT_LIST_NODE_T(fil_space_t) rotation_list;
  314. /*!< list of spaces needing
  315. key rotation */
  316. bool is_in_rotation_list;
  317. /*!< true if this space is
  318. currently in key rotation list */
  319. ulint magic_n;/*!< FIL_SPACE_MAGIC_N */
  320. /** @return whether the tablespace is about to be dropped or truncated */
  321. bool is_stopping() const
  322. {
  323. return stop_new_ops;
  324. }
  325. };
  326. /** Value of fil_space_t::magic_n */
  327. #define FIL_SPACE_MAGIC_N 89472
  328. /** The tablespace memory cache; also the totality of logs (the log
  329. data space) is stored here; below we talk about tablespaces, but also
  330. the ib_logfiles form a 'space' and it is handled here */
  331. struct fil_system_t {
  332. #ifndef UNIV_HOTBACKUP
  333. ib_mutex_t mutex; /*!< The mutex protecting the cache */
  334. #endif /* !UNIV_HOTBACKUP */
  335. hash_table_t* spaces; /*!< The hash table of spaces in the
  336. system; they are hashed on the space
  337. id */
  338. hash_table_t* name_hash; /*!< hash table based on the space
  339. name */
  340. UT_LIST_BASE_NODE_T(fil_node_t) LRU;
  341. /*!< base node for the LRU list of the
  342. most recently used open files with no
  343. pending i/o's; if we start an i/o on
  344. the file, we first remove it from this
  345. list, and return it to the start of
  346. the list when the i/o ends;
  347. log files and the system tablespace are
  348. not put to this list: they are opened
  349. after the startup, and kept open until
  350. shutdown */
  351. UT_LIST_BASE_NODE_T(fil_space_t) unflushed_spaces;
  352. /*!< base node for the list of those
  353. tablespaces whose files contain
  354. unflushed writes; those spaces have
  355. at least one file node where
  356. modification_counter > flush_counter */
  357. ulint n_open; /*!< number of files currently open */
  358. ulint max_n_open; /*!< n_open is not allowed to exceed
  359. this */
  360. ib_int64_t modification_counter;/*!< when we write to a file we
  361. increment this by one */
  362. ulint max_assigned_id;/*!< maximum space id in the existing
  363. tables, or assigned during the time
  364. mysqld has been up; at an InnoDB
  365. startup we scan the data dictionary
  366. and set here the maximum of the
  367. space id's of the tables there */
  368. ib_int64_t tablespace_version;
  369. /*!< a counter which is incremented for
  370. every space object memory creation;
  371. every space mem object gets a
  372. 'timestamp' from this; in DISCARD/
  373. IMPORT this is used to check if we
  374. should ignore an insert buffer merge
  375. request */
  376. UT_LIST_BASE_NODE_T(fil_space_t) space_list;
  377. /*!< list of all file spaces */
  378. UT_LIST_BASE_NODE_T(fil_space_t) rotation_list;
  379. /*!< list of all file spaces needing
  380. key rotation.*/
  381. ibool space_id_reuse_warned;
  382. /* !< TRUE if fil_space_create()
  383. has issued a warning about
  384. potential space_id reuse */
  385. };
  386. /** The tablespace memory cache. This variable is NULL before the module is
  387. initialized. */
  388. extern fil_system_t* fil_system;
  389. #ifndef UNIV_HOTBACKUP
  390. /*******************************************************************//**
  391. Returns the version number of a tablespace, -1 if not found.
  392. @return version number, -1 if the tablespace does not exist in the
  393. memory cache */
  394. UNIV_INTERN
  395. ib_int64_t
  396. fil_space_get_version(
  397. /*==================*/
  398. ulint id); /*!< in: space id */
  399. /*******************************************************************//**
  400. Returns the latch of a file space.
  401. @return latch protecting storage allocation */
  402. UNIV_INTERN
  403. prio_rw_lock_t*
  404. fil_space_get_latch(
  405. /*================*/
  406. ulint id, /*!< in: space id */
  407. ulint* zip_size);/*!< out: compressed page size, or
  408. 0 for uncompressed tablespaces */
  409. /*******************************************************************//**
  410. Returns the type of a file space.
  411. @return FIL_TABLESPACE or FIL_LOG */
  412. UNIV_INTERN
  413. ulint
  414. fil_space_get_type(
  415. /*===============*/
  416. ulint id); /*!< in: space id */
  417. #endif /* !UNIV_HOTBACKUP */
  418. /*******************************************************************//**
  419. Appends a new file to the chain of files of a space. File must be closed.
  420. @return pointer to the file name, or NULL on error */
  421. UNIV_INTERN
  422. char*
  423. fil_node_create(
  424. /*============*/
  425. const char* name, /*!< in: file name (file must be closed) */
  426. ulint size, /*!< in: file size in database blocks, rounded
  427. downwards to an integer */
  428. ulint id, /*!< in: space id where to append */
  429. ibool is_raw) /*!< in: TRUE if a raw device or
  430. a raw disk partition */
  431. MY_ATTRIBUTE((nonnull, warn_unused_result));
  432. #ifdef UNIV_LOG_ARCHIVE
  433. /****************************************************************//**
  434. Drops files from the start of a file space, so that its size is cut by
  435. the amount given. */
  436. UNIV_INTERN
  437. void
  438. fil_space_truncate_start(
  439. /*=====================*/
  440. ulint id, /*!< in: space id */
  441. ulint trunc_len); /*!< in: truncate by this much; it is an error
  442. if this does not equal to the combined size of
  443. some initial files in the space */
  444. /****************************************************************//**
  445. Check is there node in file space with given name. */
  446. UNIV_INTERN
  447. ibool
  448. fil_space_contains_node(
  449. /*====================*/
  450. ulint id, /*!< in: space id */
  451. char* node_name); /*!< in: node name */
  452. #endif /* UNIV_LOG_ARCHIVE */
  453. /*******************************************************************//**
  454. Creates a space memory object and puts it to the 'fil system' hash table.
  455. If there is an error, prints an error message to the .err log.
  456. @param[in] name Space name
  457. @param[in] id Space id
  458. @param[in] flags Tablespace flags
  459. @param[in] purpose FIL_TABLESPACE or FIL_LOG if log
  460. @param[in] crypt_data Encryption information
  461. @param[in] create_table True if this is create table
  462. @param[in] mode Encryption mode
  463. @return TRUE if success */
  464. UNIV_INTERN
  465. bool
  466. fil_space_create(
  467. const char* name,
  468. ulint id,
  469. ulint flags,
  470. ulint purpose,
  471. fil_space_crypt_t* crypt_data,
  472. bool create_table,
  473. fil_encryption_t mode = FIL_ENCRYPTION_DEFAULT);
  474. /*******************************************************************//**
  475. Assigns a new space id for a new single-table tablespace. This works simply by
  476. incrementing the global counter. If 4 billion id's is not enough, we may need
  477. to recycle id's.
  478. @return TRUE if assigned, FALSE if not */
  479. UNIV_INTERN
  480. ibool
  481. fil_assign_new_space_id(
  482. /*====================*/
  483. ulint* space_id); /*!< in/out: space id */
  484. /*******************************************************************//**
  485. Returns the path from the first fil_node_t found for the space ID sent.
  486. The caller is responsible for freeing the memory allocated here for the
  487. value returned.
  488. @return a copy of fil_node_t::path, NULL if space is zero or not found. */
  489. UNIV_INTERN
  490. char*
  491. fil_space_get_first_path(
  492. /*=====================*/
  493. ulint id); /*!< in: space id */
  494. /** Set the recovered size of a tablespace in pages.
  495. @param id tablespace ID
  496. @param size recovered size in pages */
  497. UNIV_INTERN
  498. void
  499. fil_space_set_recv_size(ulint id, ulint size);
  500. /*******************************************************************//**
  501. Returns the size of the space in pages. The tablespace must be cached in the
  502. memory cache.
  503. @return space size, 0 if space not found */
  504. UNIV_INTERN
  505. ulint
  506. fil_space_get_size(
  507. /*===============*/
  508. ulint id); /*!< in: space id */
  509. /*******************************************************************//**
  510. Returns the flags of the space. The tablespace must be cached
  511. in the memory cache.
  512. @return flags, ULINT_UNDEFINED if space not found */
  513. UNIV_INTERN
  514. ulint
  515. fil_space_get_flags(
  516. /*================*/
  517. ulint id); /*!< in: space id */
  518. /*******************************************************************//**
  519. Returns the compressed page size of the space, or 0 if the space
  520. is not compressed. The tablespace must be cached in the memory cache.
  521. @return compressed page size, ULINT_UNDEFINED if space not found */
  522. UNIV_INTERN
  523. ulint
  524. fil_space_get_zip_size(
  525. /*===================*/
  526. ulint id); /*!< in: space id */
  527. /*******************************************************************//**
  528. Checks if the pair space, page_no refers to an existing page in a tablespace
  529. file space. The tablespace must be cached in the memory cache.
  530. @return TRUE if the address is meaningful */
  531. UNIV_INTERN
  532. ibool
  533. fil_check_adress_in_tablespace(
  534. /*===========================*/
  535. ulint id, /*!< in: space id */
  536. ulint page_no);/*!< in: page number */
  537. /****************************************************************//**
  538. Initializes the tablespace memory cache. */
  539. UNIV_INTERN
  540. void
  541. fil_init(
  542. /*=====*/
  543. ulint hash_size, /*!< in: hash table size */
  544. ulint max_n_open); /*!< in: max number of open files */
  545. /*******************************************************************//**
  546. Initializes the tablespace memory cache. */
  547. UNIV_INTERN
  548. void
  549. fil_close(void);
  550. /*===========*/
  551. /*******************************************************************//**
  552. Opens all log files and system tablespace data files. They stay open until the
  553. database server shutdown. This should be called at a server startup after the
  554. space objects for the log and the system tablespace have been created. The
  555. purpose of this operation is to make sure we never run out of file descriptors
  556. if we need to read from the insert buffer or to write to the log. */
  557. UNIV_INTERN
  558. void
  559. fil_open_log_and_system_tablespace_files(void);
  560. /*==========================================*/
  561. /*******************************************************************//**
  562. Closes all open files. There must not be any pending i/o's or not flushed
  563. modifications in the files. */
  564. UNIV_INTERN
  565. void
  566. fil_close_all_files(void);
  567. /*=====================*/
  568. /*******************************************************************//**
  569. Closes the redo log files. There must not be any pending i/o's or not
  570. flushed modifications in the files. */
  571. UNIV_INTERN
  572. void
  573. fil_close_log_files(
  574. /*================*/
  575. bool free); /*!< in: whether to free the memory object */
  576. /*******************************************************************//**
  577. Sets the max tablespace id counter if the given number is bigger than the
  578. previous value. */
  579. UNIV_INTERN
  580. void
  581. fil_set_max_space_id_if_bigger(
  582. /*===========================*/
  583. ulint max_id);/*!< in: maximum known id */
  584. #ifndef UNIV_HOTBACKUP
  585. /****************************************************************//**
  586. Writes the flushed lsn and the latest archived log number to the page
  587. header of the first page of each data file in the system tablespace.
  588. @return DB_SUCCESS or error number */
  589. UNIV_INTERN
  590. dberr_t
  591. fil_write_flushed_lsn_to_data_files(
  592. /*================================*/
  593. lsn_t lsn, /*!< in: lsn to write */
  594. ulint arch_log_no); /*!< in: latest archived log file number */
  595. /** Acquire a tablespace when it could be dropped concurrently.
  596. Used by background threads that do not necessarily hold proper locks
  597. for concurrency control.
  598. @param[in] id tablespace ID
  599. @param[in] for_io whether to look up the tablespace while performing I/O
  600. (possibly executing TRUNCATE)
  601. @return the tablespace
  602. @retval NULL if missing or being deleted or truncated */
  603. fil_space_t*
  604. fil_space_acquire(ulint id, bool for_io = false)
  605. MY_ATTRIBUTE((warn_unused_result));
  606. /** Acquire a tablespace that may not exist.
  607. Used by background threads that do not necessarily hold proper locks
  608. for concurrency control.
  609. @param[in] id tablespace ID
  610. @return the tablespace
  611. @retval NULL if missing or being deleted */
  612. fil_space_t*
  613. fil_space_acquire_silent(ulint id)
  614. MY_ATTRIBUTE((warn_unused_result));
  615. /** Release a tablespace acquired with fil_space_acquire().
  616. @param[in,out] space tablespace to release */
  617. void
  618. fil_space_release(fil_space_t* space);
  619. /** Return the next fil_space_t.
  620. Once started, the caller must keep calling this until it returns NULL.
  621. fil_space_acquire() and fil_space_release() are invoked here which
  622. blocks a concurrent operation from dropping the tablespace.
  623. @param[in,out] prev_space Pointer to the previous fil_space_t.
  624. If NULL, use the first fil_space_t on fil_system->space_list.
  625. @return pointer to the next fil_space_t.
  626. @retval NULL if this was the last */
  627. fil_space_t*
  628. fil_space_next(
  629. fil_space_t* prev_space)
  630. MY_ATTRIBUTE((warn_unused_result));
  631. /** Return the next fil_space_t from key rotation list.
  632. Once started, the caller must keep calling this until it returns NULL.
  633. fil_space_acquire() and fil_space_release() are invoked here which
  634. blocks a concurrent operation from dropping the tablespace.
  635. @param[in,out] prev_space Pointer to the previous fil_space_t.
  636. If NULL, use the first fil_space_t on fil_system->space_list.
  637. @return pointer to the next fil_space_t.
  638. @retval NULL if this was the last*/
  639. fil_space_t*
  640. fil_space_keyrotate_next(
  641. fil_space_t* prev_space)
  642. MY_ATTRIBUTE((warn_unused_result));
  643. /*******************************************************************//**
  644. Reads the flushed lsn, arch no, and tablespace flag fields from a data
  645. file at database startup.
  646. @retval NULL on success, or if innodb_force_recovery is set
  647. @return pointer to an error message string */
  648. UNIV_INTERN
  649. const char*
  650. fil_read_first_page(
  651. /*================*/
  652. os_file_t data_file, /*!< in: open data file */
  653. ibool one_read_already, /*!< in: TRUE if min and max
  654. parameters below already
  655. contain sensible data */
  656. ulint* flags, /*!< out: FSP_SPACE_FLAGS */
  657. ulint* space_id, /*!< out: tablespace ID */
  658. lsn_t* min_flushed_lsn, /*!< out: min of flushed
  659. lsn values in data files */
  660. lsn_t* max_flushed_lsn, /*!< out: max of flushed
  661. lsn values in data files */
  662. fil_space_crypt_t** crypt_data) /*!< out: crypt data */
  663. __attribute__((warn_unused_result));
  664. /*******************************************************************//**
  665. Increments the count of pending operation, if space is not being deleted.
  666. @return TRUE if being deleted, and operation should be skipped */
  667. UNIV_INTERN
  668. ibool
  669. fil_inc_pending_ops(
  670. /*================*/
  671. ulint id, /*!< in: space id */
  672. ibool print_err); /*!< in: need to print error or not */
  673. /*******************************************************************//**
  674. Decrements the count of pending operations. */
  675. UNIV_INTERN
  676. void
  677. fil_decr_pending_ops(
  678. /*=================*/
  679. ulint id); /*!< in: space id */
  680. #endif /* !UNIV_HOTBACKUP */
  681. /*******************************************************************//**
  682. Parses the body of a log record written about an .ibd file operation. That is,
  683. the log record part after the standard (type, space id, page no) header of the
  684. log record.
  685. If desired, also replays the delete or rename operation if the .ibd file
  686. exists and the space id in it matches. Replays the create operation if a file
  687. at that path does not exist yet. If the database directory for the file to be
  688. created does not exist, then we create the directory, too.
  689. Note that mysqlbackup --apply-log sets fil_path_to_mysql_datadir to point to
  690. the datadir that we should use in replaying the file operations.
  691. @return end of log record, or NULL if the record was not completely
  692. contained between ptr and end_ptr */
  693. UNIV_INTERN
  694. byte*
  695. fil_op_log_parse_or_replay(
  696. /*=======================*/
  697. byte* ptr, /*!< in: buffer containing the log record body,
  698. or an initial segment of it, if the record does
  699. not fir completely between ptr and end_ptr */
  700. byte* end_ptr, /*!< in: buffer end */
  701. ulint type, /*!< in: the type of this log record */
  702. ulint space_id, /*!< in: the space id of the tablespace in
  703. question, or 0 if the log record should
  704. only be parsed but not replayed */
  705. ulint log_flags); /*!< in: redo log flags
  706. (stored in the page number parameter) */
  707. /*******************************************************************//**
  708. Deletes a single-table tablespace. The tablespace must be cached in the
  709. memory cache.
  710. @return TRUE if success */
  711. UNIV_INTERN
  712. dberr_t
  713. fil_delete_tablespace(
  714. /*==================*/
  715. ulint id, /*!< in: space id */
  716. buf_remove_t buf_remove); /*!< in: specify the action to take
  717. on the tables pages in the buffer
  718. pool */
  719. /*******************************************************************//**
  720. Closes a single-table tablespace. The tablespace must be cached in the
  721. memory cache. Free all pages used by the tablespace.
  722. @return DB_SUCCESS or error */
  723. UNIV_INTERN
  724. dberr_t
  725. fil_close_tablespace(
  726. /*=================*/
  727. trx_t* trx, /*!< in/out: Transaction covering the close */
  728. ulint id); /*!< in: space id */
  729. #ifndef UNIV_HOTBACKUP
  730. /*******************************************************************//**
  731. Discards a single-table tablespace. The tablespace must be cached in the
  732. memory cache. Discarding is like deleting a tablespace, but
  733. 1. We do not drop the table from the data dictionary;
  734. 2. We remove all insert buffer entries for the tablespace immediately;
  735. in DROP TABLE they are only removed gradually in the background;
  736. 3. When the user does IMPORT TABLESPACE, the tablespace will have the
  737. same id as it originally had.
  738. 4. Free all the pages in use by the tablespace if rename=TRUE.
  739. @return DB_SUCCESS or error */
  740. UNIV_INTERN
  741. dberr_t
  742. fil_discard_tablespace(
  743. /*===================*/
  744. ulint id) /*!< in: space id */
  745. MY_ATTRIBUTE((warn_unused_result));
  746. #endif /* !UNIV_HOTBACKUP */
  747. /** Test if a tablespace file can be renamed to a new filepath by checking
  748. if that the old filepath exists and the new filepath does not exist.
  749. @param[in] space_id tablespace id
  750. @param[in] old_path old filepath
  751. @param[in] new_path new filepath
  752. @param[in] is_discarded whether the tablespace is discarded
  753. @return innodb error code */
  754. dberr_t
  755. fil_rename_tablespace_check(
  756. ulint space_id,
  757. const char* old_path,
  758. const char* new_path,
  759. bool is_discarded);
  760. /*******************************************************************//**
  761. Renames a single-table tablespace. The tablespace must be cached in the
  762. tablespace memory cache.
  763. @return TRUE if success */
  764. UNIV_INTERN
  765. ibool
  766. fil_rename_tablespace(
  767. /*==================*/
  768. const char* old_name_in, /*!< in: old table name in the
  769. standard databasename/tablename
  770. format of InnoDB, or NULL if we
  771. do the rename based on the space
  772. id only */
  773. ulint id, /*!< in: space id */
  774. const char* new_name, /*!< in: new table name in the
  775. standard databasename/tablename
  776. format of InnoDB */
  777. const char* new_path); /*!< in: new full datafile path
  778. if the tablespace is remotely
  779. located, or NULL if it is located
  780. in the normal data directory. */
  781. /*******************************************************************//**
  782. Allocates a file name for a single-table tablespace. The string must be freed
  783. by caller with mem_free().
  784. @return own: file name */
  785. UNIV_INTERN
  786. char*
  787. fil_make_ibd_name(
  788. /*==============*/
  789. const char* name, /*!< in: table name or a dir path */
  790. bool is_full_path); /*!< in: TRUE if it is a dir path */
  791. /*******************************************************************//**
  792. Allocates a file name for a tablespace ISL file (InnoDB Symbolic Link).
  793. The string must be freed by caller with mem_free().
  794. @return own: file name */
  795. UNIV_INTERN
  796. char*
  797. fil_make_isl_name(
  798. /*==============*/
  799. const char* name); /*!< in: table name */
  800. /*******************************************************************//**
  801. Creates a new InnoDB Symbolic Link (ISL) file. It is always created
  802. under the 'datadir' of MySQL. The datadir is the directory of a
  803. running mysqld program. We can refer to it by simply using the path '.'.
  804. @return DB_SUCCESS or error code */
  805. UNIV_INTERN
  806. dberr_t
  807. fil_create_link_file(
  808. /*=================*/
  809. const char* tablename, /*!< in: tablename */
  810. const char* filepath); /*!< in: pathname of tablespace */
  811. /*******************************************************************//**
  812. Deletes an InnoDB Symbolic Link (ISL) file. */
  813. UNIV_INTERN
  814. void
  815. fil_delete_link_file(
  816. /*==================*/
  817. const char* tablename); /*!< in: name of table */
  818. /*******************************************************************//**
  819. Reads an InnoDB Symbolic Link (ISL) file.
  820. It is always created under the 'datadir' of MySQL. The name is of the
  821. form {databasename}/{tablename}. and the isl file is expected to be in a
  822. '{databasename}' directory called '{tablename}.isl'. The caller must free
  823. the memory of the null-terminated path returned if it is not null.
  824. @return own: filepath found in link file, NULL if not found. */
  825. UNIV_INTERN
  826. char*
  827. fil_read_link_file(
  828. /*===============*/
  829. const char* name); /*!< in: tablespace name */
  830. #include "fil0crypt.h"
  831. /*******************************************************************//**
  832. Creates a new single-table tablespace to a database directory of MySQL.
  833. Database directories are under the 'datadir' of MySQL. The datadir is the
  834. directory of a running mysqld program. We can refer to it by simply the
  835. path '.'. Tables created with CREATE TEMPORARY TABLE we place in the temp
  836. dir of the mysqld server.
  837. @return DB_SUCCESS or error code */
  838. UNIV_INTERN
  839. dberr_t
  840. fil_create_new_single_table_tablespace(
  841. /*===================================*/
  842. ulint space_id, /*!< in: space id */
  843. const char* tablename, /*!< in: the table name in the usual
  844. databasename/tablename format
  845. of InnoDB */
  846. const char* dir_path, /*!< in: NULL or a dir path */
  847. ulint flags, /*!< in: tablespace flags */
  848. ulint flags2, /*!< in: table flags2 */
  849. ulint size, /*!< in: the initial size of the
  850. tablespace file in pages,
  851. must be >= FIL_IBD_FILE_INITIAL_SIZE */
  852. fil_encryption_t mode, /*!< in: encryption mode */
  853. ulint key_id) /*!< in: encryption key_id */
  854. __attribute__((nonnull, warn_unused_result));
  855. #ifndef UNIV_HOTBACKUP
  856. /** Try to adjust FSP_SPACE_FLAGS if they differ from the expectations.
  857. (Typically when upgrading from MariaDB 10.1.0..10.1.20.)
  858. @param[in] space_id tablespace ID
  859. @param[in] flags desired tablespace flags */
  860. UNIV_INTERN
  861. void
  862. fsp_flags_try_adjust(ulint space_id, ulint flags);
  863. /********************************************************************//**
  864. Tries to open a single-table tablespace and optionally checks the space id is
  865. right in it. If does not succeed, prints an error message to the .err log. This
  866. function is used to open a tablespace when we start up mysqld, and also in
  867. IMPORT TABLESPACE.
  868. NOTE that we assume this operation is used either at the database startup
  869. or under the protection of the dictionary mutex, so that two users cannot
  870. race here. This operation does not leave the file associated with the
  871. tablespace open, but closes it after we have looked at the space id in it.
  872. If the validate boolean is set, we read the first page of the file and
  873. check that the space id in the file is what we expect. We assume that
  874. this function runs much faster if no check is made, since accessing the
  875. file inode probably is much faster (the OS caches them) than accessing
  876. the first page of the file. This boolean may be initially FALSE, but if
  877. a remote tablespace is found it will be changed to true.
  878. If the fix_dict boolean is set, then it is safe to use an internal SQL
  879. statement to update the dictionary tables if they are incorrect.
  880. @return DB_SUCCESS or error code */
  881. UNIV_INTERN
  882. dberr_t
  883. fil_open_single_table_tablespace(
  884. /*=============================*/
  885. bool validate, /*!< in: Do we validate tablespace? */
  886. bool fix_dict, /*!< in: Can we fix the dictionary? */
  887. ulint id, /*!< in: space id */
  888. ulint flags, /*!< in: expected FSP_SPACE_FLAGS */
  889. const char* tablename, /*!< in: table name in the
  890. databasename/tablename format */
  891. const char* filepath, /*!< in: tablespace filepath */
  892. dict_table_t* table) /*!< in: table */
  893. __attribute__((nonnull(5), warn_unused_result));
  894. #endif /* !UNIV_HOTBACKUP */
  895. /********************************************************************//**
  896. At the server startup, if we need crash recovery, scans the database
  897. directories under the MySQL datadir, looking for .ibd files. Those files are
  898. single-table tablespaces. We need to know the space id in each of them so that
  899. we know into which file we should look to check the contents of a page stored
  900. in the doublewrite buffer, also to know where to apply log records where the
  901. space id is != 0.
  902. @return DB_SUCCESS or error number */
  903. UNIV_INTERN
  904. dberr_t
  905. fil_load_single_table_tablespaces(void);
  906. /*===================================*/
  907. /*******************************************************************//**
  908. Returns TRUE if a single-table tablespace does not exist in the memory cache,
  909. or is being deleted there.
  910. @return TRUE if does not exist or is being deleted */
  911. UNIV_INTERN
  912. ibool
  913. fil_tablespace_deleted_or_being_deleted_in_mem(
  914. /*===========================================*/
  915. ulint id, /*!< in: space id */
  916. ib_int64_t version);/*!< in: tablespace_version should be this; if
  917. you pass -1 as the value of this, then this
  918. parameter is ignored */
  919. /*******************************************************************//**
  920. Returns TRUE if a single-table tablespace exists in the memory cache.
  921. @return TRUE if exists */
  922. UNIV_INTERN
  923. ibool
  924. fil_tablespace_exists_in_mem(
  925. /*=========================*/
  926. ulint id); /*!< in: space id */
  927. #ifndef UNIV_HOTBACKUP
  928. /** Check if a matching tablespace exists in the InnoDB tablespace memory
  929. cache. Note that if we have not done a crash recovery at the database startup,
  930. there may be many tablespaces which are not yet in the memory cache.
  931. @return whether a matching tablespace exists in the memory cache */
  932. UNIV_INTERN
  933. bool
  934. fil_space_for_table_exists_in_mem(
  935. /*==============================*/
  936. ulint id, /*!< in: space id */
  937. const char* name, /*!< in: table name in the standard
  938. 'databasename/tablename' format */
  939. bool print_error_if_does_not_exist,
  940. /*!< in: print detailed error
  941. information to the .err log if a
  942. matching tablespace is not found from
  943. memory */
  944. bool adjust_space, /*!< in: whether to adjust space id
  945. when find table space mismatch */
  946. mem_heap_t* heap, /*!< in: heap memory */
  947. table_id_t table_id, /*!< in: table id */
  948. ulint table_flags); /*!< in: table flags */
  949. #else /* !UNIV_HOTBACKUP */
  950. /********************************************************************//**
  951. Extends all tablespaces to the size stored in the space header. During the
  952. mysqlbackup --apply-log phase we extended the spaces on-demand so that log
  953. records could be appllied, but that may have left spaces still too small
  954. compared to the size stored in the space header. */
  955. UNIV_INTERN
  956. void
  957. fil_extend_tablespaces_to_stored_len(void);
  958. /*======================================*/
  959. #endif /* !UNIV_HOTBACKUP */
  960. /**********************************************************************//**
  961. Tries to extend a data file so that it would accommodate the number of pages
  962. given. The tablespace must be cached in the memory cache. If the space is big
  963. enough already, does nothing.
  964. @return TRUE if success */
  965. UNIV_INTERN
  966. ibool
  967. fil_extend_space_to_desired_size(
  968. /*=============================*/
  969. ulint* actual_size, /*!< out: size of the space after extension;
  970. if we ran out of disk space this may be lower
  971. than the desired size */
  972. ulint space_id, /*!< in: space id */
  973. ulint size_after_extend);/*!< in: desired size in pages after the
  974. extension; if the current space size is bigger
  975. than this already, the function does nothing */
  976. /*******************************************************************//**
  977. Tries to reserve free extents in a file space.
  978. @return TRUE if succeed */
  979. UNIV_INTERN
  980. ibool
  981. fil_space_reserve_free_extents(
  982. /*===========================*/
  983. ulint id, /*!< in: space id */
  984. ulint n_free_now, /*!< in: number of free extents now */
  985. ulint n_to_reserve); /*!< in: how many one wants to reserve */
  986. /*******************************************************************//**
  987. Releases free extents in a file space. */
  988. UNIV_INTERN
  989. void
  990. fil_space_release_free_extents(
  991. /*===========================*/
  992. ulint id, /*!< in: space id */
  993. ulint n_reserved); /*!< in: how many one reserved */
  994. /*******************************************************************//**
  995. Gets the number of reserved extents. If the database is silent, this number
  996. should be zero. */
  997. UNIV_INTERN
  998. ulint
  999. fil_space_get_n_reserved_extents(
  1000. /*=============================*/
  1001. ulint id); /*!< in: space id */
  1002. /********************************************************************//**
  1003. Reads or writes data. This operation is asynchronous (aio).
  1004. @return DB_SUCCESS, or DB_TABLESPACE_DELETED if we are trying to do
  1005. i/o on a tablespace which does not exist */
  1006. UNIV_INTERN
  1007. dberr_t
  1008. _fil_io(
  1009. /*===*/
  1010. ulint type, /*!< in: OS_FILE_READ or OS_FILE_WRITE,
  1011. ORed to OS_FILE_LOG, if a log i/o
  1012. and ORed to OS_AIO_SIMULATED_WAKE_LATER
  1013. if simulated aio and we want to post a
  1014. batch of i/os; NOTE that a simulated batch
  1015. may introduce hidden chances of deadlocks,
  1016. because i/os are not actually handled until
  1017. all have been posted: use with great
  1018. caution! */
  1019. bool sync, /*!< in: true if synchronous aio is desired */
  1020. ulint space_id, /*!< in: space id */
  1021. ulint zip_size, /*!< in: compressed page size in bytes;
  1022. 0 for uncompressed pages */
  1023. ulint block_offset, /*!< in: offset in number of blocks */
  1024. ulint byte_offset, /*!< in: remainder of offset in bytes; in
  1025. aio this must be divisible by the OS block
  1026. size */
  1027. ulint len, /*!< in: how many bytes to read or write; this
  1028. must not cross a file boundary; in aio this
  1029. must be a block size multiple */
  1030. void* buf, /*!< in/out: buffer where to store read data
  1031. or from where to write; in aio this must be
  1032. appropriately aligned */
  1033. void* message, /*!< in: message for aio handler if non-sync
  1034. aio used, else ignored */
  1035. ulint* write_size, /*!< in/out: Actual write size initialized
  1036. after fist successfull trim
  1037. operation for this page and if
  1038. initialized we do not trim again if
  1039. actual page size does not decrease. */
  1040. trx_t* trx) /*!< in: trx */
  1041. __attribute__((nonnull(8)));
  1042. #define fil_io(type, sync, space_id, zip_size, block_offset, byte_offset, len, buf, message, write_size) \
  1043. _fil_io(type, sync, space_id, zip_size, block_offset, byte_offset, len, buf, message, write_size, NULL)
  1044. /** Determine the block size of the data file.
  1045. @param[in] space tablespace
  1046. @param[in] offset page number
  1047. @return block size */
  1048. UNIV_INTERN
  1049. ulint
  1050. fil_space_get_block_size(const fil_space_t* space, unsigned offset);
  1051. /**********************************************************************//**
  1052. Waits for an aio operation to complete. This function is used to write the
  1053. handler for completed requests. The aio array of pending requests is divided
  1054. into segments (see os0file.cc for more info). The thread specifies which
  1055. segment it wants to wait for. */
  1056. UNIV_INTERN
  1057. void
  1058. fil_aio_wait(
  1059. /*=========*/
  1060. ulint segment); /*!< in: the number of the segment in the aio
  1061. array to wait for */
  1062. /**********************************************************************//**
  1063. Flushes to disk possible writes cached by the OS. If the space does not exist
  1064. or is being dropped, does not do anything. */
  1065. UNIV_INTERN
  1066. void
  1067. fil_flush(
  1068. /*======*/
  1069. ulint space_id); /*!< in: file space id (this can be a group of
  1070. log files or a tablespace of the database) */
  1071. /** Flush a tablespace.
  1072. @param[in,out] space tablespace to flush */
  1073. UNIV_INTERN
  1074. void
  1075. fil_flush(fil_space_t* space);
  1076. /** Flush to disk the writes in file spaces of the given type
  1077. possibly cached by the OS.
  1078. @param[in] purpose FIL_TYPE_TABLESPACE or FIL_TYPE_LOG */
  1079. UNIV_INTERN
  1080. void
  1081. fil_flush_file_spaces(ulint purpose);
  1082. /******************************************************************//**
  1083. Checks the consistency of the tablespace cache.
  1084. @return TRUE if ok */
  1085. UNIV_INTERN
  1086. ibool
  1087. fil_validate(void);
  1088. /*==============*/
  1089. /********************************************************************//**
  1090. Returns TRUE if file address is undefined.
  1091. @return TRUE if undefined */
  1092. UNIV_INTERN
  1093. ibool
  1094. fil_addr_is_null(
  1095. /*=============*/
  1096. fil_addr_t addr); /*!< in: address */
  1097. /********************************************************************//**
  1098. Get the predecessor of a file page.
  1099. @return FIL_PAGE_PREV */
  1100. UNIV_INTERN
  1101. ulint
  1102. fil_page_get_prev(
  1103. /*==============*/
  1104. const byte* page); /*!< in: file page */
  1105. /********************************************************************//**
  1106. Get the successor of a file page.
  1107. @return FIL_PAGE_NEXT */
  1108. UNIV_INTERN
  1109. ulint
  1110. fil_page_get_next(
  1111. /*==============*/
  1112. const byte* page); /*!< in: file page */
  1113. /*********************************************************************//**
  1114. Sets the file page type. */
  1115. UNIV_INTERN
  1116. void
  1117. fil_page_set_type(
  1118. /*==============*/
  1119. byte* page, /*!< in/out: file page */
  1120. ulint type); /*!< in: type */
  1121. /*********************************************************************//**
  1122. Gets the file page type.
  1123. @return type; NOTE that if the type has not been written to page, the
  1124. return value not defined */
  1125. UNIV_INTERN
  1126. ulint
  1127. fil_page_get_type(
  1128. /*==============*/
  1129. const byte* page); /*!< in: file page */
  1130. /*******************************************************************//**
  1131. Returns TRUE if a single-table tablespace is being deleted.
  1132. @return TRUE if being deleted */
  1133. UNIV_INTERN
  1134. ibool
  1135. fil_tablespace_is_being_deleted(
  1136. /*============================*/
  1137. ulint id); /*!< in: space id */
  1138. /********************************************************************//**
  1139. Delete the tablespace file and any related files like .cfg.
  1140. This should not be called for temporary tables. */
  1141. UNIV_INTERN
  1142. void
  1143. fil_delete_file(
  1144. /*============*/
  1145. const char* path); /*!< in: filepath of the ibd tablespace */
  1146. /** Callback functor. */
  1147. struct PageCallback {
  1148. /**
  1149. Default constructor */
  1150. PageCallback()
  1151. :
  1152. m_zip_size(),
  1153. m_page_size(),
  1154. m_filepath() UNIV_NOTHROW {}
  1155. virtual ~PageCallback() UNIV_NOTHROW {}
  1156. /**
  1157. Called for page 0 in the tablespace file at the start.
  1158. @param file_size - size of the file in bytes
  1159. @param block - contents of the first page in the tablespace file
  1160. @retval DB_SUCCESS or error code.*/
  1161. virtual dberr_t init(
  1162. os_offset_t file_size,
  1163. const buf_block_t* block) UNIV_NOTHROW = 0;
  1164. /**
  1165. Called for every page in the tablespace. If the page was not
  1166. updated then its state must be set to BUF_PAGE_NOT_USED. For
  1167. compressed tables the page descriptor memory will be at offset:
  1168. block->frame + UNIV_PAGE_SIZE;
  1169. @param offset - physical offset within the file
  1170. @param block - block read from file, note it is not from the buffer pool
  1171. @retval DB_SUCCESS or error code. */
  1172. virtual dberr_t operator()(
  1173. os_offset_t offset,
  1174. buf_block_t* block) UNIV_NOTHROW = 0;
  1175. /**
  1176. Set the name of the physical file and the file handle that is used
  1177. to open it for the file that is being iterated over.
  1178. @param filename - then physical name of the tablespace file.
  1179. @param file - OS file handle */
  1180. void set_file(const char* filename, os_file_t file) UNIV_NOTHROW
  1181. {
  1182. m_file = file;
  1183. m_filepath = filename;
  1184. }
  1185. /**
  1186. @return the space id of the tablespace */
  1187. virtual ulint get_space_id() const UNIV_NOTHROW = 0;
  1188. /** The compressed page size
  1189. @return the compressed page size */
  1190. ulint get_zip_size() const
  1191. {
  1192. return(m_zip_size);
  1193. }
  1194. /**
  1195. Set the tablespace compressed table size.
  1196. @return DB_SUCCESS if it is valie or DB_CORRUPTION if not */
  1197. dberr_t set_zip_size(const buf_frame_t* page) UNIV_NOTHROW;
  1198. /** The compressed page size
  1199. @return the compressed page size */
  1200. ulint get_page_size() const
  1201. {
  1202. return(m_page_size);
  1203. }
  1204. /** Compressed table page size */
  1205. ulint m_zip_size;
  1206. /** The tablespace page size. */
  1207. ulint m_page_size;
  1208. /** File handle to the tablespace */
  1209. os_file_t m_file;
  1210. /** Physical file path. */
  1211. const char* m_filepath;
  1212. protected:
  1213. // Disable copying
  1214. PageCallback(const PageCallback&);
  1215. PageCallback& operator=(const PageCallback&);
  1216. };
  1217. /********************************************************************//**
  1218. Iterate over all the pages in the tablespace.
  1219. @param table - the table definiton in the server
  1220. @param n_io_buffers - number of blocks to read and write together
  1221. @param callback - functor that will do the page updates
  1222. @return DB_SUCCESS or error code */
  1223. UNIV_INTERN
  1224. dberr_t
  1225. fil_tablespace_iterate(
  1226. /*===================*/
  1227. dict_table_t* table,
  1228. ulint n_io_buffers,
  1229. PageCallback& callback)
  1230. MY_ATTRIBUTE((nonnull, warn_unused_result));
  1231. /*******************************************************************//**
  1232. Checks if a single-table tablespace for a given table name exists in the
  1233. tablespace memory cache.
  1234. @return space id, ULINT_UNDEFINED if not found */
  1235. UNIV_INTERN
  1236. ulint
  1237. fil_get_space_id_for_table(
  1238. /*=======================*/
  1239. const char* name); /*!< in: table name in the standard
  1240. 'databasename/tablename' format */
  1241. /**
  1242. Iterate over all the spaces in the space list and fetch the
  1243. tablespace names. It will return a copy of the name that must be
  1244. freed by the caller using: delete[].
  1245. @return DB_SUCCESS if all OK. */
  1246. UNIV_INTERN
  1247. dberr_t
  1248. fil_get_space_names(
  1249. /*================*/
  1250. space_name_list_t& space_name_list)
  1251. /*!< in/out: Vector for collecting the names. */
  1252. MY_ATTRIBUTE((warn_unused_result));
  1253. /** Generate redo log for swapping two .ibd files
  1254. @param[in] old_table old table
  1255. @param[in] new_table new table
  1256. @param[in] tmp_name temporary table name
  1257. @param[in,out] mtr mini-transaction
  1258. @return innodb error code */
  1259. UNIV_INTERN
  1260. dberr_t
  1261. fil_mtr_rename_log(
  1262. const dict_table_t* old_table,
  1263. const dict_table_t* new_table,
  1264. const char* tmp_name,
  1265. mtr_t* mtr)
  1266. MY_ATTRIBUTE((nonnull));
  1267. /*******************************************************************//**
  1268. Finds the given page_no of the given space id from the double write buffer,
  1269. and copies it to the corresponding .ibd file.
  1270. @return true if copy was successful, or false. */
  1271. bool
  1272. fil_user_tablespace_restore_page(
  1273. /*==============================*/
  1274. fsp_open_info* fsp, /* in: contains space id and .ibd
  1275. file information */
  1276. ulint page_no); /* in: page_no to obtain from double
  1277. write buffer */
  1278. /*******************************************************************//**
  1279. Returns a pointer to the file_space_t that is in the memory cache
  1280. associated with a space id.
  1281. @return file_space_t pointer, NULL if space not found */
  1282. fil_space_t*
  1283. fil_space_get(
  1284. /*==========*/
  1285. ulint id); /*!< in: space id */
  1286. #endif /* !UNIV_INNOCHECKSUM */
  1287. /*************************************************************************
  1288. Return local hash table informations. */
  1289. ulint
  1290. fil_system_hash_cells(void);
  1291. /*========================*/
  1292. ulint
  1293. fil_system_hash_nodes(void);
  1294. /*========================*/
  1295. /*************************************************************************
  1296. functions to access is_corrupt flag of fil_space_t*/
  1297. void
  1298. fil_space_set_corrupt(
  1299. /*==================*/
  1300. ulint space_id);
  1301. /** Acquire the fil_system mutex. */
  1302. #define fil_system_enter() mutex_enter(&fil_system->mutex)
  1303. /** Release the fil_system mutex. */
  1304. #define fil_system_exit() mutex_exit(&fil_system->mutex)
  1305. #ifndef UNIV_INNOCHECKSUM
  1306. /*******************************************************************//**
  1307. Returns the table space by a given id, NULL if not found. */
  1308. fil_space_t*
  1309. fil_space_found_by_id(
  1310. /*==================*/
  1311. ulint id); /*!< in: space id */
  1312. /*******************************************************************//**
  1313. Returns the table space by a given id, NULL if not found. */
  1314. fil_space_t*
  1315. fil_space_get_by_id(
  1316. /*================*/
  1317. ulint id); /*!< in: space id */
  1318. /******************************************************************
  1319. Get id of first tablespace or ULINT_UNDEFINED if none */
  1320. UNIV_INTERN
  1321. ulint
  1322. fil_get_first_space();
  1323. /*=================*/
  1324. /******************************************************************
  1325. Get id of next tablespace or ULINT_UNDEFINED if none */
  1326. UNIV_INTERN
  1327. ulint
  1328. fil_get_next_space(
  1329. /*===============*/
  1330. ulint id); /*!< in: space id */
  1331. /******************************************************************
  1332. Get id of first tablespace that has node or ULINT_UNDEFINED if none */
  1333. UNIV_INTERN
  1334. ulint
  1335. fil_get_first_space_safe();
  1336. /*======================*/
  1337. /******************************************************************
  1338. Get id of next tablespace that has node or ULINT_UNDEFINED if none */
  1339. UNIV_INTERN
  1340. ulint
  1341. fil_get_next_space_safe(
  1342. /*====================*/
  1343. ulint id); /*!< in: previous space id */
  1344. #endif /* UNIV_INNOCHECKSUM */
  1345. /*******************************************************************//**
  1346. Return space flags */
  1347. UNIV_INLINE
  1348. ulint
  1349. fil_space_flags(
  1350. /*===========*/
  1351. fil_space_t* space); /*!< in: space */
  1352. /****************************************************************//**
  1353. Does error handling when a file operation fails.
  1354. @return TRUE if we should retry the operation */
  1355. ibool
  1356. os_file_handle_error_no_exit(
  1357. /*=========================*/
  1358. const char* name, /*!< in: name of a file or NULL */
  1359. const char* operation, /*!< in: operation */
  1360. ibool on_error_silent,/*!< in: if TRUE then don't print
  1361. any message to the log. */
  1362. const char* file, /*!< in: file name */
  1363. const ulint line); /*!< in: line */
  1364. /*******************************************************************//**
  1365. Return page type name */
  1366. UNIV_INLINE
  1367. const char*
  1368. fil_get_page_type_name(
  1369. /*===================*/
  1370. ulint page_type); /*!< in: FIL_PAGE_TYPE */
  1371. #ifndef UNIV_NONINL
  1372. #include "fil0fil.ic"
  1373. #endif
  1374. #endif /* fil0fil_h */