mariadb

Tree: a0d396fd3f

Author	SHA1	Message	Date
Jan Lindström	a0d396fd3f	MDEV-11684: post-10.1-merge fixes 10.1 is merged into 10.2 now. Two issues are left to fix: (1) encryption.innochecksum test (2) read_page0 vs page_0_crypt_read (1) innochecksum tool did not compile after merge because buf_page_is_corrupted uses fil_crypt_t that has been changed. extra/CMakeLists.txt: Added fil/fil0crypt.cc as dependency as we need to use fil_crypt_verify_checksum for encrypted pages. innochecksum.cc: If we think page is encrypted i.e. FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION != 0 we call fil_crypt_verify_checksum() function to compare calculated checksum to stored checksum calculated after encryption (this is stored on different offset i.e. FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION + 4). If checksum does not match we call normal buf_page_is_corrupted to compare calculated checksum to stored checksum. fil0crypt.cc: add #ifdef UNIV_INNOCHECKSUM to be able to compile this file for innochecksum tool. (2) read_page0 is not needed and thus removed.	9 years ago
Marko Mäkelä	8a04b8cade	MDEV-11688 fil_crypt_threads_end() tries to create threads after aborted InnoDB startup This bug was repeatable by starting MariaDB 10.2 with an invalid option, such as --innodb-flush-method=foo. It is not repeatable in MariaDB 10.1 in the same way, but the problem exists already there.	9 years ago
Marko Mäkelä	f2fe65106f	MDEV-11679 Remove redundant function fsp_header_get_crypt_offset() fsp_header_get_crypt_offset(): Remove. xdes_arr_size(): Remove. fsp_header_get_encryption_offset(): Make this an inline function. The correctness of this change was ensured with the following patch that ensures that the two functions returned the same value, only differing by FSP_HEADER_OFFSET (38 bytes): diff --git a/storage/innobase/fsp/fsp0fsp.cc b/storage/innobase/fsp/fsp0fsp.cc index f2a4c6bf218..e96c788b7df 100644 --- a/storage/innobase/fsp/fsp0fsp.cc +++ b/storage/innobase/fsp/fsp0fsp.cc @@ -850,6 +850,7 @@ fsp_parse_init_file_page( return(ptr); } +static ulint fsp_header_get_encryption_offset(const page_size_t&); /********************************************************************// Initializes the fsp system. / void @@ -868,6 +869,31 @@ fsp_init(void) #endif / Does nothing at the moment / + + for (ulint sz = 4096; sz <= 65536; sz = 2) { + ulint m; + if (sz <= 16384) { + for (ulint ph = 1024; ph <= sz; ph = 2) { + const page_size_t ps(ph, sz, true); + ulint maria = fsp_header_get_crypt_offset(ps, &m), + oracle = fsp_header_get_encryption_offset(ps); + if (maria != oracle + 38) { + ib::error() << "zip size mismatch: " + << maria << "!=" << oracle + << "(" << ph <<","<<sz<<")" + << m; + } + } + } + const page_size_t p(sz, sz, false); + ulint maria = fsp_header_get_crypt_offset(p, &m), + oracle = fsp_header_get_encryption_offset(p); + if (maria != oracle + 38) { + ib::error() << "size mismatch: " + << maria << "!=" << oracle + << "(" <<sz<<")" << m; + } + } } /*******************************************************************//	9 years ago
Sergei Golubchik	48dc7cc66e	cleanup: redundant memcmp()	9 years ago
Marko Mäkelä	d6a1f9f10f	MDEV-11630 Call mutex_free() before freeing the mutex list fil_space_crypt_cleanup(): Call mutex_free() to pair with fil_space_crypt_init(). fil_space_destroy_crypt_data(): Call mutex_free() to pair with fil_space_create_crypt_data() and fil_space_read_crypt_data(). fil_crypt_threads_cleanup(): Call mutex_free() to pair with fil_crypt_threads_init(). fil_space_free_low(): Invoke fil_space_destroy_crypt_data(). fil_close(): Invoke fil_space_crypt_cleanup(), just like fil_init() invoked fil_space_crypt_init(). Datafile::shutdown(): Set m_crypt_info=NULL without dereferencing the pointer. The object will be freed along with the fil_space_t in fil_space_free_low(). Remove some unnecessary conditions (ut_free(NULL) is OK). srv_shutdown_all_bg_threads(): Shut down the encryption threads by calling fil_crypt_threads_end(). srv_shutdown_bg_undo_sources(): Do not prematurely call fil_crypt_threads_end(). Many pages can still be written by change buffer merge, rollback of incomplete transactions, and purge, especially in slow shutdown (innodb_fast_shutdown=0). innobase_shutdown_for_mysql(): Call fil_crypt_threads_cleanup() also when innodb_read_only=1, because the threads will have been created also in that case. sync_check_close(): Re-enable the invocation of sync_latch_meta_destroy() to free the mutex list.	9 years ago
Marko Mäkelä	545c912696	Remove an unnecessary comparison.	9 years ago
Marko Mäkelä	7e02fd1f71	MDEV-11630 Call mutex_free() before freeing the mutex list Make some global fil_crypt_ variables static. fil_close(): Call mutex_free(&fil_system->mutex) also in InnoDB, not only in XtraDB. In InnoDB, sync_close() was called before fil_close(). innobase_shutdown_for_mysql(): Call fil_close() before sync_close(), similar to XtraDB shutdown. fil_space_crypt_cleanup(): Call mutex_free() to pair with fil_space_crypt_init(). fil_crypt_threads_cleanup(): Call mutex_free() to pair with fil_crypt_threads_init().	9 years ago
Jan Lindström	72cc73cea2	MDEV-10368: get_latest_version() called too often Reduce the number of calls to encryption_get_key_get_latest_version when doing key rotation with two different methods: (1) We need to fetch key information when tablespace not yet have a encryption information, invalid keys are handled now differently (see below). There was extra call to detect if key_id is not found on key rotation. (2) If key_id is not found from encryption plugin, do not try fetching new key_version for it as it will fail anyway. We store return value from encryption_get_key_get_latest_version call and if it returns ENCRYPTION_KEY_VERSION_INVALID there is no need to call it again.	9 years ago
Jan Lindström	885577fb10	MDEV-11004: Unable to start (Segfault or os error 2) when encryption key missing Two problems: (1) When pushing warning to sql-layer we need to check that thd != NULL to avoid NULL-pointer reference. (2) At tablespace key rotation if used key_id is not found from encryption plugin tablespace should not be rotated.	9 years ago
Jan Lindström	2bedc3978b	MDEV-9931: InnoDB reads first page of every .ibd file at startup Analysis: By design InnoDB was reading first page of every .ibd file at startup to find out is tablespace encrypted or not. This is because tablespace could have been encrypted always, not encrypted newer or encrypted based on configuration and this information can be find realible only from first page of .ibd file. Fix: Do not read first page of every .ibd file at startup. Instead whenever tablespace is first time accedded we will read the first page to find necessary information about tablespace encryption status. TODO: Add support for SYS_TABLEOPTIONS where all table options encryption information included will be stored.	9 years ago
Jan Lindström	fec844aca8	Merge InnoDB 5.7 from mysql-5.7.14. Contains also: MDEV-10549 mysqld: sql/handler.cc:2692: int handler::ha_index_first(uchar*): Assertion `table_share->tmp_table != NO_TMP_TABLE \|\| m_lock_type != 2' failed. (branch bb-10.2-jan) Unlike MySQL, InnoDB still uses THR_LOCK in MariaDB MDEV-10548 Some of the debug sync waits do not work with InnoDB 5.7 (branch bb-10.2-jan) enable tests that were fixed in MDEV-10549 MDEV-10548 Some of the debug sync waits do not work with InnoDB 5.7 (branch bb-10.2-jan) fix main.innodb_mysql_sync - re-enable online alter for partitioned innodb tables	9 years ago
Jan Lindström	2e814d4702	Merge InnoDB 5.7 from mysql-5.7.9. Contains also MDEV-10547: Test multi_update_innodb fails with InnoDB 5.7 The failure happened because 5.7 has changed the signature of the bool handler::primary_key_is_clustered() const virtual function ("const" was added). InnoDB was using the old signature which caused the function not to be used. MDEV-10550: Parallel replication lock waits/deadlock handling does not work with InnoDB 5.7 Fixed mutexing problem on lock_trx_handle_wait. Note that rpl_parallel and rpl_optimistic_parallel tests still fail. MDEV-10156 : Group commit tests fail on 10.2 InnoDB (branch bb-10.2-jan) Reason: incorrect merge MDEV-10550: Parallel replication can't sync with master in InnoDB 5.7 (branch bb-10.2-jan) Reason: incorrect merge	9 years ago
Monty	4dc5075860	Fixed compiler warnings and test failures found by buildbot Fixed ccfilter to detect errors where the column is included in the error message	9 years ago
Jan Lindström	ee768d8e0e	MDEV-9640: Add used key_id to INFORMATION_SCHEMA.INNODB_TABLESPACES_ENCRYPTION	10 years ago
Jan Lindström	a12326458d	Add check to avoid NULL-pointer access if encryption information is not available. Clarify system tablespace page 0 check.	10 years ago
Jan Lindström	f341d94423	MDEV-9549: Trying to decrypt a not encrypted page Make sure that on decrypt we do not try to reference NULL pointer and if page contains undefined FIL_PAGE_FILE_FLUSH_LSN field on when page is not the first page or page is not in system tablespace, clear it.	10 years ago
Jan Lindström	36ca65b73b	MDEV-9559: Server without encryption configs crashes if selecting from an implicitly encrypted table There was two problems. Firstly, if page in ibuf is encrypted but decrypt failed we should not allow InnoDB to start because this means that system tablespace is encrypted and not usable. Secondly, if page decrypt is detected we should return false from buf_page_decrypt_after_read.	10 years ago
Jan Lindström	111acb721f	MDEV-9359: encryption.create_or_replace fails sporadically in buildbot: failing assertion: mutex->magic_n == MUTEX_MAGIC_N Make sure that encryption threads mutex is initialized before starting encryption threads.	10 years ago
Jan Lindström	0cf39f401c	MDEV-8817: Failing assertion: new_state->key_version != ENCRYPTION_KEY_VERSION_INVALID Analysis: Problem sees to be the fact that we allow creating or altering table to use encryption_key_id that does not exists in case where original table is not encrypted currently. Secondly we should not do key rotation to tables that are not encrypted or tablespaces that can't be found from tablespace cache. Fix: Do not allow creating unencrypted table with nondefault encryption key and do not rotate tablespaces that are not encrypted (FIL_SPACE_ENCRYPTION_OFF) or can't be found from tablespace cache.	10 years ago
Jan Lindström	89af0f11a8	MDEV-8770: Incorrect error message when importing page compressed tablespace Added decompression and after page update recompression support for import.	10 years ago
Jan Lindström	161db7c18f	MDEV-8773: InnoDB innochecksum does not work with encrypted or page compressed tables	10 years ago
Jan Lindström	4d3f680c95	MDEV-8772: Assertion failure in file ha_innodb.cc line 20027 when importing page compressed and encrypted tablespace using incorrect keys Add error handling to decryp function when decrypt fails during import.	10 years ago
Jan Lindström	509b836623	MDEV-8708: InnoDB temp file encryption Added encryption support for online alter table where InnoDB temporary files are used. Added similar support also for tables containing full text-indexes. Made sure that table remains encrypted during discard and import tablespace.	10 years ago
Jan Lindström	7e916bb86f	MDEV-8588: Assertion failure in file ha_innodb.cc line 21140 if at least one encrypted table exists and encryption service is not available Analysis: Problem was that in fil_read_first_page we do find that table has encryption information and that encryption service or used key_id is not available. But, then we just printed fatal error message that causes above assertion. Fix: When we open single table tablespace if it has encryption information (crypt_data) store this crypt data to the table structure. When we open a table and we find out that tablespace is not available, check has table a encryption information and from there is encryption service or used key_id is not available. If it is, add additional warning for SQL-layer.	10 years ago
Jan Lindström	05bcb088bf	MDEV-8583: Empty lines in encryption logging Removed extra line break.	10 years ago
Jan Lindström	62b5a56191	MDEV-8501: encryption.create_or_replace fails in buildbot on P8 builders Analysis: There is race between drop table and encryption threads that could cause encryption thread to enter mutex that has been already released. Fix: When destroying crypt_data first enter the mutex and set crypt data unavailable, then release the memory and clean up the data. This should make the race more unprobable. Additionally, added big_test for create_or_replace as it could fail testcase timeout if you have slow I/O (tested that testcase passes with --mem).	10 years ago
Jan Lindström	84cefe2455	MDEV-8396: InnoDB: Assertion failure in file fil0crypt.cc line 2052 Logic error, NULL pointer was referenced.	10 years ago
Jan Lindström	79af0b3823	MDEV-8393: InnoDB: Assertion failure in file fil0crypt.cc line 2109 Crypt data was used when it was already freed. Delete crypt data more thorougly when freed.	10 years ago
Jan Lindström	b94eaff8d9	MDEV-8310: Encryption bogus message still in 10.1.5 Moved warning print to debug builds only because on production these messages are unnecessary.	11 years ago
Jan Lindström	70b82efd20	MDEV-8273: InnoDB: Assertion failure in file fil0pagecompress.cc line 532 Analysis: Problem was that actual payload size (page size) after compression was handled incorrectly on encryption. Additionally, some of the variables were not initialized. Fixed by encrypting/decrypting only the actual compressed page size.	11 years ago
Jan Lindström	d7f3d889de	MDEV-8272: Encryption performance: Reduce the number of unused memcpy's Removed memcpy's on cases when page is not encrypted and make sure we use the correct buffer for reading/writing.	11 years ago
Jan Lindström	f7002c05ae	MDEV-8250: InnoDB: Page compressed tables are not compressed and compressed+encrypted tables cause crash Analysis: Problem is that both encrypted tables and compressed tables use FIL header offset FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION to store required metadata. Furhermore, for only compressed tables currently code skips compression. Fixes: - Only encrypted pages store key_version to FIL header offset FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION, no need to fix - Only compressed pages store compression algorithm to FIL header offset FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION, no need to fix as they have different page type FIL_PAGE_PAGE_COMPRESSED - Compressed and encrypted pages now use a new page type FIL_PAGE_PAGE_COMPRESSED_ENCRYPTED and key_version is stored on FIL header offset FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION and compression method is stored after FIL header similar way as compressed size, so that first FIL_PAGE_COMPRESSED_SIZE is stored followed by FIL_PAGE_COMPRESSION_METHOD - Fix buf_page_encrypt_before_write function to really compress pages if compression is enabled - Fix buf_page_decrypt_after_read function to really decompress pages if compression is used - Small style fixes	11 years ago
Jan Lindström	a25ccd4f83	MDEV-8238: Tables with encryption=yes using file_key_management plugin are not encrypted Analysis: Problem was that encryption was skipped. Fixed by making sure that tables with ENCRYPTED=YES are encrypted.	11 years ago
Jan Lindström	2bea4bd9ed	MDEV-8233: InnoDB: Assertion failure in fil_page_decompress with encrypted tables Analysis: Problem was that used compression method needs to be stored to the page. Fixed by storing compression method after key_version to the page.	11 years ago
Jan Lindström	536112dd30	MDEV-8195: InnoDB: Error: trying to access tablespace 11262 page no. 7, InnoDB: but the tablespace does not exist or is just being dropped. Analysis: Problem was that we did try to read from tablespace that was being dropped. Fixed by introducing a new function to find a tablespace only if it is not being dropped currently and adding this check before trying to read pages from tablespace.	11 years ago
Jan Lindström	925b64124a	MDEV-8182: Failing assertion: 1 == UT_LIST_GET_LEN(space->chain) Analysis: At fil_spage_get_space there is small change that space is found from tablespace list but we have not yet created node for it (and added it to space->chain) and as we hold fil_system mutex here fil_node_create can't continue. Fixed by allowing UT_LIST_GET_LEN(space->chain) == 0\|\| 1 and introducint two new functions that access filespace list and before returning space check that node is also created.	11 years ago
Jan Lindström	3e55ef26d4	MDEV-8173: InnoDB; Failing assertion: crypt_data->type == 1 Make sure that when we publish the crypt_data we access the memory cache of the tablespace crypt_data. Make sure that crypt_data is stored whenever it is really needed. All this is not yet enough in my opinion because: sql/encryption.cc has DBUG_ASSERT(scheme->type == 1) i.e. crypt_data->type == CRYPT_SCHEME_1 However, for InnoDB point of view we have global crypt_data for every tablespace. When we change variables on crypt_data we take mutex. However, when we use crypt_data for encryption/decryption we use pointer to this global structure and no mutex to protect against changes on crypt_data. Tablespace encryption starts in fil_crypt_start_encrypting_space from crypt_data that has crypt_data->type = CRYPT_SCHEME_UNENCRYPTED and later we write page 0 CRYPT_SCHEME_1 and finally whe publish that to memory cache.	11 years ago
Jan Lindström	20c23048c1	MDEV-8164: Server crashes in pfs_mutex_enter_func after fil_crypt_is_closing or alike Analysis: Problem was that tablespaces not encrypted might not have crypt_data stored on disk. Fixed by always creating crypt_data to memory cache of the tablespace. MDEV-8138: strange results from encrypt-and-grep test Analysis: crypt_data->type is not updated correctly on memory cache. This caused problem with state tranfer on encrypted => unencrypted => encrypted. Fixed by updating memory cache of crypt_data->type correctly based on current srv_encrypt_tables value to either CRYPT_SCHEME_1 or CRYPT_SCHEME_UNENCRYPTED.	11 years ago
Sergei Golubchik	54672a4f1e	MDEV-8043 innodb tablespace encryption "use after free" bug, when a thread replaces space->crypt_data and frees the old crypt_data object while it's being used by another thread.	11 years ago
Sergei Golubchik	8258a34800	InnoDB: check if scrubbing is enabled before scanning the tablespace	11 years ago
Sergei Golubchik	a94cabd503	MDEV-8159 InnoDB: Failing assertion: key_state->key_id to simplify various checks, set crypt_data->key_id even for not encrypted tablespaces	11 years ago
Sergei Golubchik	2300fe2e0e	Identical key derivation code in XtraDB/InnoDB/Aria * Extract it into the "encryption_scheme" service. * Make these engines to use the service, remove duplicate code. * Change MY_AES_xxx error codes, to return them safely from encryption_scheme_encrypt/decrypt without conflicting with ENCRYPTION_SCHEME_KEY_INVALID error	11 years ago
Jan Lindström	f8cacd03a7	MDEV-8143: InnoDB: Database page corruption on disk or a failed file read Analysis: Problem was that we did create crypt data for encrypted table but this new crypt data was not written to page 0. Instead a default crypt data was written to page 0 at table creation. Fixed by explicitly writing new crypt data to page 0 after successfull table creation.	11 years ago
Sergei Golubchik	def48e623a	MDEV-8141 InnoDB: background encryption thread uses FIL_DEFAULT_ENCRYPTION_KEY * check key version per key id (that is, per tablespace). * wake encryption thread when a tablespace needs re-encryption	11 years ago
Sergei Golubchik	5fdb14542a	MDEV-8021 "InnoDB: Tablespace id 4 encrypted but encryption service not available. Can't continue opening tablespace" on server restart when there are encrypted tables key id was written in the wrong place on the tablespace first page (thus its value could not be read back later)	11 years ago
Jan Lindström	2f25c653ad	MDEV-8074: Failing assertion: mutex->magic_n == MUTEX_MAGIC_N in file sync0sync.cc line 508 Problem was that e.g. on crash recovery fil_space_crypt_close_tablespace and fil_space_crypt_mark_space_closing access mutex that is not yet initialized. Mutex is naturally initialized only if encryption is configured.	11 years ago
Sergei Golubchik	966b236612	add support for --innodb-encrypt-tables=FORCE this will prevent user from creating tables with ENCRYPTED=OFF	11 years ago
Sergei Golubchik	0a9052f591	Store the key id in the tablespace and read it back	11 years ago
Sergei Golubchik	97d5de4ccf	Add encryption key id to the API as a distinct concept which is separate from the encryption key version	11 years ago
Sergei Golubchik	4d40a7d133	remove now-empty my_aes.{h,cc} move remaning defines to my_crypt, add MY_ namespace prefix	11 years ago

1 2

78 Commits (a0d396fd3f5f86e09f602ac9f76b8d9f2e739221)