mariadb

Tree: 8511f04fdb

Author	SHA1	Message	Date
Marko Mäkelä	8511f04fdb	Cleanup: Remove srv_start_lsn Most of the time, we can refer to recv_sys.recovered_lsn.	6 years ago
Marko Mäkelä	25e2a556de	MDEV-21133 Optimize access to InnoDB page header fields Introduce memcpy_aligned<N>(), memcmp_aligned<N>(), memset_aligned<N>() and use them for accessing InnoDB page header fields that are known to be aligned. MY_ASSUME_ALIGNED(): Wrapper for the GCC/clang __builtin_assume_aligned(). Nothing similar seems to exist in Microsoft Visual Studio, and the C++20 std::assume_aligned is not available to us yet. Explicitly specified alignment guarantees allow compilers to generate faster code on platforms with strict alignment rules, instead of emitting calls to potentially unaligned memcpy(), memcmp(), or memset().	6 years ago
Marko Mäkelä	d67ea8151f	Rename LOG_HEADER_FORMAT_ to log_t::FORMAT_	6 years ago
Vladislav Vaintroub	f61a980686	Update WolfSSL, remove older workarounds.	6 years ago
Thirunarayanan Balathandayuthapani	723a4b1d78	MDEV-17228 Encrypted temporary tables are not encrypted - Introduce a new variable called innodb_encrypt_temporary_tables which is a boolean variable. It decides whether to encrypt the temporary tablespace. - Encrypts the temporary tablespace based on full checksum format. - Introduced a new counter to track encrypted and decrypted temporary tablespace pages. - Warnings issued if temporary table creation has conflict value with innodb_encrypt_temporary_tables - Added a new test case which reads and writes the pages from/to temporary tablespace.	6 years ago
Vladislav Vaintroub	88b7926ff8	MDEV-19582 WolfSSL decyption function can read memory out-of-bounds. MDEV-19581 Valgrind error with WolfSSL and encrypted binlog WolfSSL can read memory out of bounds in EVP_CipherUpdate() in decrypt/NOPAD mode, when the input length is not multiple of AES block size. The workaround ensures that input will have some padding at the end by having slightly larger allocated buffer, or padding the structures with 16 more bytes.	7 years ago
Marko Mäkelä	f6d4f624eb	MDEV-12041: innodb_encrypt_log key rotation This will change the InnoDB encrypted redo log format only. Unencrypted redo log will keep using the MariaDB 10.3 format. In the new encrypted redo log format, 4 additional bytes will be reserved in the redo log block trailer for storing the encryption key version. For performance reasons, the encryption key rotation (checking if the latest encryption key version is being used) is only done at log_checkpoint(). LOG_HEADER_FORMAT_CURRENT: Remove. LOG_HEADER_FORMAT_ENC_10_4: The encrypted 10.4 format. LOG_BLOCK_KEY: The encryption key version field. LOG_BLOCK_TRL_SIZE: Remove. log_t: Add accessors framing_size(), payload_size(), trailer_offset(), to be used instead of referring to LOG_BLOCK_TRL_SIZE. log_crypt_t: An operation passed to log_crypt(). log_crypt(): Perform decryption, encryption, or encryption with key rotation. Return an error if key rotation at decryption fails. On encryption, keep using the previous key if the rotation fails. At startup, old-format encrypted redo log may be written before the redo log is upgraded (rebuilt) to the latest format. log_write_up_to(): Add the parameter rotate_key=false. log_checkpoint(): Invoke log_write_up_to() with rotate_key=true.	7 years ago
Marko Mäkelä	5932a4e77d	MDEV-13834: Upgrade failure from 10.1 innodb_encrypt_log log_crypt_101_read_block(): Mimic MariaDB 10.1, and use the first encryption key if the key for the checkpoint cannot be found. Redo log encryption key rotation was ultimately disabled in MDEV-9422 (MariaDB 10.1.13) due to design issues. So, from MariaDB 10.1.13 onwards only one log encryption key should matter. recv_log_format_0_recover(): Add the parameter 'bool crypt'. Indicate when the log cannot be decrypted for upgrade, instead of making a possibly false claim that the log requires crash recovery. init_crypt_key(): Remove extra space from a message.	7 years ago
Marko Mäkelä	8a42ad7a5d	MDEV-13834 10.2 wrongly recognizes 10.1.10 innodb_encrypt_log=ON data as after-crash and refuses to start infos[]: Allocate enough entries to accommodate all keys from both checkpoint pages. infos_used: The size of infos[]. get_crypt_info(): Merge to the only caller, log_crypt_101_read_block(). log_crypt_101_read_block(): Do not validate the log block checksum, because it will not be valid when upgrading from MariaDB 10.1.10. Instead, check that the encryption key exists. log_crypt_101_read_checkpoint(): Append to infos[] instead of overwriting.	8 years ago
Marko Mäkelä	d73a898d64	MDEV-16045: Allocate log_sys statically There is only one redo log subsystem in InnoDB. Allocate the object statically, to avoid unnecessary dereferencing of the pointer. log_t::create(): Renamed from log_sys_init(). log_t::close(): Renamed from log_shutdown(). log_t::checkpoint_buf_ptr: Remove. Allocate log_t::checkpoint_buf statically.	8 years ago
Vladislav Vaintroub	6c279ad6a7	MDEV-15091 : Windows, 64bit: reenable and fix warning C4267 (conversion from 'size_t' to 'type', possible loss of data) Handle string length as size_t, consistently (almost always:)) Change function prototypes to accept size_t, where in the past ulong or uint were used. change local/member variables to size_t when appropriate. This fix excludes rocksdb, spider,spider, sphinx and connect for now.	8 years ago
Marko Mäkelä	16d308e21d	MDEV-14874 innodb_encrypt_log corrupts the log when the LSN crosses 32-bit boundary This bug affects both writing and reading encrypted redo log in MariaDB 10.1, starting from version 10.1.3 which added support for innodb_encrypt_log. That is, InnoDB crash recovery and Mariabackup will sometimes fail when innodb_encrypt_log is used. MariaDB 10.2 or Mariabackup 10.2 or later versions are not affected. log_block_get_start_lsn(): Remove. This function would cause trouble if a log segment that is being read is crossing a 32-bit boundary of the LSN, because this function does not allow the most significant 32 bits of the LSN to change. log_blocks_crypt(), log_encrypt_before_write(), log_decrypt_after_read(): Add the parameter "lsn" for the start LSN of the block. log_blocks_encrypt(): Remove (unused function).	8 years ago
Jan Lindström	4d01dd79a1	MDEV-12634: Uninitialised ROW_MERGE_RESERVE_SIZE bytes written to temporary file After review cleanup.	8 years ago
Jan Lindström	fa2701c6f7	MDEV-12634: Uninitialised ROW_MERGE_RESERVE_SIZE bytes written to tem… …porary file Fixed by removing writing key version to start of every block that was encrypted. Instead we will use single key version from log_sys crypt info. After this MDEV also blocks writen to row log are encrypted and blocks read from row log aren decrypted if encryption is configured for the table. innodb_status_variables[], struct srv_stats_t Added status variables for merge block and row log block encryption and decryption amounts. Removed ROW_MERGE_RESERVE_SIZE define. row_merge_fts_doc_tokenize Remove ROW_MERGE_RESERVE_SIZE row_log_t Add index, crypt_tail, crypt_head to be used in case of encryption. row_log_online_op, row_log_table_close_func Before writing a block encrypt it if encryption is enabled row_log_table_apply_ops, row_log_apply_ops After reading a block decrypt it if encryption is enabled row_log_allocate Allocate temporary buffers crypt_head and crypt_tail if needed. row_log_free Free temporary buffers crypt_head and crypt_tail if they exist. row_merge_encrypt_buf, row_merge_decrypt_buf Removed. row_merge_buf_create, row_merge_buf_write Remove ROW_MERGE_RESERVE_SIZE row_merge_build_indexes Allocate temporary buffer used in decryption and encryption if needed. log_tmp_blocks_crypt, log_tmp_block_encrypt, log_temp_block_decrypt New functions used in block encryption and decryption log_tmp_is_encrypted New function to check is encryption enabled. Added test case innodb-rowlog to force creating a row log and verify that operations are done using introduced status variables.	8 years ago
Marko Mäkelä	66a09bd6ab	MDEV-13318 Crash recovery failure after the server is killed during innodb_encrypt_log startup This fixes several InnoDB bugs related to innodb_encrypt_log and two Mariabackup --backup bugs. log_crypt(): Properly derive the initialization vector from the start LSN of each block. Add a debug assertion. log_crypt_init(): Note that the function should only be used when creating redo log files and that the information is persisted in the checkpoint pages. xtrabackup_copy_log(): Validate data_len. xtrabackup_backup_func(): Always use the chosen checkpoint buffer. log_group_write_buf(), log_write_up_to(): Only log_crypt() the redo log payload, not the padding bytes. innobase_start_or_create_for_mysql(): Do not invoke log_crypt_init() or initiate a redo log checkpoint. recv_find_max_checkpoint(): Return the contents of LOG_CHECKPOINT_NO to xtrabackup_backup_func() in log_sys->next_checkpoint_no.	8 years ago
Michael Widenius	4aaa38d26e	Enusure that my_global.h is included first - Added sql/mariadb.h file that should be included first by files in sql directory, if sql_plugin.h is not used (sql_plugin.h adds SHOW variables that must be done before my_global.h is included) - Removed a lot of include my_global.h from include files - Removed include's of some files that my_global.h automatically includes - Removed duplicated include's of my_sys.h - Replaced include my_config.h with my_global.h	8 years ago
Marko Mäkelä	a4efeabc43	MDEV-13061 innodb_encrypt_log recovery is spamming the error log log_crypt(): Remove the useless error log output that was accidentally introduced in MDEV-11782. These messages could be emitted to the server error log during crash recovery.	8 years ago
Marko Mäkelä	9756440fb5	Fix a compilation warning introduced by MDEV-11782 log_crypt(): Do not cast byte* to uint32_t*, because it may break strict aliasing rules. Instead, cast in the opposite direction.	9 years ago
Vladislav Vaintroub	db39107413	MDEV-11663 Create services for functionality used by plugins Added service for - encryption (AES) - error reporting, e.g my_printf_error()	9 years ago
Sergei Golubchik	b6a1d6538b	compiler warnings	9 years ago
Marko Mäkelä	498f4a825b	Fix InnoDB/XtraDB compilation warnings on 32-bit builds.	9 years ago
Marko Mäkelä	2af28a363c	MDEV-11782: Redefine the innodb_encrypt_log format Write only one encryption key to the checkpoint page. Use 4 bytes of nonce. Encrypt more of each redo log block, only skipping the 4-byte field LOG_BLOCK_HDR_NO which the initialization vector is derived from. Issue notes, not warning messages for rewriting the redo log files. recv_recovery_from_checkpoint_finish(): Do not generate any redo log, because we must avoid that before rewriting the redo log files, or otherwise a crash during a redo log rewrite (removing or adding encryption) may end up making the database unrecoverable. Instead, do these tasks in innobase_start_or_create_for_mysql(). Issue a firm "Missing MLOG_CHECKPOINT" error message. Remove some unreachable code and duplicated error messages for log corruption. LOG_HEADER_FORMAT_ENCRYPTED: A flag for identifying an encrypted redo log format. log_group_t::is_encrypted(), log_t::is_encrypted(): Determine if the redo log is in encrypted format. recv_find_max_checkpoint(): Interpret LOG_HEADER_FORMAT_ENCRYPTED. srv_prepare_to_delete_redo_log_files(): Display NOTE messages about adding or removing encryption. Do not issue warnings for redo log resizing any more. innobase_start_or_create_for_mysql(): Rebuild the redo logs also when the encryption changes. innodb_log_checksums_func_update(): Always use the CRC-32C checksum if innodb_encrypt_log. If needed, issue a warning that innodb_encrypt_log implies innodb_log_checksums. log_group_write_buf(): Compute the checksum on the encrypted block contents, so that transmission errors or incomplete blocks can be detected without decrypting. Rewrite most of the redo log encryption code. Only remember one encryption key at a time (but remember up to 5 when upgrading from the MariaDB 10.1 format.)	9 years ago
Marko Mäkelä	3272a19741	MDEV-11782 preparation: Add separate code for validating the 10.1 redo log. log_crypt_101_read_checkpoint(): Read the encryption information from a MariaDB 10.1 checkpoint page. log_crypt_101_read_block(): Attempt to decrypt a MariaDB 10.1 redo log page. recv_log_format_0_recover(): Only attempt decryption on checksum mismatch. NOTE: With the MariaDB 10.1 innodb_encrypt_log format, we can actually determine from the cleartext portion of the redo log whether the redo log is empty. We do not really have to decrypt the redo log here, if we did not want to determine if the checksum is valid.	9 years ago
Jan Lindström	71495a1748	MDEV-11849: Fix storage/innobase/* compile warnings	9 years ago
Jan Lindström	fec844aca8	Merge InnoDB 5.7 from mysql-5.7.14. Contains also: MDEV-10549 mysqld: sql/handler.cc:2692: int handler::ha_index_first(uchar*): Assertion `table_share->tmp_table != NO_TMP_TABLE \|\| m_lock_type != 2' failed. (branch bb-10.2-jan) Unlike MySQL, InnoDB still uses THR_LOCK in MariaDB MDEV-10548 Some of the debug sync waits do not work with InnoDB 5.7 (branch bb-10.2-jan) enable tests that were fixed in MDEV-10549 MDEV-10548 Some of the debug sync waits do not work with InnoDB 5.7 (branch bb-10.2-jan) fix main.innodb_mysql_sync - re-enable online alter for partitioned innodb tables	9 years ago
Jan Lindström	2e814d4702	Merge InnoDB 5.7 from mysql-5.7.9. Contains also MDEV-10547: Test multi_update_innodb fails with InnoDB 5.7 The failure happened because 5.7 has changed the signature of the bool handler::primary_key_is_clustered() const virtual function ("const" was added). InnoDB was using the old signature which caused the function not to be used. MDEV-10550: Parallel replication lock waits/deadlock handling does not work with InnoDB 5.7 Fixed mutexing problem on lock_trx_handle_wait. Note that rpl_parallel and rpl_optimistic_parallel tests still fail. MDEV-10156 : Group commit tests fail on 10.2 InnoDB (branch bb-10.2-jan) Reason: incorrect merge MDEV-10550: Parallel replication can't sync with master in InnoDB 5.7 (branch bb-10.2-jan) Reason: incorrect merge	9 years ago
Monty	4dc5075860	Fixed compiler warnings and test failures found by buildbot Fixed ccfilter to detect errors where the column is included in the error message	9 years ago
Galina Shalygina	be1d06c8a5	Merge branch '10.2' into 10.2-mdev9864	10 years ago
Jan Lindström	c395aad668	MDEV-9840: Test encryption.innodb-log-encrypt-crash fails on buildbot Problem: We created more than 5 encryption keys for redo-logs. Idea was that we do not anymore create more than one encryption key for redo-logs but if existing checkpoint from earlier MariaDB contains more keys, we should read all of them. Fix: Add new encryption key to memory structure only if there currently has none or if we are reading checkpoint from the log. Checkpoint from older MariaDB version could contain more than one key.	10 years ago
Jan Lindström	37a65e3335	MDEV-9793: getting mysqld crypto key from key version failed Make sure that we read all possible encryption keys from checkpoint and if log block checksum does not match, print all found checkpoint encryption keys.	10 years ago
Jan Lindström	7cb16dc2a3	MDEV-9422: Checksum errors on restart when killing busy instance that uses encrypted XtraDB tables Fix incorrectly merged files on innodb_plugin.	10 years ago
Jan Lindström	f448a800e1	MDEV-9422: Checksum errors on restart when killing busy instance that uses encrypted XtraDB tables Analysis: -- InnoDB has n (>0) redo-log files. -- In the first page of redo-log there is 2 checkpoint records on fixed location (checkpoint is not encrypted) -- On every checkpoint record there is up to 5 crypt_keys containing the keys used for encryption/decryption -- On crash recovery we read all checkpoints on every file -- Recovery starts by reading from the latest checkpoint forward -- Problem is that latest checkpoint might not always contain the key we need to decrypt all the redo-log blocks (see MDEV-9422 for one example) -- Furthermore, there is no way to identify is the log block corrupted or encrypted For example checkpoint can contain following keys : write chk: 4 [ chk key ]: [ 5 1 ] [ 4 1 ] [ 3 1 ] [ 2 1 ] [ 1 1 ] so over time we could have a checkpoint write chk: 13 [ chk key ]: [ 14 1 ] [ 13 1 ] [ 12 1 ] [ 11 1 ] [ 10 1 ] killall -9 mysqld causes crash recovery and on crash recovery we read as many checkpoints as there is log files, e.g. read [ chk key ]: [ 13 1 ] [ 12 1 ] [ 11 1 ] [ 10 1 ] [ 9 1 ] read [ chk key ]: [ 14 1 ] [ 13 1 ] [ 12 1 ] [ 11 1 ] [ 10 1 ] [ 9 1 ] This is problematic, as we could still scan log blocks e.g. from checkpoint 4 and we do not know anymore the correct key. CRYPT INFO: for checkpoint 14 search 4 CRYPT INFO: for checkpoint 13 search 4 CRYPT INFO: for checkpoint 12 search 4 CRYPT INFO: for checkpoint 11 search 4 CRYPT INFO: for checkpoint 10 search 4 CRYPT INFO: for checkpoint 9 search 4 (NOTE: NOT FOUND) For every checkpoint, code generated a new encrypted key based on key from encryption plugin and random numbers. Only random numbers are stored on checkpoint. Fix: Generate only one key for every log file. If checkpoint contains only one key, use that key to encrypt/decrypt all log blocks. If checkpoint contains more than one key (this is case for databases created using MariaDB server version 10.1.0 - 10.1.12 if log encryption was used). If looked checkpoint_no is found from keys on checkpoint we use that key to decrypt the log block. For encryption we use always the first key. If the looked checkpoint_no is not found from keys on checkpoint we use the first key. Modified code also so that if log is not encrypted, we do not generate any empty keys. If we have a log block and no keys is found from checkpoint we assume that log block is unencrypted. Log corruption or missing keys is found by comparing log block checksums. If we have a keys but current log block checksum is correct we again assume log block to be unencrypted. This is because current implementation stores checksum only before encryption and new checksum after encryption but before disk write is not stored anywhere.	10 years ago
Jan Lindström	81d35841bd	MDEV-9011: Redo log encryption does not work Redo log encryption used too short buffer when getting encryption keys.	10 years ago
Sergei Golubchik	66b9a9409c	New encryption API. Piece-wise encryption. Instead of encrypt(src, dst, key, iv) that encrypts all data in one go, now we have encrypt_init(key,iv), encrypt_update(src,dst), and encrypt_finish(dst). This also causes collateral changes in the internal my_crypt.cc encryption functions and in the encryption service. There are wrappers to provide the old all-at-once encryption functionality. But binlog events are often written piecewise, they'll need the new api.	10 years ago
Jan Lindström	18b0176a05	MDEV-8410: Changing file-key-management to example-key-management causes crash and no real error MDEV-8409: Changing file-key-management-encryption-algorithm causes crash and no real info why Analysis: Both bugs has two different error cases. Firstly, at startup when server reads latest checkpoint but requested key_version, key management plugin or encryption algorithm or method is not found leading corrupted log entry. Secondly, similarly when reading system tablespace if requested key_version, key management plugin or encryption algorithm or method is not found leading buffer pool page corruption. Fix: Firsly, when reading checkpoint at startup check if the log record may be encrypted and if we find that it could be encrypted, print error message and do not start server. Secondly, if page is buffer pool seems corrupted but we find out that there is crypt_info, print additional error message before asserting.	10 years ago
Sergei Golubchik	ff7a1ff6d7	fix printf format string	10 years ago
Jan Lindström	f5ddffd83e	MDEV-8156: Assertion failure in file log0crypt.cc line 220 on server restart Instead of asserting print informative error message to error log and return failure from innodb_init causing the server to shutdown.	11 years ago
Jan Lindström	ecfc3de57e	MDEV-8129: Compilation warnings in log0crypt.cc Fix incorrect types and compiler warnings.	11 years ago
Jan Lindström	d259376fd8	MDEV-8041: InnoDB redo log encryption Merged new version of InnoDB/XtraDB redo log encryption from Google provided by Jonas Oreland.	11 years ago
Sergei Golubchik	97d5de4ccf	Add encryption key id to the API as a distinct concept which is separate from the encryption key version	11 years ago
Sergei Golubchik	129e960179	fix log_blocks_crypt() to actually decrypt the encrypted log It used to double-encrypt it, relying on the fact that second encrypt() call was (like XOR) negating the effect of the first one.	11 years ago
Sergei Golubchik	4d40a7d133	remove now-empty my_aes.{h,cc} move remaning defines to my_crypt, add MY_ namespace prefix	11 years ago
Sergei Golubchik	65e7826070	renames to follow single consistent naming style with namespace prefixes	11 years ago
Sergei Golubchik	bb1b61b312	encryption plugin controls the encryption * no --encryption-algorithm option anymore * encrypt/decrypt methods in the encryption plugin * ecnrypt/decrypt methods in the encryption_km service * file_km plugin has --file-key-management-encryption-algorithm * debug_km always uses aes_cbc * example_km changes between aes_cbc and aes_ecb for different key versions	11 years ago
Sergei Golubchik	9ccafffc29	rename "encryption key management plugin" to "encryption plugin" because it's going to do more than just key management	11 years ago
Sergei Golubchik	ef5b4889c2	optimize encryption api only one encryption key lookup in most cases instead of three (has_key, get_key_size, get_key).	11 years ago
Jan Lindström	71ec0463af	Encryption cleanup Step 3: -- Make encrytion_algorithm changeable by SUPER -- Remove AES_ECB method from encryption_algorithms -- Support AES method change by storing used method on InnoDB/XtraDB objects -- Store used AES method to crypt_data as different crypt types -- Store used AES method to redo/undo logs and checkpoint -- Store used AES method on every encrypted page after key_version -- Add test	11 years ago
Jan Lindström	7047bef1ef	Use standard InnoDB error mechanism on compression and encryption error messages.	11 years ago
Sergei Golubchik	c3f80a2bff	fix new innodb warnings to use the standard innodb warning syntax	11 years ago
Jan Lindström	2eae6848d9	MDEV-7572: InnoDB: Assertion failure in log_init_crypt_key if file_key_management_plugin is used Fixed error handling and added disabling InnoDB redo log encryption if encryption key management plugin is not there.	11 years ago

1 2

65 Commits (8511f04fdb4cc69a6c90b543624305ef715f9e53)