mariadb

Tree: 6c5c1f0b2f

Author	SHA1	Message	Date
Eugene Kosov	6c5c1f0b2f	MDEV-19231 make DB_SUCCESS equal to 0 It's a micro optimization. On most platforms CPUs has instructions to compare with 0 fast. DB_SUCCESS is the most popular outcome of functions and this patch optimized code like (err == DB_SUCCESS) BtrBulk::finish(): bogus assertion fixed fil_node_t::read_page0(): corrected usage of os_file_read() que_eval_sql(): bugus assertion removed. Apparently it checked that the field was assigned after having been zero-initialized at object creation. It turns out that the return type of os_file_read_func() was changed in mysql/mysql-server@98909cefbc37e54efc6452c7e95bccbf64ac9213 (MySQL 5.7) from ibool to dberr_t. The reviewer (if there was any) failed to point out that because of future merges, it could be a bad idea to change the return type of a function without changing the function name. This change was applied to MariaDB 10.2.2 in commit `2e814d4702` but the MariaDB-specific code was not fully adjusted accordingly, e.g. in fil_node_open_file(). Essentially, code like !os_file_read(...) became dead code in MariaDB and later in Mariabackup 10.2, and we could be dealing with an uninitialized buffer after a failed page read.	7 years ago
Marko Mäkelä	b761211685	MDEV-18659: Fix string truncation/overflow in InnoDB and XtraDB Fix the warnings issued by GCC 8 -Wstringop-truncation and -Wstringop-overflow in InnoDB and XtraDB. This work is motivated by Jan Lindström. The patch mainly differs from his original one as follows: (1) We remove explicit initialization of stack-allocated string buffers. The minimum amount of initialization that is needed is a terminating NUL character. (2) GCC issues a warning for invoking strncpy(dest, src, sizeof dest) because if strlen(src) >= sizeof dest, there would be no terminating NUL byte in dest. We avoid this problem by invoking strncpy() with a limit that is 1 less than the buffer size, and by always writing NUL to the last byte of the buffer. (3) We replace strncpy() with memcpy() or strcpy() in those cases when the result is functionally equivalent. Note: fts_fetch_index_words() never deals with len==UNIV_SQL_NULL. This was enforced by an assertion that limits the maximum length to FTS_MAX_WORD_LEN. Also, the encoding that InnoDB uses for the compressed fulltext index is not byte-order agnostic, that is, InnoDB data files that use FULLTEXT INDEX are not portable between big-endian and little-endian systems.	7 years ago
Vladislav Vaintroub	5c159c9037	MDEV-18356 Renamed backup-encrypted option introduced in `7158edcba3` Rename it because it caused parser warning whenever --backup was used.	7 years ago
Vladislav Vaintroub	a8a27e65a8	MDEV-18212 mariabackup: Make output format uniform whenever possible	7 years ago
Marko Mäkelä	23e4446adc	Fix a merge error in the parent commit Fix an inadvertently inverted condition that caused galera.galera_sst_mariabackup_table_options test failure.	7 years ago
Marko Mäkelä	7158edcba3	MDEV-18129 Backup fails for encrypted tables: mariabackup: Database page corruption detected at page 1 If an encrypted table is created during backup, then mariabackup --backup could wrongly fail. This caused a failure of the test mariabackup.huge_lsn once on buildbot. This is due to the way how InnoDB creates .ibd files. It would first write a dummy page 0 with no encryption information. Due to this, xb_fil_cur_open() could wrongly interpret that the table is not encrypted. Subsequently, page_is_corrupted() would compare the computed page checksum to the wrong checksum. (There are both "before" and "after" checksums for encrypted pages.) To work around this problem, we introduce a Boolean option --backup-encrypted that is enabled by default. With this option, Mariabackup will assume that a nonzero key_version implies that the page is encrypted. We need this option in order to be able to copy encrypted tables from MariaDB 10.1 or 10.2, because unencrypted pages that were originally created before MySQL 5.1.48 could contain nonzero garbage in the fields that were repurposed for encryption. Later, MDEV-18128 would clean up the way how .ibd files are created, to remove the need for this option. page_is_corrupted(): Add missing const qualifiers, and do not check space->crypt_data unless --skip-backup-encrypted has been specified. xb_fil_cur_read(): After a failed page read, output a page dump.	7 years ago
Marko Mäkelä	50c9469be8	MDEV-18105 Mariabackup fails to copy encrypted InnoDB system tablespace if LSN>4G This is a regression caused by commit `8c43f96388` that was part of the MDEV-12112 fixes. page_is_corrupted(): Never interpret page_no=0 as encrypted.	7 years ago
Vladislav Vaintroub	9f4a4cb401	Cleanup recent mariabackup validation patches. - Refactor code to isolate page validation in page_is_corrupted() function. - Introduce --extended-validation parameter(default OFF) for mariabackup --backup to enable decryption of encrypted uncompressed pages during backup. - mariabackup would still always check checksum on encrypted data, it is needed to detect partially written pages.	7 years ago
Marko Mäkelä	dd72d7d561	MDEV-18025: Improve test case and consistency checks Write a test case that computes valid crc32 checksums for an encrypted page, but zeroes out the payload area, so that the checksum after decryption fails. xb_fil_cur_read(): Validate the page number before trying any checksum calculation or decrypting or decompression. Also, skip zero-filled pages. For page_compressed pages, ensure that the FIL_PAGE_TYPE was changed. Also, reject FIL_PAGE_PAGE_COMPRESSED_ENCRYPTED if no decryption was attempted.	7 years ago
Thirunarayanan Balathandayuthapani	171271edf8	MDEV-18025 Mariabackup fails to detect corrupted page_compressed=1 tables Problem: ======= Mariabackup seems to fail to verify the pages of compressed tables. The reason is that both fil_space_verify_crypt_checksum() and buf_page_is_corrupted() will skip the validation for compressed pages. Fix: ==== Mariabackup should call fil_page_decompress() for compressed and encrypted compressed page. After that, call buf_page_is_corrupted() to check the page corruption.	7 years ago
Marko Mäkelä	8c43f96388	Follow-up to MDEV-12112: corruption in encrypted table may be overlooked The initial fix only covered a part of Mariabackup. This fix hardens InnoDB and XtraDB in a similar way, in order to reduce the probability of mistaking a corrupted encrypted page for a valid unencrypted one. This is based on work by Thirunarayanan Balathandayuthapani. fil_space_verify_crypt_checksum(): Assert that key_version!=0. Let the callers guarantee that. Now that we have this assertion, we also know that buf_page_is_zeroes() cannot hold. Also, remove all diagnostic output and related parameters, and let the relevant callers emit such messages. Last but not least, validate the post-encryption checksum according to the innodb_checksum_algorithm (only accepting one checksum for the strict variants), and no longer try to validate the page as if it was unencrypted. buf_page_is_zeroes(): Move to the compilation unit of the only callers, and declare static. xb_fil_cur_read(), buf_page_check_corrupt(): Add a condition before calling fil_space_verify_crypt_checksum(). This is a non-functional change. buf_dblwr_process(): Validate the page only as encrypted or unencrypted, but not both.	7 years ago
Marko Mäkelä	fb252f70c1	MDEV-12112 corruption in encrypted table may be overlooked After validating the post-encryption checksum on an encrypted page, Mariabackup should decrypt the page and validate the pre-encryption checksum as well. This should reduce the probability of accepting invalid pages as valid ones. This is a backport and refactoring of a patch that was originally written by Thirunarayanan Balathandayuthapani for the 10.2 branch.	7 years ago
Marko Mäkelä	e82e216e37	MDEV-17849 Undo tablespace truncation recovery fails to shrink file fil_space_t::add(): Replaces fil_node_create(), fil_node_create_low(). Let the caller pass fil_node_t::handle, to avoid having to close and re-open files. fil_node_t::read_page0(): Refactored from fil_node_open_file(). Read the first page of a data file. fil_node_open_file(): Open the file only once. srv_undo_tablespace_open(): Set the file handle for the opened undo tablespace. This should ensure that ut_ad(file->is_open()) no longer fails in recv_add_trim(). xtrabackup_backup_func(): Remove some dead code. xb_fil_cur_open(): Open files only if needed. Undo tablespaces should already have been opened.	7 years ago
Marko Mäkelä	eb6364619f	Remove the redundant variable fil_n_file_opened	7 years ago
Sergei Golubchik	8bee7c16c8	compiler warnings (clang 4.0.1 on i386) extra/mariabackup/fil_cur.cc:361:42: warning: format specifies type 'unsigned long' but the argument has type 'ib_int64_t' (aka 'long long') [-Wformat] extra/mariabackup/fil_cur.cc:376:9: warning: format specifies type 'unsigned long' but the argument has type 'ib_int64_t' (aka 'long long') [-Wformat] sql/handler.cc:6196:45: warning: format specifies type 'unsigned long' but the argument has type 'wsrep_trx_id_t' (aka 'unsigned long long') [-Wformat] sql/log.cc:1681:16: warning: format specifies type 'unsigned long' but the argument has type 'size_t' (aka 'unsigned int') [-Wformat] sql/log.cc:1687:16: warning: format specifies type 'unsigned long' but the argument has type 'size_t' (aka 'unsigned int') [-Wformat] sql/wsrep_sst.cc:1388:86: warning: format specifies type 'long' but the argument has type 'wsrep_seqno_t' (aka 'long long') [-Wformat] sql/wsrep_sst.cc:232:86: warning: format specifies type 'long' but the argument has type 'wsrep_seqno_t' (aka 'long long') [-Wformat] storage/connect/filamdbf.cpp:450:47: warning: format specifies type 'short' but the argument has type 'int' [-Wformat] storage/connect/filamdbf.cpp:970:47: warning: format specifies type 'short' but the argument has type 'int' [-Wformat] storage/connect/inihandl.cpp:197:16: warning: address of array 'key->name' will always evaluate to 'true' [-Wpointer-bool-conversion] storage/innobase/btr/btr0scrub.cc:151:17: warning: format specifies type 'long' but the argument has type 'int' [-Wformat] storage/innobase/buf/buf0buf.cc:5085:8: warning: nonnull parameter 'bpage' will evaluate to 'true' on first encounter [-Wpointer-bool-conversion] storage/innobase/fil/fil0crypt.cc:2454:5: warning: format specifies type 'long' but the argument has type 'int' [-Wformat] storage/innobase/handler/ha_innodb.cc:18685:7: warning: format specifies type 'unsigned long' but the argument has type 'wsrep_trx_id_t' (aka 'unsigned long long') [-Wformat] storage/innobase/row/row0mysql.cc:3319:5: warning: format specifies type 'long' but the argument has type 'int' [-Wformat] storage/innobase/row/row0mysql.cc:3327:5: warning: format specifies type 'long' but the argument has type 'int' [-Wformat] storage/maria/ma_norec.c:35:10: warning: implicit conversion from 'int' to 'my_bool' (aka 'char') changes value from 131 to -125 [-Wconstant-conversion] storage/maria/ma_norec.c:42:10: warning: implicit conversion from 'int' to 'my_bool' (aka 'char') changes value from 131 to -125 [-Wconstant-conversion] storage/maria/ma_test2.c:1009:12: warning: format specifies type 'unsigned long' but the argument has type 'size_t' (aka 'unsigned int') [-Wformat] storage/maria/ma_test2.c:1010:12: warning: format specifies type 'unsigned long' but the argument has type 'size_t' (aka 'unsigned int') [-Wformat] storage/mroonga/ha_mroonga.cpp:9189:44: warning: use of logical '&&' with constant operand [-Wconstant-logical-operand] storage/mroonga/vendor/groonga/lib/expr.c:4987:22: warning: comparison of constant -1 with expression of type 'grn_operator' is always false [-Wtautological-constant-out-of-range-compare] storage/xtradb/btr/btr0scrub.cc:151:17: warning: format specifies type 'long' but the argument has type 'int' [-Wformat] storage/xtradb/buf/buf0buf.cc:5047:8: warning: nonnull parameter 'bpage' will evaluate to 'true' on first encounter [-Wpointer-bool-conversion] storage/xtradb/fil/fil0crypt.cc:2454:5: warning: format specifies type 'long' but the argument has type 'int' [-Wformat] storage/xtradb/row/row0mysql.cc:3324:5: warning: format specifies type 'long' but the argument has type 'int' [-Wformat] storage/xtradb/row/row0mysql.cc:3332:5: warning: format specifies type 'long' but the argument has type 'int' [-Wformat] unittest/sql/mf_iocache-t.cc:120:35: warning: format specifies type 'unsigned long' but the argument has type 'int' [-Wformat] unittest/sql/mf_iocache-t.cc:96:35: note: expanded from macro 'INFO_TAIL'	7 years ago
Vladislav Vaintroub	922e7badfc	MDEV-16791 mariabackup : Support DDL commands during backup	7 years ago
Marko Mäkelä	6979d20426	MDEV-14499 Mariabackup 10.2 fails to back up a multi-file InnoDB system tablespace When Mariabackup is invoked on an instance that uses a multi-file InnoDB system tablespace, it may fail to other files of the system tablespace than the first one. This was revealed by the MDEV-14447 test case. The offending code is assuming that the first page of each data file is page 0. But, in multi-file system tablespaces that is not the case. xb_fil_cur_open(): Instead of re-reading the first page of the file, rely on the fil_space_t metadata that already exists in memory. xb_get_space_flags(): Remove.	8 years ago
Marko Mäkelä	909cdafd35	MDEV-13496 Use "mariabackup" rather than "xtrabackup" in console output Replace the remaining "xtrabackup:" references in the code.	8 years ago
Marko Mäkelä	e9e051d297	Follow-up fix to MDEV-12988 backup fails if innodb_undo_tablespaces>0 The fix broke mariabackup --prepare --incremental. The restore of an incremental backup starts up (parts of) InnoDB twice. First, all data files are discovered for applying .delta files. Then, after the .delta files have been applied, InnoDB will be restarted more completely, so that the redo log records will be applied via the buffer pool. During the first startup, the buffer pool is not initialized, and thus trx_rseg_get_n_undo_tablespaces() must not be invoked. The apply of the .delta files will currently assume that the --innodb-undo-tablespaces option correctly specifies the number of undo tablespace files, just like --backup does. The second InnoDB startup of --prepare for applying the redo log will properly invoke trx_rseg_get_n_undo_tablespaces(). enum srv_operation_mode: Add SRV_OPERATION_RESTORE_DELTA for distinguishing the apply of .delta files from SRV_OPERATION_RESTORE. srv_undo_tablespaces_init(): In mariabackup --prepare --incremental, in the initial SRV_OPERATION_RESTORE_DELTA phase, do not invoke trx_rseg_get_n_undo_tablespaces() because the buffer pool or the redo logs are not available. Instead, blindly rely on the parameter --innodb-undo-tablespaces.	8 years ago
Marko Mäkelä	e3d3147792	MDEV-13105 InnoDB fails to load a table with PAGE_COMPRESSION_LEVEL after upgrade from 10.1.20 When using innodb_page_size=16k, InnoDB tables that were created in MariaDB 10.1.0 to 10.1.20 with PAGE_COMPRESSED=1 and PAGE_COMPRESSION_LEVEL=2 or PAGE_COMPRESSION_LEVEL=3 would fail to load. fsp_flags_is_valid(): When using innodb_page_size=16k, use a more strict check for .ibd files, with the assumption that nobody would try to use different-page-size files.	8 years ago
Marko Mäkelä	8c71c6aa8b	MDEV-12548 Initial implementation of Mariabackup for MariaDB 10.2 InnoDB I/O and buffer pool interfaces and the redo log format have been changed between MariaDB 10.1 and 10.2, and the backup code has to be adjusted accordingly. The code has been simplified, and many memory leaks have been fixed. Instead of the file name xtrabackup_logfile, the file name ib_logfile0 is being used for the copy of the redo log. Unnecessary InnoDB startup and shutdown and some unnecessary threads have been removed. Some help was provided by Vladislav Vaintroub. Parameters have been cleaned up and aligned with those of MariaDB 10.2. The --dbug option has been added, so that in debug builds, --dbug=d,ib_log can be specified to enable diagnostic messages for processing redo log entries. By default, innodb_doublewrite=OFF, so that --prepare works faster. If more crash-safety for --prepare is needed, double buffering can be enabled. The parameter innodb_log_checksums=OFF can be used to ignore redo log checksums in --backup. Some messages have been cleaned up. Unless --export is specified, Mariabackup will not deal with undo log. The InnoDB mini-transaction redo log is not only about user-level transactions; it is actually about mini-transactions. To avoid confusion, call it the redo log, not transaction log. We disable any undo log processing in --prepare. Because MariaDB 10.2 supports indexed virtual columns, the undo log processing would need to be able to evaluate virtual column expressions. To reduce the amount of code dependencies, we will not process any undo log in prepare. This means that the --export option must be disabled for now. This also means that the following options are redundant and have been removed: xtrabackup --apply-log-only innobackupex --redo-only In addition to disabling any undo log processing, we will disable any further changes to data pages during --prepare, including the change buffer merge. This means that restoring incremental backups should reliably work even when change buffering is being used on the server. Because of this, preparing a backup will not generate any further redo log, and the redo log file can be safely deleted. (If the --export option is enabled in the future, it must generate redo log when processing undo logs and buffered changes.) In --prepare, we cannot easily know if a partial backup was used, especially when restoring a series of incremental backups. So, we simply warn about any missing files, and ignore the redo log for them. FIXME: Enable the --export option. FIXME: Improve the handling of the MLOG_INDEX_LOAD record, and write a test that initiates a backup while an ALGORITHM=INPLACE operation is creating indexes or rebuilding a table. An error should be detected when preparing the backup. FIXME: In --incremental --prepare, xtrabackup_apply_delta() should ensure that if FSP_SIZE is modified, the file size will be adjusted accordingly.	8 years ago
Vladislav Vaintroub	ce4c56db0c	MDEV-9566 Port Percona Xtrabackup to MariaDB as mariabackup - Modify backup code to work with MariaDB's 10.1 xtradb - Remove crypt encryption, version_check.pl, "compact backup" - Add encryption plugin	9 years ago
Vladislav Vaintroub	d7714308e0	MDEV-9566 Add Percona Xtrabackup 2.3.7	9 years ago

33 Commits (6c5c1f0b2fc71ab7e90dc7798180135e7967a264)