mariadb

Tree: 420f8e24ab

Author	SHA1	Message	Date
Vladislav Vaintroub	1faaaa9718	MDEV-15680 xb_aws_key_management fails in buildbot. aws_key_management needs current directory to be datadir during initalization, it scans current directory for encrypted keys. Fix is to ensure, that plugin initialization in mariabackup happens after the call to my_setwd(mysql_real_data_home).	7 years ago
Marko Mäkelä	f6d4f624eb	MDEV-12041: innodb_encrypt_log key rotation This will change the InnoDB encrypted redo log format only. Unencrypted redo log will keep using the MariaDB 10.3 format. In the new encrypted redo log format, 4 additional bytes will be reserved in the redo log block trailer for storing the encryption key version. For performance reasons, the encryption key rotation (checking if the latest encryption key version is being used) is only done at log_checkpoint(). LOG_HEADER_FORMAT_CURRENT: Remove. LOG_HEADER_FORMAT_ENC_10_4: The encrypted 10.4 format. LOG_BLOCK_KEY: The encryption key version field. LOG_BLOCK_TRL_SIZE: Remove. log_t: Add accessors framing_size(), payload_size(), trailer_offset(), to be used instead of referring to LOG_BLOCK_TRL_SIZE. log_crypt_t: An operation passed to log_crypt(). log_crypt(): Perform decryption, encryption, or encryption with key rotation. Return an error if key rotation at decryption fails. On encryption, keep using the previous key if the rotation fails. At startup, old-format encrypted redo log may be written before the redo log is upgraded (rebuilt) to the latest format. log_write_up_to(): Add the parameter rotate_key=false. log_checkpoint(): Invoke log_write_up_to() with rotate_key=true.	7 years ago
Marko Mäkelä	7b145fae13	mariabackup: Use snprintf() instead of sprintf()	7 years ago
Vladislav Vaintroub	95ef8de891	mariabackup - rename backup-rocksdb option to rocksdb-backup to avoid "using unique option prefix 'backup' is error-prone", there is already --backup option there.	7 years ago
Marko Mäkelä	b8514c94f6	MDEV-16496 Mariabackup: Implement --verbose option to instrument InnoDB log apply srv_print_verbose_log: Introduce the value 2 to refer to mariabackup --verbose. recv_recover_page(), recv_parse_log_recs(): Add output for mariabackup --verbose.	7 years ago
Marko Mäkelä	ff317fe08e	Follow-up to MDEV-16367 mariabackup: error: failed to copy enough redo log Commit `dc9c555415` moved the final phase of the redo log copying to the background thread. This would sometimes cause too little redo log to be copied at the end of the backup. We would only guarantee copying up to the latest redo log checkpoint. This would produce a consistent backup, but it could refer to a too old point of time. xtrabackup_copy_log(), xtrabackup_copy_logfile(): Add the parameter 'last'. xtrabackup_backup_low(): Copy any remaining part of the log after the backup threads have terminated.	7 years ago
Marko Mäkelä	a79b033b35	MDEV-16457 mariabackup 10.2+ should default to innodb_checksum_algorithm=crc32 Since MariaDB Server 10.2.2 (and MySQL 5.7), the default value of innodb_checksum_algorithm is crc32 (CRC-32C), not the inefficient "innodb" checksum. Change Mariabackup to use the same default, so that checksum validation (when using the default algorithm on the server) will take less time during mariabackup --backup. Also, mariabackup --prepare should be a little faster, and the server should read backups faster, because the page checksums would only be validated against CRC-32C.	7 years ago
Vladislav Vaintroub	aba2d7301f	MDEV-13122 Backup myrocksdb with mariabackup.	7 years ago
Marko Mäkelä	dc9c555415	MDEV-16367 mariabackup: error: failed to copy enough redo log log_copying_thread(): Keep copying redo log until the end has been reached. (Previously, we would stop copying as soon as the first batch of xtrabackup_copy_logfile() returned.) log_copying: Remove. Use log_copying_running instead. copy_logfile: Remove. Log copying will now only be invoked from 2 places: from xtrabackup_backup_func() for the initial batch, and from log_copying_thread() until all of the log has been read. Use the global variable metadata_to_lsn for determining if the final part of the log is being copied. xtrabackup_copy_log(): Add diagnostic messages for terminating the copying. These messages should be dead code, because log_group_read_log_seg() should be checking for the same. xtrabackup_copy_logfile(): Correct the retrying logic. If anything was successfully read, process the portion that was read. On failure, let the caller close dst_log_file. io_watching_thread(): Stop throttling during the last phase of copying the log (metadata_to_lsn!=0). The final copying of the log will now be performed in log_copying_thread(). stop_backup_threads(): Clean up the message about stopping the log copying thread. xtrabackup_backup_low(): Read metadata_to_lsn from the latest checkpoint header page, even if it is the first page. Let the log_copying_thread take care of copying all of the redo log.	7 years ago
Marko Mäkelä	619c277a6c	Mariabackup: Make some globals static	7 years ago
Vladislav Vaintroub	f98496da96	MDEV-16105: Mariabackup does not support SSL The reason is the missing HAVE_OPENSSL define for mariabackup.	8 years ago
Marko Mäkelä	8bbcc0d505	MDEV-12218: Put back mariabackup --innodb-flush-method Implement innodb_flush_method as an enum parameter in Mariabackup, instead of ignoring the option and hard-wiring it to a default value. xb0xb.h: Remove. Only xtrabackup.cc refers to the enum parameters. innodb_flush_method_names[], innodb_flush_method_typelib[]: Define as non-static, so that mariabackup can share the definitions. srv_file_flush_method: Change the type to ulong, to match the assignment in init_one_value() and handle_options() in mariabackup.	8 years ago
Marko Mäkelä	baa5a43d8c	MDEV-16045: Replace log_group_t with log_t::files There is only one log_sys and only one log_sys.log. log_t::files::create(): Replaces log_init(). log_t::files::close(): Replaces log_group_close(), log_group_close_all(). fil_close_log_files(): if (free) log_sys.log_close(); The callers that passed free=true used to call log_group_close_all(). log_header_read(): Replaces log_group_header_read(). log_t::files::file_header_bufs_ptr: Use a single allocation. log_t::files::file_header_bufs[]: Statically allocate the pointers. log_t::files::set_fields(): Replaces log_group_set_fields(). log_t::files::calc_lsn_offset(): Replaces log_group_calc_lsn_offset(). Simplify the computation by using fewer variables. log_t::files::read_log_seg(): Replaces log_group_read_log_seg(). log_sys_t::complete_checkpoint(): Replaces log_io_complete_checkpoint(). fil_aio_wait(): Move the logic from log_io_complete().	8 years ago
Marko Mäkelä	d73a898d64	MDEV-16045: Allocate log_sys statically There is only one redo log subsystem in InnoDB. Allocate the object statically, to avoid unnecessary dereferencing of the pointer. log_t::create(): Renamed from log_sys_init(). log_t::close(): Renamed from log_shutdown(). log_t::checkpoint_buf_ptr: Remove. Allocate log_t::checkpoint_buf statically.	8 years ago
Marko Mäkelä	715e4f4320	MDEV-12218 Clean up InnoDB parameter validation Bind more InnoDB parameters directly to MYSQL_SYSVAR and remove "shadow variables". innodb_change_buffering: Declare as ENUM, not STRING. innodb_flush_method: Declare as ENUM, not STRING. innodb_log_buffer_size: Bind directly to srv_log_buffer_size, without rounding it to a multiple of innodb_page_size. LOG_BUFFER_SIZE: Remove. SysTablespace::normalize_size(): Renamed from normalize(). innodb_init_params(): A new function to initialize and validate InnoDB startup parameters. innodb_init(): Renamed from innobase_init(). Invoke innodb_init_params() before actually trying to start up InnoDB. srv_start(bool): Renamed from innobase_start_or_create_for_mysql(). Added the input parameter create_new_db. SRV_ALL_O_DIRECT_FSYNC: Define only for _WIN32. xb_normalize_init_values(): Merge to innodb_init_param().	8 years ago
Marko Mäkelä	9ed2b2b2b8	Do not divide or multiply by srv_page_size Instead, shift by srv_page_size_shift.	8 years ago
Marko Mäkelä	a90100d756	Replace univ_page_size and UNIV_PAGE_SIZE Try to use one variable (srv_page_size) for innodb_page_size. Also, replace UNIV_PAGE_SIZE_SHIFT with srv_page_size_shift.	8 years ago
Marko Mäkelä	5b79303b40	MDEV-15988 Crash in ./mtr mariabackup.undo_space_id xb_assign_undo_space_start(): Correctly pass the length of the buffer, so that the file name will not be truncated.	8 years ago
Marko Mäkelä	6c64101bf0	MDEV-12266 follow-up fix to Mariabackup xtrabackup_apply_delta(): Refer to fil_system.sys_space directly.	8 years ago
Michael Widenius	ddc5764303	Remove compiler warnings - Remove unused variables - Mark variables unused - Fix wrong types - Add no-strict-aliasing to BUILD scripts	8 years ago
Vladislav Vaintroub	c2dc72c0c3	MDEV-15779 - mariabackup incremental prepare fails on CIFS mount. CIFS does not like O_DIRECT flag (it is set successfully, but pread would fail). The fix is not to use O_DIRECT, there is not need for it. posix_fadvise() was used already that should prevent buffer cache pollution on Linux. As recommended by documentation of posix_fadvise(), we'll also fsync() tablespaces after a batch of writes.	8 years ago
Vladislav Vaintroub	ecf6675cfc	MDEV-15713 mariabackup: throw warning, if --stream is used without --backup	8 years ago
Vladislav Vaintroub	37f24806fc	MDEV-15825 Mariabackup help mentions Percona and PXC but not MariaDB	8 years ago
Marko Mäkelä	bc2501453c	Remove an unused variable	8 years ago
Thirunarayanan Balathandayuthapani	d9c5a46678	MDEV-15737 assertion in mariabackup.exe!recv_calc_lsn_on_data_add() - recovered_lsn shouldn't be initialized during xtrabackup_copy_logfile(). If partial redo log read during the end of xtrabackup_copy_logfile() then recovered_lsn will be different from scanned_lsn. Re-initialization of recovered_lsn could lead to partial read again. It is a regression of MDEV-14545	8 years ago
Vladislav Vaintroub	27c24808f7	MDEV-15636 mariabackup --lock-ddl-per-table hangs if ALTER table is running concurrently. There is a deadlock between C1 mariabackup's connection that holds MDL locks C2 Online ALTER TABLE that wants to have MDL exclusively and tries to upgrade its mdl lock. C3 another mariabackup's connection that does FLUSH TABLES (or FTWRL) C3 waits waits for C2, which waits for C1, which waits for C3, thus the deadlock. MDL locks cannot be released until FLUSH succeeds, because otherwise it would allow ALTER to sneak in, causing backup to abort and breaking lock-ddl-per-table's promise. The fix here workarounds the deadlock, by killing connections in "Waiting for metadata lock" status (i.e ALTER). This killing continues until FTWRL succeeds. Killing connections is skipped in case --no-locks parameter was passed to backup, because there won't be a FLUSH. For the reference,in Percona's xtrabackup --lock-ddl-per-connection silently implies --no-lock ie FLUSH is always skipped there. A rather large part of fix is introducing DBUG capability to start a query the new connection at the right moment of backup compensating somewhat for mariabackup' lack of send_query or DBUG_SYNC.	8 years ago
Marko Mäkelä	4cad42392a	MDEV-12266: Change dict_table_t::space to fil_space_t* InnoDB always keeps all tablespaces in the fil_system cache. The fil_system.LRU is only for closing file handles; the fil_space_t and fil_node_t for all data files will remain in main memory. Between startup to shutdown, they can only be created and removed by DDL statements. Therefore, we can let dict_table_t::space point directly to the fil_space_t. dict_table_t::space_id: A numeric tablespace ID for the corner cases where we do not have a tablespace. The most prominent examples are ALTER TABLE...DISCARD TABLESPACE or a missing or corrupted file. There are a few functional differences; most notably: (1) DROP TABLE will delete matching .ibd and .cfg files, even if they were not attached to the data dictionary. (2) Some error messages will report file names instead of numeric IDs. There still are many functions that use numeric tablespace IDs instead of fil_space_t, and many functions could be converted to fil_space_t member functions. Also, Tablespace and Datafile should be merged with fil_space_t and fil_node_t. page_id_t and buf_page_get_gen() could use fil_space_t& instead of a numeric ID, and after moving to a single buffer pool (MDEV-15058), buf_pool_t::page_hash could be moved to fil_space_t::page_hash. FilSpace: Remove. Only few calls to fil_space_acquire() will remain, and gradually they should be removed. mtr_t::set_named_space_id(ulint): Renamed from set_named_space(), to prevent accidental calls to this slower function. Very few callers remain. fseg_create(), fsp_reserve_free_extents(): Take fil_space_t as a parameter instead of a space_id. fil_space_t::rename(): Wrapper for fil_rename_tablespace_check(), fil_name_write_rename(), fil_rename_tablespace(). Mariabackup passes the parameter log=false; InnoDB passes log=true. dict_mem_table_create(): Take fil_space_t* instead of space_id as parameter. dict_process_sys_tables_rec_and_mtr_commit(): Replace the parameter 'status' with 'bool cached'. dict_get_and_save_data_dir_path(): Avoid copying the fil_node_t::name. fil_ibd_open(): Return the tablespace. fil_space_t::set_imported(): Replaces fil_space_set_imported(). truncate_t: Change many member function parameters to fil_space_t*, and remove page_size parameters. row_truncate_prepare(): Merge to its only caller. row_drop_table_from_cache(): Assert that the table is persistent. dict_create_sys_indexes_tuple(): Write SYS_INDEXES.SPACE=FIL_NULL if the tablespace has been discarded. row_import_update_discarded_flag(): Remove a constant parameter.	8 years ago
Marko Mäkelä	05863142ad	MDEV-12266: Remove fil_system_t::named_spaces fil_space_get_by_name(): Remove. (Implement differently in mariabackup.) fil_ibd_open(): Check if the tablespace by the same ID already exists. If it is the same name, return success, else failure.	8 years ago
Marko Mäkelä	2ac8b1a907	MDEV-12266: Add fil_system.sys_space, temp_space Add fil_system_t::sys_space, fil_system_t::temp_space. These will replace lookups for TRX_SYS_SPACE or SRV_TMP_SPACE_ID. mtr_t::m_undo_space, mtr_t::m_sys_space: Remove. mtr_t::set_sys_modified(): Remove. fil_space_get_type(), fil_space_get_n_reserved_extents(): Remove. fsp_header_get_tablespace_size(), fsp_header_inc_size(): Merge to the only caller, innobase_start_or_create_for_mysql().	8 years ago
Marko Mäkelä	600c85e85a	Allocate the singleton fil_system statically fil_system_t::create(): Replaces fil_init(), fsp_init(). fil_system_t::close(): Replaces fil_close(). fil_system_t::max_n_open: Remove. Use srv_max_n_open_files directly.	8 years ago
Thirunarayanan Balathandayuthapani	b6d68c6aa3	MDEV-13561 Mariabackup is incompatible with retroactively created innodb_undo_tablespaces - Mariabackup supports starting undo tablespace id which is greater than 1.	8 years ago
Thirunarayanan Balathandayuthapani	6d1d5c3aeb	MDEV-14545 Backup fails due to MLOG_INDEX_LOAD record - Fixed the asan failure of the unsupported_redo test case	8 years ago
Marko Mäkelä	bd7ed1b923	MDEV-13935 INSERT stuck at state Unlocking tables Revert the dead code for MySQL 5.7 multi-master replication (GCS), also known as WL#6835: InnoDB: GCS Replication: Deterministic Deadlock Handling (High Prio Transactions in InnoDB). Also, make innodb_lock_schedule_algorithm=vats skip SPATIAL INDEX, because the code does not seem to be compatible with them. Add FIXME comments to some SPATIAL INDEX locking code. It looks like Galera write-set replication might not work with SPATIAL INDEX.	8 years ago
Marko Mäkelä	84129fb1b5	After-merge fix for commit `98eb9518db` The merge only covered 10.1 up to commit `4d248974e0`. Actually merge the changes up to commit `0a534348c7`. Also, remove the unused InnoDB field trx_t::abort_type.	8 years ago
Sergey Vojtovich	0a534348c7	MDEV-14265 - RPMLint warning: shared-lib-calls-exit Eliminated last exit() call from libmysqld.	8 years ago
Thirunarayanan Balathandayuthapani	ff909acfa4	MDEV-14545 Backup fails due to MLOG_INDEX_LOAD record Problem: ======= Mariabackup exits during prepare phase if it encounters MLOG_INDEX_LOAD redo log record. MLOG_INDEX_LOAD record informs Mariabackup that the backup cannot be completed based on the redo log scan, because some information is purposely omitted due to bulk index creation in ALTER TABLE. Solution: ======== Detect the MLOG_INDEX_LOAD redo record during backup phase and exit the mariabackup with the proper error message.	8 years ago
Marko Mäkelä	54765aaa4d	MDEV-15524 Do not disable page checksums for temporary tables buf_flush_init_for_writing(): Remove the parameter skip_checksum.	8 years ago
Sergey Vojtovich	131d9a5d0c	Allocate lock_sys statically There is only one lock_sys. Allocate it statically in order to avoid dereferencing a pointer whenever accessing it. Also, align some members to their own cache line in order to avoid false sharing. lock_sys_t::create(): The deferred constructor. lock_sys_t::close(): The early destructor.	8 years ago
Vladislav Vaintroub	56e7b7eaed	Make possible to use clang on Windows (clang-cl) -DWITH_ASAN can be used as well now, on x64 Fix many clang-cl warnings.	8 years ago
Marko Mäkelä	947efe17ed	MDEV-15158 On commit, do not write to the TRX_SYS page This is based on a prototype by Thirunarayanan Balathandayuthapani <thiru@mariadb.com>. Binlog and Galera write-set replication information was written into TRX_SYS page on each commit. Instead of writing to the TRX_SYS during normal operation, InnoDB can make use of rollback segment header pages, which are already being written to during a commit. The following list of fields in rollback segment header page are added: TRX_RSEG_BINLOG_OFFSET TRX_RSEG_BINLOG_NAME (NUL-terminated; empty name = not present) TRX_RSEG_WSREP_XID_FORMAT (0=not present; 1=present) TRX_RSEG_WSREP_XID_GTRID TRX_RSEG_WSREP_XID_BQUAL TRX_RSEG_WSREP_XID_DATA trx_sys_t: Introduce the fields recovered_binlog_filename, recovered_binlog_offset, recovered_wsrep_xid. To facilitate upgrade from older mysql or mariaDB versions, we will read the information in TRX_SYS page. It will be overridden by the information that we find in rollback segment header pages. Mariabackup --prepare will read the metadata from the rollback segment header pages via trx_rseg_array_init(). It will still not read any undo log pages or recover any transactions.	8 years ago
Vladislav Vaintroub	2129eab7e2	MDEV-15071 backup does not store xtrabackup_info in the --extra-lsndir directory	8 years ago
Vladislav Vaintroub	a08121c978	MDEV-14997 mariabackup crashes with invalid --innodb-flush-method Avoid NULL pointer dereference when shutting down after invalid parameters were parsed.	8 years ago
Marko Mäkelä	00f0c039d2	MDEV-15270 Mariabackup should not try to use doublewrite buffer When Mariabackup gets a bad read of the first page of the system tablespace file, it would inappropriately try to apply the doublewrite buffer and write changes back to the data file (to the source file)! This is very wrong and must be prevented. The correct action would be to retry reading the system tablespace as well as any other files whose first page was read incorrectly. Fixing this was not attempted. xb_load_tablespaces(): Shorten a bogus message to be more relevant. The message can be displayed by --backup or --prepare. xtrabackup_backup_func(), os_file_write_func(): Add a missing space to a message. Datafile::restore_from_doublewrite(): Do not even attempt the operation in Mariabackup. recv_init_crash_recovery_spaces(): Do not attempt to restore the doublewrite buffer in Mariabackup (--prepare or --export), because all pages should have been copied correctly in --backup already, and because --backup should ignore the doublewrite buffer. SysTablespace::read_lsn_and_check_flags(): Do not attempt to initialize the doublewrite buffer in Mariabackup. innodb_make_page_dirty(): Correct the bounds check. Datafile::read_first_page(): Correct the name of the parameter.	8 years ago
Vladislav Vaintroub	6c279ad6a7	MDEV-15091 : Windows, 64bit: reenable and fix warning C4267 (conversion from 'size_t' to 'type', possible loss of data) Handle string length as size_t, consistently (almost always:)) Change function prototypes to accept size_t, where in the past ulong or uint were used. change local/member variables to size_t when appropriate. This fix excludes rocksdb, spider,spider, sphinx and connect for now.	8 years ago
Marko Mäkelä	d7d910d08b	Fix a warning about possibly unused variable xtrabackup_init_temp_log(): Remove a redundant variable and limit scope.	8 years ago
Daniel Black	7eb084fe2c	MariaBackup: gcc7 - snprintf output overflow warning extra/mariabackup/xtrabackup.cc: In function ‘ulint xb_process_datadir(const char, const char, handle_datadir_entry_func_t)’: extra/mariabackup/xtrabackup.cc:4534:1: warning: ‘snprintf’ output may be truncated before the last format character [-Wformat-truncation=] xb_process_datadir( ^~~~~~~~~~~~~~~~~~ mariabackup/xtrabackup.cc:4607:11: note: ‘snprintf’ output 2 or more bytes (assuming 4001) into a destination of size 4000 snprintf(dbpath, sizeof(dbpath), "%s/%s", path, dbinfo.name); ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~	8 years ago
Marko Mäkelä	8d1d38f953	Simplify access to the TRX_SYS page trx_sysf_t: Remove. trx_sysf_get(): Return the TRX_SYS page, not a pointer within it. trx_sysf_rseg_get_space(), trx_sysf_rseg_get_page_no(): Remove a parameter, and merge the declaration and definition. Take the TRX_SYS page as a parameter. TRX_SYS_N_RSEGS: Correct the comment. trx_sysf_rseg_find_free(), trx_sys_update_mysql_binlog_offset(), trx_sys_update_wsrep_checkpoint(): Take the TRX_SYS page as a parameter. trx_rseg_header_create(): Add a parameter for the TRX_SYS page. trx_sysf_rseg_set_space(), trx_sysf_rseg_set_page_no(): Remove; merge to the only caller, trx_rseg_header_create().	8 years ago
Vladislav Vaintroub	7cdf759c86	MDEV-14485 Server hangs on startup in THD::init Solve 3 way deadlock between plugin_initialiaze(), THD::init() and mysql_sys_var_char(). The deadlock exists because of the lock order inversion between LOCK_global_system_variables mutex and LOCK_system_variables_hash read-write lock- In this case, it is enough to change LOCK_system_variables_hash to prefer reads to fix the deadlock, i.e change it to mysql_prlock_t	8 years ago
Daniel Black	12f900228f	mariadbbackup: use defaults-group-suffix even if no --defaults-file Signed-off-by: Daniel Black <daniel@linux.vnet.ibm.com>	8 years ago
Vladislav Vaintroub	87db5eb813	MDEV-13825 mariabackup --lock-ddl-per-table does not properly lock FULLTEXT auxiliary tables Change the logic to take mdl lock on all tables before tablespaces are copied, rather than lock every single tablespace just before it is copied.	8 years ago

... 2 3 4 5 6 ...

461 Commits (420f8e24ab73ef00b323aaa9423f365fb38e9306)