mariadb

Commit Graph

Author	SHA1	Message	Date
Julius Goryavsky	888010d9dd	MDEV-21951: mariabackup SST fail if data-directory have lost+found directory To fix this, it is necessary to add an option to exclude the database with the name "lost+found" from processing (the database name will be checked by the check_if_skip_database_by_path() or by the check_if_skip_database() function, and as a result "lost+found" will be skipped). In addition, it is necessary to slightly modify the verification logic in the check_if_skip_database() function. Also added a new test galera_sst_mariabackup_lost_found.test	6 years ago
Julius Goryavsky	31201dcbaf	MDEV-21770: `galera_3nodes.galera_ipv6_mariabackup` fails This patch fixes several flaws in the SST scripts that cause failures while running tests that use version 6 IP addresses for cluster nodes. First, if the netcat utility is used for streaming (but not socat), then in accordance with its command line syntax, we need to remove the square brackets around the IPv6 address. However, for socat, the address must contain square brackets, as before. Secondly, if an IPv6 address is used, then from the joiner side for a number of systems (such as Debian) we need to explicitly specify the "-6" option, otherwise a listening socket with an IPv6 address may not be created. This patch also contains code improvements in the wsrep_sst_common. Changed the code that pars the connection address - fixed the shortcomings that sometimes led to incorrect parsing of parameters when using shells other than the latest versions of bash. Also, this patch removes the duplicate code that is intended for parsing the connection address and which was located in the wsrep_sst_mariabackup file, since all the necessary actions have already been done in wsrep_sst_common and there they are done in such a way that any shell is supported, not just bash. The fix does not require separate tests, since all the necessary tests are already present in the galera_3nodes suite. On the contrary, after this fix, tests using IPv6 addresses can be removed from the disabled list (this will be done in a separate commit related to MDEV-23659).	5 years ago
Jan Lindström	9aea50616c	Increase mariabackup SST initial timeout to avoid timeouts.	5 years ago
Julius Goryavsky	7ad4709a3b	MDEV-21526: mysqld_multi no longer works with different server binaries The problem is caused by the fact that adding the --defaults-group-suffix option to fix MDEV-18863 causes mysqld to read all options from the appropriate sections of the config file, including options specific to mysqld_multi. Reading unknown options (which are not supported by mysqld) causes mysqld to terminate with an error. However, the MDEV-18863 problem has been completely fixed by passing options on the command line, and now there is no need to specify the --defaults-group-suffix option (we just need to give priority to options passed through the command line, so as not to break MDEV-18863).	5 years ago
Julius Goryavsky	f9d8571f38	MDEV-22554: galera_sst_mariabackup fails with "Failed to start mysqld.2" The problem is caused by the operation of netcat streamer and does not appear on systems where socat is installed. We need to add the "-N" option for netcat to call shutdown() on the socket when receiving EOF from STDIN.	6 years ago
Julius Goryavsky	982294ac16	MDEV-17601: MariaDB Galera does not expect 'mbstream' as streamfmt Setting "streamfmt=mbstream" in the "[sst]" section causes SST to fail because the format automatically switches to 'tar' by default (insead of mbstream). To fix this, we need to add mbstream to the list of valid values for the format, making it synonymous with xbstream. This must be done both in the SST script and when parsing the options of the corresponding utilities.	6 years ago
Hartmut Holzgraefe	c4a844ca51	MDEV-20981 wsrep_sst_mariabackup fails silently when mariabackup is not installed (#1406 ) Make sure failure to find mariabackup binary does not terminate the script silently, terminate with a clear error message instead	6 years ago
Julius Goryavsky	4ba20e0a14	Improved handling of subdirectories in the xtrabackup-v2 SST scripts (similar to MDEV-18863) for more predictable test results (related to xtrabackup-v2 SST)	6 years ago
Julius Goryavsky	de0f93fb0d	MDEV-20420: SST failed after MDEV-18863 in some test configurations After applying MDEV-18863, in some test configurations, SST may fails due to duplication of some parameters (in particular "--port") in the main part of the command line and after "--mysqld-args", as well as due to incorrect interpretation of the parameter "--port" passed after "--mysqld-args" when the SST script is invoked without explicitly specifying a port for SST. In addition, it is necessary to correctly handle spaces, quotation marks and special characters when copying original arguments from the argv[] array to a new command line (after "--mysqld-args"). This patch resolves these shortcomings.	6 years ago
Julius Goryavsky	ff6d3075d5	MDEV-18863: Galera SST scripts can't read [mysqldN] option groups Some users and some scripts (for example, mysqld_multi.sh) use special option groups with names like [mysqld1], [mysqld2], ..., [mysqldN]. But SST scripts can't currently fully support these option groups. The only option group-related value it gets from the server is --defaults-group-suffix, if that option was set for mysqld when the server was started. However, the SST scripts does not get told by the server to read these option groups, so this means that the SST script will fail to read options like innodb-data-home-dir when it is in a option group like [mysqld1]...[mysqldN]. Moreover, SST scripts ignore many parameters that can be passed to them explicitly and cannot transfer them further, for example, to the input of mariabackup utility. Ideally, we want to transfer all the parameters of the original mysqld call to utilities such as mariabackup, however the SST script does not receive these parameters from the server and therefore cannot transfer them to mariabackup. To correct these shortcomings, we need to transfer to the scripts all of the parameters of the original mysqld call, and in the SST scripts themselves provide for the transfer all of these parameters to utilities such as mariabackup. To prevent these parameters from mixing with the script's own parameters, they should be transferred to SST script after the special option "--mysqld-args", followed by the string argument with the original parameters, as it received by the mysqld call at the time of launch (further all these parameters will be passed to mariabackup, for example). In addition, the SST scripts themselves must be refined so that they can read the parameters from the user-selected group, not just from the global mysqld configuration group. And also so that they can receive the parameters (which important for their work) as command-line arguments.	6 years ago
Julius Goryavsky	457dc9d64d	MDEV-18863: Galera SST scripts can't read [mysqldN] option groups Some users and some scripts (for example, mysqld_multi.sh) use special option groups with names like [mysqld1], [mysqld2], ..., [mysqldN]. But SST scripts can't currently fully support these option groups. The only option group-related value it gets from the server is --defaults-group-suffix, if that option was set for mysqld when the server was started. However, the SST scripts does not get told by the server to read these option groups, so this means that the SST script will fail to read options like innodb-data-home-dir when it is in a option group like [mysqld1]...[mysqldN]. Moreover, SST scripts ignore many parameters that can be passed to them explicitly and cannot transfer them further, for example, to the input of mariabackup utility. Ideally, we want to transfer all the parameters of the original mysqld call to utilities such as mariabackup, however the SST script does not receive these parameters from the server and therefore cannot transfer them to mariabackup. To correct these shortcomings, we need to transfer to the scripts all of the parameters of the original mysqld call, and in the SST scripts themselves provide for the transfer all of these parameters to utilities such as mariabackup. To prevent these parameters from mixing with the script's own parameters, they should be transferred to SST script after the special option "--mysqld-args", followed by the string argument with the original parameters, as it received by the mysqld call at the time of launch (further all these parameters will be passed to mariabackup, for example). In addition, the SST scripts themselves must be refined so that they can read the parameters from the user-selected group, not just from the global mysqld configuration group. And also so that they can receive the parameters (which important for their work) as command-line arguments.	6 years ago
Daniel Black	85a18526bc	wsrep_sst: remove WSREP_SST_OPT_SUFFIX_VALUE, checking [mysqld] is covered in the parse_cnf --mysqld case	7 years ago
Jan Lindström	0957d25781	MDEV-18830: Port SST fixes from 10.4 to 10.1 modified: mysql-test/suite/galera/disabled.def modified: mysql-test/suite/galera/r/galera_many_rows.result modified: mysql-test/suite/galera/t/galera_kill_nochanges.test new file: mysql-test/suite/galera/t/galera_many_rows.cnf modified: mysql-test/suite/galera/t/galera_many_rows.test modified: mysql-test/suite/galera/t/galera_var_dirty_reads.test modified: scripts/wsrep_sst_mariabackup.sh	7 years ago
Julius Goryavsky	5b82751111	MDEV-18426: Most of the mtr tests in the galera_3nodes suite fail Most of the mtr tests in the galera_3nodes suite fail for a variety of reasons with a variety of errors. This patch fixes several substantial flaws in the galera_3nodes suite tests and in the mtr framework service files, adapting the tests from galera_3nodes for the current version of MariaDB. This patch also synchronizes some galera_3nodes-related files with the latest changes made for MDEV-17835 (v2 patch) and for MDEV-18379 in other branches (10.2 and 10.3). Closes #1161	7 years ago
Julius Goryavsky	dfc9bff5a9	MDEV-18379: IPv6 compatibility changes/Unification of check for IPv6 This patch contains the port of the MDEV-18379 patch for 10.2 branch, but also includes a number of changes made within MDEV-17835, which are necessary for the normal operation of tests that use IPv6: 1) Currently, the three-node mtr suite for Galera (galera_3nodes) uses a separate IPv6 availability check using the "have_ipv6.inc" file. This check duplicates a more accurate check at suite.pm level, which can be used by including the file "check_ipv6.inc". This patch removes this discrepancy between suites. 2) Fixed numerous bugs in the SST scripts and in the mtr test files (galera_3nodes mtr suite) that prevented the use of Galera with IPv6 addresses. 3) Fixed flaws in the galera_3nodes mtr suite control scripts, because of which they could not work with mariabackup. 4) Fixed flaws in the rsync and mysqldump tests (for galera_3nodes mtr tests suite). These tests were not performed successfully without these fixes. 5) GAL-501 test in the galera_3nodes suite does not contain the option "--bind-address=::" that is needed for the test to work correctly with IPv6 (at least on some systems), since without it the server will not wait for connections on the IPv6 interface. https://jira.mariadb.org/browse/MDEV-18379 and partially https://jira.mariadb.org/browse/MDEV-17835	7 years ago
Julius Goryavsky	4aea6b3e3f	MDEV-18379: Unification of check for IPv6 This patch contains the port of the MDEV-18379 patch for 10.1 branch, but also includes a number of changes made within MDEV-17835, which are necessary for the normal operation of tests that use IPv6: 1) Fixed flaws in the galera_3nodes mtr suite control scripts, because of which they could not work with mariabackup. 2) Fixed numerous bugs in the SST scripts and in the mtr test files (galera_3nodes mtr suite) that prevented the use of Galera with IPv6 addresses. 3) Fixed flaws in tests for rsync and mysqldump (for galera_3nodes mtr tests suite). These tests were not performed successfully without these fixes. 4) Currently, the three-node mtr suite for Galera (galera_3nodes) uses a separate IPv6 availability check using the "have_ipv6.inc" file. This check duplicates a more accurate check at suite.pm level, which can be used by including the file "check_ipv6.inc". This patch removes this discrepancy between suites. 5) GAL-501 test in the galera_3nodes suite does not contain the option "--bind-address=::" which is needed for the test to work correctly with IPv6 (at least on some systems), since without it the server will not wait for connections on the IPv6 interface. https://jira.mariadb.org/browse/MDEV-18379 and partially https://jira.mariadb.org/browse/MDEV-17835	7 years ago
Geoff Montee	2084cd5422	MDEV-17973: Don't overwrite xtrabackup-v2/mariabackup SST logs by default	7 years ago
Tao ZHOU	cb85803c45	Use absolute path for mariabackup binary	7 years ago
Tao ZHOU	216d5f7899	use `ps -p` instead of `ps --pid` make it compatible on FreeBSD. If the parent process does not exist, there's no need to kill it.	7 years ago
angeloudy	0e794c6a69	Make mariabackup.sh compatible on FreeBSD	7 years ago
mkaruza	7a7a61998c	MDEV-15442 xtrabackup-v2 SST donor stuck in DONOR/DESYNCED state when joiner is killed SST transfer scripts rsync and maribackup now monitor parent process and will cleanup if process dies	7 years ago
sjaakola	15c6d6a94a	MDEV-14612 wsrep_sst_mariabackup unnecessarily converts address to host name SST script for mariabackup (wsrep_sst_mariabackup.sh) does reverse DNS to convert the joiner's IP address to domain name, and this is not always a wanted feature. It is not understood why this remote host DNS conversion is in the script, in the first place, but conversion was originally added by commit: commit `3d8aacba86` Author: Nirbhay Choubey <nirbhay.choubey@gmail.com> Date: Wed Feb 22 15:58:45 2017 -0500 In this commit, REMOTEHOST variable is removed and replaced by REMOTEIP. All references to joiner is therefore by IP address only.	8 years ago
Sergei Golubchik	a15ab358fc	wsrep_sst scripts: support traditional netcat	8 years ago
Marko Mäkelä	8c71c6aa8b	MDEV-12548 Initial implementation of Mariabackup for MariaDB 10.2 InnoDB I/O and buffer pool interfaces and the redo log format have been changed between MariaDB 10.1 and 10.2, and the backup code has to be adjusted accordingly. The code has been simplified, and many memory leaks have been fixed. Instead of the file name xtrabackup_logfile, the file name ib_logfile0 is being used for the copy of the redo log. Unnecessary InnoDB startup and shutdown and some unnecessary threads have been removed. Some help was provided by Vladislav Vaintroub. Parameters have been cleaned up and aligned with those of MariaDB 10.2. The --dbug option has been added, so that in debug builds, --dbug=d,ib_log can be specified to enable diagnostic messages for processing redo log entries. By default, innodb_doublewrite=OFF, so that --prepare works faster. If more crash-safety for --prepare is needed, double buffering can be enabled. The parameter innodb_log_checksums=OFF can be used to ignore redo log checksums in --backup. Some messages have been cleaned up. Unless --export is specified, Mariabackup will not deal with undo log. The InnoDB mini-transaction redo log is not only about user-level transactions; it is actually about mini-transactions. To avoid confusion, call it the redo log, not transaction log. We disable any undo log processing in --prepare. Because MariaDB 10.2 supports indexed virtual columns, the undo log processing would need to be able to evaluate virtual column expressions. To reduce the amount of code dependencies, we will not process any undo log in prepare. This means that the --export option must be disabled for now. This also means that the following options are redundant and have been removed: xtrabackup --apply-log-only innobackupex --redo-only In addition to disabling any undo log processing, we will disable any further changes to data pages during --prepare, including the change buffer merge. This means that restoring incremental backups should reliably work even when change buffering is being used on the server. Because of this, preparing a backup will not generate any further redo log, and the redo log file can be safely deleted. (If the --export option is enabled in the future, it must generate redo log when processing undo logs and buffered changes.) In --prepare, we cannot easily know if a partial backup was used, especially when restoring a series of incremental backups. So, we simply warn about any missing files, and ignore the redo log for them. FIXME: Enable the --export option. FIXME: Improve the handling of the MLOG_INDEX_LOAD record, and write a test that initiates a backup while an ALGORITHM=INPLACE operation is creating indexes or rebuilding a table. An error should be detected when preparing the backup. FIXME: In --incremental --prepare, xtrabackup_apply_delta() should ensure that if FSP_SIZE is modified, the file size will be adjusted accordingly.	8 years ago
Nirbhay Choubey	3d8aacba86	SST script for mariabackup. Execute mariabackup in innobackupex mode to avoid "unknown argument" error.	9 years ago
Hartmut Holzgraefe	c9ded859a6	MDEV-10853 netcat help output in error log when running xtrabackup SST	9 years ago
Hartmut Holzgraefe	04f92dde67	MDEV-10853 netcat help output in error log when running xtrabackup SST	9 years ago
sjaakola	58386ca04d	refs codership/mysql-wsrep#239 Synced xtrabackup SST scripts from PXC source tree as of PXC 5.6.27-25.13 - PXC#480: xtrabackup-v2 SST fails with multiple log_bin directives in my.cn - PXC#460: wsrep_sst_auth don't work in Percona-XtraDB-Cluster-56-5.6.25-25. - PXC-416: Fix SST related issues. - PXC-389: Merge remote-tracking branch 'wsrep/5.6' into 5.6-wsrep-pxc389 - Bug #1431101: SST does not clobber backup-my.cnf	10 years ago
Martin Stefany	64c115b835	Use hostname instead of IP in donor's socat Using IP address in donor's socat with TLS/SSL and certificate which doesn't contain IP address in CN or SubjectAltName causes transfer to fail with message: socat[5799] E certificate is valid but its commonName does not match hostname. This patch tries to reverse resolve IP address to hostname and use it for transfer. If reverse resolution fails, IP address is still used as fall-back, so proper A/AAAA and PTR records are important, but not mandatory. Certain certificates cannot contain IP addresses, e.g. FreeIPA's Dogtag doesn't allow it, so in my case I would need to use self- signed certificates instead, use verify=0 with socat or don't use TLS/SSL at all. Issue is mentioned in MDEV-9403.	10 years ago
Nirbhay Choubey	51a32ebeb3	MDEV-9884: Existing /var/lib/mysql/.sst directory (with contents) causes SST to fail with xtrabackup-v2 [Fix taken from https://github.com/percona/percona-xtradb- cluster/commit/b3ee75949ed82b88f355ca2e26431350cc1c89ac] During SST, the receiver node creates .sst directory under datadir to process/prepare the received data and removes it at the end of the process. In case of error, this directory, however, was not removed, which later caused subsequent SSTs to fail. Fixed by removing this directory at the beginning of SST if it existed.	10 years ago
Krunal Bauskar	f67d6fccac	- PXC#480: xtrabackup-v2 SST fails with multiple log_bin directives in my.cnf If any given variable the xtrabackup-v2 sst script looks for is specified multiple times in cnf file then it tend to pick both of them causing some of the follow-up command to fail. Avoid this programatic mistake by honoring only the last variable assigned setting as done by mysqld too. Check https://bugs.launchpad.net/percona-xtradb-cluster/+bug/1362830	10 years ago
Krunal Bauskar	0cf66e49db	- PXC#460: wsrep_sst_auth don't work in Percona-XtraDB-Cluster-56-5.6.25-25.12.1.el7 Semantics: --------- * Generally end-user will create a separate user with needed privileges for performing DONOR action. * This user credentials are specified using wsrep_sst_auth. * Along with this user there could be other user(s) created on the server that sysadmin may use for normal or other operations * Credentials for these user(s) can be specified in same cluster/server cnf file as part of [client] section When cluster act as DONOR and if wsrep_sst_auth is provided then it should strictly use it for performing SST based action. What if end-user has same credentials for performing both SST action and normal admin work ? * Then end-user can simply specify these credentials as part of [client] section in cnf file and skip providing wsrep_sst_auth. Issue: ----- MySQL client user/password parsing preference order is as follows: * command line (through --user/--password) * cnf file * MYSQL_PWD enviornment variable. Recent change tried passing sst user password through MYSQL_PWD (and user though --user command line param as before). On the system where-in admin had another user for performing non-SST actions, credentials for such user were present in cnf file under [client] section. Due to mysql client preference order, SST user name was used (as it was passed through command line) but password of other user (meant for non-SST) action was being used as it was passed through cnf file. Password passed through MYSQL_PWD was completely ignored causing user-name/password mismatch. Solution: --------- * If user has specified credentials for SST then pass them through command line so that they are used in priority. (There could be security concern on passing things through command line but when I tried passing user-name and password through command line to mysql client and then did ps I saw this ./bin/mysql --user=sstuser --password=x xxxxxxxx -S /tmp/n1.sock so seems like password is not shown)	10 years ago
Nirbhay Choubey	ab9a488dec	MDEV-9141 : [PATCH] Add CA validation to wsrep_sst_xtrabackup-v2.sh - Add CA validation to wsrep_sst_xtrabackup-v2.sh. - Also added a few {} around tpem for consistency. - Abort if encryption is requested but socat is not ssl-enabled. Patch contributed by : Klaas Demter	10 years ago
Alexey Yurchenko	a7ea3ec34d	Synced xtrabackup SST fixes from Percona tree (as of PXC 5.6.24-25.11 release). This fixes/adresses the following LP bugs: - LP1380697: wsrep_sst_xtrabackup-v2 doesn't stop when mysql is SIGKILLed. (full fix for this (as engineeered by Percona) requires Linux-specific patch that we don't carry, but keep xtrabackup scripts as close as possible) - LP1399134: Log the innobackupex/SST logs in SST to syslog if possible. (fixed) - LP1405668: Race condition between donor and joiner in PXB SST. (fixed) - LP1405985: Fail early if xtrabackup_checkkpoints is missing. (fixed) - LP1407599: wsrep_sst_xtrabackup-v2 script causes innobackupex to print a false positive stack trace into the log. (fixed) - LP1441762: IST Fails with SST script error. (fixed) - LP1451670: Fail when move-back fails in xtrabackup SST. (fixed)	11 years ago
Alexey Yurchenko	d78110e7fa	Refs codership/mysql-wsrep#141 : this commit 1. Passes wsrep_sst_auth_value to SST scripts via WSREP_SST_OPT_AUTH envronmental variable, so it never appears on the command line 2. In mysqldump and xtrabackup* SST scripts which rely on MySQL authentication, instead of passing password on the command line, SST script sets MYSQL_PWD environment variable, so that password also never appears on the mysqldump/innobackupex command line.	11 years ago
Nirbhay Choubey	dced5146bd	Merge branch '10.0-galera' into 10.1	10 years ago
Nirbhay Choubey	5467b12dc3	MDEV-7903 : xtrabackup SST failing with maria-10.0-galera Remove master-bin.state file on joiner node during SST.	11 years ago
Nirbhay Choubey	8fdf8f00c7	mysql-wsrep#38 : wsrep_sst_xtrabackup-v2 script causes innobackupex to ..print a false positive stack trace into the log Suppress the stacktrace that innobackupex dumps on receiving SIGPIPE.	11 years ago
Philip Stoev	7d550c76be	LP1378355 - Pass the value of --defaults-group-suffix to xtrabackup during SST	11 years ago
Alexey Yurchenko	14b6172075	Going more MTR-friendly - for SST prefer to use mysql client, mysqldump and my_print_defaults from the local build tree.	11 years ago
dirtysalt	9fdae3eb60	Refs #8 : preserve gvwstate.dat for pc recovery feature	11 years ago
Alexey Yurchenko	822c00536d	Going more MTR-friendly - for SST prefer to use mysql client, mysqldump and my_print_defaults from the local build tree.	11 years ago
dirtysalt	51744b3f80	Refs #8 : [5.5] preserve gvwstate.dat for pc recovery feature	11 years ago
Nirbhay Choubey	34d86ac9ff	MDEV-6594: Use separate domain_id for Galera transactions	11 years ago
Nirbhay Choubey	abf400e810	MDEV-6771 : Incorrect Size for Transfer Reported to pv xargs may split the input and execute them in batches for large number of files. As a result xtrabackup sst script may feed incorrect size to pv. Fixed by piping the files to du directly (du --files0-from=-). [Based on suggestions from Brad Jorgensen]	11 years ago
Nirbhay Choubey	33b4fab867	MDEV-6771 : Incorrect Size for Transfer Reported to pv xargs may split the input and execute them in batches for large number of files. As a result xtrabackup sst script may feed incorrect size to pv. Fixed by piping the files to du directly (du --files0-from=-). [Based on suggestions from Brad Jorgensen]	11 years ago
Sergei Golubchik	4bb49d84a9	correct handling on defaults[-extra]-file is SST scripts pass --defaults-file and --defaults-extra-file (whatever was specified, or none) from mysqld down to SST scripts. parse these options in SST scripts and pass them down to mysqldump, my_print_defaults, and xtrabackup	11 years ago
Jan Lindström	df4dd593f2	MDEV-6247: Merge 10.0-galera to 10.1. Merged lp:maria/maria-10.0-galera up to revision 3879. Added a new functions to handler API to forcefully abort_transaction, producing fake_trx_id, get_checkpoint and set_checkpoint for XA. These were added for future possiblity to add more storage engines that could use galera replication.	11 years ago
Jan Lindström	5ebb396638	MDEV-6247: Merge 10.0-galera to 10.1. Merged lp:maria/maria-10.0-galera up to revision 3867.	11 years ago
Nirbhay Choubey	c29cbac2eb	bzr merge -r3985..3997 codership/5.5	12 years ago

37 Commits (888010d9dd5409c48caf01fb0f465c98caa232c7)