You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

235 lines
7.2 KiB

WL#3072 Maria Recovery - new program maria_read_log to display and apply log records found in a Maria log (see file's revision comment) - minor, misc fixes storage/maria/Makefile.am: new program maria_read_log storage/maria/ha_maria.cc: create control file if missing storage/maria/ma_blockrec.c: 0 -> LSN_IMPOSSIBLE; comments storage/maria/ma_checkpoint.h: preparations for Checkpoint module storage/maria/ma_close.c: comment storage/maria/ma_control_file.c: renaming constants. Possibility to say "open control file but don't create it if it's missing" (used by maria_read_log which does not want to create anything) storage/maria/ma_control_file.h: renaming constants storage/maria/ma_create.c: I had duplicated "linkname" and "linkname_ptr", now I see it's not needed, reverting. Indeed those variables don't contain interesting information; fixing log record accordingly (the links are in ci->data/index_file_name). Storing keystart in log record is needed, to know at which size we must extend the file if we replay LOGREC_CREATE_TABLE. storage/maria/ma_loghandler.c: some structures need to be known to maria_read_log.c, taking them to ma_loghandler.h storage/maria/ma_loghandler.h: we have page_store, adding page_korr. translog_lock() made public, because Checkpoint will need it (to write to control file). Some structures moved from ma_loghandler.c because maria_read_log.c needs them (needs to know the execute-in-REDO-phase hooks of each record). storage/maria/ma_loghandler_lsn.h: constants defined in ma_control_file.h serve everywhere, and they relate to LSNs, so putting them in ma_loghandler_lsn.h. Stronger constraints in LSN_VALID(). storage/maria/ma_pagecache.c: renaming constants storage/maria/ma_recovery.h: copyright storage/maria/ma_test1.c: new prototype storage/maria/ma_test2.c: new prototype storage/maria/trnman_public.h: double-inclusion safe storage/maria/unittest/ma_control_file-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multigroup-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multithread-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_pagecache-t.c: constants renamed, new prototype storage/myisam/mi_close.c: comment storage/maria/maria_read_log.c: program to read and print log records from a Maria transaction log, and optionally apply them to tables. Very basic, early version. Should serve as a base for Recovery's code. Designed to be idempotent. Create a log by running maria.test, then cd to var/master-data and run "maria_read_log --only-display" to see info about records; run "maria_read_log --display-and-apply" to also apply the records to tables (it's more interesting if you first wipe out the tables in var/master-data/test, to see how they get re-created). Only a few records are handled by now: LONG_TRANSACTION_ID, COMMIT, FILE_ID, REDO_CREATE_TABLE; place is ready for REDO_INSERT_ROW_HEAD where I could use Monty's help (search for "Monty" in the file). Note: changes to the index pages, index's header and bitmap pages are not properly logged yet, so don't expect the program to work with that.
19 years ago
WL#3072 - Maria recovery Unit test for recovery: runs ma_test1 and ma_test2 (both only with INSERTs and DELETEs; UPDATEs disabled as not handled by recovery) then moves the tables elswhere; recreates tables from the log, and compares and fails if there is a difference. Passes now. Most of maria_read_log.c moved to ma_recovery.c, as it will be re-used for recovery-from-ha_maria. Bugfixes of applying of REDO_INSERT, REDO_PURGE_ROW. Applying of REDO_PURGE_BLOCKS, REDO_DELETE_ALL, REDO_DROP_TABLE, UNDO_ROW_INSERT (in REDO phase only, i.e. just doing records++), UNDO_ROW_DELETE, UNDO_ROW_PURGE. Code cleanups. Monty: please look for "QQ". Sanja: please look for "Sanja". Future tasks: recovery of the bitmap (easy), recovery of the state (make it idempotent), more REDOs (Monty to work on REDO_UPDATE?), UNDO phase... Pushing this cset as it looks safe, contains test and bugfixes which will help Monty implement applying of REDO_UPDATE. sql/handler.cc: typo storage/maria/Makefile.am: Adding ma_test_recovery (which ma_test_all invokes, and which can also be run alone). Most of maria_read_log.c moved to ma_recovery.c storage/maria/ha_maria.cc: comments storage/maria/ma_bitmap.c: fixing comments. 2 -> sizeof(maria_bitmap_marker). Bitmap-related part of _ma_initialize_datafile() moves in bitmap module. Now putting the "bm" signature when creating the first bitmap page (it used to happen only at next open, but that caused an annoying difference when testing Recovery if the original run didn't open the table, and it looks more logical like this: it goes to disk only with its signature correct); see the "QQ" comment towards the _ma_initialize_data_file() call in ma_create.c for more). When reading a bitmap page, verify its signature (happens when normally using the table or when CHECKing it; not when REPAIRing it). storage/maria/ma_blockrec.c: * no need to sync the data file if table is not transactional * Comments, code cleanup (log-related data moved to log-related code block, int5store->page_store). * Store the table's short id into LOGREC_UNDO_ROW_PURGE, like we do for other records (though this record will soon be replaced with a CLR). * If "page" is 1 it means the page which extends from byte page*block_size+1 to (page+1)*block_size (byte number 1 being the first byte of the file). The last byte of the file is data_file_length (same convention). A new page needs to be created if the last byte of the page is beyond the last byte of the file, i.e. (page+1)*block_size+1 > data_file_length, so we correct the test (bug found when testing log applying for ma_test1 -M -T --skip-update). * update the page's LSN when removing a row from it during execution of a REDO_PURGE_ROW record (bug found when testing log applying for ma_test1 -M -T --skip-update). * applying of REDO_PURGE_BLOCKs (limited to a one-page range for now). storage/maria/ma_blockrec.h: new functions. maria_bitmap_marker does not need to be exported. storage/maria/ma_close.c: we can always flush the table's state when closing the last instance of the table. And it is needed for maria_read_log (as it does not use maria_lock_database()). storage/maria/ma_control_file.c: when in Recovery, some assertions should not be used. storage/maria/ma_control_file.h: double-inclusion safe storage/maria/ma_create.c: during recovery, don't log records. Comments. Moving the creation of the first bitmap page to ma_bitmap.c storage/maria/ma_delete_table.c: during recovery, don't log records. Log the end-zero of the dropped table's name, so that recovery can use the string in place without extending it to fit an end zero. storage/maria/ma_loghandler.c: * inwrite_rec_hook also needs access to the MARIA_SHARE, like prewrite_rec_hook. This will be needed to update share->records_diff (in the upcoming patch "recovery of the state"). * LOG_DESC::record_ends_group changed to an enum. * LOG_DESC for LOGREC_REDO_PURGE_BLOCKS and LOGREC_UNDO_ROW_PURGE corrected * Sanja please see the @todo LOG BUG * avoiding DBUG_RETURN(func()) as it gives confusing debug traces. storage/maria/ma_loghandler.h: - log write hooks called while the log's lock is held (inwrite_rec_hook) now need the MARIA_SHARE, like prewrite_rec_hook already had - instead of a bool saying if this record's type ends groups or not, we refine: it may not end a group, it may end a group, or it may be a group in itself. Imagine that we had a physical write failure to a table before we log the UNDO, we still end up in external_lock(F_UNLCK) and then we log a COMMIT: we don't want to consider this COMMIT as ending the group of REDOs (don't want to execute those REDOs during Recovery), that's why we say "COMMIT is a group in itself, it aborts any previous group". This also gives one more sanity check in maria_read_log. storage/maria/ma_recovery.c: New Recovery code, replacing the old pseudocode. Most of maria_read_log moved here. Call-able from ha_maria, but not enabled yet. Compared to the previous version of maria_read_log, some bugs have been fixed, debugging output can go to stdout or a disk file (for now it's useful for me, later it can be changed), execution of REDO_DROP_TABLE, REDO_DELETE_ALL, REDO_PURGE_BLOCKS has been added. Duplicate code has been factored into functions. We abort an unfinished group of records if we see a record which is a group in itself (like COMMIT). No need for maria_panic() after a bug (which caused tables to not be closed) was fixed; if there is yet another bug I prefer to see it. When opening a table for Recovery, set data_file_length and key_file_length to their real physical value (these are the easiest state members to restore :). Warn us if the last page was truncated (but Recovery handles it). MARIA_SHARE::state::state::records is now partly recovered (not idempotent, but works if recreating tables from scracth). When applying a REDO to a page, stamp it with the UNDO's LSN (current_group_end_lsn), not with the REDO's LSN; it makes the table more identical to the original table (easier to compare the two tables in the end). Big thing missing: some types of REDOs are not handled, and the UNDO phase does not exist (missing functions to execute UNDOs to actually rollback). So for now tests are only inserting/deleting a few 100 rows, closing the table and seeing if the log is applied ok; it works. UPDATE not handled. storage/maria/ma_recovery.h: new functions: ma_recover() for recovery from inside ha_maria; _ma_apply_log() for maria_read_log (ma_recover() calls _ma_apply_log()). Btw, we need to not use the word "recover" for REPAIR/maria_chk anymore. storage/maria/ma_rename.c: don't write log records during recovery storage/maria/ma_test2.c: - fail if maria_info() or other subtests find some wrong information - new option -g to skip updates. - init the translog before creating the table, so that log applying can work. - in "#if 0" you'll see some fixed bugs (will be removed). storage/maria/ma_test_all.sh: cleanup files. Test log applying. storage/maria/maria_read_log.c: most of the logic moves to ma_recovery.c to be shared between maria_read_log and recovery-from-inside-mysqld. See ma_recovery.c for additional changes made to the moved code. storage/maria/ma_test_recovery: unit test for Recovery. Tests insert and delete, REDO_UPDATE not yet coded. Script is called from ma_test_all. Can run standalone.
19 years ago
WL#3072 Maria Recovery - new program maria_read_log to display and apply log records found in a Maria log (see file's revision comment) - minor, misc fixes storage/maria/Makefile.am: new program maria_read_log storage/maria/ha_maria.cc: create control file if missing storage/maria/ma_blockrec.c: 0 -> LSN_IMPOSSIBLE; comments storage/maria/ma_checkpoint.h: preparations for Checkpoint module storage/maria/ma_close.c: comment storage/maria/ma_control_file.c: renaming constants. Possibility to say "open control file but don't create it if it's missing" (used by maria_read_log which does not want to create anything) storage/maria/ma_control_file.h: renaming constants storage/maria/ma_create.c: I had duplicated "linkname" and "linkname_ptr", now I see it's not needed, reverting. Indeed those variables don't contain interesting information; fixing log record accordingly (the links are in ci->data/index_file_name). Storing keystart in log record is needed, to know at which size we must extend the file if we replay LOGREC_CREATE_TABLE. storage/maria/ma_loghandler.c: some structures need to be known to maria_read_log.c, taking them to ma_loghandler.h storage/maria/ma_loghandler.h: we have page_store, adding page_korr. translog_lock() made public, because Checkpoint will need it (to write to control file). Some structures moved from ma_loghandler.c because maria_read_log.c needs them (needs to know the execute-in-REDO-phase hooks of each record). storage/maria/ma_loghandler_lsn.h: constants defined in ma_control_file.h serve everywhere, and they relate to LSNs, so putting them in ma_loghandler_lsn.h. Stronger constraints in LSN_VALID(). storage/maria/ma_pagecache.c: renaming constants storage/maria/ma_recovery.h: copyright storage/maria/ma_test1.c: new prototype storage/maria/ma_test2.c: new prototype storage/maria/trnman_public.h: double-inclusion safe storage/maria/unittest/ma_control_file-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multigroup-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multithread-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_pagecache-t.c: constants renamed, new prototype storage/myisam/mi_close.c: comment storage/maria/maria_read_log.c: program to read and print log records from a Maria transaction log, and optionally apply them to tables. Very basic, early version. Should serve as a base for Recovery's code. Designed to be idempotent. Create a log by running maria.test, then cd to var/master-data and run "maria_read_log --only-display" to see info about records; run "maria_read_log --display-and-apply" to also apply the records to tables (it's more interesting if you first wipe out the tables in var/master-data/test, to see how they get re-created). Only a few records are handled by now: LONG_TRANSACTION_ID, COMMIT, FILE_ID, REDO_CREATE_TABLE; place is ready for REDO_INSERT_ROW_HEAD where I could use Monty's help (search for "Monty" in the file). Note: changes to the index pages, index's header and bitmap pages are not properly logged yet, so don't expect the program to work with that.
19 years ago
WL#3072 Maria Recovery - new program maria_read_log to display and apply log records found in a Maria log (see file's revision comment) - minor, misc fixes storage/maria/Makefile.am: new program maria_read_log storage/maria/ha_maria.cc: create control file if missing storage/maria/ma_blockrec.c: 0 -> LSN_IMPOSSIBLE; comments storage/maria/ma_checkpoint.h: preparations for Checkpoint module storage/maria/ma_close.c: comment storage/maria/ma_control_file.c: renaming constants. Possibility to say "open control file but don't create it if it's missing" (used by maria_read_log which does not want to create anything) storage/maria/ma_control_file.h: renaming constants storage/maria/ma_create.c: I had duplicated "linkname" and "linkname_ptr", now I see it's not needed, reverting. Indeed those variables don't contain interesting information; fixing log record accordingly (the links are in ci->data/index_file_name). Storing keystart in log record is needed, to know at which size we must extend the file if we replay LOGREC_CREATE_TABLE. storage/maria/ma_loghandler.c: some structures need to be known to maria_read_log.c, taking them to ma_loghandler.h storage/maria/ma_loghandler.h: we have page_store, adding page_korr. translog_lock() made public, because Checkpoint will need it (to write to control file). Some structures moved from ma_loghandler.c because maria_read_log.c needs them (needs to know the execute-in-REDO-phase hooks of each record). storage/maria/ma_loghandler_lsn.h: constants defined in ma_control_file.h serve everywhere, and they relate to LSNs, so putting them in ma_loghandler_lsn.h. Stronger constraints in LSN_VALID(). storage/maria/ma_pagecache.c: renaming constants storage/maria/ma_recovery.h: copyright storage/maria/ma_test1.c: new prototype storage/maria/ma_test2.c: new prototype storage/maria/trnman_public.h: double-inclusion safe storage/maria/unittest/ma_control_file-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multigroup-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multithread-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_pagecache-t.c: constants renamed, new prototype storage/myisam/mi_close.c: comment storage/maria/maria_read_log.c: program to read and print log records from a Maria transaction log, and optionally apply them to tables. Very basic, early version. Should serve as a base for Recovery's code. Designed to be idempotent. Create a log by running maria.test, then cd to var/master-data and run "maria_read_log --only-display" to see info about records; run "maria_read_log --display-and-apply" to also apply the records to tables (it's more interesting if you first wipe out the tables in var/master-data/test, to see how they get re-created). Only a few records are handled by now: LONG_TRANSACTION_ID, COMMIT, FILE_ID, REDO_CREATE_TABLE; place is ready for REDO_INSERT_ROW_HEAD where I could use Monty's help (search for "Monty" in the file). Note: changes to the index pages, index's header and bitmap pages are not properly logged yet, so don't expect the program to work with that.
19 years ago
Remove SAFE_MODE for opt_range as it disables UPDATE to use keys REDO optimization (Bascily avoid moving blocks from/to pagecache) More command line arguments to maria_read_log Fixed recovery bug when recreating table sql/opt_range.cc: Remove SAFE_MODE for opt_range as it disables UPDATE to use keys storage/maria/ma_blockrec.c: REDO optimization Use new interface for pagecache_reads to avoid copying page buffers storage/maria/ma_loghandler.c: Patch from Sanja: - Added new parameter to translog_get_page to use direct links to pagecache - Changed scanner to be able to use direct links This avoids a lot of calls to bmove512() in page cache. storage/maria/ma_loghandler.h: Added direct link to pagecache objects storage/maria/ma_open.c: Added const to parameter Added missing braces storage/maria/ma_pagecache.c: From Sanja: - Added direct links to pagecache (from pagecache_read()) Dirrect link means that on pagecache_read we get back a pointer to the pagecache buffer From Monty: - Fixed arguments to init_page_cache to handle big page caches - Fixed compiler warnings - Replaced PAGECACHE_PAGE_LINK with PAGECACHE_BLOCK_LINK * to catch errors storage/maria/ma_pagecache.h: Changed block numbers from int to long to be able to handle big page caches Changed some PAGECACHE_PAGE_LINK to PAGECACHE_BLOCK_LINK storage/maria/ma_recovery.c: Fixed recovery bug when recreating table (table was kept open) Moved some variables to function start (portability) Added space to some print messages storage/maria/maria_chk.c: key_buffer_size -> page_buffer_size storage/maria/maria_def.h: Changed default page_buffer_size to 10M storage/maria/maria_read_log.c: Added more startup options: --version --undo (apply undo) --page_cache_size (to run with big cache sizes) --silent (to not get any output from --apply) storage/maria/unittest/ma_control_file-t.c: Fixed compiler warning storage/maria/unittest/ma_test_loghandler-t.c: Added new argument to translog_init_scanner() storage/maria/unittest/ma_test_loghandler_multigroup-t.c: Added new argument to translog_init_scanner() storage/maria/unittest/ma_test_loghandler_multithread-t.c: Added new argument to translog_init_scanner()
18 years ago
WL#3072 Maria recovery. * Thanks to Serg's tip, we fix here the compilation issue of REDO_REPAIR_TABLE's execution, by defining versions of _ma_killed_ptr() and _ma_check_print_info|warning|error() in maria_read_log.c (we move those of maria_chk.c into an include file and include it in maria_chk.c and maria_read_log.c). Execution of such record looks like working from my tests (it only happens in maria_read_log; recovery-from-mysqld skips DDLs and REPAIR is considered DDL here as it bypasses logging): tested ALTER TABLE ENABLE KEYS and then remove table, apply log: that did a repair. * Recent changes broke maria_read_log a bit: -a became default and -o caused error; fixing this. storage/maria/Makefile.am: addind new file storage/maria/ma_recovery.c: enable execution of REDO_REPAIR_TABLE by maria_read_log now that it compiles. Now reason to keep only T_QUICK from testflag. storage/maria/maria_chk.c: moving these functions to ma_check_standalone.h for reusability storage/maria/maria_def.h: comment storage/maria/maria_read_log.c: ma_check_standalone.h needs my_progname_short. Fixing bug where "maria_read_log" would default to -a and "maria_read_log -o" would throw an error. Implemented behaviour is: - no options: usage() - -a : applys, applys UNDOs by default unless --disable-undo - -o : only prints storage/maria/ma_check_standalone.h: All standalone programs which need to use functions from ma_check.c (like maria_repair()) must define their version of _ma_killed_ptr() and _ma_check_print_info|warning|error(). Indeed, linking with ma_check.o brings in the dependencies of ma_check.o which are definitions of the above functions; if the program does not define them then the ones of ha_maria.o are used i.e. ha_maria.o is linked into the program, and this brings dependencies of ha_maria.o on mysqld.o into the program's linking which thus fails, as the program is not linked with mysqld.o. We put in this file the functions which maria_chk.c uses, so that they can be reused by maria_read_log (when it replays REDO_REPAIR_TABLE) as they are good enough (they just print to stdout/stderr like maria_read_log already does).
18 years ago
WL#3072 Maria Recovery - new program maria_read_log to display and apply log records found in a Maria log (see file's revision comment) - minor, misc fixes storage/maria/Makefile.am: new program maria_read_log storage/maria/ha_maria.cc: create control file if missing storage/maria/ma_blockrec.c: 0 -> LSN_IMPOSSIBLE; comments storage/maria/ma_checkpoint.h: preparations for Checkpoint module storage/maria/ma_close.c: comment storage/maria/ma_control_file.c: renaming constants. Possibility to say "open control file but don't create it if it's missing" (used by maria_read_log which does not want to create anything) storage/maria/ma_control_file.h: renaming constants storage/maria/ma_create.c: I had duplicated "linkname" and "linkname_ptr", now I see it's not needed, reverting. Indeed those variables don't contain interesting information; fixing log record accordingly (the links are in ci->data/index_file_name). Storing keystart in log record is needed, to know at which size we must extend the file if we replay LOGREC_CREATE_TABLE. storage/maria/ma_loghandler.c: some structures need to be known to maria_read_log.c, taking them to ma_loghandler.h storage/maria/ma_loghandler.h: we have page_store, adding page_korr. translog_lock() made public, because Checkpoint will need it (to write to control file). Some structures moved from ma_loghandler.c because maria_read_log.c needs them (needs to know the execute-in-REDO-phase hooks of each record). storage/maria/ma_loghandler_lsn.h: constants defined in ma_control_file.h serve everywhere, and they relate to LSNs, so putting them in ma_loghandler_lsn.h. Stronger constraints in LSN_VALID(). storage/maria/ma_pagecache.c: renaming constants storage/maria/ma_recovery.h: copyright storage/maria/ma_test1.c: new prototype storage/maria/ma_test2.c: new prototype storage/maria/trnman_public.h: double-inclusion safe storage/maria/unittest/ma_control_file-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multigroup-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multithread-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_pagecache-t.c: constants renamed, new prototype storage/myisam/mi_close.c: comment storage/maria/maria_read_log.c: program to read and print log records from a Maria transaction log, and optionally apply them to tables. Very basic, early version. Should serve as a base for Recovery's code. Designed to be idempotent. Create a log by running maria.test, then cd to var/master-data and run "maria_read_log --only-display" to see info about records; run "maria_read_log --display-and-apply" to also apply the records to tables (it's more interesting if you first wipe out the tables in var/master-data/test, to see how they get re-created). Only a few records are handled by now: LONG_TRANSACTION_ID, COMMIT, FILE_ID, REDO_CREATE_TABLE; place is ready for REDO_INSERT_ROW_HEAD where I could use Monty's help (search for "Monty" in the file). Note: changes to the index pages, index's header and bitmap pages are not properly logged yet, so don't expect the program to work with that.
19 years ago
WL#3072 - Maria recovery maria_read_log used to always print a warning message at startup to say it is unsafe if ALTER TABLE was used. Now it prints it only if the log does show the problem (=ALTER TABLE or CREATE SELECT, which both disable logging of REDO_INSERT*). For that, when ha_maria::external_lock() disables transactionality it writes a LOGREC_INCOMPLETE_LOG to the log, which "maria_read_log -a" picks up to write a warning. REPAIR TABLE also disables those REDO_INSERT* but as maria_read_log executes LOGREC_REDO_REPAIR no warning is needed. storage/maria/ha_maria.cc: as we now log a record when disabling transactionility, we need the TRN to be set up first storage/maria/ma_blockrec.c: comment storage/maria/ma_loghandler.c: new type of log record storage/maria/ma_loghandler.h: new type of log record storage/maria/ma_recovery.c: * maria_apply_log() now returns a count of warnings. What currently produces warnings is: - skipping applying UNDOs though there are some (=> inconsistent table) - replaying log (in maria_read_log) though the log contains some ALTER TABLE or CREATE SELECT (log misses REDO_INSERT* for those and is so incomplete). Count of warnings affects the final message of maria_read_log and recovery (though in recovery none of the two conditions above should happen). * maria_read_log used to always print a warning message at startup to say it is unsafe if ALTER TABLE was used. Now it prints it only if the log does show the problem, i.e. ALTER TABLE or CREATE SELECT was used (both disable logging of REDO_INSERT* as those records are not needed for recovery; those missing records in turn make recreation-from-scratch, via maria_read_log, impossible). For that, when ha_maria::external_lock() disables transactionality, _ma_tmp_disable_logging_for_table() writes a LOGREC_INCOMPLETE_LOG to the log, which maria_apply_log() picks up to write a warning. storage/maria/ma_recovery.h: maria_apply_log() returns a count of warnings storage/maria/maria_def.h: _ma_tmp_disable_logging_for_table() grows so becomes a function storage/maria/maria_read_log.c: maria_apply_log can now return a count of warnings, to temper the "SUCCESS" message printed in the end by maria_read_log. Advise users to make a backup first.
18 years ago
WL#3072 Maria Recovery - new program maria_read_log to display and apply log records found in a Maria log (see file's revision comment) - minor, misc fixes storage/maria/Makefile.am: new program maria_read_log storage/maria/ha_maria.cc: create control file if missing storage/maria/ma_blockrec.c: 0 -> LSN_IMPOSSIBLE; comments storage/maria/ma_checkpoint.h: preparations for Checkpoint module storage/maria/ma_close.c: comment storage/maria/ma_control_file.c: renaming constants. Possibility to say "open control file but don't create it if it's missing" (used by maria_read_log which does not want to create anything) storage/maria/ma_control_file.h: renaming constants storage/maria/ma_create.c: I had duplicated "linkname" and "linkname_ptr", now I see it's not needed, reverting. Indeed those variables don't contain interesting information; fixing log record accordingly (the links are in ci->data/index_file_name). Storing keystart in log record is needed, to know at which size we must extend the file if we replay LOGREC_CREATE_TABLE. storage/maria/ma_loghandler.c: some structures need to be known to maria_read_log.c, taking them to ma_loghandler.h storage/maria/ma_loghandler.h: we have page_store, adding page_korr. translog_lock() made public, because Checkpoint will need it (to write to control file). Some structures moved from ma_loghandler.c because maria_read_log.c needs them (needs to know the execute-in-REDO-phase hooks of each record). storage/maria/ma_loghandler_lsn.h: constants defined in ma_control_file.h serve everywhere, and they relate to LSNs, so putting them in ma_loghandler_lsn.h. Stronger constraints in LSN_VALID(). storage/maria/ma_pagecache.c: renaming constants storage/maria/ma_recovery.h: copyright storage/maria/ma_test1.c: new prototype storage/maria/ma_test2.c: new prototype storage/maria/trnman_public.h: double-inclusion safe storage/maria/unittest/ma_control_file-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multigroup-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multithread-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_pagecache-t.c: constants renamed, new prototype storage/myisam/mi_close.c: comment storage/maria/maria_read_log.c: program to read and print log records from a Maria transaction log, and optionally apply them to tables. Very basic, early version. Should serve as a base for Recovery's code. Designed to be idempotent. Create a log by running maria.test, then cd to var/master-data and run "maria_read_log --only-display" to see info about records; run "maria_read_log --display-and-apply" to also apply the records to tables (it's more interesting if you first wipe out the tables in var/master-data/test, to see how they get re-created). Only a few records are handled by now: LONG_TRANSACTION_ID, COMMIT, FILE_ID, REDO_CREATE_TABLE; place is ready for REDO_INSERT_ROW_HEAD where I could use Monty's help (search for "Monty" in the file). Note: changes to the index pages, index's header and bitmap pages are not properly logged yet, so don't expect the program to work with that.
19 years ago
WL#3072 Maria recovery. * Thanks to Serg's tip, we fix here the compilation issue of REDO_REPAIR_TABLE's execution, by defining versions of _ma_killed_ptr() and _ma_check_print_info|warning|error() in maria_read_log.c (we move those of maria_chk.c into an include file and include it in maria_chk.c and maria_read_log.c). Execution of such record looks like working from my tests (it only happens in maria_read_log; recovery-from-mysqld skips DDLs and REPAIR is considered DDL here as it bypasses logging): tested ALTER TABLE ENABLE KEYS and then remove table, apply log: that did a repair. * Recent changes broke maria_read_log a bit: -a became default and -o caused error; fixing this. storage/maria/Makefile.am: addind new file storage/maria/ma_recovery.c: enable execution of REDO_REPAIR_TABLE by maria_read_log now that it compiles. Now reason to keep only T_QUICK from testflag. storage/maria/maria_chk.c: moving these functions to ma_check_standalone.h for reusability storage/maria/maria_def.h: comment storage/maria/maria_read_log.c: ma_check_standalone.h needs my_progname_short. Fixing bug where "maria_read_log" would default to -a and "maria_read_log -o" would throw an error. Implemented behaviour is: - no options: usage() - -a : applys, applys UNDOs by default unless --disable-undo - -o : only prints storage/maria/ma_check_standalone.h: All standalone programs which need to use functions from ma_check.c (like maria_repair()) must define their version of _ma_killed_ptr() and _ma_check_print_info|warning|error(). Indeed, linking with ma_check.o brings in the dependencies of ma_check.o which are definitions of the above functions; if the program does not define them then the ones of ha_maria.o are used i.e. ha_maria.o is linked into the program, and this brings dependencies of ha_maria.o on mysqld.o into the program's linking which thus fails, as the program is not linked with mysqld.o. We put in this file the functions which maria_chk.c uses, so that they can be reused by maria_read_log (when it replays REDO_REPAIR_TABLE) as they are good enough (they just print to stdout/stderr like maria_read_log already does).
18 years ago
WL#3072 Maria Recovery - new program maria_read_log to display and apply log records found in a Maria log (see file's revision comment) - minor, misc fixes storage/maria/Makefile.am: new program maria_read_log storage/maria/ha_maria.cc: create control file if missing storage/maria/ma_blockrec.c: 0 -> LSN_IMPOSSIBLE; comments storage/maria/ma_checkpoint.h: preparations for Checkpoint module storage/maria/ma_close.c: comment storage/maria/ma_control_file.c: renaming constants. Possibility to say "open control file but don't create it if it's missing" (used by maria_read_log which does not want to create anything) storage/maria/ma_control_file.h: renaming constants storage/maria/ma_create.c: I had duplicated "linkname" and "linkname_ptr", now I see it's not needed, reverting. Indeed those variables don't contain interesting information; fixing log record accordingly (the links are in ci->data/index_file_name). Storing keystart in log record is needed, to know at which size we must extend the file if we replay LOGREC_CREATE_TABLE. storage/maria/ma_loghandler.c: some structures need to be known to maria_read_log.c, taking them to ma_loghandler.h storage/maria/ma_loghandler.h: we have page_store, adding page_korr. translog_lock() made public, because Checkpoint will need it (to write to control file). Some structures moved from ma_loghandler.c because maria_read_log.c needs them (needs to know the execute-in-REDO-phase hooks of each record). storage/maria/ma_loghandler_lsn.h: constants defined in ma_control_file.h serve everywhere, and they relate to LSNs, so putting them in ma_loghandler_lsn.h. Stronger constraints in LSN_VALID(). storage/maria/ma_pagecache.c: renaming constants storage/maria/ma_recovery.h: copyright storage/maria/ma_test1.c: new prototype storage/maria/ma_test2.c: new prototype storage/maria/trnman_public.h: double-inclusion safe storage/maria/unittest/ma_control_file-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multigroup-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multithread-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_pagecache-t.c: constants renamed, new prototype storage/myisam/mi_close.c: comment storage/maria/maria_read_log.c: program to read and print log records from a Maria transaction log, and optionally apply them to tables. Very basic, early version. Should serve as a base for Recovery's code. Designed to be idempotent. Create a log by running maria.test, then cd to var/master-data and run "maria_read_log --only-display" to see info about records; run "maria_read_log --display-and-apply" to also apply the records to tables (it's more interesting if you first wipe out the tables in var/master-data/test, to see how they get re-created). Only a few records are handled by now: LONG_TRANSACTION_ID, COMMIT, FILE_ID, REDO_CREATE_TABLE; place is ready for REDO_INSERT_ROW_HEAD where I could use Monty's help (search for "Monty" in the file). Note: changes to the index pages, index's header and bitmap pages are not properly logged yet, so don't expect the program to work with that.
19 years ago
WL#3072 - Maria recovery Unit test for recovery: runs ma_test1 and ma_test2 (both only with INSERTs and DELETEs; UPDATEs disabled as not handled by recovery) then moves the tables elswhere; recreates tables from the log, and compares and fails if there is a difference. Passes now. Most of maria_read_log.c moved to ma_recovery.c, as it will be re-used for recovery-from-ha_maria. Bugfixes of applying of REDO_INSERT, REDO_PURGE_ROW. Applying of REDO_PURGE_BLOCKS, REDO_DELETE_ALL, REDO_DROP_TABLE, UNDO_ROW_INSERT (in REDO phase only, i.e. just doing records++), UNDO_ROW_DELETE, UNDO_ROW_PURGE. Code cleanups. Monty: please look for "QQ". Sanja: please look for "Sanja". Future tasks: recovery of the bitmap (easy), recovery of the state (make it idempotent), more REDOs (Monty to work on REDO_UPDATE?), UNDO phase... Pushing this cset as it looks safe, contains test and bugfixes which will help Monty implement applying of REDO_UPDATE. sql/handler.cc: typo storage/maria/Makefile.am: Adding ma_test_recovery (which ma_test_all invokes, and which can also be run alone). Most of maria_read_log.c moved to ma_recovery.c storage/maria/ha_maria.cc: comments storage/maria/ma_bitmap.c: fixing comments. 2 -> sizeof(maria_bitmap_marker). Bitmap-related part of _ma_initialize_datafile() moves in bitmap module. Now putting the "bm" signature when creating the first bitmap page (it used to happen only at next open, but that caused an annoying difference when testing Recovery if the original run didn't open the table, and it looks more logical like this: it goes to disk only with its signature correct); see the "QQ" comment towards the _ma_initialize_data_file() call in ma_create.c for more). When reading a bitmap page, verify its signature (happens when normally using the table or when CHECKing it; not when REPAIRing it). storage/maria/ma_blockrec.c: * no need to sync the data file if table is not transactional * Comments, code cleanup (log-related data moved to log-related code block, int5store->page_store). * Store the table's short id into LOGREC_UNDO_ROW_PURGE, like we do for other records (though this record will soon be replaced with a CLR). * If "page" is 1 it means the page which extends from byte page*block_size+1 to (page+1)*block_size (byte number 1 being the first byte of the file). The last byte of the file is data_file_length (same convention). A new page needs to be created if the last byte of the page is beyond the last byte of the file, i.e. (page+1)*block_size+1 > data_file_length, so we correct the test (bug found when testing log applying for ma_test1 -M -T --skip-update). * update the page's LSN when removing a row from it during execution of a REDO_PURGE_ROW record (bug found when testing log applying for ma_test1 -M -T --skip-update). * applying of REDO_PURGE_BLOCKs (limited to a one-page range for now). storage/maria/ma_blockrec.h: new functions. maria_bitmap_marker does not need to be exported. storage/maria/ma_close.c: we can always flush the table's state when closing the last instance of the table. And it is needed for maria_read_log (as it does not use maria_lock_database()). storage/maria/ma_control_file.c: when in Recovery, some assertions should not be used. storage/maria/ma_control_file.h: double-inclusion safe storage/maria/ma_create.c: during recovery, don't log records. Comments. Moving the creation of the first bitmap page to ma_bitmap.c storage/maria/ma_delete_table.c: during recovery, don't log records. Log the end-zero of the dropped table's name, so that recovery can use the string in place without extending it to fit an end zero. storage/maria/ma_loghandler.c: * inwrite_rec_hook also needs access to the MARIA_SHARE, like prewrite_rec_hook. This will be needed to update share->records_diff (in the upcoming patch "recovery of the state"). * LOG_DESC::record_ends_group changed to an enum. * LOG_DESC for LOGREC_REDO_PURGE_BLOCKS and LOGREC_UNDO_ROW_PURGE corrected * Sanja please see the @todo LOG BUG * avoiding DBUG_RETURN(func()) as it gives confusing debug traces. storage/maria/ma_loghandler.h: - log write hooks called while the log's lock is held (inwrite_rec_hook) now need the MARIA_SHARE, like prewrite_rec_hook already had - instead of a bool saying if this record's type ends groups or not, we refine: it may not end a group, it may end a group, or it may be a group in itself. Imagine that we had a physical write failure to a table before we log the UNDO, we still end up in external_lock(F_UNLCK) and then we log a COMMIT: we don't want to consider this COMMIT as ending the group of REDOs (don't want to execute those REDOs during Recovery), that's why we say "COMMIT is a group in itself, it aborts any previous group". This also gives one more sanity check in maria_read_log. storage/maria/ma_recovery.c: New Recovery code, replacing the old pseudocode. Most of maria_read_log moved here. Call-able from ha_maria, but not enabled yet. Compared to the previous version of maria_read_log, some bugs have been fixed, debugging output can go to stdout or a disk file (for now it's useful for me, later it can be changed), execution of REDO_DROP_TABLE, REDO_DELETE_ALL, REDO_PURGE_BLOCKS has been added. Duplicate code has been factored into functions. We abort an unfinished group of records if we see a record which is a group in itself (like COMMIT). No need for maria_panic() after a bug (which caused tables to not be closed) was fixed; if there is yet another bug I prefer to see it. When opening a table for Recovery, set data_file_length and key_file_length to their real physical value (these are the easiest state members to restore :). Warn us if the last page was truncated (but Recovery handles it). MARIA_SHARE::state::state::records is now partly recovered (not idempotent, but works if recreating tables from scracth). When applying a REDO to a page, stamp it with the UNDO's LSN (current_group_end_lsn), not with the REDO's LSN; it makes the table more identical to the original table (easier to compare the two tables in the end). Big thing missing: some types of REDOs are not handled, and the UNDO phase does not exist (missing functions to execute UNDOs to actually rollback). So for now tests are only inserting/deleting a few 100 rows, closing the table and seeing if the log is applied ok; it works. UPDATE not handled. storage/maria/ma_recovery.h: new functions: ma_recover() for recovery from inside ha_maria; _ma_apply_log() for maria_read_log (ma_recover() calls _ma_apply_log()). Btw, we need to not use the word "recover" for REPAIR/maria_chk anymore. storage/maria/ma_rename.c: don't write log records during recovery storage/maria/ma_test2.c: - fail if maria_info() or other subtests find some wrong information - new option -g to skip updates. - init the translog before creating the table, so that log applying can work. - in "#if 0" you'll see some fixed bugs (will be removed). storage/maria/ma_test_all.sh: cleanup files. Test log applying. storage/maria/maria_read_log.c: most of the logic moves to ma_recovery.c to be shared between maria_read_log and recovery-from-inside-mysqld. See ma_recovery.c for additional changes made to the moved code. storage/maria/ma_test_recovery: unit test for Recovery. Tests insert and delete, REDO_UPDATE not yet coded. Script is called from ma_test_all. Can run standalone.
19 years ago
WL#3072 Maria Recovery - new program maria_read_log to display and apply log records found in a Maria log (see file's revision comment) - minor, misc fixes storage/maria/Makefile.am: new program maria_read_log storage/maria/ha_maria.cc: create control file if missing storage/maria/ma_blockrec.c: 0 -> LSN_IMPOSSIBLE; comments storage/maria/ma_checkpoint.h: preparations for Checkpoint module storage/maria/ma_close.c: comment storage/maria/ma_control_file.c: renaming constants. Possibility to say "open control file but don't create it if it's missing" (used by maria_read_log which does not want to create anything) storage/maria/ma_control_file.h: renaming constants storage/maria/ma_create.c: I had duplicated "linkname" and "linkname_ptr", now I see it's not needed, reverting. Indeed those variables don't contain interesting information; fixing log record accordingly (the links are in ci->data/index_file_name). Storing keystart in log record is needed, to know at which size we must extend the file if we replay LOGREC_CREATE_TABLE. storage/maria/ma_loghandler.c: some structures need to be known to maria_read_log.c, taking them to ma_loghandler.h storage/maria/ma_loghandler.h: we have page_store, adding page_korr. translog_lock() made public, because Checkpoint will need it (to write to control file). Some structures moved from ma_loghandler.c because maria_read_log.c needs them (needs to know the execute-in-REDO-phase hooks of each record). storage/maria/ma_loghandler_lsn.h: constants defined in ma_control_file.h serve everywhere, and they relate to LSNs, so putting them in ma_loghandler_lsn.h. Stronger constraints in LSN_VALID(). storage/maria/ma_pagecache.c: renaming constants storage/maria/ma_recovery.h: copyright storage/maria/ma_test1.c: new prototype storage/maria/ma_test2.c: new prototype storage/maria/trnman_public.h: double-inclusion safe storage/maria/unittest/ma_control_file-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multigroup-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multithread-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_pagecache-t.c: constants renamed, new prototype storage/myisam/mi_close.c: comment storage/maria/maria_read_log.c: program to read and print log records from a Maria transaction log, and optionally apply them to tables. Very basic, early version. Should serve as a base for Recovery's code. Designed to be idempotent. Create a log by running maria.test, then cd to var/master-data and run "maria_read_log --only-display" to see info about records; run "maria_read_log --display-and-apply" to also apply the records to tables (it's more interesting if you first wipe out the tables in var/master-data/test, to see how they get re-created). Only a few records are handled by now: LONG_TRANSACTION_ID, COMMIT, FILE_ID, REDO_CREATE_TABLE; place is ready for REDO_INSERT_ROW_HEAD where I could use Monty's help (search for "Monty" in the file). Note: changes to the index pages, index's header and bitmap pages are not properly logged yet, so don't expect the program to work with that.
19 years ago
WL#3072 Maria recovery * create page cache before initializing engine and not after, because Maria's recovery needs a page cache * make the creation of a bitmap page more crash-resistent * bugfix (see ma_blockrec.c) * back to old way: create an 8k bitmap page when creating table * preparations for the UNDO phase: recreate TRNs * preparations for Checkpoint: list of dirty pages, testing of rec_lsn to know if page should be skipped during Recovery (unused in this patch as no Checkpoint module pushed yet) * maria_chk tags repaired table with a special LSN * reworking all around in ma_recovery.c (less duplication) mysys/my_realloc.c: noted an issue in my_realloc() sql/mysqld.cc: page cache needs to be created before engines are initialized, because Maria's initialization may do a recovery which needs the page cache. storage/maria/ha_maria.cc: update to new prototype storage/maria/ma_bitmap.c: when creating the first bitmap page we used chsize to 8192 bytes then pwrite (overwrite) the last 2 bytes (8191-8192). If crash between the two operations, this leaves a bitmap page full without its end marker. A later recovery may try to read this page and find it exists and misses a marker and conclude it's corrupted and fail. Changing the chsize to only 8190 bytes: recovery will then find the page is too short and recreate it entirely. storage/maria/ma_blockrec.c: Fix for a bug: when executing a REDO, if the data page is created, data_file_length was increased before _ma_bitmap_set(): _ma_bitmap_set() called _ma_read_bitmap_page() which, due to the increased data_file_length, expected to find a bitmap page on disk with a correct end marker; if the bitmap page didn't exist already in fact, this failed. Fixed by increasing data_file_length only after _ma_read_bitmap_page() has created the new bitmap page correctly. This bug could happen every time a REDO is about creating a new bitmap page. storage/maria/ma_check.c: empty data file has a bitmap page storage/maria/ma_control_file.c: useless parameter to ma_control_file_create_or_open(), just test if this is recovery. storage/maria/ma_control_file.h: new prototype storage/maria/ma_create.c: Back to how it was before: maria_create() creates an 8k bitmap page. Thus (bugfix) data_file_length needs to reflect this instead of being 0. storage/maria/ma_loghandler.c: as ma_test1 and ma_test2 now use real transactions and not dummy_transaction_object, REDO for INSERT/UPDATE/DELETE are always about real transactions, can assert this. A function for Recovery to assign a short id to a table. storage/maria/ma_loghandler.h: new function storage/maria/ma_loghandler_lsn.h: maria_chk tags repaired tables with this LSN storage/maria/ma_open.c: * enforce that DMLs on transactional tables use real transactions and not dummy_transaction_object. * test if table was repaired with maria_chk (which has to been seen as an import of an external table into the server), test validity of create_rename_lsn (header corruption detection) * comments. storage/maria/ma_recovery.c: * preparations for the UNDO phase: recreate TRNs * preparations for Checkpoint: list of dirty pages, testing of rec_lsn to know if page should be skipped during Recovery (unused in this patch as no Checkpoint module pushed yet) * reworking all around (less duplication) storage/maria/ma_recovery.h: a parameter to say if the UNDO phase should be skipped storage/maria/maria_chk.c: tag repaired tables with a special LSN storage/maria/maria_read_log.c: * update to new prototype * no UNDO phase in maria_read_log for now storage/maria/trnman.c: * a function for Recovery to create a transaction (TRN), needed in the UNDO phase * a function for Recovery to grab an existing transaction, needed in the UNDO phase (rollback all existing transactions) storage/maria/trnman_public.h: new functions
19 years ago
WL#3072 Maria Recovery - new program maria_read_log to display and apply log records found in a Maria log (see file's revision comment) - minor, misc fixes storage/maria/Makefile.am: new program maria_read_log storage/maria/ha_maria.cc: create control file if missing storage/maria/ma_blockrec.c: 0 -> LSN_IMPOSSIBLE; comments storage/maria/ma_checkpoint.h: preparations for Checkpoint module storage/maria/ma_close.c: comment storage/maria/ma_control_file.c: renaming constants. Possibility to say "open control file but don't create it if it's missing" (used by maria_read_log which does not want to create anything) storage/maria/ma_control_file.h: renaming constants storage/maria/ma_create.c: I had duplicated "linkname" and "linkname_ptr", now I see it's not needed, reverting. Indeed those variables don't contain interesting information; fixing log record accordingly (the links are in ci->data/index_file_name). Storing keystart in log record is needed, to know at which size we must extend the file if we replay LOGREC_CREATE_TABLE. storage/maria/ma_loghandler.c: some structures need to be known to maria_read_log.c, taking them to ma_loghandler.h storage/maria/ma_loghandler.h: we have page_store, adding page_korr. translog_lock() made public, because Checkpoint will need it (to write to control file). Some structures moved from ma_loghandler.c because maria_read_log.c needs them (needs to know the execute-in-REDO-phase hooks of each record). storage/maria/ma_loghandler_lsn.h: constants defined in ma_control_file.h serve everywhere, and they relate to LSNs, so putting them in ma_loghandler_lsn.h. Stronger constraints in LSN_VALID(). storage/maria/ma_pagecache.c: renaming constants storage/maria/ma_recovery.h: copyright storage/maria/ma_test1.c: new prototype storage/maria/ma_test2.c: new prototype storage/maria/trnman_public.h: double-inclusion safe storage/maria/unittest/ma_control_file-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multigroup-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multithread-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_pagecache-t.c: constants renamed, new prototype storage/myisam/mi_close.c: comment storage/maria/maria_read_log.c: program to read and print log records from a Maria transaction log, and optionally apply them to tables. Very basic, early version. Should serve as a base for Recovery's code. Designed to be idempotent. Create a log by running maria.test, then cd to var/master-data and run "maria_read_log --only-display" to see info about records; run "maria_read_log --display-and-apply" to also apply the records to tables (it's more interesting if you first wipe out the tables in var/master-data/test, to see how they get re-created). Only a few records are handled by now: LONG_TRANSACTION_ID, COMMIT, FILE_ID, REDO_CREATE_TABLE; place is ready for REDO_INSERT_ROW_HEAD where I could use Monty's help (search for "Monty" in the file). Note: changes to the index pages, index's header and bitmap pages are not properly logged yet, so don't expect the program to work with that.
19 years ago
WL#3072 - Maria recovery Unit test for recovery: runs ma_test1 and ma_test2 (both only with INSERTs and DELETEs; UPDATEs disabled as not handled by recovery) then moves the tables elswhere; recreates tables from the log, and compares and fails if there is a difference. Passes now. Most of maria_read_log.c moved to ma_recovery.c, as it will be re-used for recovery-from-ha_maria. Bugfixes of applying of REDO_INSERT, REDO_PURGE_ROW. Applying of REDO_PURGE_BLOCKS, REDO_DELETE_ALL, REDO_DROP_TABLE, UNDO_ROW_INSERT (in REDO phase only, i.e. just doing records++), UNDO_ROW_DELETE, UNDO_ROW_PURGE. Code cleanups. Monty: please look for "QQ". Sanja: please look for "Sanja". Future tasks: recovery of the bitmap (easy), recovery of the state (make it idempotent), more REDOs (Monty to work on REDO_UPDATE?), UNDO phase... Pushing this cset as it looks safe, contains test and bugfixes which will help Monty implement applying of REDO_UPDATE. sql/handler.cc: typo storage/maria/Makefile.am: Adding ma_test_recovery (which ma_test_all invokes, and which can also be run alone). Most of maria_read_log.c moved to ma_recovery.c storage/maria/ha_maria.cc: comments storage/maria/ma_bitmap.c: fixing comments. 2 -> sizeof(maria_bitmap_marker). Bitmap-related part of _ma_initialize_datafile() moves in bitmap module. Now putting the "bm" signature when creating the first bitmap page (it used to happen only at next open, but that caused an annoying difference when testing Recovery if the original run didn't open the table, and it looks more logical like this: it goes to disk only with its signature correct); see the "QQ" comment towards the _ma_initialize_data_file() call in ma_create.c for more). When reading a bitmap page, verify its signature (happens when normally using the table or when CHECKing it; not when REPAIRing it). storage/maria/ma_blockrec.c: * no need to sync the data file if table is not transactional * Comments, code cleanup (log-related data moved to log-related code block, int5store->page_store). * Store the table's short id into LOGREC_UNDO_ROW_PURGE, like we do for other records (though this record will soon be replaced with a CLR). * If "page" is 1 it means the page which extends from byte page*block_size+1 to (page+1)*block_size (byte number 1 being the first byte of the file). The last byte of the file is data_file_length (same convention). A new page needs to be created if the last byte of the page is beyond the last byte of the file, i.e. (page+1)*block_size+1 > data_file_length, so we correct the test (bug found when testing log applying for ma_test1 -M -T --skip-update). * update the page's LSN when removing a row from it during execution of a REDO_PURGE_ROW record (bug found when testing log applying for ma_test1 -M -T --skip-update). * applying of REDO_PURGE_BLOCKs (limited to a one-page range for now). storage/maria/ma_blockrec.h: new functions. maria_bitmap_marker does not need to be exported. storage/maria/ma_close.c: we can always flush the table's state when closing the last instance of the table. And it is needed for maria_read_log (as it does not use maria_lock_database()). storage/maria/ma_control_file.c: when in Recovery, some assertions should not be used. storage/maria/ma_control_file.h: double-inclusion safe storage/maria/ma_create.c: during recovery, don't log records. Comments. Moving the creation of the first bitmap page to ma_bitmap.c storage/maria/ma_delete_table.c: during recovery, don't log records. Log the end-zero of the dropped table's name, so that recovery can use the string in place without extending it to fit an end zero. storage/maria/ma_loghandler.c: * inwrite_rec_hook also needs access to the MARIA_SHARE, like prewrite_rec_hook. This will be needed to update share->records_diff (in the upcoming patch "recovery of the state"). * LOG_DESC::record_ends_group changed to an enum. * LOG_DESC for LOGREC_REDO_PURGE_BLOCKS and LOGREC_UNDO_ROW_PURGE corrected * Sanja please see the @todo LOG BUG * avoiding DBUG_RETURN(func()) as it gives confusing debug traces. storage/maria/ma_loghandler.h: - log write hooks called while the log's lock is held (inwrite_rec_hook) now need the MARIA_SHARE, like prewrite_rec_hook already had - instead of a bool saying if this record's type ends groups or not, we refine: it may not end a group, it may end a group, or it may be a group in itself. Imagine that we had a physical write failure to a table before we log the UNDO, we still end up in external_lock(F_UNLCK) and then we log a COMMIT: we don't want to consider this COMMIT as ending the group of REDOs (don't want to execute those REDOs during Recovery), that's why we say "COMMIT is a group in itself, it aborts any previous group". This also gives one more sanity check in maria_read_log. storage/maria/ma_recovery.c: New Recovery code, replacing the old pseudocode. Most of maria_read_log moved here. Call-able from ha_maria, but not enabled yet. Compared to the previous version of maria_read_log, some bugs have been fixed, debugging output can go to stdout or a disk file (for now it's useful for me, later it can be changed), execution of REDO_DROP_TABLE, REDO_DELETE_ALL, REDO_PURGE_BLOCKS has been added. Duplicate code has been factored into functions. We abort an unfinished group of records if we see a record which is a group in itself (like COMMIT). No need for maria_panic() after a bug (which caused tables to not be closed) was fixed; if there is yet another bug I prefer to see it. When opening a table for Recovery, set data_file_length and key_file_length to their real physical value (these are the easiest state members to restore :). Warn us if the last page was truncated (but Recovery handles it). MARIA_SHARE::state::state::records is now partly recovered (not idempotent, but works if recreating tables from scracth). When applying a REDO to a page, stamp it with the UNDO's LSN (current_group_end_lsn), not with the REDO's LSN; it makes the table more identical to the original table (easier to compare the two tables in the end). Big thing missing: some types of REDOs are not handled, and the UNDO phase does not exist (missing functions to execute UNDOs to actually rollback). So for now tests are only inserting/deleting a few 100 rows, closing the table and seeing if the log is applied ok; it works. UPDATE not handled. storage/maria/ma_recovery.h: new functions: ma_recover() for recovery from inside ha_maria; _ma_apply_log() for maria_read_log (ma_recover() calls _ma_apply_log()). Btw, we need to not use the word "recover" for REPAIR/maria_chk anymore. storage/maria/ma_rename.c: don't write log records during recovery storage/maria/ma_test2.c: - fail if maria_info() or other subtests find some wrong information - new option -g to skip updates. - init the translog before creating the table, so that log applying can work. - in "#if 0" you'll see some fixed bugs (will be removed). storage/maria/ma_test_all.sh: cleanup files. Test log applying. storage/maria/maria_read_log.c: most of the logic moves to ma_recovery.c to be shared between maria_read_log and recovery-from-inside-mysqld. See ma_recovery.c for additional changes made to the moved code. storage/maria/ma_test_recovery: unit test for Recovery. Tests insert and delete, REDO_UPDATE not yet coded. Script is called from ma_test_all. Can run standalone.
19 years ago
Remove SAFE_MODE for opt_range as it disables UPDATE to use keys REDO optimization (Bascily avoid moving blocks from/to pagecache) More command line arguments to maria_read_log Fixed recovery bug when recreating table sql/opt_range.cc: Remove SAFE_MODE for opt_range as it disables UPDATE to use keys storage/maria/ma_blockrec.c: REDO optimization Use new interface for pagecache_reads to avoid copying page buffers storage/maria/ma_loghandler.c: Patch from Sanja: - Added new parameter to translog_get_page to use direct links to pagecache - Changed scanner to be able to use direct links This avoids a lot of calls to bmove512() in page cache. storage/maria/ma_loghandler.h: Added direct link to pagecache objects storage/maria/ma_open.c: Added const to parameter Added missing braces storage/maria/ma_pagecache.c: From Sanja: - Added direct links to pagecache (from pagecache_read()) Dirrect link means that on pagecache_read we get back a pointer to the pagecache buffer From Monty: - Fixed arguments to init_page_cache to handle big page caches - Fixed compiler warnings - Replaced PAGECACHE_PAGE_LINK with PAGECACHE_BLOCK_LINK * to catch errors storage/maria/ma_pagecache.h: Changed block numbers from int to long to be able to handle big page caches Changed some PAGECACHE_PAGE_LINK to PAGECACHE_BLOCK_LINK storage/maria/ma_recovery.c: Fixed recovery bug when recreating table (table was kept open) Moved some variables to function start (portability) Added space to some print messages storage/maria/maria_chk.c: key_buffer_size -> page_buffer_size storage/maria/maria_def.h: Changed default page_buffer_size to 10M storage/maria/maria_read_log.c: Added more startup options: --version --undo (apply undo) --page_cache_size (to run with big cache sizes) --silent (to not get any output from --apply) storage/maria/unittest/ma_control_file-t.c: Fixed compiler warning storage/maria/unittest/ma_test_loghandler-t.c: Added new argument to translog_init_scanner() storage/maria/unittest/ma_test_loghandler_multigroup-t.c: Added new argument to translog_init_scanner() storage/maria/unittest/ma_test_loghandler_multithread-t.c: Added new argument to translog_init_scanner()
18 years ago
WL#3072 Maria Recovery - new program maria_read_log to display and apply log records found in a Maria log (see file's revision comment) - minor, misc fixes storage/maria/Makefile.am: new program maria_read_log storage/maria/ha_maria.cc: create control file if missing storage/maria/ma_blockrec.c: 0 -> LSN_IMPOSSIBLE; comments storage/maria/ma_checkpoint.h: preparations for Checkpoint module storage/maria/ma_close.c: comment storage/maria/ma_control_file.c: renaming constants. Possibility to say "open control file but don't create it if it's missing" (used by maria_read_log which does not want to create anything) storage/maria/ma_control_file.h: renaming constants storage/maria/ma_create.c: I had duplicated "linkname" and "linkname_ptr", now I see it's not needed, reverting. Indeed those variables don't contain interesting information; fixing log record accordingly (the links are in ci->data/index_file_name). Storing keystart in log record is needed, to know at which size we must extend the file if we replay LOGREC_CREATE_TABLE. storage/maria/ma_loghandler.c: some structures need to be known to maria_read_log.c, taking them to ma_loghandler.h storage/maria/ma_loghandler.h: we have page_store, adding page_korr. translog_lock() made public, because Checkpoint will need it (to write to control file). Some structures moved from ma_loghandler.c because maria_read_log.c needs them (needs to know the execute-in-REDO-phase hooks of each record). storage/maria/ma_loghandler_lsn.h: constants defined in ma_control_file.h serve everywhere, and they relate to LSNs, so putting them in ma_loghandler_lsn.h. Stronger constraints in LSN_VALID(). storage/maria/ma_pagecache.c: renaming constants storage/maria/ma_recovery.h: copyright storage/maria/ma_test1.c: new prototype storage/maria/ma_test2.c: new prototype storage/maria/trnman_public.h: double-inclusion safe storage/maria/unittest/ma_control_file-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multigroup-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multithread-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_pagecache-t.c: constants renamed, new prototype storage/myisam/mi_close.c: comment storage/maria/maria_read_log.c: program to read and print log records from a Maria transaction log, and optionally apply them to tables. Very basic, early version. Should serve as a base for Recovery's code. Designed to be idempotent. Create a log by running maria.test, then cd to var/master-data and run "maria_read_log --only-display" to see info about records; run "maria_read_log --display-and-apply" to also apply the records to tables (it's more interesting if you first wipe out the tables in var/master-data/test, to see how they get re-created). Only a few records are handled by now: LONG_TRANSACTION_ID, COMMIT, FILE_ID, REDO_CREATE_TABLE; place is ready for REDO_INSERT_ROW_HEAD where I could use Monty's help (search for "Monty" in the file). Note: changes to the index pages, index's header and bitmap pages are not properly logged yet, so don't expect the program to work with that.
19 years ago
WL#3072 Maria Recovery - new program maria_read_log to display and apply log records found in a Maria log (see file's revision comment) - minor, misc fixes storage/maria/Makefile.am: new program maria_read_log storage/maria/ha_maria.cc: create control file if missing storage/maria/ma_blockrec.c: 0 -> LSN_IMPOSSIBLE; comments storage/maria/ma_checkpoint.h: preparations for Checkpoint module storage/maria/ma_close.c: comment storage/maria/ma_control_file.c: renaming constants. Possibility to say "open control file but don't create it if it's missing" (used by maria_read_log which does not want to create anything) storage/maria/ma_control_file.h: renaming constants storage/maria/ma_create.c: I had duplicated "linkname" and "linkname_ptr", now I see it's not needed, reverting. Indeed those variables don't contain interesting information; fixing log record accordingly (the links are in ci->data/index_file_name). Storing keystart in log record is needed, to know at which size we must extend the file if we replay LOGREC_CREATE_TABLE. storage/maria/ma_loghandler.c: some structures need to be known to maria_read_log.c, taking them to ma_loghandler.h storage/maria/ma_loghandler.h: we have page_store, adding page_korr. translog_lock() made public, because Checkpoint will need it (to write to control file). Some structures moved from ma_loghandler.c because maria_read_log.c needs them (needs to know the execute-in-REDO-phase hooks of each record). storage/maria/ma_loghandler_lsn.h: constants defined in ma_control_file.h serve everywhere, and they relate to LSNs, so putting them in ma_loghandler_lsn.h. Stronger constraints in LSN_VALID(). storage/maria/ma_pagecache.c: renaming constants storage/maria/ma_recovery.h: copyright storage/maria/ma_test1.c: new prototype storage/maria/ma_test2.c: new prototype storage/maria/trnman_public.h: double-inclusion safe storage/maria/unittest/ma_control_file-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multigroup-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multithread-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_pagecache-t.c: constants renamed, new prototype storage/myisam/mi_close.c: comment storage/maria/maria_read_log.c: program to read and print log records from a Maria transaction log, and optionally apply them to tables. Very basic, early version. Should serve as a base for Recovery's code. Designed to be idempotent. Create a log by running maria.test, then cd to var/master-data and run "maria_read_log --only-display" to see info about records; run "maria_read_log --display-and-apply" to also apply the records to tables (it's more interesting if you first wipe out the tables in var/master-data/test, to see how they get re-created). Only a few records are handled by now: LONG_TRANSACTION_ID, COMMIT, FILE_ID, REDO_CREATE_TABLE; place is ready for REDO_INSERT_ROW_HEAD where I could use Monty's help (search for "Monty" in the file). Note: changes to the index pages, index's header and bitmap pages are not properly logged yet, so don't expect the program to work with that.
19 years ago
WL#3072 Maria Recovery - new program maria_read_log to display and apply log records found in a Maria log (see file's revision comment) - minor, misc fixes storage/maria/Makefile.am: new program maria_read_log storage/maria/ha_maria.cc: create control file if missing storage/maria/ma_blockrec.c: 0 -> LSN_IMPOSSIBLE; comments storage/maria/ma_checkpoint.h: preparations for Checkpoint module storage/maria/ma_close.c: comment storage/maria/ma_control_file.c: renaming constants. Possibility to say "open control file but don't create it if it's missing" (used by maria_read_log which does not want to create anything) storage/maria/ma_control_file.h: renaming constants storage/maria/ma_create.c: I had duplicated "linkname" and "linkname_ptr", now I see it's not needed, reverting. Indeed those variables don't contain interesting information; fixing log record accordingly (the links are in ci->data/index_file_name). Storing keystart in log record is needed, to know at which size we must extend the file if we replay LOGREC_CREATE_TABLE. storage/maria/ma_loghandler.c: some structures need to be known to maria_read_log.c, taking them to ma_loghandler.h storage/maria/ma_loghandler.h: we have page_store, adding page_korr. translog_lock() made public, because Checkpoint will need it (to write to control file). Some structures moved from ma_loghandler.c because maria_read_log.c needs them (needs to know the execute-in-REDO-phase hooks of each record). storage/maria/ma_loghandler_lsn.h: constants defined in ma_control_file.h serve everywhere, and they relate to LSNs, so putting them in ma_loghandler_lsn.h. Stronger constraints in LSN_VALID(). storage/maria/ma_pagecache.c: renaming constants storage/maria/ma_recovery.h: copyright storage/maria/ma_test1.c: new prototype storage/maria/ma_test2.c: new prototype storage/maria/trnman_public.h: double-inclusion safe storage/maria/unittest/ma_control_file-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multigroup-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multithread-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_pagecache-t.c: constants renamed, new prototype storage/myisam/mi_close.c: comment storage/maria/maria_read_log.c: program to read and print log records from a Maria transaction log, and optionally apply them to tables. Very basic, early version. Should serve as a base for Recovery's code. Designed to be idempotent. Create a log by running maria.test, then cd to var/master-data and run "maria_read_log --only-display" to see info about records; run "maria_read_log --display-and-apply" to also apply the records to tables (it's more interesting if you first wipe out the tables in var/master-data/test, to see how they get re-created). Only a few records are handled by now: LONG_TRANSACTION_ID, COMMIT, FILE_ID, REDO_CREATE_TABLE; place is ready for REDO_INSERT_ROW_HEAD where I could use Monty's help (search for "Monty" in the file). Note: changes to the index pages, index's header and bitmap pages are not properly logged yet, so don't expect the program to work with that.
19 years ago
WL#3072 - Maria recovery Unit test for recovery: runs ma_test1 and ma_test2 (both only with INSERTs and DELETEs; UPDATEs disabled as not handled by recovery) then moves the tables elswhere; recreates tables from the log, and compares and fails if there is a difference. Passes now. Most of maria_read_log.c moved to ma_recovery.c, as it will be re-used for recovery-from-ha_maria. Bugfixes of applying of REDO_INSERT, REDO_PURGE_ROW. Applying of REDO_PURGE_BLOCKS, REDO_DELETE_ALL, REDO_DROP_TABLE, UNDO_ROW_INSERT (in REDO phase only, i.e. just doing records++), UNDO_ROW_DELETE, UNDO_ROW_PURGE. Code cleanups. Monty: please look for "QQ". Sanja: please look for "Sanja". Future tasks: recovery of the bitmap (easy), recovery of the state (make it idempotent), more REDOs (Monty to work on REDO_UPDATE?), UNDO phase... Pushing this cset as it looks safe, contains test and bugfixes which will help Monty implement applying of REDO_UPDATE. sql/handler.cc: typo storage/maria/Makefile.am: Adding ma_test_recovery (which ma_test_all invokes, and which can also be run alone). Most of maria_read_log.c moved to ma_recovery.c storage/maria/ha_maria.cc: comments storage/maria/ma_bitmap.c: fixing comments. 2 -> sizeof(maria_bitmap_marker). Bitmap-related part of _ma_initialize_datafile() moves in bitmap module. Now putting the "bm" signature when creating the first bitmap page (it used to happen only at next open, but that caused an annoying difference when testing Recovery if the original run didn't open the table, and it looks more logical like this: it goes to disk only with its signature correct); see the "QQ" comment towards the _ma_initialize_data_file() call in ma_create.c for more). When reading a bitmap page, verify its signature (happens when normally using the table or when CHECKing it; not when REPAIRing it). storage/maria/ma_blockrec.c: * no need to sync the data file if table is not transactional * Comments, code cleanup (log-related data moved to log-related code block, int5store->page_store). * Store the table's short id into LOGREC_UNDO_ROW_PURGE, like we do for other records (though this record will soon be replaced with a CLR). * If "page" is 1 it means the page which extends from byte page*block_size+1 to (page+1)*block_size (byte number 1 being the first byte of the file). The last byte of the file is data_file_length (same convention). A new page needs to be created if the last byte of the page is beyond the last byte of the file, i.e. (page+1)*block_size+1 > data_file_length, so we correct the test (bug found when testing log applying for ma_test1 -M -T --skip-update). * update the page's LSN when removing a row from it during execution of a REDO_PURGE_ROW record (bug found when testing log applying for ma_test1 -M -T --skip-update). * applying of REDO_PURGE_BLOCKs (limited to a one-page range for now). storage/maria/ma_blockrec.h: new functions. maria_bitmap_marker does not need to be exported. storage/maria/ma_close.c: we can always flush the table's state when closing the last instance of the table. And it is needed for maria_read_log (as it does not use maria_lock_database()). storage/maria/ma_control_file.c: when in Recovery, some assertions should not be used. storage/maria/ma_control_file.h: double-inclusion safe storage/maria/ma_create.c: during recovery, don't log records. Comments. Moving the creation of the first bitmap page to ma_bitmap.c storage/maria/ma_delete_table.c: during recovery, don't log records. Log the end-zero of the dropped table's name, so that recovery can use the string in place without extending it to fit an end zero. storage/maria/ma_loghandler.c: * inwrite_rec_hook also needs access to the MARIA_SHARE, like prewrite_rec_hook. This will be needed to update share->records_diff (in the upcoming patch "recovery of the state"). * LOG_DESC::record_ends_group changed to an enum. * LOG_DESC for LOGREC_REDO_PURGE_BLOCKS and LOGREC_UNDO_ROW_PURGE corrected * Sanja please see the @todo LOG BUG * avoiding DBUG_RETURN(func()) as it gives confusing debug traces. storage/maria/ma_loghandler.h: - log write hooks called while the log's lock is held (inwrite_rec_hook) now need the MARIA_SHARE, like prewrite_rec_hook already had - instead of a bool saying if this record's type ends groups or not, we refine: it may not end a group, it may end a group, or it may be a group in itself. Imagine that we had a physical write failure to a table before we log the UNDO, we still end up in external_lock(F_UNLCK) and then we log a COMMIT: we don't want to consider this COMMIT as ending the group of REDOs (don't want to execute those REDOs during Recovery), that's why we say "COMMIT is a group in itself, it aborts any previous group". This also gives one more sanity check in maria_read_log. storage/maria/ma_recovery.c: New Recovery code, replacing the old pseudocode. Most of maria_read_log moved here. Call-able from ha_maria, but not enabled yet. Compared to the previous version of maria_read_log, some bugs have been fixed, debugging output can go to stdout or a disk file (for now it's useful for me, later it can be changed), execution of REDO_DROP_TABLE, REDO_DELETE_ALL, REDO_PURGE_BLOCKS has been added. Duplicate code has been factored into functions. We abort an unfinished group of records if we see a record which is a group in itself (like COMMIT). No need for maria_panic() after a bug (which caused tables to not be closed) was fixed; if there is yet another bug I prefer to see it. When opening a table for Recovery, set data_file_length and key_file_length to their real physical value (these are the easiest state members to restore :). Warn us if the last page was truncated (but Recovery handles it). MARIA_SHARE::state::state::records is now partly recovered (not idempotent, but works if recreating tables from scracth). When applying a REDO to a page, stamp it with the UNDO's LSN (current_group_end_lsn), not with the REDO's LSN; it makes the table more identical to the original table (easier to compare the two tables in the end). Big thing missing: some types of REDOs are not handled, and the UNDO phase does not exist (missing functions to execute UNDOs to actually rollback). So for now tests are only inserting/deleting a few 100 rows, closing the table and seeing if the log is applied ok; it works. UPDATE not handled. storage/maria/ma_recovery.h: new functions: ma_recover() for recovery from inside ha_maria; _ma_apply_log() for maria_read_log (ma_recover() calls _ma_apply_log()). Btw, we need to not use the word "recover" for REPAIR/maria_chk anymore. storage/maria/ma_rename.c: don't write log records during recovery storage/maria/ma_test2.c: - fail if maria_info() or other subtests find some wrong information - new option -g to skip updates. - init the translog before creating the table, so that log applying can work. - in "#if 0" you'll see some fixed bugs (will be removed). storage/maria/ma_test_all.sh: cleanup files. Test log applying. storage/maria/maria_read_log.c: most of the logic moves to ma_recovery.c to be shared between maria_read_log and recovery-from-inside-mysqld. See ma_recovery.c for additional changes made to the moved code. storage/maria/ma_test_recovery: unit test for Recovery. Tests insert and delete, REDO_UPDATE not yet coded. Script is called from ma_test_all. Can run standalone.
19 years ago
WL#3072 - Maria recovery maria_read_log used to always print a warning message at startup to say it is unsafe if ALTER TABLE was used. Now it prints it only if the log does show the problem (=ALTER TABLE or CREATE SELECT, which both disable logging of REDO_INSERT*). For that, when ha_maria::external_lock() disables transactionality it writes a LOGREC_INCOMPLETE_LOG to the log, which "maria_read_log -a" picks up to write a warning. REPAIR TABLE also disables those REDO_INSERT* but as maria_read_log executes LOGREC_REDO_REPAIR no warning is needed. storage/maria/ha_maria.cc: as we now log a record when disabling transactionility, we need the TRN to be set up first storage/maria/ma_blockrec.c: comment storage/maria/ma_loghandler.c: new type of log record storage/maria/ma_loghandler.h: new type of log record storage/maria/ma_recovery.c: * maria_apply_log() now returns a count of warnings. What currently produces warnings is: - skipping applying UNDOs though there are some (=> inconsistent table) - replaying log (in maria_read_log) though the log contains some ALTER TABLE or CREATE SELECT (log misses REDO_INSERT* for those and is so incomplete). Count of warnings affects the final message of maria_read_log and recovery (though in recovery none of the two conditions above should happen). * maria_read_log used to always print a warning message at startup to say it is unsafe if ALTER TABLE was used. Now it prints it only if the log does show the problem, i.e. ALTER TABLE or CREATE SELECT was used (both disable logging of REDO_INSERT* as those records are not needed for recovery; those missing records in turn make recreation-from-scratch, via maria_read_log, impossible). For that, when ha_maria::external_lock() disables transactionality, _ma_tmp_disable_logging_for_table() writes a LOGREC_INCOMPLETE_LOG to the log, which maria_apply_log() picks up to write a warning. storage/maria/ma_recovery.h: maria_apply_log() returns a count of warnings storage/maria/maria_def.h: _ma_tmp_disable_logging_for_table() grows so becomes a function storage/maria/maria_read_log.c: maria_apply_log can now return a count of warnings, to temper the "SUCCESS" message printed in the end by maria_read_log. Advise users to make a backup first.
18 years ago
WL#3072 Maria Recovery - new program maria_read_log to display and apply log records found in a Maria log (see file's revision comment) - minor, misc fixes storage/maria/Makefile.am: new program maria_read_log storage/maria/ha_maria.cc: create control file if missing storage/maria/ma_blockrec.c: 0 -> LSN_IMPOSSIBLE; comments storage/maria/ma_checkpoint.h: preparations for Checkpoint module storage/maria/ma_close.c: comment storage/maria/ma_control_file.c: renaming constants. Possibility to say "open control file but don't create it if it's missing" (used by maria_read_log which does not want to create anything) storage/maria/ma_control_file.h: renaming constants storage/maria/ma_create.c: I had duplicated "linkname" and "linkname_ptr", now I see it's not needed, reverting. Indeed those variables don't contain interesting information; fixing log record accordingly (the links are in ci->data/index_file_name). Storing keystart in log record is needed, to know at which size we must extend the file if we replay LOGREC_CREATE_TABLE. storage/maria/ma_loghandler.c: some structures need to be known to maria_read_log.c, taking them to ma_loghandler.h storage/maria/ma_loghandler.h: we have page_store, adding page_korr. translog_lock() made public, because Checkpoint will need it (to write to control file). Some structures moved from ma_loghandler.c because maria_read_log.c needs them (needs to know the execute-in-REDO-phase hooks of each record). storage/maria/ma_loghandler_lsn.h: constants defined in ma_control_file.h serve everywhere, and they relate to LSNs, so putting them in ma_loghandler_lsn.h. Stronger constraints in LSN_VALID(). storage/maria/ma_pagecache.c: renaming constants storage/maria/ma_recovery.h: copyright storage/maria/ma_test1.c: new prototype storage/maria/ma_test2.c: new prototype storage/maria/trnman_public.h: double-inclusion safe storage/maria/unittest/ma_control_file-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multigroup-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multithread-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_pagecache-t.c: constants renamed, new prototype storage/myisam/mi_close.c: comment storage/maria/maria_read_log.c: program to read and print log records from a Maria transaction log, and optionally apply them to tables. Very basic, early version. Should serve as a base for Recovery's code. Designed to be idempotent. Create a log by running maria.test, then cd to var/master-data and run "maria_read_log --only-display" to see info about records; run "maria_read_log --display-and-apply" to also apply the records to tables (it's more interesting if you first wipe out the tables in var/master-data/test, to see how they get re-created). Only a few records are handled by now: LONG_TRANSACTION_ID, COMMIT, FILE_ID, REDO_CREATE_TABLE; place is ready for REDO_INSERT_ROW_HEAD where I could use Monty's help (search for "Monty" in the file). Note: changes to the index pages, index's header and bitmap pages are not properly logged yet, so don't expect the program to work with that.
19 years ago
WL#3072 - Maria recovery maria_read_log used to always print a warning message at startup to say it is unsafe if ALTER TABLE was used. Now it prints it only if the log does show the problem (=ALTER TABLE or CREATE SELECT, which both disable logging of REDO_INSERT*). For that, when ha_maria::external_lock() disables transactionality it writes a LOGREC_INCOMPLETE_LOG to the log, which "maria_read_log -a" picks up to write a warning. REPAIR TABLE also disables those REDO_INSERT* but as maria_read_log executes LOGREC_REDO_REPAIR no warning is needed. storage/maria/ha_maria.cc: as we now log a record when disabling transactionility, we need the TRN to be set up first storage/maria/ma_blockrec.c: comment storage/maria/ma_loghandler.c: new type of log record storage/maria/ma_loghandler.h: new type of log record storage/maria/ma_recovery.c: * maria_apply_log() now returns a count of warnings. What currently produces warnings is: - skipping applying UNDOs though there are some (=> inconsistent table) - replaying log (in maria_read_log) though the log contains some ALTER TABLE or CREATE SELECT (log misses REDO_INSERT* for those and is so incomplete). Count of warnings affects the final message of maria_read_log and recovery (though in recovery none of the two conditions above should happen). * maria_read_log used to always print a warning message at startup to say it is unsafe if ALTER TABLE was used. Now it prints it only if the log does show the problem, i.e. ALTER TABLE or CREATE SELECT was used (both disable logging of REDO_INSERT* as those records are not needed for recovery; those missing records in turn make recreation-from-scratch, via maria_read_log, impossible). For that, when ha_maria::external_lock() disables transactionality, _ma_tmp_disable_logging_for_table() writes a LOGREC_INCOMPLETE_LOG to the log, which maria_apply_log() picks up to write a warning. storage/maria/ma_recovery.h: maria_apply_log() returns a count of warnings storage/maria/maria_def.h: _ma_tmp_disable_logging_for_table() grows so becomes a function storage/maria/maria_read_log.c: maria_apply_log can now return a count of warnings, to temper the "SUCCESS" message printed in the end by maria_read_log. Advise users to make a backup first.
18 years ago
WL#3072 Maria Recovery - new program maria_read_log to display and apply log records found in a Maria log (see file's revision comment) - minor, misc fixes storage/maria/Makefile.am: new program maria_read_log storage/maria/ha_maria.cc: create control file if missing storage/maria/ma_blockrec.c: 0 -> LSN_IMPOSSIBLE; comments storage/maria/ma_checkpoint.h: preparations for Checkpoint module storage/maria/ma_close.c: comment storage/maria/ma_control_file.c: renaming constants. Possibility to say "open control file but don't create it if it's missing" (used by maria_read_log which does not want to create anything) storage/maria/ma_control_file.h: renaming constants storage/maria/ma_create.c: I had duplicated "linkname" and "linkname_ptr", now I see it's not needed, reverting. Indeed those variables don't contain interesting information; fixing log record accordingly (the links are in ci->data/index_file_name). Storing keystart in log record is needed, to know at which size we must extend the file if we replay LOGREC_CREATE_TABLE. storage/maria/ma_loghandler.c: some structures need to be known to maria_read_log.c, taking them to ma_loghandler.h storage/maria/ma_loghandler.h: we have page_store, adding page_korr. translog_lock() made public, because Checkpoint will need it (to write to control file). Some structures moved from ma_loghandler.c because maria_read_log.c needs them (needs to know the execute-in-REDO-phase hooks of each record). storage/maria/ma_loghandler_lsn.h: constants defined in ma_control_file.h serve everywhere, and they relate to LSNs, so putting them in ma_loghandler_lsn.h. Stronger constraints in LSN_VALID(). storage/maria/ma_pagecache.c: renaming constants storage/maria/ma_recovery.h: copyright storage/maria/ma_test1.c: new prototype storage/maria/ma_test2.c: new prototype storage/maria/trnman_public.h: double-inclusion safe storage/maria/unittest/ma_control_file-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multigroup-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multithread-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_pagecache-t.c: constants renamed, new prototype storage/myisam/mi_close.c: comment storage/maria/maria_read_log.c: program to read and print log records from a Maria transaction log, and optionally apply them to tables. Very basic, early version. Should serve as a base for Recovery's code. Designed to be idempotent. Create a log by running maria.test, then cd to var/master-data and run "maria_read_log --only-display" to see info about records; run "maria_read_log --display-and-apply" to also apply the records to tables (it's more interesting if you first wipe out the tables in var/master-data/test, to see how they get re-created). Only a few records are handled by now: LONG_TRANSACTION_ID, COMMIT, FILE_ID, REDO_CREATE_TABLE; place is ready for REDO_INSERT_ROW_HEAD where I could use Monty's help (search for "Monty" in the file). Note: changes to the index pages, index's header and bitmap pages are not properly logged yet, so don't expect the program to work with that.
19 years ago
WL#3072 Maria recovery. * Thanks to Serg's tip, we fix here the compilation issue of REDO_REPAIR_TABLE's execution, by defining versions of _ma_killed_ptr() and _ma_check_print_info|warning|error() in maria_read_log.c (we move those of maria_chk.c into an include file and include it in maria_chk.c and maria_read_log.c). Execution of such record looks like working from my tests (it only happens in maria_read_log; recovery-from-mysqld skips DDLs and REPAIR is considered DDL here as it bypasses logging): tested ALTER TABLE ENABLE KEYS and then remove table, apply log: that did a repair. * Recent changes broke maria_read_log a bit: -a became default and -o caused error; fixing this. storage/maria/Makefile.am: addind new file storage/maria/ma_recovery.c: enable execution of REDO_REPAIR_TABLE by maria_read_log now that it compiles. Now reason to keep only T_QUICK from testflag. storage/maria/maria_chk.c: moving these functions to ma_check_standalone.h for reusability storage/maria/maria_def.h: comment storage/maria/maria_read_log.c: ma_check_standalone.h needs my_progname_short. Fixing bug where "maria_read_log" would default to -a and "maria_read_log -o" would throw an error. Implemented behaviour is: - no options: usage() - -a : applys, applys UNDOs by default unless --disable-undo - -o : only prints storage/maria/ma_check_standalone.h: All standalone programs which need to use functions from ma_check.c (like maria_repair()) must define their version of _ma_killed_ptr() and _ma_check_print_info|warning|error(). Indeed, linking with ma_check.o brings in the dependencies of ma_check.o which are definitions of the above functions; if the program does not define them then the ones of ha_maria.o are used i.e. ha_maria.o is linked into the program, and this brings dependencies of ha_maria.o on mysqld.o into the program's linking which thus fails, as the program is not linked with mysqld.o. We put in this file the functions which maria_chk.c uses, so that they can be reused by maria_read_log (when it replays REDO_REPAIR_TABLE) as they are good enough (they just print to stdout/stderr like maria_read_log already does).
18 years ago
WL#3072 Maria Recovery - new program maria_read_log to display and apply log records found in a Maria log (see file's revision comment) - minor, misc fixes storage/maria/Makefile.am: new program maria_read_log storage/maria/ha_maria.cc: create control file if missing storage/maria/ma_blockrec.c: 0 -> LSN_IMPOSSIBLE; comments storage/maria/ma_checkpoint.h: preparations for Checkpoint module storage/maria/ma_close.c: comment storage/maria/ma_control_file.c: renaming constants. Possibility to say "open control file but don't create it if it's missing" (used by maria_read_log which does not want to create anything) storage/maria/ma_control_file.h: renaming constants storage/maria/ma_create.c: I had duplicated "linkname" and "linkname_ptr", now I see it's not needed, reverting. Indeed those variables don't contain interesting information; fixing log record accordingly (the links are in ci->data/index_file_name). Storing keystart in log record is needed, to know at which size we must extend the file if we replay LOGREC_CREATE_TABLE. storage/maria/ma_loghandler.c: some structures need to be known to maria_read_log.c, taking them to ma_loghandler.h storage/maria/ma_loghandler.h: we have page_store, adding page_korr. translog_lock() made public, because Checkpoint will need it (to write to control file). Some structures moved from ma_loghandler.c because maria_read_log.c needs them (needs to know the execute-in-REDO-phase hooks of each record). storage/maria/ma_loghandler_lsn.h: constants defined in ma_control_file.h serve everywhere, and they relate to LSNs, so putting them in ma_loghandler_lsn.h. Stronger constraints in LSN_VALID(). storage/maria/ma_pagecache.c: renaming constants storage/maria/ma_recovery.h: copyright storage/maria/ma_test1.c: new prototype storage/maria/ma_test2.c: new prototype storage/maria/trnman_public.h: double-inclusion safe storage/maria/unittest/ma_control_file-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multigroup-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multithread-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_pagecache-t.c: constants renamed, new prototype storage/myisam/mi_close.c: comment storage/maria/maria_read_log.c: program to read and print log records from a Maria transaction log, and optionally apply them to tables. Very basic, early version. Should serve as a base for Recovery's code. Designed to be idempotent. Create a log by running maria.test, then cd to var/master-data and run "maria_read_log --only-display" to see info about records; run "maria_read_log --display-and-apply" to also apply the records to tables (it's more interesting if you first wipe out the tables in var/master-data/test, to see how they get re-created). Only a few records are handled by now: LONG_TRANSACTION_ID, COMMIT, FILE_ID, REDO_CREATE_TABLE; place is ready for REDO_INSERT_ROW_HEAD where I could use Monty's help (search for "Monty" in the file). Note: changes to the index pages, index's header and bitmap pages are not properly logged yet, so don't expect the program to work with that.
19 years ago
WL#3072 - Maria recovery Unit test for recovery: runs ma_test1 and ma_test2 (both only with INSERTs and DELETEs; UPDATEs disabled as not handled by recovery) then moves the tables elswhere; recreates tables from the log, and compares and fails if there is a difference. Passes now. Most of maria_read_log.c moved to ma_recovery.c, as it will be re-used for recovery-from-ha_maria. Bugfixes of applying of REDO_INSERT, REDO_PURGE_ROW. Applying of REDO_PURGE_BLOCKS, REDO_DELETE_ALL, REDO_DROP_TABLE, UNDO_ROW_INSERT (in REDO phase only, i.e. just doing records++), UNDO_ROW_DELETE, UNDO_ROW_PURGE. Code cleanups. Monty: please look for "QQ". Sanja: please look for "Sanja". Future tasks: recovery of the bitmap (easy), recovery of the state (make it idempotent), more REDOs (Monty to work on REDO_UPDATE?), UNDO phase... Pushing this cset as it looks safe, contains test and bugfixes which will help Monty implement applying of REDO_UPDATE. sql/handler.cc: typo storage/maria/Makefile.am: Adding ma_test_recovery (which ma_test_all invokes, and which can also be run alone). Most of maria_read_log.c moved to ma_recovery.c storage/maria/ha_maria.cc: comments storage/maria/ma_bitmap.c: fixing comments. 2 -> sizeof(maria_bitmap_marker). Bitmap-related part of _ma_initialize_datafile() moves in bitmap module. Now putting the "bm" signature when creating the first bitmap page (it used to happen only at next open, but that caused an annoying difference when testing Recovery if the original run didn't open the table, and it looks more logical like this: it goes to disk only with its signature correct); see the "QQ" comment towards the _ma_initialize_data_file() call in ma_create.c for more). When reading a bitmap page, verify its signature (happens when normally using the table or when CHECKing it; not when REPAIRing it). storage/maria/ma_blockrec.c: * no need to sync the data file if table is not transactional * Comments, code cleanup (log-related data moved to log-related code block, int5store->page_store). * Store the table's short id into LOGREC_UNDO_ROW_PURGE, like we do for other records (though this record will soon be replaced with a CLR). * If "page" is 1 it means the page which extends from byte page*block_size+1 to (page+1)*block_size (byte number 1 being the first byte of the file). The last byte of the file is data_file_length (same convention). A new page needs to be created if the last byte of the page is beyond the last byte of the file, i.e. (page+1)*block_size+1 > data_file_length, so we correct the test (bug found when testing log applying for ma_test1 -M -T --skip-update). * update the page's LSN when removing a row from it during execution of a REDO_PURGE_ROW record (bug found when testing log applying for ma_test1 -M -T --skip-update). * applying of REDO_PURGE_BLOCKs (limited to a one-page range for now). storage/maria/ma_blockrec.h: new functions. maria_bitmap_marker does not need to be exported. storage/maria/ma_close.c: we can always flush the table's state when closing the last instance of the table. And it is needed for maria_read_log (as it does not use maria_lock_database()). storage/maria/ma_control_file.c: when in Recovery, some assertions should not be used. storage/maria/ma_control_file.h: double-inclusion safe storage/maria/ma_create.c: during recovery, don't log records. Comments. Moving the creation of the first bitmap page to ma_bitmap.c storage/maria/ma_delete_table.c: during recovery, don't log records. Log the end-zero of the dropped table's name, so that recovery can use the string in place without extending it to fit an end zero. storage/maria/ma_loghandler.c: * inwrite_rec_hook also needs access to the MARIA_SHARE, like prewrite_rec_hook. This will be needed to update share->records_diff (in the upcoming patch "recovery of the state"). * LOG_DESC::record_ends_group changed to an enum. * LOG_DESC for LOGREC_REDO_PURGE_BLOCKS and LOGREC_UNDO_ROW_PURGE corrected * Sanja please see the @todo LOG BUG * avoiding DBUG_RETURN(func()) as it gives confusing debug traces. storage/maria/ma_loghandler.h: - log write hooks called while the log's lock is held (inwrite_rec_hook) now need the MARIA_SHARE, like prewrite_rec_hook already had - instead of a bool saying if this record's type ends groups or not, we refine: it may not end a group, it may end a group, or it may be a group in itself. Imagine that we had a physical write failure to a table before we log the UNDO, we still end up in external_lock(F_UNLCK) and then we log a COMMIT: we don't want to consider this COMMIT as ending the group of REDOs (don't want to execute those REDOs during Recovery), that's why we say "COMMIT is a group in itself, it aborts any previous group". This also gives one more sanity check in maria_read_log. storage/maria/ma_recovery.c: New Recovery code, replacing the old pseudocode. Most of maria_read_log moved here. Call-able from ha_maria, but not enabled yet. Compared to the previous version of maria_read_log, some bugs have been fixed, debugging output can go to stdout or a disk file (for now it's useful for me, later it can be changed), execution of REDO_DROP_TABLE, REDO_DELETE_ALL, REDO_PURGE_BLOCKS has been added. Duplicate code has been factored into functions. We abort an unfinished group of records if we see a record which is a group in itself (like COMMIT). No need for maria_panic() after a bug (which caused tables to not be closed) was fixed; if there is yet another bug I prefer to see it. When opening a table for Recovery, set data_file_length and key_file_length to their real physical value (these are the easiest state members to restore :). Warn us if the last page was truncated (but Recovery handles it). MARIA_SHARE::state::state::records is now partly recovered (not idempotent, but works if recreating tables from scracth). When applying a REDO to a page, stamp it with the UNDO's LSN (current_group_end_lsn), not with the REDO's LSN; it makes the table more identical to the original table (easier to compare the two tables in the end). Big thing missing: some types of REDOs are not handled, and the UNDO phase does not exist (missing functions to execute UNDOs to actually rollback). So for now tests are only inserting/deleting a few 100 rows, closing the table and seeing if the log is applied ok; it works. UPDATE not handled. storage/maria/ma_recovery.h: new functions: ma_recover() for recovery from inside ha_maria; _ma_apply_log() for maria_read_log (ma_recover() calls _ma_apply_log()). Btw, we need to not use the word "recover" for REPAIR/maria_chk anymore. storage/maria/ma_rename.c: don't write log records during recovery storage/maria/ma_test2.c: - fail if maria_info() or other subtests find some wrong information - new option -g to skip updates. - init the translog before creating the table, so that log applying can work. - in "#if 0" you'll see some fixed bugs (will be removed). storage/maria/ma_test_all.sh: cleanup files. Test log applying. storage/maria/maria_read_log.c: most of the logic moves to ma_recovery.c to be shared between maria_read_log and recovery-from-inside-mysqld. See ma_recovery.c for additional changes made to the moved code. storage/maria/ma_test_recovery: unit test for Recovery. Tests insert and delete, REDO_UPDATE not yet coded. Script is called from ma_test_all. Can run standalone.
19 years ago
WL#3072 Maria Recovery - new program maria_read_log to display and apply log records found in a Maria log (see file's revision comment) - minor, misc fixes storage/maria/Makefile.am: new program maria_read_log storage/maria/ha_maria.cc: create control file if missing storage/maria/ma_blockrec.c: 0 -> LSN_IMPOSSIBLE; comments storage/maria/ma_checkpoint.h: preparations for Checkpoint module storage/maria/ma_close.c: comment storage/maria/ma_control_file.c: renaming constants. Possibility to say "open control file but don't create it if it's missing" (used by maria_read_log which does not want to create anything) storage/maria/ma_control_file.h: renaming constants storage/maria/ma_create.c: I had duplicated "linkname" and "linkname_ptr", now I see it's not needed, reverting. Indeed those variables don't contain interesting information; fixing log record accordingly (the links are in ci->data/index_file_name). Storing keystart in log record is needed, to know at which size we must extend the file if we replay LOGREC_CREATE_TABLE. storage/maria/ma_loghandler.c: some structures need to be known to maria_read_log.c, taking them to ma_loghandler.h storage/maria/ma_loghandler.h: we have page_store, adding page_korr. translog_lock() made public, because Checkpoint will need it (to write to control file). Some structures moved from ma_loghandler.c because maria_read_log.c needs them (needs to know the execute-in-REDO-phase hooks of each record). storage/maria/ma_loghandler_lsn.h: constants defined in ma_control_file.h serve everywhere, and they relate to LSNs, so putting them in ma_loghandler_lsn.h. Stronger constraints in LSN_VALID(). storage/maria/ma_pagecache.c: renaming constants storage/maria/ma_recovery.h: copyright storage/maria/ma_test1.c: new prototype storage/maria/ma_test2.c: new prototype storage/maria/trnman_public.h: double-inclusion safe storage/maria/unittest/ma_control_file-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multigroup-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multithread-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_pagecache-t.c: constants renamed, new prototype storage/myisam/mi_close.c: comment storage/maria/maria_read_log.c: program to read and print log records from a Maria transaction log, and optionally apply them to tables. Very basic, early version. Should serve as a base for Recovery's code. Designed to be idempotent. Create a log by running maria.test, then cd to var/master-data and run "maria_read_log --only-display" to see info about records; run "maria_read_log --display-and-apply" to also apply the records to tables (it's more interesting if you first wipe out the tables in var/master-data/test, to see how they get re-created). Only a few records are handled by now: LONG_TRANSACTION_ID, COMMIT, FILE_ID, REDO_CREATE_TABLE; place is ready for REDO_INSERT_ROW_HEAD where I could use Monty's help (search for "Monty" in the file). Note: changes to the index pages, index's header and bitmap pages are not properly logged yet, so don't expect the program to work with that.
19 years ago
WL#3072 Maria recovery. * Thanks to Serg's tip, we fix here the compilation issue of REDO_REPAIR_TABLE's execution, by defining versions of _ma_killed_ptr() and _ma_check_print_info|warning|error() in maria_read_log.c (we move those of maria_chk.c into an include file and include it in maria_chk.c and maria_read_log.c). Execution of such record looks like working from my tests (it only happens in maria_read_log; recovery-from-mysqld skips DDLs and REPAIR is considered DDL here as it bypasses logging): tested ALTER TABLE ENABLE KEYS and then remove table, apply log: that did a repair. * Recent changes broke maria_read_log a bit: -a became default and -o caused error; fixing this. storage/maria/Makefile.am: addind new file storage/maria/ma_recovery.c: enable execution of REDO_REPAIR_TABLE by maria_read_log now that it compiles. Now reason to keep only T_QUICK from testflag. storage/maria/maria_chk.c: moving these functions to ma_check_standalone.h for reusability storage/maria/maria_def.h: comment storage/maria/maria_read_log.c: ma_check_standalone.h needs my_progname_short. Fixing bug where "maria_read_log" would default to -a and "maria_read_log -o" would throw an error. Implemented behaviour is: - no options: usage() - -a : applys, applys UNDOs by default unless --disable-undo - -o : only prints storage/maria/ma_check_standalone.h: All standalone programs which need to use functions from ma_check.c (like maria_repair()) must define their version of _ma_killed_ptr() and _ma_check_print_info|warning|error(). Indeed, linking with ma_check.o brings in the dependencies of ma_check.o which are definitions of the above functions; if the program does not define them then the ones of ha_maria.o are used i.e. ha_maria.o is linked into the program, and this brings dependencies of ha_maria.o on mysqld.o into the program's linking which thus fails, as the program is not linked with mysqld.o. We put in this file the functions which maria_chk.c uses, so that they can be reused by maria_read_log (when it replays REDO_REPAIR_TABLE) as they are good enough (they just print to stdout/stderr like maria_read_log already does).
18 years ago
WL#3072 Maria Recovery - new program maria_read_log to display and apply log records found in a Maria log (see file's revision comment) - minor, misc fixes storage/maria/Makefile.am: new program maria_read_log storage/maria/ha_maria.cc: create control file if missing storage/maria/ma_blockrec.c: 0 -> LSN_IMPOSSIBLE; comments storage/maria/ma_checkpoint.h: preparations for Checkpoint module storage/maria/ma_close.c: comment storage/maria/ma_control_file.c: renaming constants. Possibility to say "open control file but don't create it if it's missing" (used by maria_read_log which does not want to create anything) storage/maria/ma_control_file.h: renaming constants storage/maria/ma_create.c: I had duplicated "linkname" and "linkname_ptr", now I see it's not needed, reverting. Indeed those variables don't contain interesting information; fixing log record accordingly (the links are in ci->data/index_file_name). Storing keystart in log record is needed, to know at which size we must extend the file if we replay LOGREC_CREATE_TABLE. storage/maria/ma_loghandler.c: some structures need to be known to maria_read_log.c, taking them to ma_loghandler.h storage/maria/ma_loghandler.h: we have page_store, adding page_korr. translog_lock() made public, because Checkpoint will need it (to write to control file). Some structures moved from ma_loghandler.c because maria_read_log.c needs them (needs to know the execute-in-REDO-phase hooks of each record). storage/maria/ma_loghandler_lsn.h: constants defined in ma_control_file.h serve everywhere, and they relate to LSNs, so putting them in ma_loghandler_lsn.h. Stronger constraints in LSN_VALID(). storage/maria/ma_pagecache.c: renaming constants storage/maria/ma_recovery.h: copyright storage/maria/ma_test1.c: new prototype storage/maria/ma_test2.c: new prototype storage/maria/trnman_public.h: double-inclusion safe storage/maria/unittest/ma_control_file-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multigroup-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multithread-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_pagecache-t.c: constants renamed, new prototype storage/myisam/mi_close.c: comment storage/maria/maria_read_log.c: program to read and print log records from a Maria transaction log, and optionally apply them to tables. Very basic, early version. Should serve as a base for Recovery's code. Designed to be idempotent. Create a log by running maria.test, then cd to var/master-data and run "maria_read_log --only-display" to see info about records; run "maria_read_log --display-and-apply" to also apply the records to tables (it's more interesting if you first wipe out the tables in var/master-data/test, to see how they get re-created). Only a few records are handled by now: LONG_TRANSACTION_ID, COMMIT, FILE_ID, REDO_CREATE_TABLE; place is ready for REDO_INSERT_ROW_HEAD where I could use Monty's help (search for "Monty" in the file). Note: changes to the index pages, index's header and bitmap pages are not properly logged yet, so don't expect the program to work with that.
19 years ago
Remove SAFE_MODE for opt_range as it disables UPDATE to use keys REDO optimization (Bascily avoid moving blocks from/to pagecache) More command line arguments to maria_read_log Fixed recovery bug when recreating table sql/opt_range.cc: Remove SAFE_MODE for opt_range as it disables UPDATE to use keys storage/maria/ma_blockrec.c: REDO optimization Use new interface for pagecache_reads to avoid copying page buffers storage/maria/ma_loghandler.c: Patch from Sanja: - Added new parameter to translog_get_page to use direct links to pagecache - Changed scanner to be able to use direct links This avoids a lot of calls to bmove512() in page cache. storage/maria/ma_loghandler.h: Added direct link to pagecache objects storage/maria/ma_open.c: Added const to parameter Added missing braces storage/maria/ma_pagecache.c: From Sanja: - Added direct links to pagecache (from pagecache_read()) Dirrect link means that on pagecache_read we get back a pointer to the pagecache buffer From Monty: - Fixed arguments to init_page_cache to handle big page caches - Fixed compiler warnings - Replaced PAGECACHE_PAGE_LINK with PAGECACHE_BLOCK_LINK * to catch errors storage/maria/ma_pagecache.h: Changed block numbers from int to long to be able to handle big page caches Changed some PAGECACHE_PAGE_LINK to PAGECACHE_BLOCK_LINK storage/maria/ma_recovery.c: Fixed recovery bug when recreating table (table was kept open) Moved some variables to function start (portability) Added space to some print messages storage/maria/maria_chk.c: key_buffer_size -> page_buffer_size storage/maria/maria_def.h: Changed default page_buffer_size to 10M storage/maria/maria_read_log.c: Added more startup options: --version --undo (apply undo) --page_cache_size (to run with big cache sizes) --silent (to not get any output from --apply) storage/maria/unittest/ma_control_file-t.c: Fixed compiler warning storage/maria/unittest/ma_test_loghandler-t.c: Added new argument to translog_init_scanner() storage/maria/unittest/ma_test_loghandler_multigroup-t.c: Added new argument to translog_init_scanner() storage/maria/unittest/ma_test_loghandler_multithread-t.c: Added new argument to translog_init_scanner()
18 years ago
WL#3072 - Maria recovery maria_read_log used to always print a warning message at startup to say it is unsafe if ALTER TABLE was used. Now it prints it only if the log does show the problem (=ALTER TABLE or CREATE SELECT, which both disable logging of REDO_INSERT*). For that, when ha_maria::external_lock() disables transactionality it writes a LOGREC_INCOMPLETE_LOG to the log, which "maria_read_log -a" picks up to write a warning. REPAIR TABLE also disables those REDO_INSERT* but as maria_read_log executes LOGREC_REDO_REPAIR no warning is needed. storage/maria/ha_maria.cc: as we now log a record when disabling transactionility, we need the TRN to be set up first storage/maria/ma_blockrec.c: comment storage/maria/ma_loghandler.c: new type of log record storage/maria/ma_loghandler.h: new type of log record storage/maria/ma_recovery.c: * maria_apply_log() now returns a count of warnings. What currently produces warnings is: - skipping applying UNDOs though there are some (=> inconsistent table) - replaying log (in maria_read_log) though the log contains some ALTER TABLE or CREATE SELECT (log misses REDO_INSERT* for those and is so incomplete). Count of warnings affects the final message of maria_read_log and recovery (though in recovery none of the two conditions above should happen). * maria_read_log used to always print a warning message at startup to say it is unsafe if ALTER TABLE was used. Now it prints it only if the log does show the problem, i.e. ALTER TABLE or CREATE SELECT was used (both disable logging of REDO_INSERT* as those records are not needed for recovery; those missing records in turn make recreation-from-scratch, via maria_read_log, impossible). For that, when ha_maria::external_lock() disables transactionality, _ma_tmp_disable_logging_for_table() writes a LOGREC_INCOMPLETE_LOG to the log, which maria_apply_log() picks up to write a warning. storage/maria/ma_recovery.h: maria_apply_log() returns a count of warnings storage/maria/maria_def.h: _ma_tmp_disable_logging_for_table() grows so becomes a function storage/maria/maria_read_log.c: maria_apply_log can now return a count of warnings, to temper the "SUCCESS" message printed in the end by maria_read_log. Advise users to make a backup first.
18 years ago
Remove SAFE_MODE for opt_range as it disables UPDATE to use keys REDO optimization (Bascily avoid moving blocks from/to pagecache) More command line arguments to maria_read_log Fixed recovery bug when recreating table sql/opt_range.cc: Remove SAFE_MODE for opt_range as it disables UPDATE to use keys storage/maria/ma_blockrec.c: REDO optimization Use new interface for pagecache_reads to avoid copying page buffers storage/maria/ma_loghandler.c: Patch from Sanja: - Added new parameter to translog_get_page to use direct links to pagecache - Changed scanner to be able to use direct links This avoids a lot of calls to bmove512() in page cache. storage/maria/ma_loghandler.h: Added direct link to pagecache objects storage/maria/ma_open.c: Added const to parameter Added missing braces storage/maria/ma_pagecache.c: From Sanja: - Added direct links to pagecache (from pagecache_read()) Dirrect link means that on pagecache_read we get back a pointer to the pagecache buffer From Monty: - Fixed arguments to init_page_cache to handle big page caches - Fixed compiler warnings - Replaced PAGECACHE_PAGE_LINK with PAGECACHE_BLOCK_LINK * to catch errors storage/maria/ma_pagecache.h: Changed block numbers from int to long to be able to handle big page caches Changed some PAGECACHE_PAGE_LINK to PAGECACHE_BLOCK_LINK storage/maria/ma_recovery.c: Fixed recovery bug when recreating table (table was kept open) Moved some variables to function start (portability) Added space to some print messages storage/maria/maria_chk.c: key_buffer_size -> page_buffer_size storage/maria/maria_def.h: Changed default page_buffer_size to 10M storage/maria/maria_read_log.c: Added more startup options: --version --undo (apply undo) --page_cache_size (to run with big cache sizes) --silent (to not get any output from --apply) storage/maria/unittest/ma_control_file-t.c: Fixed compiler warning storage/maria/unittest/ma_test_loghandler-t.c: Added new argument to translog_init_scanner() storage/maria/unittest/ma_test_loghandler_multigroup-t.c: Added new argument to translog_init_scanner() storage/maria/unittest/ma_test_loghandler_multithread-t.c: Added new argument to translog_init_scanner()
18 years ago
Remove SAFE_MODE for opt_range as it disables UPDATE to use keys REDO optimization (Bascily avoid moving blocks from/to pagecache) More command line arguments to maria_read_log Fixed recovery bug when recreating table sql/opt_range.cc: Remove SAFE_MODE for opt_range as it disables UPDATE to use keys storage/maria/ma_blockrec.c: REDO optimization Use new interface for pagecache_reads to avoid copying page buffers storage/maria/ma_loghandler.c: Patch from Sanja: - Added new parameter to translog_get_page to use direct links to pagecache - Changed scanner to be able to use direct links This avoids a lot of calls to bmove512() in page cache. storage/maria/ma_loghandler.h: Added direct link to pagecache objects storage/maria/ma_open.c: Added const to parameter Added missing braces storage/maria/ma_pagecache.c: From Sanja: - Added direct links to pagecache (from pagecache_read()) Dirrect link means that on pagecache_read we get back a pointer to the pagecache buffer From Monty: - Fixed arguments to init_page_cache to handle big page caches - Fixed compiler warnings - Replaced PAGECACHE_PAGE_LINK with PAGECACHE_BLOCK_LINK * to catch errors storage/maria/ma_pagecache.h: Changed block numbers from int to long to be able to handle big page caches Changed some PAGECACHE_PAGE_LINK to PAGECACHE_BLOCK_LINK storage/maria/ma_recovery.c: Fixed recovery bug when recreating table (table was kept open) Moved some variables to function start (portability) Added space to some print messages storage/maria/maria_chk.c: key_buffer_size -> page_buffer_size storage/maria/maria_def.h: Changed default page_buffer_size to 10M storage/maria/maria_read_log.c: Added more startup options: --version --undo (apply undo) --page_cache_size (to run with big cache sizes) --silent (to not get any output from --apply) storage/maria/unittest/ma_control_file-t.c: Fixed compiler warning storage/maria/unittest/ma_test_loghandler-t.c: Added new argument to translog_init_scanner() storage/maria/unittest/ma_test_loghandler_multigroup-t.c: Added new argument to translog_init_scanner() storage/maria/unittest/ma_test_loghandler_multithread-t.c: Added new argument to translog_init_scanner()
18 years ago
Added maria_commit() and maria_begin() to be used with external tests Now ma_test1 -M -T and ma_test2 -M -T produces readable, applyable logs Note: The .MAD file is not binary identical after applying redo compare to a an original file. (This is becasue we don't have full information which function called PURGE_REDO_BLOCKS). To verify if a file was correctly applied, we now instead compare row checksums BitKeeper/etc/ignore: added storage/maria/tmp/* include/maria.h: Added maria_commit() and maria_begin() to be used with external tests storage/maria/ha_maria.cc: Ensure maria_def. is read in C mode storage/maria/ma_blockrec.c: Fixed redo handling. _ma_apply_redo_purge_blocks() updated to handle any number of purged blocks Removed code to make data file idenitcal after redo (can't easily be done). See changeset comments Now ma_test1 -M -T and ma_test2 -M -T produces readable, applyable logs storage/maria/ma_commit.c: More DBUG statements Moved variable declaration to start of function (portability fix) Added helper functions 'maria_commit()' and 'maria_begin()' storage/maria/ma_loghandler.c: Fixed wrong REDO_PURGE_BLOCKS initialization storage/maria/ma_recovery.c: Added UNDO_ROW_UPDATE Removed wrong setting of lsn (there was no lsn at the used position) Fixed REDO_PURGE_BLOCKS to handle any number of blocks storage/maria/ma_test1.c: Added transaction support (via maria_begin() & maria_commit()) to get a log that can be applied with maria_read_log storage/maria/ma_test2.c: Added transaction support (via maria_begin() & maria_commit()) to get a log that can be applied with maria_read_log storage/maria/ma_test_recovery: Create temporary files in maria/tmp Verify files with checksums instead of byte comparisons storage/maria/maria_chk.c: When using with -dss we only get filename, records and checksum. This is useful to do a quick comparision if a files is identical to another one. storage/maria/maria_def.h: Added ma_commit() storage/maria/maria_read_log.c: Added --help
19 years ago
WL#3072 - Maria recovery maria_read_log used to always print a warning message at startup to say it is unsafe if ALTER TABLE was used. Now it prints it only if the log does show the problem (=ALTER TABLE or CREATE SELECT, which both disable logging of REDO_INSERT*). For that, when ha_maria::external_lock() disables transactionality it writes a LOGREC_INCOMPLETE_LOG to the log, which "maria_read_log -a" picks up to write a warning. REPAIR TABLE also disables those REDO_INSERT* but as maria_read_log executes LOGREC_REDO_REPAIR no warning is needed. storage/maria/ha_maria.cc: as we now log a record when disabling transactionility, we need the TRN to be set up first storage/maria/ma_blockrec.c: comment storage/maria/ma_loghandler.c: new type of log record storage/maria/ma_loghandler.h: new type of log record storage/maria/ma_recovery.c: * maria_apply_log() now returns a count of warnings. What currently produces warnings is: - skipping applying UNDOs though there are some (=> inconsistent table) - replaying log (in maria_read_log) though the log contains some ALTER TABLE or CREATE SELECT (log misses REDO_INSERT* for those and is so incomplete). Count of warnings affects the final message of maria_read_log and recovery (though in recovery none of the two conditions above should happen). * maria_read_log used to always print a warning message at startup to say it is unsafe if ALTER TABLE was used. Now it prints it only if the log does show the problem, i.e. ALTER TABLE or CREATE SELECT was used (both disable logging of REDO_INSERT* as those records are not needed for recovery; those missing records in turn make recreation-from-scratch, via maria_read_log, impossible). For that, when ha_maria::external_lock() disables transactionality, _ma_tmp_disable_logging_for_table() writes a LOGREC_INCOMPLETE_LOG to the log, which maria_apply_log() picks up to write a warning. storage/maria/ma_recovery.h: maria_apply_log() returns a count of warnings storage/maria/maria_def.h: _ma_tmp_disable_logging_for_table() grows so becomes a function storage/maria/maria_read_log.c: maria_apply_log can now return a count of warnings, to temper the "SUCCESS" message printed in the end by maria_read_log. Advise users to make a backup first.
18 years ago
post-merge fixes, and fixes for some of the 16 compiler warnings found in pushbuild on sapsrv1. Some not fixed as not repeatable on my machine (32/64 bit issue?). Fixes for some test failures: - "maria-connect" now passes; - "maria": after fixing the obvious reasons for failures, the test went further and hit a more complex issues: difference in the output of EXPLAIN output; not fixed; - "ps_maria" still crashes in assertion mysqld: ha_maria.cc:1627: virtual int ha_maria::index_read(uchar*, const uchar*, uint, ha_rkey_function): Ass ertion `inited == INDEX' failed, as already observable in pushbuild. All this might just be due to an incomplete merge of MyISAM changes into Maria when 5.1 was last merged to mysql-maria. include/my_global.h: temporary fix until next merge of 5.1; without this it does not build mysql-test/r/maria-connect.result: position changed mysql-test/t/maria-connect.test: If one wants to use the binlog it has to ask for it. 1582 is not used for dup entry error anymore (it was in older 5.1). Size of first event in binlog was increased by 4 (when the new type of event "gap" was added). mysql-test/t/maria.test: 1582 not used anymore in this case storage/maria/ha_maria.cc: engine now has to say what binlogging it supports storage/maria/ma_blockrec.c: fix for compiler warnings ("comparison is always true" or "always false") storage/maria/ma_loghandler.c: fix for compiler warnings (comparing char* to uchar*) storage/maria/ma_packrec.c: fix for compiler warning (fix simply merged from MyISAM) storage/maria/ma_pagecache.c: info_check_pin() was not used so gave a compiler warning. storage/maria/ma_pagecache.h: fixing typo from the last 5.1->maria merge. storage/maria/ma_recovery.c: my_free() has a void* argument, so why cast. byte->uchar. storage/maria/ma_search.c: fix for compiler warning (fix simply merged from MyISAM) storage/maria/maria_read_log.c: gptr->uchar* storage/maria/trnman.c: probable fix for warning found in pushbuild (but not on my machine): storage/maria/trnman.c: 142 passing argument 6 of \u2018lf_hash_init\u2019 from incompatible pointer type on sapsrv1.
19 years ago
Remove SAFE_MODE for opt_range as it disables UPDATE to use keys REDO optimization (Bascily avoid moving blocks from/to pagecache) More command line arguments to maria_read_log Fixed recovery bug when recreating table sql/opt_range.cc: Remove SAFE_MODE for opt_range as it disables UPDATE to use keys storage/maria/ma_blockrec.c: REDO optimization Use new interface for pagecache_reads to avoid copying page buffers storage/maria/ma_loghandler.c: Patch from Sanja: - Added new parameter to translog_get_page to use direct links to pagecache - Changed scanner to be able to use direct links This avoids a lot of calls to bmove512() in page cache. storage/maria/ma_loghandler.h: Added direct link to pagecache objects storage/maria/ma_open.c: Added const to parameter Added missing braces storage/maria/ma_pagecache.c: From Sanja: - Added direct links to pagecache (from pagecache_read()) Dirrect link means that on pagecache_read we get back a pointer to the pagecache buffer From Monty: - Fixed arguments to init_page_cache to handle big page caches - Fixed compiler warnings - Replaced PAGECACHE_PAGE_LINK with PAGECACHE_BLOCK_LINK * to catch errors storage/maria/ma_pagecache.h: Changed block numbers from int to long to be able to handle big page caches Changed some PAGECACHE_PAGE_LINK to PAGECACHE_BLOCK_LINK storage/maria/ma_recovery.c: Fixed recovery bug when recreating table (table was kept open) Moved some variables to function start (portability) Added space to some print messages storage/maria/maria_chk.c: key_buffer_size -> page_buffer_size storage/maria/maria_def.h: Changed default page_buffer_size to 10M storage/maria/maria_read_log.c: Added more startup options: --version --undo (apply undo) --page_cache_size (to run with big cache sizes) --silent (to not get any output from --apply) storage/maria/unittest/ma_control_file-t.c: Fixed compiler warning storage/maria/unittest/ma_test_loghandler-t.c: Added new argument to translog_init_scanner() storage/maria/unittest/ma_test_loghandler_multigroup-t.c: Added new argument to translog_init_scanner() storage/maria/unittest/ma_test_loghandler_multithread-t.c: Added new argument to translog_init_scanner()
18 years ago
WL#3072 Maria Recovery - new program maria_read_log to display and apply log records found in a Maria log (see file's revision comment) - minor, misc fixes storage/maria/Makefile.am: new program maria_read_log storage/maria/ha_maria.cc: create control file if missing storage/maria/ma_blockrec.c: 0 -> LSN_IMPOSSIBLE; comments storage/maria/ma_checkpoint.h: preparations for Checkpoint module storage/maria/ma_close.c: comment storage/maria/ma_control_file.c: renaming constants. Possibility to say "open control file but don't create it if it's missing" (used by maria_read_log which does not want to create anything) storage/maria/ma_control_file.h: renaming constants storage/maria/ma_create.c: I had duplicated "linkname" and "linkname_ptr", now I see it's not needed, reverting. Indeed those variables don't contain interesting information; fixing log record accordingly (the links are in ci->data/index_file_name). Storing keystart in log record is needed, to know at which size we must extend the file if we replay LOGREC_CREATE_TABLE. storage/maria/ma_loghandler.c: some structures need to be known to maria_read_log.c, taking them to ma_loghandler.h storage/maria/ma_loghandler.h: we have page_store, adding page_korr. translog_lock() made public, because Checkpoint will need it (to write to control file). Some structures moved from ma_loghandler.c because maria_read_log.c needs them (needs to know the execute-in-REDO-phase hooks of each record). storage/maria/ma_loghandler_lsn.h: constants defined in ma_control_file.h serve everywhere, and they relate to LSNs, so putting them in ma_loghandler_lsn.h. Stronger constraints in LSN_VALID(). storage/maria/ma_pagecache.c: renaming constants storage/maria/ma_recovery.h: copyright storage/maria/ma_test1.c: new prototype storage/maria/ma_test2.c: new prototype storage/maria/trnman_public.h: double-inclusion safe storage/maria/unittest/ma_control_file-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multigroup-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multithread-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_pagecache-t.c: constants renamed, new prototype storage/myisam/mi_close.c: comment storage/maria/maria_read_log.c: program to read and print log records from a Maria transaction log, and optionally apply them to tables. Very basic, early version. Should serve as a base for Recovery's code. Designed to be idempotent. Create a log by running maria.test, then cd to var/master-data and run "maria_read_log --only-display" to see info about records; run "maria_read_log --display-and-apply" to also apply the records to tables (it's more interesting if you first wipe out the tables in var/master-data/test, to see how they get re-created). Only a few records are handled by now: LONG_TRANSACTION_ID, COMMIT, FILE_ID, REDO_CREATE_TABLE; place is ready for REDO_INSERT_ROW_HEAD where I could use Monty's help (search for "Monty" in the file). Note: changes to the index pages, index's header and bitmap pages are not properly logged yet, so don't expect the program to work with that.
19 years ago
WL#3072 - Maria recovery maria_read_log used to always print a warning message at startup to say it is unsafe if ALTER TABLE was used. Now it prints it only if the log does show the problem (=ALTER TABLE or CREATE SELECT, which both disable logging of REDO_INSERT*). For that, when ha_maria::external_lock() disables transactionality it writes a LOGREC_INCOMPLETE_LOG to the log, which "maria_read_log -a" picks up to write a warning. REPAIR TABLE also disables those REDO_INSERT* but as maria_read_log executes LOGREC_REDO_REPAIR no warning is needed. storage/maria/ha_maria.cc: as we now log a record when disabling transactionility, we need the TRN to be set up first storage/maria/ma_blockrec.c: comment storage/maria/ma_loghandler.c: new type of log record storage/maria/ma_loghandler.h: new type of log record storage/maria/ma_recovery.c: * maria_apply_log() now returns a count of warnings. What currently produces warnings is: - skipping applying UNDOs though there are some (=> inconsistent table) - replaying log (in maria_read_log) though the log contains some ALTER TABLE or CREATE SELECT (log misses REDO_INSERT* for those and is so incomplete). Count of warnings affects the final message of maria_read_log and recovery (though in recovery none of the two conditions above should happen). * maria_read_log used to always print a warning message at startup to say it is unsafe if ALTER TABLE was used. Now it prints it only if the log does show the problem, i.e. ALTER TABLE or CREATE SELECT was used (both disable logging of REDO_INSERT* as those records are not needed for recovery; those missing records in turn make recreation-from-scratch, via maria_read_log, impossible). For that, when ha_maria::external_lock() disables transactionality, _ma_tmp_disable_logging_for_table() writes a LOGREC_INCOMPLETE_LOG to the log, which maria_apply_log() picks up to write a warning. storage/maria/ma_recovery.h: maria_apply_log() returns a count of warnings storage/maria/maria_def.h: _ma_tmp_disable_logging_for_table() grows so becomes a function storage/maria/maria_read_log.c: maria_apply_log can now return a count of warnings, to temper the "SUCCESS" message printed in the end by maria_read_log. Advise users to make a backup first.
18 years ago
Remove SAFE_MODE for opt_range as it disables UPDATE to use keys REDO optimization (Bascily avoid moving blocks from/to pagecache) More command line arguments to maria_read_log Fixed recovery bug when recreating table sql/opt_range.cc: Remove SAFE_MODE for opt_range as it disables UPDATE to use keys storage/maria/ma_blockrec.c: REDO optimization Use new interface for pagecache_reads to avoid copying page buffers storage/maria/ma_loghandler.c: Patch from Sanja: - Added new parameter to translog_get_page to use direct links to pagecache - Changed scanner to be able to use direct links This avoids a lot of calls to bmove512() in page cache. storage/maria/ma_loghandler.h: Added direct link to pagecache objects storage/maria/ma_open.c: Added const to parameter Added missing braces storage/maria/ma_pagecache.c: From Sanja: - Added direct links to pagecache (from pagecache_read()) Dirrect link means that on pagecache_read we get back a pointer to the pagecache buffer From Monty: - Fixed arguments to init_page_cache to handle big page caches - Fixed compiler warnings - Replaced PAGECACHE_PAGE_LINK with PAGECACHE_BLOCK_LINK * to catch errors storage/maria/ma_pagecache.h: Changed block numbers from int to long to be able to handle big page caches Changed some PAGECACHE_PAGE_LINK to PAGECACHE_BLOCK_LINK storage/maria/ma_recovery.c: Fixed recovery bug when recreating table (table was kept open) Moved some variables to function start (portability) Added space to some print messages storage/maria/maria_chk.c: key_buffer_size -> page_buffer_size storage/maria/maria_def.h: Changed default page_buffer_size to 10M storage/maria/maria_read_log.c: Added more startup options: --version --undo (apply undo) --page_cache_size (to run with big cache sizes) --silent (to not get any output from --apply) storage/maria/unittest/ma_control_file-t.c: Fixed compiler warning storage/maria/unittest/ma_test_loghandler-t.c: Added new argument to translog_init_scanner() storage/maria/unittest/ma_test_loghandler_multigroup-t.c: Added new argument to translog_init_scanner() storage/maria/unittest/ma_test_loghandler_multithread-t.c: Added new argument to translog_init_scanner()
18 years ago
WL#3072 Maria Recovery - new program maria_read_log to display and apply log records found in a Maria log (see file's revision comment) - minor, misc fixes storage/maria/Makefile.am: new program maria_read_log storage/maria/ha_maria.cc: create control file if missing storage/maria/ma_blockrec.c: 0 -> LSN_IMPOSSIBLE; comments storage/maria/ma_checkpoint.h: preparations for Checkpoint module storage/maria/ma_close.c: comment storage/maria/ma_control_file.c: renaming constants. Possibility to say "open control file but don't create it if it's missing" (used by maria_read_log which does not want to create anything) storage/maria/ma_control_file.h: renaming constants storage/maria/ma_create.c: I had duplicated "linkname" and "linkname_ptr", now I see it's not needed, reverting. Indeed those variables don't contain interesting information; fixing log record accordingly (the links are in ci->data/index_file_name). Storing keystart in log record is needed, to know at which size we must extend the file if we replay LOGREC_CREATE_TABLE. storage/maria/ma_loghandler.c: some structures need to be known to maria_read_log.c, taking them to ma_loghandler.h storage/maria/ma_loghandler.h: we have page_store, adding page_korr. translog_lock() made public, because Checkpoint will need it (to write to control file). Some structures moved from ma_loghandler.c because maria_read_log.c needs them (needs to know the execute-in-REDO-phase hooks of each record). storage/maria/ma_loghandler_lsn.h: constants defined in ma_control_file.h serve everywhere, and they relate to LSNs, so putting them in ma_loghandler_lsn.h. Stronger constraints in LSN_VALID(). storage/maria/ma_pagecache.c: renaming constants storage/maria/ma_recovery.h: copyright storage/maria/ma_test1.c: new prototype storage/maria/ma_test2.c: new prototype storage/maria/trnman_public.h: double-inclusion safe storage/maria/unittest/ma_control_file-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multigroup-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multithread-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_pagecache-t.c: constants renamed, new prototype storage/myisam/mi_close.c: comment storage/maria/maria_read_log.c: program to read and print log records from a Maria transaction log, and optionally apply them to tables. Very basic, early version. Should serve as a base for Recovery's code. Designed to be idempotent. Create a log by running maria.test, then cd to var/master-data and run "maria_read_log --only-display" to see info about records; run "maria_read_log --display-and-apply" to also apply the records to tables (it's more interesting if you first wipe out the tables in var/master-data/test, to see how they get re-created). Only a few records are handled by now: LONG_TRANSACTION_ID, COMMIT, FILE_ID, REDO_CREATE_TABLE; place is ready for REDO_INSERT_ROW_HEAD where I could use Monty's help (search for "Monty" in the file). Note: changes to the index pages, index's header and bitmap pages are not properly logged yet, so don't expect the program to work with that.
19 years ago
Remove SAFE_MODE for opt_range as it disables UPDATE to use keys REDO optimization (Bascily avoid moving blocks from/to pagecache) More command line arguments to maria_read_log Fixed recovery bug when recreating table sql/opt_range.cc: Remove SAFE_MODE for opt_range as it disables UPDATE to use keys storage/maria/ma_blockrec.c: REDO optimization Use new interface for pagecache_reads to avoid copying page buffers storage/maria/ma_loghandler.c: Patch from Sanja: - Added new parameter to translog_get_page to use direct links to pagecache - Changed scanner to be able to use direct links This avoids a lot of calls to bmove512() in page cache. storage/maria/ma_loghandler.h: Added direct link to pagecache objects storage/maria/ma_open.c: Added const to parameter Added missing braces storage/maria/ma_pagecache.c: From Sanja: - Added direct links to pagecache (from pagecache_read()) Dirrect link means that on pagecache_read we get back a pointer to the pagecache buffer From Monty: - Fixed arguments to init_page_cache to handle big page caches - Fixed compiler warnings - Replaced PAGECACHE_PAGE_LINK with PAGECACHE_BLOCK_LINK * to catch errors storage/maria/ma_pagecache.h: Changed block numbers from int to long to be able to handle big page caches Changed some PAGECACHE_PAGE_LINK to PAGECACHE_BLOCK_LINK storage/maria/ma_recovery.c: Fixed recovery bug when recreating table (table was kept open) Moved some variables to function start (portability) Added space to some print messages storage/maria/maria_chk.c: key_buffer_size -> page_buffer_size storage/maria/maria_def.h: Changed default page_buffer_size to 10M storage/maria/maria_read_log.c: Added more startup options: --version --undo (apply undo) --page_cache_size (to run with big cache sizes) --silent (to not get any output from --apply) storage/maria/unittest/ma_control_file-t.c: Fixed compiler warning storage/maria/unittest/ma_test_loghandler-t.c: Added new argument to translog_init_scanner() storage/maria/unittest/ma_test_loghandler_multigroup-t.c: Added new argument to translog_init_scanner() storage/maria/unittest/ma_test_loghandler_multithread-t.c: Added new argument to translog_init_scanner()
18 years ago
WL#3072 Maria recovery. * Thanks to Serg's tip, we fix here the compilation issue of REDO_REPAIR_TABLE's execution, by defining versions of _ma_killed_ptr() and _ma_check_print_info|warning|error() in maria_read_log.c (we move those of maria_chk.c into an include file and include it in maria_chk.c and maria_read_log.c). Execution of such record looks like working from my tests (it only happens in maria_read_log; recovery-from-mysqld skips DDLs and REPAIR is considered DDL here as it bypasses logging): tested ALTER TABLE ENABLE KEYS and then remove table, apply log: that did a repair. * Recent changes broke maria_read_log a bit: -a became default and -o caused error; fixing this. storage/maria/Makefile.am: addind new file storage/maria/ma_recovery.c: enable execution of REDO_REPAIR_TABLE by maria_read_log now that it compiles. Now reason to keep only T_QUICK from testflag. storage/maria/maria_chk.c: moving these functions to ma_check_standalone.h for reusability storage/maria/maria_def.h: comment storage/maria/maria_read_log.c: ma_check_standalone.h needs my_progname_short. Fixing bug where "maria_read_log" would default to -a and "maria_read_log -o" would throw an error. Implemented behaviour is: - no options: usage() - -a : applys, applys UNDOs by default unless --disable-undo - -o : only prints storage/maria/ma_check_standalone.h: All standalone programs which need to use functions from ma_check.c (like maria_repair()) must define their version of _ma_killed_ptr() and _ma_check_print_info|warning|error(). Indeed, linking with ma_check.o brings in the dependencies of ma_check.o which are definitions of the above functions; if the program does not define them then the ones of ha_maria.o are used i.e. ha_maria.o is linked into the program, and this brings dependencies of ha_maria.o on mysqld.o into the program's linking which thus fails, as the program is not linked with mysqld.o. We put in this file the functions which maria_chk.c uses, so that they can be reused by maria_read_log (when it replays REDO_REPAIR_TABLE) as they are good enough (they just print to stdout/stderr like maria_read_log already does).
18 years ago
WL#3072 Maria Recovery - new program maria_read_log to display and apply log records found in a Maria log (see file's revision comment) - minor, misc fixes storage/maria/Makefile.am: new program maria_read_log storage/maria/ha_maria.cc: create control file if missing storage/maria/ma_blockrec.c: 0 -> LSN_IMPOSSIBLE; comments storage/maria/ma_checkpoint.h: preparations for Checkpoint module storage/maria/ma_close.c: comment storage/maria/ma_control_file.c: renaming constants. Possibility to say "open control file but don't create it if it's missing" (used by maria_read_log which does not want to create anything) storage/maria/ma_control_file.h: renaming constants storage/maria/ma_create.c: I had duplicated "linkname" and "linkname_ptr", now I see it's not needed, reverting. Indeed those variables don't contain interesting information; fixing log record accordingly (the links are in ci->data/index_file_name). Storing keystart in log record is needed, to know at which size we must extend the file if we replay LOGREC_CREATE_TABLE. storage/maria/ma_loghandler.c: some structures need to be known to maria_read_log.c, taking them to ma_loghandler.h storage/maria/ma_loghandler.h: we have page_store, adding page_korr. translog_lock() made public, because Checkpoint will need it (to write to control file). Some structures moved from ma_loghandler.c because maria_read_log.c needs them (needs to know the execute-in-REDO-phase hooks of each record). storage/maria/ma_loghandler_lsn.h: constants defined in ma_control_file.h serve everywhere, and they relate to LSNs, so putting them in ma_loghandler_lsn.h. Stronger constraints in LSN_VALID(). storage/maria/ma_pagecache.c: renaming constants storage/maria/ma_recovery.h: copyright storage/maria/ma_test1.c: new prototype storage/maria/ma_test2.c: new prototype storage/maria/trnman_public.h: double-inclusion safe storage/maria/unittest/ma_control_file-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multigroup-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multithread-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_pagecache-t.c: constants renamed, new prototype storage/myisam/mi_close.c: comment storage/maria/maria_read_log.c: program to read and print log records from a Maria transaction log, and optionally apply them to tables. Very basic, early version. Should serve as a base for Recovery's code. Designed to be idempotent. Create a log by running maria.test, then cd to var/master-data and run "maria_read_log --only-display" to see info about records; run "maria_read_log --display-and-apply" to also apply the records to tables (it's more interesting if you first wipe out the tables in var/master-data/test, to see how they get re-created). Only a few records are handled by now: LONG_TRANSACTION_ID, COMMIT, FILE_ID, REDO_CREATE_TABLE; place is ready for REDO_INSERT_ROW_HEAD where I could use Monty's help (search for "Monty" in the file). Note: changes to the index pages, index's header and bitmap pages are not properly logged yet, so don't expect the program to work with that.
19 years ago
WL#3072 Maria recovery. * Thanks to Serg's tip, we fix here the compilation issue of REDO_REPAIR_TABLE's execution, by defining versions of _ma_killed_ptr() and _ma_check_print_info|warning|error() in maria_read_log.c (we move those of maria_chk.c into an include file and include it in maria_chk.c and maria_read_log.c). Execution of such record looks like working from my tests (it only happens in maria_read_log; recovery-from-mysqld skips DDLs and REPAIR is considered DDL here as it bypasses logging): tested ALTER TABLE ENABLE KEYS and then remove table, apply log: that did a repair. * Recent changes broke maria_read_log a bit: -a became default and -o caused error; fixing this. storage/maria/Makefile.am: addind new file storage/maria/ma_recovery.c: enable execution of REDO_REPAIR_TABLE by maria_read_log now that it compiles. Now reason to keep only T_QUICK from testflag. storage/maria/maria_chk.c: moving these functions to ma_check_standalone.h for reusability storage/maria/maria_def.h: comment storage/maria/maria_read_log.c: ma_check_standalone.h needs my_progname_short. Fixing bug where "maria_read_log" would default to -a and "maria_read_log -o" would throw an error. Implemented behaviour is: - no options: usage() - -a : applys, applys UNDOs by default unless --disable-undo - -o : only prints storage/maria/ma_check_standalone.h: All standalone programs which need to use functions from ma_check.c (like maria_repair()) must define their version of _ma_killed_ptr() and _ma_check_print_info|warning|error(). Indeed, linking with ma_check.o brings in the dependencies of ma_check.o which are definitions of the above functions; if the program does not define them then the ones of ha_maria.o are used i.e. ha_maria.o is linked into the program, and this brings dependencies of ha_maria.o on mysqld.o into the program's linking which thus fails, as the program is not linked with mysqld.o. We put in this file the functions which maria_chk.c uses, so that they can be reused by maria_read_log (when it replays REDO_REPAIR_TABLE) as they are good enough (they just print to stdout/stderr like maria_read_log already does).
18 years ago
WL#3072 Maria Recovery - new program maria_read_log to display and apply log records found in a Maria log (see file's revision comment) - minor, misc fixes storage/maria/Makefile.am: new program maria_read_log storage/maria/ha_maria.cc: create control file if missing storage/maria/ma_blockrec.c: 0 -> LSN_IMPOSSIBLE; comments storage/maria/ma_checkpoint.h: preparations for Checkpoint module storage/maria/ma_close.c: comment storage/maria/ma_control_file.c: renaming constants. Possibility to say "open control file but don't create it if it's missing" (used by maria_read_log which does not want to create anything) storage/maria/ma_control_file.h: renaming constants storage/maria/ma_create.c: I had duplicated "linkname" and "linkname_ptr", now I see it's not needed, reverting. Indeed those variables don't contain interesting information; fixing log record accordingly (the links are in ci->data/index_file_name). Storing keystart in log record is needed, to know at which size we must extend the file if we replay LOGREC_CREATE_TABLE. storage/maria/ma_loghandler.c: some structures need to be known to maria_read_log.c, taking them to ma_loghandler.h storage/maria/ma_loghandler.h: we have page_store, adding page_korr. translog_lock() made public, because Checkpoint will need it (to write to control file). Some structures moved from ma_loghandler.c because maria_read_log.c needs them (needs to know the execute-in-REDO-phase hooks of each record). storage/maria/ma_loghandler_lsn.h: constants defined in ma_control_file.h serve everywhere, and they relate to LSNs, so putting them in ma_loghandler_lsn.h. Stronger constraints in LSN_VALID(). storage/maria/ma_pagecache.c: renaming constants storage/maria/ma_recovery.h: copyright storage/maria/ma_test1.c: new prototype storage/maria/ma_test2.c: new prototype storage/maria/trnman_public.h: double-inclusion safe storage/maria/unittest/ma_control_file-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multigroup-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multithread-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_pagecache-t.c: constants renamed, new prototype storage/myisam/mi_close.c: comment storage/maria/maria_read_log.c: program to read and print log records from a Maria transaction log, and optionally apply them to tables. Very basic, early version. Should serve as a base for Recovery's code. Designed to be idempotent. Create a log by running maria.test, then cd to var/master-data and run "maria_read_log --only-display" to see info about records; run "maria_read_log --display-and-apply" to also apply the records to tables (it's more interesting if you first wipe out the tables in var/master-data/test, to see how they get re-created). Only a few records are handled by now: LONG_TRANSACTION_ID, COMMIT, FILE_ID, REDO_CREATE_TABLE; place is ready for REDO_INSERT_ROW_HEAD where I could use Monty's help (search for "Monty" in the file). Note: changes to the index pages, index's header and bitmap pages are not properly logged yet, so don't expect the program to work with that.
19 years ago
Added maria_commit() and maria_begin() to be used with external tests Now ma_test1 -M -T and ma_test2 -M -T produces readable, applyable logs Note: The .MAD file is not binary identical after applying redo compare to a an original file. (This is becasue we don't have full information which function called PURGE_REDO_BLOCKS). To verify if a file was correctly applied, we now instead compare row checksums BitKeeper/etc/ignore: added storage/maria/tmp/* include/maria.h: Added maria_commit() and maria_begin() to be used with external tests storage/maria/ha_maria.cc: Ensure maria_def. is read in C mode storage/maria/ma_blockrec.c: Fixed redo handling. _ma_apply_redo_purge_blocks() updated to handle any number of purged blocks Removed code to make data file idenitcal after redo (can't easily be done). See changeset comments Now ma_test1 -M -T and ma_test2 -M -T produces readable, applyable logs storage/maria/ma_commit.c: More DBUG statements Moved variable declaration to start of function (portability fix) Added helper functions 'maria_commit()' and 'maria_begin()' storage/maria/ma_loghandler.c: Fixed wrong REDO_PURGE_BLOCKS initialization storage/maria/ma_recovery.c: Added UNDO_ROW_UPDATE Removed wrong setting of lsn (there was no lsn at the used position) Fixed REDO_PURGE_BLOCKS to handle any number of blocks storage/maria/ma_test1.c: Added transaction support (via maria_begin() & maria_commit()) to get a log that can be applied with maria_read_log storage/maria/ma_test2.c: Added transaction support (via maria_begin() & maria_commit()) to get a log that can be applied with maria_read_log storage/maria/ma_test_recovery: Create temporary files in maria/tmp Verify files with checksums instead of byte comparisons storage/maria/maria_chk.c: When using with -dss we only get filename, records and checksum. This is useful to do a quick comparision if a files is identical to another one. storage/maria/maria_def.h: Added ma_commit() storage/maria/maria_read_log.c: Added --help
19 years ago
Remove SAFE_MODE for opt_range as it disables UPDATE to use keys REDO optimization (Bascily avoid moving blocks from/to pagecache) More command line arguments to maria_read_log Fixed recovery bug when recreating table sql/opt_range.cc: Remove SAFE_MODE for opt_range as it disables UPDATE to use keys storage/maria/ma_blockrec.c: REDO optimization Use new interface for pagecache_reads to avoid copying page buffers storage/maria/ma_loghandler.c: Patch from Sanja: - Added new parameter to translog_get_page to use direct links to pagecache - Changed scanner to be able to use direct links This avoids a lot of calls to bmove512() in page cache. storage/maria/ma_loghandler.h: Added direct link to pagecache objects storage/maria/ma_open.c: Added const to parameter Added missing braces storage/maria/ma_pagecache.c: From Sanja: - Added direct links to pagecache (from pagecache_read()) Dirrect link means that on pagecache_read we get back a pointer to the pagecache buffer From Monty: - Fixed arguments to init_page_cache to handle big page caches - Fixed compiler warnings - Replaced PAGECACHE_PAGE_LINK with PAGECACHE_BLOCK_LINK * to catch errors storage/maria/ma_pagecache.h: Changed block numbers from int to long to be able to handle big page caches Changed some PAGECACHE_PAGE_LINK to PAGECACHE_BLOCK_LINK storage/maria/ma_recovery.c: Fixed recovery bug when recreating table (table was kept open) Moved some variables to function start (portability) Added space to some print messages storage/maria/maria_chk.c: key_buffer_size -> page_buffer_size storage/maria/maria_def.h: Changed default page_buffer_size to 10M storage/maria/maria_read_log.c: Added more startup options: --version --undo (apply undo) --page_cache_size (to run with big cache sizes) --silent (to not get any output from --apply) storage/maria/unittest/ma_control_file-t.c: Fixed compiler warning storage/maria/unittest/ma_test_loghandler-t.c: Added new argument to translog_init_scanner() storage/maria/unittest/ma_test_loghandler_multigroup-t.c: Added new argument to translog_init_scanner() storage/maria/unittest/ma_test_loghandler_multithread-t.c: Added new argument to translog_init_scanner()
18 years ago
WL#3072 Maria Recovery - new program maria_read_log to display and apply log records found in a Maria log (see file's revision comment) - minor, misc fixes storage/maria/Makefile.am: new program maria_read_log storage/maria/ha_maria.cc: create control file if missing storage/maria/ma_blockrec.c: 0 -> LSN_IMPOSSIBLE; comments storage/maria/ma_checkpoint.h: preparations for Checkpoint module storage/maria/ma_close.c: comment storage/maria/ma_control_file.c: renaming constants. Possibility to say "open control file but don't create it if it's missing" (used by maria_read_log which does not want to create anything) storage/maria/ma_control_file.h: renaming constants storage/maria/ma_create.c: I had duplicated "linkname" and "linkname_ptr", now I see it's not needed, reverting. Indeed those variables don't contain interesting information; fixing log record accordingly (the links are in ci->data/index_file_name). Storing keystart in log record is needed, to know at which size we must extend the file if we replay LOGREC_CREATE_TABLE. storage/maria/ma_loghandler.c: some structures need to be known to maria_read_log.c, taking them to ma_loghandler.h storage/maria/ma_loghandler.h: we have page_store, adding page_korr. translog_lock() made public, because Checkpoint will need it (to write to control file). Some structures moved from ma_loghandler.c because maria_read_log.c needs them (needs to know the execute-in-REDO-phase hooks of each record). storage/maria/ma_loghandler_lsn.h: constants defined in ma_control_file.h serve everywhere, and they relate to LSNs, so putting them in ma_loghandler_lsn.h. Stronger constraints in LSN_VALID(). storage/maria/ma_pagecache.c: renaming constants storage/maria/ma_recovery.h: copyright storage/maria/ma_test1.c: new prototype storage/maria/ma_test2.c: new prototype storage/maria/trnman_public.h: double-inclusion safe storage/maria/unittest/ma_control_file-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multigroup-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multithread-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_pagecache-t.c: constants renamed, new prototype storage/myisam/mi_close.c: comment storage/maria/maria_read_log.c: program to read and print log records from a Maria transaction log, and optionally apply them to tables. Very basic, early version. Should serve as a base for Recovery's code. Designed to be idempotent. Create a log by running maria.test, then cd to var/master-data and run "maria_read_log --only-display" to see info about records; run "maria_read_log --display-and-apply" to also apply the records to tables (it's more interesting if you first wipe out the tables in var/master-data/test, to see how they get re-created). Only a few records are handled by now: LONG_TRANSACTION_ID, COMMIT, FILE_ID, REDO_CREATE_TABLE; place is ready for REDO_INSERT_ROW_HEAD where I could use Monty's help (search for "Monty" in the file). Note: changes to the index pages, index's header and bitmap pages are not properly logged yet, so don't expect the program to work with that.
19 years ago
WL#3072 Maria recovery. * Thanks to Serg's tip, we fix here the compilation issue of REDO_REPAIR_TABLE's execution, by defining versions of _ma_killed_ptr() and _ma_check_print_info|warning|error() in maria_read_log.c (we move those of maria_chk.c into an include file and include it in maria_chk.c and maria_read_log.c). Execution of such record looks like working from my tests (it only happens in maria_read_log; recovery-from-mysqld skips DDLs and REPAIR is considered DDL here as it bypasses logging): tested ALTER TABLE ENABLE KEYS and then remove table, apply log: that did a repair. * Recent changes broke maria_read_log a bit: -a became default and -o caused error; fixing this. storage/maria/Makefile.am: addind new file storage/maria/ma_recovery.c: enable execution of REDO_REPAIR_TABLE by maria_read_log now that it compiles. Now reason to keep only T_QUICK from testflag. storage/maria/maria_chk.c: moving these functions to ma_check_standalone.h for reusability storage/maria/maria_def.h: comment storage/maria/maria_read_log.c: ma_check_standalone.h needs my_progname_short. Fixing bug where "maria_read_log" would default to -a and "maria_read_log -o" would throw an error. Implemented behaviour is: - no options: usage() - -a : applys, applys UNDOs by default unless --disable-undo - -o : only prints storage/maria/ma_check_standalone.h: All standalone programs which need to use functions from ma_check.c (like maria_repair()) must define their version of _ma_killed_ptr() and _ma_check_print_info|warning|error(). Indeed, linking with ma_check.o brings in the dependencies of ma_check.o which are definitions of the above functions; if the program does not define them then the ones of ha_maria.o are used i.e. ha_maria.o is linked into the program, and this brings dependencies of ha_maria.o on mysqld.o into the program's linking which thus fails, as the program is not linked with mysqld.o. We put in this file the functions which maria_chk.c uses, so that they can be reused by maria_read_log (when it replays REDO_REPAIR_TABLE) as they are good enough (they just print to stdout/stderr like maria_read_log already does).
18 years ago
Remove SAFE_MODE for opt_range as it disables UPDATE to use keys REDO optimization (Bascily avoid moving blocks from/to pagecache) More command line arguments to maria_read_log Fixed recovery bug when recreating table sql/opt_range.cc: Remove SAFE_MODE for opt_range as it disables UPDATE to use keys storage/maria/ma_blockrec.c: REDO optimization Use new interface for pagecache_reads to avoid copying page buffers storage/maria/ma_loghandler.c: Patch from Sanja: - Added new parameter to translog_get_page to use direct links to pagecache - Changed scanner to be able to use direct links This avoids a lot of calls to bmove512() in page cache. storage/maria/ma_loghandler.h: Added direct link to pagecache objects storage/maria/ma_open.c: Added const to parameter Added missing braces storage/maria/ma_pagecache.c: From Sanja: - Added direct links to pagecache (from pagecache_read()) Dirrect link means that on pagecache_read we get back a pointer to the pagecache buffer From Monty: - Fixed arguments to init_page_cache to handle big page caches - Fixed compiler warnings - Replaced PAGECACHE_PAGE_LINK with PAGECACHE_BLOCK_LINK * to catch errors storage/maria/ma_pagecache.h: Changed block numbers from int to long to be able to handle big page caches Changed some PAGECACHE_PAGE_LINK to PAGECACHE_BLOCK_LINK storage/maria/ma_recovery.c: Fixed recovery bug when recreating table (table was kept open) Moved some variables to function start (portability) Added space to some print messages storage/maria/maria_chk.c: key_buffer_size -> page_buffer_size storage/maria/maria_def.h: Changed default page_buffer_size to 10M storage/maria/maria_read_log.c: Added more startup options: --version --undo (apply undo) --page_cache_size (to run with big cache sizes) --silent (to not get any output from --apply) storage/maria/unittest/ma_control_file-t.c: Fixed compiler warning storage/maria/unittest/ma_test_loghandler-t.c: Added new argument to translog_init_scanner() storage/maria/unittest/ma_test_loghandler_multigroup-t.c: Added new argument to translog_init_scanner() storage/maria/unittest/ma_test_loghandler_multithread-t.c: Added new argument to translog_init_scanner()
18 years ago
WL#3072 Maria Recovery - new program maria_read_log to display and apply log records found in a Maria log (see file's revision comment) - minor, misc fixes storage/maria/Makefile.am: new program maria_read_log storage/maria/ha_maria.cc: create control file if missing storage/maria/ma_blockrec.c: 0 -> LSN_IMPOSSIBLE; comments storage/maria/ma_checkpoint.h: preparations for Checkpoint module storage/maria/ma_close.c: comment storage/maria/ma_control_file.c: renaming constants. Possibility to say "open control file but don't create it if it's missing" (used by maria_read_log which does not want to create anything) storage/maria/ma_control_file.h: renaming constants storage/maria/ma_create.c: I had duplicated "linkname" and "linkname_ptr", now I see it's not needed, reverting. Indeed those variables don't contain interesting information; fixing log record accordingly (the links are in ci->data/index_file_name). Storing keystart in log record is needed, to know at which size we must extend the file if we replay LOGREC_CREATE_TABLE. storage/maria/ma_loghandler.c: some structures need to be known to maria_read_log.c, taking them to ma_loghandler.h storage/maria/ma_loghandler.h: we have page_store, adding page_korr. translog_lock() made public, because Checkpoint will need it (to write to control file). Some structures moved from ma_loghandler.c because maria_read_log.c needs them (needs to know the execute-in-REDO-phase hooks of each record). storage/maria/ma_loghandler_lsn.h: constants defined in ma_control_file.h serve everywhere, and they relate to LSNs, so putting them in ma_loghandler_lsn.h. Stronger constraints in LSN_VALID(). storage/maria/ma_pagecache.c: renaming constants storage/maria/ma_recovery.h: copyright storage/maria/ma_test1.c: new prototype storage/maria/ma_test2.c: new prototype storage/maria/trnman_public.h: double-inclusion safe storage/maria/unittest/ma_control_file-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multigroup-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multithread-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_pagecache-t.c: constants renamed, new prototype storage/myisam/mi_close.c: comment storage/maria/maria_read_log.c: program to read and print log records from a Maria transaction log, and optionally apply them to tables. Very basic, early version. Should serve as a base for Recovery's code. Designed to be idempotent. Create a log by running maria.test, then cd to var/master-data and run "maria_read_log --only-display" to see info about records; run "maria_read_log --display-and-apply" to also apply the records to tables (it's more interesting if you first wipe out the tables in var/master-data/test, to see how they get re-created). Only a few records are handled by now: LONG_TRANSACTION_ID, COMMIT, FILE_ID, REDO_CREATE_TABLE; place is ready for REDO_INSERT_ROW_HEAD where I could use Monty's help (search for "Monty" in the file). Note: changes to the index pages, index's header and bitmap pages are not properly logged yet, so don't expect the program to work with that.
19 years ago
  1. /* Copyright (C) 2007 MySQL AB
  2. This program is free software; you can redistribute it and/or modify
  3. it under the terms of the GNU General Public License as published by
  4. the Free Software Foundation; version 2 of the License.
  5. This program is distributed in the hope that it will be useful,
  6. but WITHOUT ANY WARRANTY; without even the implied warranty of
  7. MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
  8. GNU General Public License for more details.
  9. You should have received a copy of the GNU General Public License
  10. along with this program; if not, write to the Free Software
  11. Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA */
  12. #include "maria_def.h"
  13. #include "ma_recovery.h"
  14. #include <my_getopt.h>
  15. #define LOG_FLAGS 0
  16. #define LOG_FILE_SIZE (1024L*1024L)
  17. static const char *load_default_groups[]= { "maria_read_log",0 };
  18. static void get_options(int *argc,char * * *argv);
  19. #ifndef DBUG_OFF
  20. #if defined(__WIN__)
  21. const char *default_dbug_option= "d:t:i:O,\\maria_read_log.trace";
  22. #else
  23. const char *default_dbug_option= "d:t:i:o,/tmp/maria_read_log.trace";
  24. #endif
  25. #endif /* DBUG_OFF */
  26. static my_bool opt_only_display, opt_apply, opt_apply_undo, opt_silent,
  27. opt_check;
  28. static ulong opt_page_buffer_size;
  29. static const char *my_progname_short;
  30. int main(int argc, char **argv)
  31. {
  32. LSN lsn;
  33. char **default_argv;
  34. uint warnings_count;
  35. MY_INIT(argv[0]);
  36. my_progname_short= my_progname+dirname_length(my_progname);
  37. load_defaults("my", load_default_groups, &argc, &argv);
  38. default_argv= argv;
  39. get_options(&argc, &argv);
  40. maria_data_root= ".";
  41. maria_in_recovery= TRUE;
  42. if (maria_init())
  43. {
  44. fprintf(stderr, "Can't init Maria engine (%d)\n", errno);
  45. goto err;
  46. }
  47. /* we don't want to create a control file, it MUST exist */
  48. if (ma_control_file_create_or_open())
  49. {
  50. fprintf(stderr, "Can't open control file (%d)\n", errno);
  51. goto err;
  52. }
  53. if (last_logno == FILENO_IMPOSSIBLE)
  54. {
  55. fprintf(stderr, "Can't find any log\n");
  56. goto err;
  57. }
  58. /* same page cache for log and data; assumes same page size... */
  59. DBUG_ASSERT(maria_block_size == TRANSLOG_PAGE_SIZE);
  60. if (init_pagecache(maria_pagecache, opt_page_buffer_size, 0, 0,
  61. TRANSLOG_PAGE_SIZE) == 0)
  62. {
  63. fprintf(stderr, "Got error in init_pagecache() (errno: %d)\n", errno);
  64. goto err;
  65. }
  66. /*
  67. If log handler does not find the "last_logno" log it will return error,
  68. which is good.
  69. But if it finds a log and this log was crashed, it will create a new log,
  70. which is useless. TODO: start log handler in read-only mode.
  71. */
  72. if (translog_init(".", LOG_FILE_SIZE, 50112, 0, maria_pagecache,
  73. TRANSLOG_DEFAULT_FLAGS))
  74. {
  75. fprintf(stderr, "Can't init loghandler (%d)\n", errno);
  76. goto err;
  77. }
  78. if (opt_only_display)
  79. printf("You are using --only-display, NOTHING will be written to disk\n");
  80. /* LSN could be also --start-from-lsn=# */
  81. lsn= translog_first_lsn_in_log();
  82. if (lsn == LSN_ERROR)
  83. {
  84. fprintf(stderr, "Opening transaction log failed\n");
  85. goto end;
  86. }
  87. if (lsn == LSN_IMPOSSIBLE)
  88. {
  89. fprintf(stdout, "The transaction log is empty\n");
  90. }
  91. fprintf(stdout, "The transaction log starts from lsn (%lu,0x%lx)\n",
  92. LSN_IN_PARTS(lsn));
  93. fprintf(stdout, "TRACE of the last maria_read_log\n");
  94. if (maria_apply_log(lsn, opt_apply ? MARIA_LOG_APPLY :
  95. (opt_check ? MARIA_LOG_CHECK :
  96. MARIA_LOG_DISPLAY_HEADER), opt_silent ? NULL : stdout,
  97. opt_apply_undo, FALSE, FALSE, &warnings_count))
  98. goto err;
  99. if (warnings_count == 0)
  100. fprintf(stdout, "%s: SUCCESS\n", my_progname_short);
  101. else
  102. fprintf(stdout, "%s: DOUBTFUL (%u warnings, check previous output)\n",
  103. my_progname_short, warnings_count);
  104. goto end;
  105. err:
  106. /* don't touch anything more, in case we hit a bug */
  107. fprintf(stderr, "%s: FAILED\n", my_progname_short);
  108. exit(1);
  109. end:
  110. maria_end();
  111. free_defaults(default_argv);
  112. my_end(0);
  113. exit(0);
  114. return 0; /* No compiler warning */
  115. }
  116. #include "ma_check_standalone.h"
  117. static struct my_option my_long_options[] =
  118. {
  119. {"apply", 'a',
  120. "Apply log to tables: modifies tables! you should make a backup first! "
  121. " Displays a lot of information if not run with --silent",
  122. (uchar **) &opt_apply, (uchar **) &opt_apply, 0,
  123. GET_BOOL, NO_ARG, 0, 0, 0, 0, 0, 0},
  124. {"check", 'c',
  125. "if --only-display, check if record is fully readable (for debugging)",
  126. (uchar **) &opt_check, (uchar **) &opt_check, 0,
  127. GET_BOOL, NO_ARG, 0, 0, 0, 0, 0, 0},
  128. #ifndef DBUG_OFF
  129. {"debug", '#', "Output debug log. Often the argument is 'd:t:o,filename'.",
  130. 0, 0, 0, GET_STR, OPT_ARG, 0, 0, 0, 0, 0, 0},
  131. #endif
  132. {"help", '?', "Display this help and exit.",
  133. 0, 0, 0, GET_NO_ARG, NO_ARG, 0, 0, 0, 0, 0, 0},
  134. {"only-display", 'o', "display brief info read from records' header",
  135. (uchar **) &opt_only_display, (uchar **) &opt_only_display, 0, GET_BOOL,
  136. NO_ARG,0, 0, 0, 0, 0, 0},
  137. { "page_buffer_size", 'P', "",
  138. (uchar**) &opt_page_buffer_size, (uchar**) &opt_page_buffer_size, 0,
  139. GET_ULONG, REQUIRED_ARG, (long) USE_BUFFER_INIT,
  140. (long) MALLOC_OVERHEAD, (long) ~(ulong) 0, (long) MALLOC_OVERHEAD,
  141. (long) IO_SIZE, 0},
  142. {"silent", 's', "Print less information during apply/undo phase",
  143. (uchar **) &opt_silent, (uchar **) &opt_silent, 0,
  144. GET_BOOL, NO_ARG, 0, 0, 0, 0, 0, 0},
  145. {"undo", 'u', "Apply UNDO records to tables. (disable with --disable-undo)",
  146. (uchar **) &opt_apply_undo, (uchar **) &opt_apply_undo, 0,
  147. GET_BOOL, NO_ARG, 1, 0, 0, 0, 0, 0},
  148. {"version", 'V', "Print version and exit.",
  149. 0, 0, 0, GET_NO_ARG, NO_ARG, 0, 0, 0, 0, 0, 0},
  150. { 0, 0, 0, 0, 0, 0, GET_NO_ARG, NO_ARG, 0, 0, 0, 0, 0, 0}
  151. };
  152. #include <help_start.h>
  153. static void print_version(void)
  154. {
  155. VOID(printf("%s Ver 1.1 for %s on %s\n",
  156. my_progname_short, SYSTEM_TYPE, MACHINE_TYPE));
  157. NETWARE_SET_SCREEN_MODE(1);
  158. }
  159. static void usage(void)
  160. {
  161. print_version();
  162. puts("Copyright (C) 2007 MySQL AB");
  163. puts("This software comes with ABSOLUTELY NO WARRANTY. This is free software,");
  164. puts("and you are welcome to modify and redistribute it under the GPL license\n");
  165. puts("Display and apply log records from a MARIA transaction log");
  166. puts("found in the current directory (for now)");
  167. VOID(printf("\nUsage: %s OPTIONS\n", my_progname_short));
  168. puts("You need to use one of -o or -a");
  169. my_print_help(my_long_options);
  170. print_defaults("my", load_default_groups);
  171. my_print_variables(my_long_options);
  172. }
  173. #include <help_end.h>
  174. static my_bool
  175. get_one_option(int optid __attribute__((unused)),
  176. const struct my_option *opt __attribute__((unused)),
  177. char *argument __attribute__((unused)))
  178. {
  179. switch (optid) {
  180. case '?':
  181. usage();
  182. exit(0);
  183. case 'V':
  184. print_version();
  185. exit(0);
  186. #ifndef DBUG_OFF
  187. case '#':
  188. DBUG_SET_INITIAL(argument ? argument : default_dbug_option);
  189. break;
  190. #endif
  191. }
  192. return 0;
  193. }
  194. static void get_options(int *argc,char ***argv)
  195. {
  196. int ho_error;
  197. if ((ho_error=handle_options(argc, argv, my_long_options, get_one_option)))
  198. exit(ho_error);
  199. if (!opt_apply)
  200. opt_apply_undo= FALSE;
  201. if ((opt_only_display + opt_apply) != 1)
  202. {
  203. usage();
  204. exit(1);
  205. }
  206. }