mariadb

Commit Graph

Author	SHA1	Message	Date
unknown	1bc5e3b971	WL#3072 - Maria recovery maria_read_log used to always print a warning message at startup to say it is unsafe if ALTER TABLE was used. Now it prints it only if the log does show the problem (=ALTER TABLE or CREATE SELECT, which both disable logging of REDO_INSERT). For that, when ha_maria::external_lock() disables transactionality it writes a LOGREC_INCOMPLETE_LOG to the log, which "maria_read_log -a" picks up to write a warning. REPAIR TABLE also disables those REDO_INSERT but as maria_read_log executes LOGREC_REDO_REPAIR no warning is needed. storage/maria/ha_maria.cc: as we now log a record when disabling transactionility, we need the TRN to be set up first storage/maria/ma_blockrec.c: comment storage/maria/ma_loghandler.c: new type of log record storage/maria/ma_loghandler.h: new type of log record storage/maria/ma_recovery.c: * maria_apply_log() now returns a count of warnings. What currently produces warnings is: - skipping applying UNDOs though there are some (=> inconsistent table) - replaying log (in maria_read_log) though the log contains some ALTER TABLE or CREATE SELECT (log misses REDO_INSERT* for those and is so incomplete). Count of warnings affects the final message of maria_read_log and recovery (though in recovery none of the two conditions above should happen). * maria_read_log used to always print a warning message at startup to say it is unsafe if ALTER TABLE was used. Now it prints it only if the log does show the problem, i.e. ALTER TABLE or CREATE SELECT was used (both disable logging of REDO_INSERT* as those records are not needed for recovery; those missing records in turn make recreation-from-scratch, via maria_read_log, impossible). For that, when ha_maria::external_lock() disables transactionality, _ma_tmp_disable_logging_for_table() writes a LOGREC_INCOMPLETE_LOG to the log, which maria_apply_log() picks up to write a warning. storage/maria/ma_recovery.h: maria_apply_log() returns a count of warnings storage/maria/maria_def.h: _ma_tmp_disable_logging_for_table() grows so becomes a function storage/maria/maria_read_log.c: maria_apply_log can now return a count of warnings, to temper the "SUCCESS" message printed in the end by maria_read_log. Advise users to make a backup first.	18 years ago
unknown	08b7152893	WL#3072 Maria recovery - fixes (in recovery, checkpoint, log handler) of bugs found during testing. - new option --check for maria_read_log: with --only-display (which only reads the header) it reads the full record, for debugging. storage/maria/ma_loghandler.c: importing patch from Sanja for bug of translog_next_LSN() found during recovery storage/maria/ma_loghandler_lsn.h: better types (0L is 4 bytes on some platforms, it causes problems when used into lsn_store(): right shift >= width of type. storage/maria/ma_pagecache.c: work around infamous "PAGECACHE_PLAIN_PAGE used for transactional tables in specialm case"; REDO phase disables logging and this causes pages to be PAGECACHE_PLAIN_PAGE, thus ignored wrongly by the checkpoint taken at the end of the REDO phase. storage/maria/ma_recovery.c: - a #ifdef which broke maria_read_log in non-debug builds (no output!) - support for maria_read_log --check - detect record corruption before opening the table - updating is_of_horizon requires writing the state - fix for wrong parsing of checkpoint record by recovery storage/maria/ma_recovery.h: support for maria_read_log --check storage/maria/maria_read_log.c: Option --check: --only-display only looks at the header; adding --check tries a translog_read_record() to see if record can be fully read (this is to find bugs).	18 years ago
unknown	63ff9877a5	WL#3072 Maria recovery Misc changes: - fix for benign Valgrind error, compiler warnings - fix for a segfault in execution of maria_delete_all_rows() and one when taking multiple checkpoints - fix for too paranoid assertion - adding ability to take checkpoints at the end of the REDO phase and at the end of recovery. - other minor changes storage/maria/ha_maria.cc: The checkpoint done after Recovery is finished, is moved to maria_recover(). storage/maria/ma_bitmap.c: fix for Valgrind error: the "shadow debug copy" of the bitmap page started unitialized and so ma_print_bitmap() would use it uninitialized storage/maria/ma_checkpoint.c: * reset pointers to NULL after freeing them, or we segfault at next checkpoint in my_realloc(). * fix for compiler warnings. storage/maria/ma_delete_all.c: info->trn is NULL for non-transactional tables storage/maria/ma_locking.c: correct assertion (it fired wrongly in execution of REDO_DROP_TABLE due to the maria_extra(HA_PREPARE_FOR_DROP)->_ma_decrement_open_count() ->maria_lock_database(F_UNLCK); another solution would have been to not call _ma_decrement_open_count() (it's ok to have a wrong open count in a table which we are dropping), but the same problem would still exist for REDO_RENAME_TABLE. storage/maria/ma_loghandler.c: fail early if UNRECOVERABLE_ERROR storage/maria/ma_recovery.c: * new argument to maria_apply_log(): should it take checkpoints (at end of REDO phase and at the very end) or no. * moving the call to translog_next_LSN() into parse_checkpoint_record() ("hide the details"). * Refining an error detection for something which could happen if there is a checkpoint record in the log. * Using close_one_table() instead of maria_extra(HA_EXTRA_PREPARE_FOR_DROP\|RENAME), as it looks safer, and also changing how close_one_table() works: it now limits itself to scanning all_tables[], thus having one loopp instead of two, which should be faster (as a result, it does not close tables not registered in this array, which is ok as there should not be any). storage/maria/ma_recovery.h: new parameter storage/maria/maria_read_log.c: update to new prototype	18 years ago
unknown	24db7ed7e1	WL#3072 Maria recovery. * Thanks to Serg's tip, we fix here the compilation issue of REDO_REPAIR_TABLE's execution, by defining versions of _ma_killed_ptr() and _ma_check_print_info\|warning\|error() in maria_read_log.c (we move those of maria_chk.c into an include file and include it in maria_chk.c and maria_read_log.c). Execution of such record looks like working from my tests (it only happens in maria_read_log; recovery-from-mysqld skips DDLs and REPAIR is considered DDL here as it bypasses logging): tested ALTER TABLE ENABLE KEYS and then remove table, apply log: that did a repair. * Recent changes broke maria_read_log a bit: -a became default and -o caused error; fixing this. storage/maria/Makefile.am: addind new file storage/maria/ma_recovery.c: enable execution of REDO_REPAIR_TABLE by maria_read_log now that it compiles. Now reason to keep only T_QUICK from testflag. storage/maria/maria_chk.c: moving these functions to ma_check_standalone.h for reusability storage/maria/maria_def.h: comment storage/maria/maria_read_log.c: ma_check_standalone.h needs my_progname_short. Fixing bug where "maria_read_log" would default to -a and "maria_read_log -o" would throw an error. Implemented behaviour is: - no options: usage() - -a : applys, applys UNDOs by default unless --disable-undo - -o : only prints storage/maria/ma_check_standalone.h: All standalone programs which need to use functions from ma_check.c (like maria_repair()) must define their version of _ma_killed_ptr() and _ma_check_print_info\|warning\|error(). Indeed, linking with ma_check.o brings in the dependencies of ma_check.o which are definitions of the above functions; if the program does not define them then the ones of ha_maria.o are used i.e. ha_maria.o is linked into the program, and this brings dependencies of ha_maria.o on mysqld.o into the program's linking which thus fails, as the program is not linked with mysqld.o. We put in this file the functions which maria_chk.c uses, so that they can be reused by maria_read_log (when it replays REDO_REPAIR_TABLE) as they are good enough (they just print to stdout/stderr like maria_read_log already does).	18 years ago
unknown	c9c58163a9	Remove SAFE_MODE for opt_range as it disables UPDATE to use keys REDO optimization (Bascily avoid moving blocks from/to pagecache) More command line arguments to maria_read_log Fixed recovery bug when recreating table sql/opt_range.cc: Remove SAFE_MODE for opt_range as it disables UPDATE to use keys storage/maria/ma_blockrec.c: REDO optimization Use new interface for pagecache_reads to avoid copying page buffers storage/maria/ma_loghandler.c: Patch from Sanja: - Added new parameter to translog_get_page to use direct links to pagecache - Changed scanner to be able to use direct links This avoids a lot of calls to bmove512() in page cache. storage/maria/ma_loghandler.h: Added direct link to pagecache objects storage/maria/ma_open.c: Added const to parameter Added missing braces storage/maria/ma_pagecache.c: From Sanja: - Added direct links to pagecache (from pagecache_read()) Dirrect link means that on pagecache_read we get back a pointer to the pagecache buffer From Monty: - Fixed arguments to init_page_cache to handle big page caches - Fixed compiler warnings - Replaced PAGECACHE_PAGE_LINK with PAGECACHE_BLOCK_LINK * to catch errors storage/maria/ma_pagecache.h: Changed block numbers from int to long to be able to handle big page caches Changed some PAGECACHE_PAGE_LINK to PAGECACHE_BLOCK_LINK storage/maria/ma_recovery.c: Fixed recovery bug when recreating table (table was kept open) Moved some variables to function start (portability) Added space to some print messages storage/maria/maria_chk.c: key_buffer_size -> page_buffer_size storage/maria/maria_def.h: Changed default page_buffer_size to 10M storage/maria/maria_read_log.c: Added more startup options: --version --undo (apply undo) --page_cache_size (to run with big cache sizes) --silent (to not get any output from --apply) storage/maria/unittest/ma_control_file-t.c: Fixed compiler warning storage/maria/unittest/ma_test_loghandler-t.c: Added new argument to translog_init_scanner() storage/maria/unittest/ma_test_loghandler_multigroup-t.c: Added new argument to translog_init_scanner() storage/maria/unittest/ma_test_loghandler_multithread-t.c: Added new argument to translog_init_scanner()	18 years ago
unknown	9c2ff270fa	WL#3072 Maria Recovery * recovery from ha_maria now skips replaying DDLs (too dangerous) * maria_read_log still replays DDLs, print warning about issues * fixes to replaying of REDO_RENAME * don't replay DDLs on corrupted tables (safer) * print a one-line message when really doing a recovery (applies to ha_maria, not maria_read_log) i.e. some REDOs or UNDOs are read. storage/maria/ma_checkpoint.c: fix for assertion failure storage/maria/ma_recovery.c: * Recovery from ha_maria now skips replaying DDLs (as the initial plan said) as this is unsafe in case of crashes during the DDL; applying the records may do harm (destroy important files) so we prefer to leave the "mess" of files untouched. A proper recovery of DDLs requires very careful thinking, probably testing separately the existence of the data and index file instead of using maria_open() which tests the existence of both, and maybe storing create_rename_lsn in the data file too. * maria_read_log still replays DDLs, we print a warning about dangers (due to ALTER TABLE not logging insertions into the tmp table; we will maybe need an option to have logging of those insertions). * fixes to replaying of REDO_RENAME (test create_rename_lsn of 'new_name' table if it exists; if that table exists and is more recent than the record, remove the 'old_name' table). * don't replay DDLs on corrupted tables (play safe) * fail also in non-debug builds if table is open when it should not be (when creating it for example, it should not be already open). * when the trace file is not stdout (i.e. when this is ha_maria), if really doing a recovery (reading REDOs or UNDOs), print a one-line message to stderr to inform about start and end of recovery (useful to know what mysqld is doing, especially if it takes long or crashes). storage/maria/ma_recovery.h: parameter to replay DDLs or not storage/maria/maria_read_log.c: replay DDLs in maria_read_log, to be able to recreate tables from scratch.	19 years ago
unknown	a303f5b2c8	Fixes of the empty log problem. storage/maria/ma_checkpoint.c: The new macro for easier printing LSN added. storage/maria/ma_loghandler.c: The assertion returned. The new macro for easier printing LSN added. storage/maria/ma_loghandler_lsn.h: The new macro for easier printing LSN added. storage/maria/ma_pagecache.c: The new macro for easier printing LSN added. storage/maria/ma_recovery.c: Recovery checks empty log state. RECHEADER_READ_ERROR means some real error. storage/maria/maria_read_log.c: Read log starts from real beggining of the log and precess error and empty log states. The new macro for easier printing LSN added. storage/maria/unittest/ma_test_loghandler-t.c: The new macro for easier printing LSN added. storage/maria/unittest/ma_test_loghandler_first_lsn-t.c: The new macro for easier printing LSN added. storage/maria/unittest/ma_test_loghandler_max_lsn-t.c: The new macro for easier printing LSN added. storage/maria/unittest/ma_test_loghandler_multigroup-t.c: The new macro for easier printing LSN added. storage/maria/unittest/ma_test_loghandler_multithread-t.c: The new macro for easier printing LSN added. storage/maria/unittest/ma_test_loghandler_noflush-t.c: The new macro for easier printing LSN added.	19 years ago
unknown	daf62687dc	fix a typo in #ifdef	19 years ago
unknown	155193a6e2	Added applying of undo for updates Fixed bug in duplicate key handling for block records during repair All read-row methods now return error number in case of error Don't calculate checksum for null fields Fixed bug when running maria_read_log with -o BUILD/SETUP.sh: Added STACK_DIRECTION BUILD/compile-pentium-debug-max: Moved STACK_DIRECTION to SETUP include/myisam.h: Added extra parameter to write_key storage/maria/ma_blockrec.c: Added applying of undo for updates Fixed indentation Removed some not needed casts Fixed wrong logging of CLR record Split ma_update_block_record to two functions to be able to reuse it from undo-applying Simplify filling of packed fields ma_record_block_record) now returns error number on failure Sligtly changed log record information for undo-update storage/maria/ma_check.c: Fixed bug in duplicate key handling for block records during repair storage/maria/ma_checksum.c: Don't calculate checksum for null fields storage/maria/ma_dynrec.c: _ma_read_dynamic_reocrd() now returns error number on error Rest of the changes are code simplification and indentation fixes storage/maria/ma_locking.c: Added comment storage/maria/ma_loghandler.c: More debugging Removed printing of total_record_length as this was always same as record_length storage/maria/ma_open.c: Allocate bitmap for changed fields storage/maria/ma_packrec.c: read_record now returns error number on error storage/maria/ma_recovery.c: Fixed wrong arguments to undo_row_update storage/maria/ma_statrec.c: read_record now returns error number on error (not 1) Code simplification storage/maria/ma_test1.c: Added exit possibility after update phase (to test undo of updates) storage/maria/maria_def.h: Include bitmap header file storage/maria/maria_read_log.c: Fixed bug when running with -o	19 years ago
unknown	ac4ad9bdba	WL#3072 Maria Recovery misc fixes of execution of UNDOs in the UNDO phase: - into the CLR_END, store the LSN of the _previous_ UNDO (we debated what was best, so far we're going with "previous"; later we can change to "current" if needed), and store the type of record which is being undone (needed to know how to update state.records when we see the CLR_END during the REDO phase). - declaring all UNDOs and CLR_END as "compressed" - when executing an UNDO in the UNDO phase, state.records is updated as a hook when writing CLR_END (needed for "recovery of the state"), and so is trn->undo_lsn (needed for when we have checkpoints). - bugfix (execution of UNDO_ROW_DELETE didn't store the correct checksum into the re-inserted row, maria_chk -r thus threw the row away). - modifications of ma_test1: where to stop is now driven by --testflag; --test-undo just tells how to stop (flush data, flush log, nothing). - ma_test_recovery: testing of the UNDO phase, more testing of the REDO phase, identification of a bug. storage/maria/ma_blockrec.c: - bugfix: execution of UNDO_ROW_DELETE didn't store the correct checksum into the row (leading to "maria_chk -r" eliminating the re-inserted row, net effect was that rollback appeared to have rolled back no deletion). Reason was that write_block_record() used info->cur_row.checksum, while "row" can be != &info->cur_row (case of UNDO_ROW_DELETE). After fixing this, problems with _ma_update_block_record() appeared; indeed checksum was computed by allocate_and_write_block_record() while _ma_update_block_record() directly calls write_block_record(). Solution is to compute checksum in write_block_record() instead. - when executing an UNDO, we now pass the LSN of the _previous_ UNDO to block_format functions. This LSN can be 0 (if the being-executed UNDO was the transaction's first UNDO), so "undo_lsn==0" cannot work anymore to indicate "this is not UNDO work". Using undo_lsn==LSN_ERROR instead (this is an impossible LSN). - store into CLR_END the type of log record which was undone (INSERT/UPDATE/DELETE); needed for Recovery to know if/how it has to update state.records if it sees this CLR_END in the REDO phase. - when writing the CLR_END in _ma_apply_undo_row_insert(), the place to store file's id is log_data+LSN_STORE_SIZE. - in _ma_apply_undo_row_insert(), the records-- is moved to a hook when writing the CLR_END (this way it is under log's mutex which is needed for "recovery of the state") storage/maria/ma_loghandler.c: - all UNDOs, and CLR_END, start with the LSN of another UNDO; so we can declare them "compressed". - write_hook_for_clr_end() to set trn->undo_lsn (to the previous UNDO's LSN) under log's lock (like UNDOs set trn->undo_lsn under log's lock), and also update, if appropriate, state.records. - reset share->id to 0 when deassigning; not useful for now but sounds logical. storage/maria/ma_recovery.c: - if no table is found for a REDO, it's not an error; for an UNDO, it is - in the REDO phase, when we see a CLR_END we must update trn->undo_lsn and sometimes state.records. - in the UNDO phase, when we execute an UNDO_ROW_INSERT: * update trn->undo_lsn only after executing the record * store the _previous_ undo_lsn into the CLR_END - at the end of the REDO phase, when we recreate TRN objects, they have already their long id in the log (either via a LOGREC_LONG_TRANSACTION_ID, or in a checkpoint record), don't write a new, useless LOGREC_LONG_TRANSACTION_ID for them. storage/maria/ma_test1.c: * where to stop execution is now driven by --testflag and not --test-undo (ma_test2 already has --testflag for the same purpose). This allows us to do a clean stop (with commit) at any point. * --test-undo=# tells how to abort (flush all pages (which implies flushing log) or only log or nothing); all such "ways of crashing" are tested in ma_test_recovery storage/maria/ma_test_recovery: * Testing execution of UNDOs, with and without BLOBs. * Testing idempotency of REDOs. * See @todo for a probable bug with BLOBs. * maria_chk -rq instead of -r, as with -q it nicely stops on any problem in the data file (like the checksum bug see comment of ma_blockrec.c). * Testing if log was written by UNDO phase (often expected), not written by REDO phase (always expected). * Less output on the screen, compares with expected output in the end. * some shell thingies like "set --" and $# are courtesy of Danny and Pekka. storage/maria/maria_read_log.c: when only displaying the records, don't do an UNDO phase storage/maria/ma_test_recovery.expected: This is the expected output of a great part of ma_test_recovery. ma_test_recovery compares its output to the expected output and tells if different. If we look at this file it mentions differences in checksum (normal, it's not recovered yet) and in records count (getting a correct records' count when recovery starts on an already existing table, like when testing rollback, is coded but not yet pushed).	19 years ago
unknown	03437ea043	Added UNDO handling of insert during recovery storage/maria/ma_blockrec.c: Added UNDO handling of insert during recovery To do this, I also had to add write locking of tail pages during undo phase (As we need to access the same page twice if extents are split over two pages) Another way to handle the undo of insert would be to store the extent information as part of the UNDO_INSERT block. storage/maria/ma_blockrec.h: Added new prototype storage/maria/ma_loghandler.c: Changed type of CLR_END (to avoid crash in log handler) Removed not used variable storage/maria/ma_loghandler.h: Added TRN argument to record_execute_in_undo_phase() storage/maria/ma_pagecache.c: Hack for undo phase of recovery. During REDO we work with PLAIN pages, but UNDO works with LSN pages, which caused an abort when trying to access a cached page. storage/maria/ma_recovery.c: Added execution of UNDO_ROW_INSERT storage/maria/ma_test1.c: Added option --test-undo for testing recovery with undo storage/maria/maria_read_log.c: Added processing of undos	19 years ago
unknown	e27890cab0	WL#3072 Maria recovery * create page cache before initializing engine and not after, because Maria's recovery needs a page cache * make the creation of a bitmap page more crash-resistent * bugfix (see ma_blockrec.c) * back to old way: create an 8k bitmap page when creating table * preparations for the UNDO phase: recreate TRNs * preparations for Checkpoint: list of dirty pages, testing of rec_lsn to know if page should be skipped during Recovery (unused in this patch as no Checkpoint module pushed yet) * maria_chk tags repaired table with a special LSN * reworking all around in ma_recovery.c (less duplication) mysys/my_realloc.c: noted an issue in my_realloc() sql/mysqld.cc: page cache needs to be created before engines are initialized, because Maria's initialization may do a recovery which needs the page cache. storage/maria/ha_maria.cc: update to new prototype storage/maria/ma_bitmap.c: when creating the first bitmap page we used chsize to 8192 bytes then pwrite (overwrite) the last 2 bytes (8191-8192). If crash between the two operations, this leaves a bitmap page full without its end marker. A later recovery may try to read this page and find it exists and misses a marker and conclude it's corrupted and fail. Changing the chsize to only 8190 bytes: recovery will then find the page is too short and recreate it entirely. storage/maria/ma_blockrec.c: Fix for a bug: when executing a REDO, if the data page is created, data_file_length was increased before _ma_bitmap_set(): _ma_bitmap_set() called _ma_read_bitmap_page() which, due to the increased data_file_length, expected to find a bitmap page on disk with a correct end marker; if the bitmap page didn't exist already in fact, this failed. Fixed by increasing data_file_length only after _ma_read_bitmap_page() has created the new bitmap page correctly. This bug could happen every time a REDO is about creating a new bitmap page. storage/maria/ma_check.c: empty data file has a bitmap page storage/maria/ma_control_file.c: useless parameter to ma_control_file_create_or_open(), just test if this is recovery. storage/maria/ma_control_file.h: new prototype storage/maria/ma_create.c: Back to how it was before: maria_create() creates an 8k bitmap page. Thus (bugfix) data_file_length needs to reflect this instead of being 0. storage/maria/ma_loghandler.c: as ma_test1 and ma_test2 now use real transactions and not dummy_transaction_object, REDO for INSERT/UPDATE/DELETE are always about real transactions, can assert this. A function for Recovery to assign a short id to a table. storage/maria/ma_loghandler.h: new function storage/maria/ma_loghandler_lsn.h: maria_chk tags repaired tables with this LSN storage/maria/ma_open.c: * enforce that DMLs on transactional tables use real transactions and not dummy_transaction_object. * test if table was repaired with maria_chk (which has to been seen as an import of an external table into the server), test validity of create_rename_lsn (header corruption detection) * comments. storage/maria/ma_recovery.c: * preparations for the UNDO phase: recreate TRNs * preparations for Checkpoint: list of dirty pages, testing of rec_lsn to know if page should be skipped during Recovery (unused in this patch as no Checkpoint module pushed yet) * reworking all around (less duplication) storage/maria/ma_recovery.h: a parameter to say if the UNDO phase should be skipped storage/maria/maria_chk.c: tag repaired tables with a special LSN storage/maria/maria_read_log.c: * update to new prototype * no UNDO phase in maria_read_log for now storage/maria/trnman.c: * a function for Recovery to create a transaction (TRN), needed in the UNDO phase * a function for Recovery to grab an existing transaction, needed in the UNDO phase (rollback all existing transactions) storage/maria/trnman_public.h: new functions	19 years ago
unknown	f7b766c029	Added maria_commit() and maria_begin() to be used with external tests Now ma_test1 -M -T and ma_test2 -M -T produces readable, applyable logs Note: The .MAD file is not binary identical after applying redo compare to a an original file. (This is becasue we don't have full information which function called PURGE_REDO_BLOCKS). To verify if a file was correctly applied, we now instead compare row checksums BitKeeper/etc/ignore: added storage/maria/tmp/* include/maria.h: Added maria_commit() and maria_begin() to be used with external tests storage/maria/ha_maria.cc: Ensure maria_def. is read in C mode storage/maria/ma_blockrec.c: Fixed redo handling. _ma_apply_redo_purge_blocks() updated to handle any number of purged blocks Removed code to make data file idenitcal after redo (can't easily be done). See changeset comments Now ma_test1 -M -T and ma_test2 -M -T produces readable, applyable logs storage/maria/ma_commit.c: More DBUG statements Moved variable declaration to start of function (portability fix) Added helper functions 'maria_commit()' and 'maria_begin()' storage/maria/ma_loghandler.c: Fixed wrong REDO_PURGE_BLOCKS initialization storage/maria/ma_recovery.c: Added UNDO_ROW_UPDATE Removed wrong setting of lsn (there was no lsn at the used position) Fixed REDO_PURGE_BLOCKS to handle any number of blocks storage/maria/ma_test1.c: Added transaction support (via maria_begin() & maria_commit()) to get a log that can be applied with maria_read_log storage/maria/ma_test2.c: Added transaction support (via maria_begin() & maria_commit()) to get a log that can be applied with maria_read_log storage/maria/ma_test_recovery: Create temporary files in maria/tmp Verify files with checksums instead of byte comparisons storage/maria/maria_chk.c: When using with -dss we only get filename, records and checksum. This is useful to do a quick comparision if a files is identical to another one. storage/maria/maria_def.h: Added ma_commit() storage/maria/maria_read_log.c: Added --help	19 years ago
unknown	4cf6756eb0	First LSN calls added for transaction log. storage/maria/ma_checkpoint.c: Definitions of LSN should be collected in the one file (ma_loghandler_lsn.h) storage/maria/ma_loghandler.c: New calls to get first theoretical and first real LSN. storage/maria/ma_loghandler.h: New calls to get first theoretical and first real LSN. storage/maria/ma_loghandler_lsn.h: Defined yet another impossible LSN to indicate error. storage/maria/ma_recovery.c: The first LSN call changed. storage/maria/maria_read_log.c: The first LSN call changed. storage/maria/unittest/Makefile.am: New unittest added. storage/maria/unittest/ma_test_loghandler_first_lsn-t.c: New BitKeeper file ``storage/maria/unittest/ma_test_loghandler_first_lsn-t.c''	19 years ago
unknown	2cccfcd8dd	Applying Sanja's patch which makes the log handler not issue errors when reading a log record which has a 0-length header (like LOGREC_REDO_DROP_TABLE). storage/maria/ma_loghandler.c: Functions reading record's header now don't use 0 to indicate error, as some valid records have a 0-length header (like REDO_DROP_TABLE). Instead, negative values are used for EOF and error. storage/maria/ma_loghandler.h: functions to read record's header now return an int (either the length of this header (>=0) or some negative values for EOF or error). storage/maria/ma_recovery.c: update to the new log handler's behaviour. Note the @todo. storage/maria/maria_read_log.c: inform when program failed storage/maria/unittest/ma_test_loghandler-t.c: update to new log handler's API storage/maria/unittest/ma_test_loghandler_multigroup-t.c: update to new log handler's API storage/maria/unittest/ma_test_loghandler_multithread-t.c: update to new log handler's API	19 years ago
unknown	662002fc8f	post-merge fixes, and fixes for some of the 16 compiler warnings found in pushbuild on sapsrv1. Some not fixed as not repeatable on my machine (32/64 bit issue?). Fixes for some test failures: - "maria-connect" now passes; - "maria": after fixing the obvious reasons for failures, the test went further and hit a more complex issues: difference in the output of EXPLAIN output; not fixed; - "ps_maria" still crashes in assertion mysqld: ha_maria.cc:1627: virtual int ha_maria::index_read(uchar, const uchar, uint, ha_rkey_function): Ass ertion `inited == INDEX' failed, as already observable in pushbuild. All this might just be due to an incomplete merge of MyISAM changes into Maria when 5.1 was last merged to mysql-maria. include/my_global.h: temporary fix until next merge of 5.1; without this it does not build mysql-test/r/maria-connect.result: position changed mysql-test/t/maria-connect.test: If one wants to use the binlog it has to ask for it. 1582 is not used for dup entry error anymore (it was in older 5.1). Size of first event in binlog was increased by 4 (when the new type of event "gap" was added). mysql-test/t/maria.test: 1582 not used anymore in this case storage/maria/ha_maria.cc: engine now has to say what binlogging it supports storage/maria/ma_blockrec.c: fix for compiler warnings ("comparison is always true" or "always false") storage/maria/ma_loghandler.c: fix for compiler warnings (comparing char* to uchar) storage/maria/ma_packrec.c: fix for compiler warning (fix simply merged from MyISAM) storage/maria/ma_pagecache.c: info_check_pin() was not used so gave a compiler warning. storage/maria/ma_pagecache.h: fixing typo from the last 5.1->maria merge. storage/maria/ma_recovery.c: my_free() has a void argument, so why cast. byte->uchar. storage/maria/ma_search.c: fix for compiler warning (fix simply merged from MyISAM) storage/maria/maria_read_log.c: gptr->uchar* storage/maria/trnman.c: probable fix for warning found in pushbuild (but not on my machine): storage/maria/trnman.c: 142 passing argument 6 of \u2018lf_hash_init\u2019 from incompatible pointer type on sapsrv1.	19 years ago
unknown	6bbca54d7d	WL#3072 - Maria recovery Unit test for recovery: runs ma_test1 and ma_test2 (both only with INSERTs and DELETEs; UPDATEs disabled as not handled by recovery) then moves the tables elswhere; recreates tables from the log, and compares and fails if there is a difference. Passes now. Most of maria_read_log.c moved to ma_recovery.c, as it will be re-used for recovery-from-ha_maria. Bugfixes of applying of REDO_INSERT, REDO_PURGE_ROW. Applying of REDO_PURGE_BLOCKS, REDO_DELETE_ALL, REDO_DROP_TABLE, UNDO_ROW_INSERT (in REDO phase only, i.e. just doing records++), UNDO_ROW_DELETE, UNDO_ROW_PURGE. Code cleanups. Monty: please look for "QQ". Sanja: please look for "Sanja". Future tasks: recovery of the bitmap (easy), recovery of the state (make it idempotent), more REDOs (Monty to work on REDO_UPDATE?), UNDO phase... Pushing this cset as it looks safe, contains test and bugfixes which will help Monty implement applying of REDO_UPDATE. sql/handler.cc: typo storage/maria/Makefile.am: Adding ma_test_recovery (which ma_test_all invokes, and which can also be run alone). Most of maria_read_log.c moved to ma_recovery.c storage/maria/ha_maria.cc: comments storage/maria/ma_bitmap.c: fixing comments. 2 -> sizeof(maria_bitmap_marker). Bitmap-related part of _ma_initialize_datafile() moves in bitmap module. Now putting the "bm" signature when creating the first bitmap page (it used to happen only at next open, but that caused an annoying difference when testing Recovery if the original run didn't open the table, and it looks more logical like this: it goes to disk only with its signature correct); see the "QQ" comment towards the _ma_initialize_data_file() call in ma_create.c for more). When reading a bitmap page, verify its signature (happens when normally using the table or when CHECKing it; not when REPAIRing it). storage/maria/ma_blockrec.c: * no need to sync the data file if table is not transactional * Comments, code cleanup (log-related data moved to log-related code block, int5store->page_store). * Store the table's short id into LOGREC_UNDO_ROW_PURGE, like we do for other records (though this record will soon be replaced with a CLR). * If "page" is 1 it means the page which extends from byte pageblock_size+1 to (page+1)block_size (byte number 1 being the first byte of the file). The last byte of the file is data_file_length (same convention). A new page needs to be created if the last byte of the page is beyond the last byte of the file, i.e. (page+1)block_size+1 > data_file_length, so we correct the test (bug found when testing log applying for ma_test1 -M -T --skip-update). update the page's LSN when removing a row from it during execution of a REDO_PURGE_ROW record (bug found when testing log applying for ma_test1 -M -T --skip-update). * applying of REDO_PURGE_BLOCKs (limited to a one-page range for now). storage/maria/ma_blockrec.h: new functions. maria_bitmap_marker does not need to be exported. storage/maria/ma_close.c: we can always flush the table's state when closing the last instance of the table. And it is needed for maria_read_log (as it does not use maria_lock_database()). storage/maria/ma_control_file.c: when in Recovery, some assertions should not be used. storage/maria/ma_control_file.h: double-inclusion safe storage/maria/ma_create.c: during recovery, don't log records. Comments. Moving the creation of the first bitmap page to ma_bitmap.c storage/maria/ma_delete_table.c: during recovery, don't log records. Log the end-zero of the dropped table's name, so that recovery can use the string in place without extending it to fit an end zero. storage/maria/ma_loghandler.c: * inwrite_rec_hook also needs access to the MARIA_SHARE, like prewrite_rec_hook. This will be needed to update share->records_diff (in the upcoming patch "recovery of the state"). * LOG_DESC::record_ends_group changed to an enum. * LOG_DESC for LOGREC_REDO_PURGE_BLOCKS and LOGREC_UNDO_ROW_PURGE corrected * Sanja please see the @todo LOG BUG * avoiding DBUG_RETURN(func()) as it gives confusing debug traces. storage/maria/ma_loghandler.h: - log write hooks called while the log's lock is held (inwrite_rec_hook) now need the MARIA_SHARE, like prewrite_rec_hook already had - instead of a bool saying if this record's type ends groups or not, we refine: it may not end a group, it may end a group, or it may be a group in itself. Imagine that we had a physical write failure to a table before we log the UNDO, we still end up in external_lock(F_UNLCK) and then we log a COMMIT: we don't want to consider this COMMIT as ending the group of REDOs (don't want to execute those REDOs during Recovery), that's why we say "COMMIT is a group in itself, it aborts any previous group". This also gives one more sanity check in maria_read_log. storage/maria/ma_recovery.c: New Recovery code, replacing the old pseudocode. Most of maria_read_log moved here. Call-able from ha_maria, but not enabled yet. Compared to the previous version of maria_read_log, some bugs have been fixed, debugging output can go to stdout or a disk file (for now it's useful for me, later it can be changed), execution of REDO_DROP_TABLE, REDO_DELETE_ALL, REDO_PURGE_BLOCKS has been added. Duplicate code has been factored into functions. We abort an unfinished group of records if we see a record which is a group in itself (like COMMIT). No need for maria_panic() after a bug (which caused tables to not be closed) was fixed; if there is yet another bug I prefer to see it. When opening a table for Recovery, set data_file_length and key_file_length to their real physical value (these are the easiest state members to restore :). Warn us if the last page was truncated (but Recovery handles it). MARIA_SHARE::state::state::records is now partly recovered (not idempotent, but works if recreating tables from scracth). When applying a REDO to a page, stamp it with the UNDO's LSN (current_group_end_lsn), not with the REDO's LSN; it makes the table more identical to the original table (easier to compare the two tables in the end). Big thing missing: some types of REDOs are not handled, and the UNDO phase does not exist (missing functions to execute UNDOs to actually rollback). So for now tests are only inserting/deleting a few 100 rows, closing the table and seeing if the log is applied ok; it works. UPDATE not handled. storage/maria/ma_recovery.h: new functions: ma_recover() for recovery from inside ha_maria; _ma_apply_log() for maria_read_log (ma_recover() calls _ma_apply_log()). Btw, we need to not use the word "recover" for REPAIR/maria_chk anymore. storage/maria/ma_rename.c: don't write log records during recovery storage/maria/ma_test2.c: - fail if maria_info() or other subtests find some wrong information - new option -g to skip updates. - init the translog before creating the table, so that log applying can work. - in "#if 0" you'll see some fixed bugs (will be removed). storage/maria/ma_test_all.sh: cleanup files. Test log applying. storage/maria/maria_read_log.c: most of the logic moves to ma_recovery.c to be shared between maria_read_log and recovery-from-inside-mysqld. See ma_recovery.c for additional changes made to the moved code. storage/maria/ma_test_recovery: unit test for Recovery. Tests insert and delete, REDO_UPDATE not yet coded. Script is called from ma_test_all. Can run standalone.	19 years ago
unknown	6ed46be70e	Fixed bytes to uchar and gptr to uchar*	19 years ago
unknown	85da513341	Implement applying of REDO entries for - LOGREC_REDO_INSERT_ROW_HEAD - LOGREC_REDO_INSERT_ROW_TAIL - LOGREC_REDO_PURGE_ROW_HEAD - LOGREC_REDO_PURGE_ROW_TAIL sql/sql_yacc.yy: Fixed typo in previous push storage/maria/ma_bitmap.c: Ensure we flush the new bitmap on close storage/maria/ma_blockrec.c: Implement applying of REDO entries for - LOGREC_REDO_INSERT_ROW_HEAD - LOGREC_REDO_INSERT_ROW_TAIL - LOGREC_REDO_PURGE_ROW_HEAD - LOGREC_REDO_PURGE_ROW_TAIL Split some functions into subfunctions to be able to reuse code storage/maria/ma_blockrec.h: Added prototypes for REDO applying functions storage/maria/ma_loghandler.h: Safety fix storage/maria/ma_loghandler_lsn.h: Avoid compiler warnings storage/maria/maria_read_log.c: Added hocks for: - REDO_INSERT_ROW_HEAD - REDO_INSERT_ROW_TAIL - REDO_PURGE_ROW_HEAD - REDO_PURGE_ROW_TAIL Added dummy hooks for: - UNDO_ROW_INSERT - UNDO_ROW_DELETE Changed to use maria_pagecache instead of own pagecache (fixed problem with unitialized share->pagecache) Use maria_panic() at end to ensure that all files are closed properly. Fixed option handling for --debug	19 years ago
unknown	388122558c	Maria: * Don't modify share->base.born_transactional; now it is a value carved in stone at creation time. share->now_transactional is what can be modified: it starts at born_transactional, can become false during ALTER TABLE (when we want no logging), and restored later. * Not resetting create_rename_lsn to 0 during delete_all or repair. * when we temporarily disable transactionality, we also change the page type to PAGECACHE_PLAIN_PAGE: it bypasses some work in the page cache (optimization), and avoids assertions related to LSNs. * Disable INSERT DELAYED for transactional tables, because durability could not be guaranteed (insertion may even not happen) mysys/mf_keycache.c: comment storage/maria/ha_maria.cc: * a transactional table cannot do INSERT DELAYED * ha_maria::save_transactional not needed anymore, as now instead we don't modify MARIA_SHARE::MARIA_BASE_INFO::born_transactional (born_transactional plays the role of save_transactional), and modify MARIA_SHARE::now_transactional. * REPAIR_TABLE log record is now logged by maria_repair() * comment why we rely on born_transactional to know if we should skipping a transaction. * putting together two if()s which test for F_UNLCK storage/maria/ha_maria.h: ha_maria::save_transactional not needed anymore (moved to the C layer) storage/maria/ma_blockrec.c: * For the block record's code (writing/updating/deleting records), all that counts is now_transactional, not born_transactional. * As we now set the page type to PAGECACHE_PLAIN_PAGE for tables which have now_transactional==FALSE, pagecache will not expect a meaningful LSN for them in pagecache_unlock_by_link(), so we can pass it LSN_IMPOSSIBLE. storage/maria/ma_check.c: * writing LOGREC_REPAIR_TABLE moves from ha_maria::repair() to maria_repair(), sounds cleaner (less functions to export). * when opening a table during REPAIR, don't use the realpath-ed name, as this may fail if the table has symlinked files (maria_open() would try to find the data and index file in the directory of unique_file_name, it would fail if data and index files are in different dirs); use the unresolved name, open_file_name, which is the argument which was passed to the maria_open() which created 'info'. storage/maria/ma_close.c: assert that when a statement is done with a table, it cleans up storage/maria/ma_create.c: new name storage/maria/ma_delete_all.c: * using now_transactional * no reason to reset create_rename_lsn during delete_all (a bug); also no reason to do it during repair: it was put there because a positive create_rename_lsn caused a call to check_and_set_lsn() which asserted in DBUG_ASSERT(block->type == PAGECACHE_LSN_PAGE); first solution was to use LSN_IMPOSSIBLE in _ma_unpin_all_pages() if not transactional; but then in the case of ALTER TABLE, with transactionality temporarily disabled, it asserted in DBUG_ASSERT(LSN_VALID(lsn)) in pagecache_fwrite() (PAGECACHE_LSN_PAGE page with zero LSN - bad). The additional solution is to use PAGECACHE_PLAIN_PAGE when we disable transactionality temporarily: this avoids checks on the LSN, and also bypasses (optimization) the "flush log up to LSN" call when the pagecache flushes our page (in other words, no WAL needed). storage/maria/ma_delete_table.c: use now_transactional storage/maria/ma_locking.c: assert that when a statement is done with a table, it cleans up. storage/maria/ma_loghandler.c: * now_transactional should be used to test if we want a log record. * Assertions to make sure dummy_transaction_object is not spoilt by its many users. storage/maria/ma_open.c: base.transactional -> base.born_transactional storage/maria/ma_pagecache.c: missing name for page's type. Comment for future. storage/maria/ma_rename.c: use now_transactional storage/maria/maria_chk.c: use born_transactional storage/maria/maria_def.h: MARIA_BASE_INFO::transactional renamed to born_transactional. MARIA_SHARE::now_transactional introduced. _ma_repair_write_log_record() is made local to ma_check.c. Macros to temporarily disable, and re-enable, transactionality for a table. storage/maria/maria_read_log.c: assertions and using the new macros. Adding a forgotten resetting when we finally close all tables.	19 years ago
unknown	ef7a757b7c	After merge fixes BitKeeper/etc/ignore: added storage/maria/maria_read_log support-files/compiler_warnings.supp: Ignore function used when debugging (can be called from gdb)	19 years ago
unknown	10bce560f6	WL#3072 - Maria recovery comments; remember the UNDO's LSN for storing it in pages when executing REDO's (to imitate what the runtime code does) storage/maria/maria_read_log.c: comments; remember the UNDO's LSN for storing it in pages when executing REDO's (to imitate what the runtime code does)	19 years ago
unknown	1e73169a82	WL#3072 - Maria recovery fixes for build failures; copyrights; small bugfixes and comments mysys/Makefile.am: missing .h breaks building from tarball storage/maria/ma_loghandler.c: applying Serg's bugfix of trnman_new_trid() to translog_assign_id_to_share() storage/maria/ma_loghandler.h: copyright storage/maria/ma_loghandler_lsn.h: copyright storage/maria/maria_read_log.c: fix for compiler warnings. Comments. Close tables when program ends.	19 years ago
unknown	79672e8c44	WL#3072 - Maria recovery: safety assertions. storage/maria/maria_read_log.c: assertions to protect against future bugs (especially, to ensure that replaying DROP TABLE, if implemented, wouldn't leave open tables behind it)	19 years ago
unknown	adac9798bf	WL#3072 Maria Recovery - new program maria_read_log to display and apply log records found in a Maria log (see file's revision comment) - minor, misc fixes storage/maria/Makefile.am: new program maria_read_log storage/maria/ha_maria.cc: create control file if missing storage/maria/ma_blockrec.c: 0 -> LSN_IMPOSSIBLE; comments storage/maria/ma_checkpoint.h: preparations for Checkpoint module storage/maria/ma_close.c: comment storage/maria/ma_control_file.c: renaming constants. Possibility to say "open control file but don't create it if it's missing" (used by maria_read_log which does not want to create anything) storage/maria/ma_control_file.h: renaming constants storage/maria/ma_create.c: I had duplicated "linkname" and "linkname_ptr", now I see it's not needed, reverting. Indeed those variables don't contain interesting information; fixing log record accordingly (the links are in ci->data/index_file_name). Storing keystart in log record is needed, to know at which size we must extend the file if we replay LOGREC_CREATE_TABLE. storage/maria/ma_loghandler.c: some structures need to be known to maria_read_log.c, taking them to ma_loghandler.h storage/maria/ma_loghandler.h: we have page_store, adding page_korr. translog_lock() made public, because Checkpoint will need it (to write to control file). Some structures moved from ma_loghandler.c because maria_read_log.c needs them (needs to know the execute-in-REDO-phase hooks of each record). storage/maria/ma_loghandler_lsn.h: constants defined in ma_control_file.h serve everywhere, and they relate to LSNs, so putting them in ma_loghandler_lsn.h. Stronger constraints in LSN_VALID(). storage/maria/ma_pagecache.c: renaming constants storage/maria/ma_recovery.h: copyright storage/maria/ma_test1.c: new prototype storage/maria/ma_test2.c: new prototype storage/maria/trnman_public.h: double-inclusion safe storage/maria/unittest/ma_control_file-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multigroup-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_multithread-t.c: constants renamed, new prototype storage/maria/unittest/ma_test_loghandler_pagecache-t.c: constants renamed, new prototype storage/myisam/mi_close.c: comment storage/maria/maria_read_log.c: program to read and print log records from a Maria transaction log, and optionally apply them to tables. Very basic, early version. Should serve as a base for Recovery's code. Designed to be idempotent. Create a log by running maria.test, then cd to var/master-data and run "maria_read_log --only-display" to see info about records; run "maria_read_log --display-and-apply" to also apply the records to tables (it's more interesting if you first wipe out the tables in var/master-data/test, to see how they get re-created). Only a few records are handled by now: LONG_TRANSACTION_ID, COMMIT, FILE_ID, REDO_CREATE_TABLE; place is ready for REDO_INSERT_ROW_HEAD where I could use Monty's help (search for "Monty" in the file). Note: changes to the index pages, index's header and bitmap pages are not properly logged yet, so don't expect the program to work with that.	19 years ago

25 Commits (422375fc1bce1ea6ffb9226cfdef02c6a05b2a0f)